Who wins Mitre Att&ck APT 29 evaluation?
Mitre based on real Russian APT29 attack created emulation of it. Well known tools were tested and results are publicly available. There were no summary, points and winners but a raw report — it can be analyzed and winners deduced. You can scroll down to the end.
—
1. User opens .doc.scr with malware (User Execution), name of the file is modified using uncommon Unicode character (Masquerading). Executed payload opens encrypted remote TCP/IP connection on non-standard port (Uncommonly used port), then executes cmd.exe and PowerShell.
User Execution:
- Carbon Black generates an alert that suspicious file is executed. It’s possible to tweak CB configuration to prevent execution.
- Bitdefender rises an Antimalware alert for an unusual executable.
- CrowdStrike rises an alert about suspicious file.
- FireEye reports technique ‘Execution from User Directory’ and first execution of a new file.
- Kaspersky rises an alert about technique ‘User Execution’.
- McAfee rises and alert ‘Running process with suspicious name’.
- Microsoft identifies exact malware.
- Palo Alto Networks identifies malware.
- SentinelOne reports tactic ‘ProcessCreationExtra’.
- Symantec reports malware using heuristics.
- Trend Micro reports suspicious file.
Masquerading:
- Bitdefender, CrowdStrike, FireEye, Kaspersky, McAfee, Microsoft, SentinelOne, Trend Micro report technique ‘Masquerading’.
- Palo Alto Networks reports screensaver file.
- Symantec reports suspicious file PE32 executable via AI.
Uncommonly Used Port:
- FireEye generates a technique alert detection called ‘Uncommon Port Connection. Similarly McAfee and SentinelOne.
- Symantec emits a general alert detection for ‘Attack, Backdoor’.
CLI:
- CarbonBlack doesn’t warn about cmd.exe, but correlates current telemetry to previous detections. The same telemetry is generated by Bitdefender, CrowdStrike, McAfee, Microsoft, Trend Micro.
- FireEye and Palo Alto Networks report a correlated detection for Technique ‘CMD Execution’.
- SentinelOne reports a correlated Tactic called ‘ProcessCreationExtra’.
- Kaspersky reports non-correlated Technique CLI.
- Symantec doesn’t report cmd.exe.
PowerShell:
- CarbonBlack, Bitdefender, CrowdStrike emit a general correlated alert.
- FireEye reports non-correlated Technique PowerShell Execution.
- Kaspersky, McAfee, SentinelOne, Symantec report correlated Technique/Tactic PowerShell Execution.
Detection of malware entrypoint into corporate network is probably the most important part of the fight with attackers. FireEye and Symantec win this round.
2. PowerShell scripts traverses filesystem for document and media files to pack them into zip archive and pushes over TCP/IP channel.
File and Directory Discovery:
- Bitdefender, Palo Alto Networks, SentinelOne, Trend Micro report a correlated technique alert.
Automated Collection:
- CrowdStrike and Trend Micro generate a correlated technique alert.
- SentinelOne generates a delayed correlated technique alert.
Data from Local System step is reported by SentinelOne generating a delayed correlated technique alert.
Data Compressed:
- CarbonBlack, Bitdefender, Palo Alto Networks, SentinelOne, Symantec report correlated Technique for compressed data.
- FireEye, Kaspersky report non-correlated Technique for compressed data.
Data Staged is reported by FireEye as general non-correlated alert.
Two open file/socket handles, one for a local zip should rise and alert.
Bitdefender, CrowdStrike, Palo Alto Networks, SentinelOne and Trend Micro generate just a telemetry event.
This time the winner is SentinelOne.
3. Screensaver drops a png file to Desktop with embedded payload (steganography), hijacts backup operations via registry manipulation to later execute PowerShell with elevated privileges, connects over HTTPS to and cleans up registry.
Remote File Copy:
- FireEye reports ‘User Directory File Write’ alert. Other products just report a telemetry.
Steganography:
- Carbon Black, CrowdStrike, Kaspersky, McAfee, Microsoft, SentinelOne, Symantec report a correlated general/technique alert.
- Bitdefender reports a non-correlated technique alert PowershellImgObfuscation.
Registry:
- Palo Alto Networks generates a correlated alert.
- Microsoft generates an alert.
- Trend Micro generates a technique alert.
Privilege Escalation:
- Bitdefender, CrowdStrike, SentinelOne report a correlated bypass UAC technique alert.
- Carbon Black, Microsoft, Palo Alto Networks report a bypass UAC technique alert
- FireEye, Kaspersky report an alert about suspicious process.
HTTPS triggers an alert in FireEye, Kaspersky, Symantec (backdoor) and Trend Micro.
4. C&C channel is used to download SysInternals suite, decompressed by PowerShell script which enumerates processes. SDelete is used to clean up evidences. Powershell.exe executes Get-WmiObject to list Class AntiVirusProduct then FireWallProduct. Netapi32.dll is used.
- Carbon Black generates a correlated technique alert about SysInternals.
- FireEye and Palo Alto Networks generate a general alert about downloaded zip.
- FireEye, Kaspersky, McAfee, SentinelOne, Symantec alert about PowerShell.
- Carbon Black, Bitdefender, FireEye, Symantec alert about unzipping.
Get-Process:
- Bitdefender, SentinelOne, Trend Micro alert about process enumeration.
Listing processes might be an administrative action as well a discovery executed by an attacker.
SDelete (3x):
- Carbon Black, Bitdefender (1x), CrowdStrike (1x), FireEye, McAfee, Symantec, Trend Micro rise an alert.
AV Discovery:
- Bitdefender and SentinelOne also Palo Alto Networks rise a technique alert. Two first emit a correlated alert.
Firewall listing triggers alert in CrowdStrike and SentinelOne.
Netapi32.dll loading triggers alert in FireEye.
No clear winner here. Carbon Black and Bitdefender do good job, but not perfect.
5–7. A new system service is created by PowerShell script. A new file is added to autorun. Google Chrome SQL database file is read to extract encrypted credentials using CryptUnprotectedData API using executable with trusted file name. Local certificate with private key is exported to PFX file. Malware dumps password hashes from the Windows Registry by injecting a malicious DLL into lsass.exe. Malware performs screen, clipboard and input capture.
- Bitdefender, FireEye, McAfee, Microsoft, SentinelOne, Symantec alarm about a new service.
- Carbon Black, Bitdefender, FireEye, McAfee, Microsoft, Palo Alto Networks, SentinelOne, Symantec, Trend Micro alarm about a startup file.
- Palo Alto Networks and SentinelOne report suspicious read activity.
File Name trick:
- Carbon Black, CrowdStrike, McAfee, Microsoft, Symantec report a malware file.
- Bitdefender, FireEye, Palo Alto Networks, Trend Micro report a masquarade technique.
Certificate:
- Carbon Black, Bitdefender, FireEye, Palo Alto Networks, Trend Micro report an alert.
Private key is a serious thing. Only these product should be considered as a list of winners.
Credentials dump:
- Bitdefender, CrowdStrike, McAfee, Palo Alto Networks, SentinelOne, Symantec, Trend Micro explicitly generate an alert about credentials dump.
- FireEye, Microsoft rise an alert about process injection.
Dumping credentials is a serious thing. Only these product should be considered as a list of winners.
Screen capture is being alarmed by by Carbon Black, Bitdefender, FireEye, Palo Alto Networks, SentinelOne, Trend Micro.
Clipboard capture is being alerted by Carbon Black, Bitdefender, Trend Micro.
GetAsyncKeyState API:
- CrowdStrike, FireEye, McAfee, Microsoft, Palo Alto Networks, SentinelOne, Trend Micro fire an alert.
- Carbon Black and Bitdefender just deliver a telemetry.
7B. Local files are gathered and pushed to WebDAV.
- Bitdefender and SentinelOne alert about reading Downloads folder.
- Bitdefender, FireEye, Palo Alto Networks and Symantec alert about 7z file.
- CrowdStrike, FireEye, Microsoft, SentinelOne are able to deliver telemetry proving that 7zip was encrypted with password.
- Bitdefender, FireEye, McAfee, Palo Alto Networks, Trend Micro detect WebDAV file transfer.
8. Remote System Discovery.
Malware performs LDAP queries to Domain Controller.
- FireEye and Trend Micro generate an alert for this activity.
WinRM is used to execute remote process listing using PowerShell.
- Carbon Black, Bitdefender, FireEye, SentinelOne, Trend Micro generate an alert for WinRM connection.
- Bitdefender, Palo Alto Networks, SentinelOne, Trend Micro generate an alert for this activity.
A python.exe file compressed with UPX is uploaded to remote machine.
- Bitdefender, FireEye, Palo Alto Networks, SentinelOne, Symantec and Trend Micro detected and alerted copy operation.
- Bitdefender, CrowdStrike, Palo Alto Networks, SentinelOne and Symantec alerted about UPX.
- FireEye and Trend Micro show just a telemetry event.
Remote logon using stolen credentials is alerted by SentinelOne and Trend Micro.
- Bitdefender, CrowdStrike, Palo Alto Networks, Trend Micro, SentinelOne, generate an alert for using IPC$ share.
- FireEye generates just an alert about SMB session.
Python is executed remotely:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Palo Alto Networks, SentinelOne, Symantec, Trend Micro rise an alert.
9. Remote operations.
- FireEye, Palo Alto Networks, Symantec alert about rar.exe written to remote %TEMP%.
- FireEye, Symantec alert about sdelete written to remote host.
- Carbon Black, FireEye, Symantec alert about PowerShell executed from python.
Malware remotely collects data.
- Bitdefender, CrowdStrike, SentinelOne, Trend Micro alert about file search.
- FireEye and Palo Alto Networks alert about file collection.
- Bitdefender, FireEye, Kaspersky alert about encrypted rar.
- CarbonBlack, Bitdefender, CrowdStrike, FireEye, Kaspersky, Palo Alto Networks, SentinelOne, Symantec alert about compressed file.
Python uploads compressed zip over TCP/IP 8443. Cleans up (SDelete x4):
- Carbon Black (4x), FireEye (3x), McAfee (3x), Symantec (3x), Trend Micro (3x), Palo Alto Networks (2x) generate alert.
10. Installed malicious system service and startup file executions are alerted only by SentinelOne.
CreateProcessWithToken API:
- Bitdefender, CrowdStrike, Palo Alto Networks, SentinelOne, Trend Micro rise an alert.
11–13. New infection using malicious payload 37486-the-shocking-truth-about-election-rigging-in-america.rtf.lnk.
- Carbon Black, Bitdefender, Kaspersky, Microsoft, Palo Alto Networks, SentinelOne, Symantec generate alerts.
NTFS alternate stream executed:
- Carbon Black, FireEye, Kaspersky, McAfee, SentinelOne, Trend Micro rise and explicit alert. Additionally Microsoft generates a generic one.
Malware checked that the BIOS version, serial number, devices, user name, domain association, processed running are not associated with VirtualBox or VMware sandbox:
- Bitdefender (4x), SentinelOne (4x), Palo Alto Networks (3x) generate alerts.
Malware adds itself to HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, Palo Alto Networks, SentinelOne, Symantec, Trend Micro generate a technique alert.
PowerShell executes payload from NTFS stream:
- Carbon Black, Bitdefender, FireEye, Kaspersky, McAfee, Symantec generate a technique alert.
This payload establishes HTTPS connection:
- Carbon Black, FireEye, Kaspersky, McAfee, Trend Micro (3x) generate an alert.
Microsoft rises and alert about enumeration about System32 directory.
Timestamp of infection marker file is modified.
- Microsoft, Palo Alto Networks, SentinelOne, Trend Micro rise an alert.
Once again AV Discovery is done using AV class:
- Bitdefender, CrowdStrike, Microsoft, SentinelOne also Palo Alto Networks rise a technique/general alert.
and Registry uninstall section:
- Microsoft and Palo Alto Networks rise an alert.
Alerts for enumerated running processes using the CreateToolhelp32Snapshot API are generated by:
- Bitdefender, SentinelOne.
COM attack against HKCU\Software\Classes\Folder\shell\open\command is alerted by:
- Bitdefender, Microsoft.
14–15. Bypassing User Account Control is alerted by:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, Microsoft, Symantec.
Usage of WMI is alerted by:
- Carbon Black, Bitdefender, FireEye, Kaspersky, Microsoft, SentinelOne, Symantec, Trend Micro.
Downloaded and dropped Mimikatz (m.exe) to disk is notified as an alert by:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, Palo Alto Networks, Symantec, Trend Micro.
Mimikatz dumped plaintext credentials. This was alerted by:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, McAfee, Microsoft, Palo Alto Networks, SentinelOne, Symantec, Trend Micro.
Encoding and writing Mimikatz output to a WMI class property using PowerShell is not alerted.
Established WMI event subscription persistence using PowerShell is alerted by:
- Carbon Black, Bitdefender, CrowdStrike, Kaspersky, McAfee, Microsoft, Palo Alto Networks, SentinelOne, Symantec, Trend Micro.
16. Malware executes LDAP queries, gets domain SID via ConvertSidToStringSid API.
LDAP:
- FireEye, Microsoft, Trend Micro rise an alert.
API call is alerted by:
- FireEye, Microsoft.
WinRM connection is established to the domain controller over TCP/IP 5985 port.
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, McAfee, SentinelOne, Trend Micro generate an alert.
- SentinelOne, Trend Micro generate an alert for successful logon to DC.
Mimikatz is dropped to DC:
- Carbon Black, Bitdefender, Microsoft, SentinelOne, Symantec, Trend Micro generate an alert.
KRBTGT hash is dumped on the domain controller:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, McAfee, Microsoft, Palo Alto Networks, SentinelOne, Symantec, Trend Micro generate an alert.
17–18. Outlook emails are dumped. HTML from local documents is read. Data to be stolen is staged and uploaded online.
Mail:
- Carbon Black, FireEye generate an alert.
HTML:
- SentinelOne rises an alert.
Staged data:
- Bitdefender, FireEye generate an alert.
Zip:
- Bitdefender, FireEye, Palo Alto Networks, SentinelOne generate an alert.
GIF header is prepended to ZIP:
- FireEye and SentinelOne generate an alert.
Network drive with public IP (OneDrive) is mapped:
- Carbon Black, Bitdefender, FireEye, Kaspersky, McAfee, Microsoft, Palo Alto Networks, Trend Micro generate an alert.
Data is uploaded:
- FireEye and Microsoft rise an alert.
20. Executed Run key persistence payload on user login using RunDll32:
- Bitdefender, FireEye, Kaspersky, McAfee, Microsoft, Symantec generate an alert.
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, Palo Alto Networks generate an alert about WMI usage.
SYSTEM level PowerShell script is executed:
- Carbon Black, Bitdefender, CrowdStrike, FireEye, Kaspersky, McAfee, Microsoft, Symantec, Trend Micro generate an alert.
Created Kerberos Golden Ticket using Invoke-Mimikatz is used in domain logon via WinRM:
- Carbon Black, Bitdefender, FireEye, Kaspersky, Microsoft, Sentinel One generate an alert about golden ticket.
- Carbon Black, Bitdefender, CrowdStrike, FireEye, McAfee, Trend Micro generate an alert about WinRM.
New local user is added to the Windows machine using net.exe. If protecting software doesn’t warn about it it’s useless.
- Carbon Black, Bitdefender, CrowdStrike, FireEye, McAfee, Palo Alto Networks, Symantec, Trend Micro generate an alert.
—
Endpoint protection tool must have a deep integration with the operating system to intercepts low level actions used by the attacker. It must know about many places and common techniques. Not a lot should be left as a telemetry for human analysis. Direct alerts are the best because the enterprise can react quickly. The winners are Bitdefender and FireEye.
