Learn how Darktrace analyzed the SolarWinds hack without signatures. Understand the techniques used to identify and mitigate this major cyber threat.
For a high-level explanation of the SolarWinds hack, watch our video below.
The SUNBURST malware attacks against SolarWinds have heightened companies’ concerns about the risk to their digital environments. Malware installed during software updates in March 2020 has allowed advanced attackers to gain unauthorized access to files that may include customer data and intellectual property.
Darktrace does not use SolarWinds, and its operations remain unaffected by this breach. However, SolarWinds is an IT discovery tool that is used by a significant number of Darktrace customers. In what follows, we explore a set of Darktrace detections that highlight and alert security teams to the types of behaviors related to this breach.
This is not an example of a SolarWinds compromise, but examples of anomalous behaviors we can expect to see from this type of breach. These examples stress the value of self-learning Cyber AI capable of understanding the evolving normal ‘patterns of life’ within an enterprise – as opposed to a signature-based approach that looks at historical data to predict today’s threat.
As Darktrace detects device activity patterns rather than known malicious signatures, detecting use of these techniques will fall into the scope of Darktrace’s capabilities without further need for configuration. The technology automatically clusters devices into ‘peer groups’, allowing it to detect cases of an individual device behaving unusually. Using a self-learning approach is the best possible mechanism to catch an attacker who gains access into your systems using a degree of stealth so as to not trigger signature-based detection.
A number of these models may fire in combination with other models in order to make a strong detection over a time-series – and this is exactly where Darktrace’s autonomous incident triage capability, Cyber AI Analyst, plays a crucial role in investigating the alerts on behalf of security teams. Cyber AI Analyst saves critical time for security teams, and its results should be treated with a high priority during this period of vigilance.
How SolarWinds was detected with AI
We want to focus on the most sophisticated details of the hands-on intrusion that in many cases followed the initial automated attack. This post-exploitation part of the attack is much more varied and stealthy. These stages are also near-impossible to predict, as they are driven by the attacker’s intentions and goals for each individual victim at this stage – making the use of signatures, threat intelligence or static use cases virtually useless.
While the automated, initial malware execution is a critical initial step to understand, the behavior was pre-configured for the malware and included the download of further payloads and the connection to domain-generation-algorithm (DGA) based subdomains of avsvmcloud[.]com. These automated first stages of the attack have been sufficiently researched in depth by the community. This post is not aiming to add anything to these findings, but instead takes a look at the potential post-infection activities.
Malware / C2 domains
The threat-actor set the hostnames on their later-stage command and control (C2) infrastructure to match a legitimate hostname found within the victim’s environment. This allowed the adversary to blend into the environment, avoid suspicion, and evade detection. They further used C2 servers in geopolitical proximity to their victims, further circumventing static geo-based trusts lists. Darktrace is unaffected by this type of tradecraft as it does not have implicit, pre-defined trust of any geo-locations.
This would be very likely to trigger the following Darktrace Cyber AI models. The models were not specifically designed to detect SolarWinds modifications but have been in place for years – they are designed to detect the subtle but significant attacker activities occurring within an organization’s network.
Compromise / Agent Beacon to New Endpoint
Compromise / SSL Beaconing to New Endpoint
Compromise / HTTP Beaconing to New Endpoint*
*The implant uses SSL, but may be identified as HTTP if using a proxy.
Lateral movement using different credentials
Once the attacker gained access to the network with compromised credentials, they moved laterally using multiple different credentials. The credentials used for lateral movement were always different from those used for remote access.
This very likely would trigger the following Cyber AI models:
User / Multiple Uncommon New Credentials on Device
Figure 1: Example breach event log showing anomalous (new) logins from a single device, with multiple user credentials
User / New Admin Credentials on Client
Figure 2: Example breach event log showing anomalous admin login
Temporary file replacement and temporary task modification
The attacker used a temporary file replacement technique to remotely execute utilities: they replaced a legitimate utility with theirs, executed their payload, and then restored the legitimate original file. They similarly manipulated scheduled tasks by updating an existing legitimate task to execute their tools and then returned the scheduled task to its original configuration. They routinely removed their tools – including the removal of backdoors once legitimate remote access was achieved.
This would be very likely to trigger the following Cyber AI models:
Anomalous Connection / New or Uncommon Service Control
Figure 3: Example breach showing uncommon service control
Anomalous Connection / High Volume of New or Uncommon Service Control
Figure 4: Example breach showing 10 uncommon service controls
Device / AT Service Scheduled Task
Figure 5: Breach event log shows new AT service scheduled task activity
Device / Multiple RPC Requests for Unknown Services
Figure 6: Breach shows multiple binds to unknown RPC services
Device / Anomalous SMB Followed By Multiple Model Breaches
Figure 10: Breach shows significant deviation in SMB activity from device
SolarWinds breach remembered
By understanding where credentials are used and which devices talk to each other, Cyber AI has an unprecedented and dynamic understanding of business systems. This empowers it to alert security teams to enterprise changes that could indicate cyber risk in real time.
These alerts demonstrate how AI learns ‘normal’ for the unique digital environment surrounding it, and then alerts operators to deviations, including those that are directly relevant to the SUNBURST compromise. It further provides insights into how the attacker exploited those networks that did not have the appropriate visibility and detection capabilities.
On top of these alerts, Cyber AI Analyst will also be automatically correlating these detections over time to identify patterns, generating comprehensive and intuitive incident summaries and significantly reducing triage time. Reviewing Cyber AI Analyst alerts should be given high priority over the next several weeks.
Like this and want more?
Receive the latest blog in your inbox
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Newsletter
Like this and want more?
Stay up to date on the latest industry news and insights.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Author
Max Heinemeyer
Chief Product Officer
Max is a cyber security expert with over a decade of experience in the field, specializing in a wide range of areas such as Penetration Testing, Red-Teaming, SIEM and SOC consulting and hunting Advanced Persistent Threat (APT) groups. At Darktrace, Max is closely involved with Darktrace’s strategic customers & prospects. He works with the R&D team at Darktrace, shaping research into new AI innovations and their various defensive and offensive applications. Max’s insights are regularly featured in international media outlets such as the BBC, Forbes and WIRED. Max holds an MSc from the University of Duisburg-Essen and a BSc from the Cooperative State University Stuttgart in International Business Information Systems.
These systems, which are exposed to the internet, are often targeted by threat actors to gain initial access to a network. They are constantly being scanned for vulnerabilities, known or unknown, by opportunistic actors hoping to exploit gaps in security. Unfortunately, this exposure remains a significant blind spot for many security teams, as monitoring edge infrastructure can be particularly challenging due to its distributed nature and the sheer volume of external traffic it processes.
In this blog, we discuss a vulnerability that was exploited in Fortinet’s FortiClient Endpoint Management Server (EMS) and the post-exploitation activity that Darktrace observed across multiple customer environments.
What is FortiClient EMS?
FortiClient is typically used for endpoint security, providing features such as virtual private networks (VPN), malware protection, and web filtering. The FortiClient EMS is a centralized platform used by administrators to enforce security policies and manage endpoint compliance. As endpoints are remote and distributed across various locations, the EMS needs to be accessible over the internet.
However, being exposed to the internet presents significant security risks, and exploiting vulnerabilities in the system may give an attacker unauthorized access. From there, they could conduct further malicious activities such as reconnaissance, establishing command-and-control (C2), moving laterally across the network, and accessing sensitive data.
CVE-2023-48788
CVE-2023-48788 is a critical SQL injection vulnerability in FortiClient EMS that can allow an attacker to gain unauthorized access to the system. It stems from improper neutralization of special elements used in SQL commands, which allows attackers to exploit the system through specially crafted requests, potentially leading to Remote Code Execution (RCE) [1]. This critical vulnerability was given a CVSS score of 9.8 and can be exploited without authentication.
The affected versions of FortiClient EMS include:
FortiClient EMS 7.2.0 to 7.2.2 (fixed in 7.2.3)
FortiClient EMS 7.0.1 to 7.0.10 (fixed in 7.0.11)
The vulnerability was publicly disclosed on March 12, 2024, and an exploit proof of concept was released by Horizon3.ai on March 21 [2]. Starting from March 24, almost two weeks after the initial disclosure, Darktrace began to observe at least six instances where the FortiClient EMS vulnerability had likely been exploited on customer networks. Seemingly exploited devices in multiple customer environments were observed performing anomalous activities, including the installation of Remote Monitoring and Management (RMM) tools, which was also reported by other security vendors around the same time [3].
Darktrace’s Coverage
Initial Access
To understand how the vulnerability can be exploited to gain initial access, we first need to explain some components of the FortiClient EMS:
The service FmcDaemon.exe is used for communication between the EMS and enrolled endpoint clients. It listens on port 8013 for incoming client connections.
Incoming requests are then sent to FCTDas.exe, which translates requests from other server components into SQL requests. This service interacts with the Microsoft SQL database.
Endpoint clients communicate with the FmcDaemon on the server on port 8013 by default.
Therefore, an SQL injection attack can be performed by crafting a malicious payload and sending it over port 8013 to the server. To carry out RCE, an attacker may send further SQL statements to enable and use the xp_cmdshell functionality of the Microsoft SQL server [2].
Shortly before post-exploitation activity began, Darktrace had observed incoming connections to some of the FortiClient EMS devices over port 8013 from the external IPs 77.246.103[.]110, 88.130.150[.]101, and 45.155.141[.]219. This likely represented the threat actors sending an SQL injection payload over port 8013 to the EMS device to validate the exploit.
Establish C2
After exploiting the vulnerability and gaining access to an EMS device on one customer network, two additional devices were seen with HTTP POST requests to 77.246.103[.]110 and 212.113.106[.]100 with a new PowerShell user agent.
Interestingly, the IP 212.113.106[.]100 has been observed in various other campaigns where threat actors have also targeted internet-facing systems and exploited other vulnerabilities. Open-source intelligence (OSINT) suggests that this indicator of compromise (IoC) is related to the Sliver C2 framework and has been used by threat actors such as APT28 (Fancy Bear) and APT29 (Cozy Bear) [4].
Unusual file downloads were also observed on four devices, including:
“SETUP.MSI” from 212.32.243[.]25 and 89.149.200[.]91 with a cURL user agent
“setup.msi” from 212.113.106[.]100 with a Windows Installer user agent
“run.zip” from 95.181.173[.]172 with a PowerShell user agent
The .msi files would typically contain the RMM tools Atera or ScreenConnect [5]. By installing RMM tools for C2, attackers can leverage their wide range of functionalities to carry out various tasks, such as file transfers, without the need to install additional tools. As RMM tools are designed to maintain a stable connection to remote systems, they may also allow the attackers to ensure persistent access to the compromised systems.
A scan of the endpoint 95.181.173[.]172 shows various other files such as “RunSchedulerTask.ps1” and “anydesk.exe” being hosted.
Figure 1: Screenshot of the endpoint 95.181.173[.]172 hosting various files [6].
Shortly after these unusual file downloads, many of the devices were also seen with usage of RMM tools such as Splashtop, Atera, and AnyDesk. The devices were seen connecting to the following endpoints:
*[.]relay.splashtop[.]com
agent-api[.]atera[.]com
api[.]playanext[.]com with user agent AnyDesk/8.0.9
RMM tools have a wide range of legitimate capabilities that allow IT administrators to remotely manage endpoints. However, they can also be repurposed for malicious activities, allowing threat actors to maintain persistent access to systems, execute commands remotely, and even exfiltrate data. As the use of RMM tools can be legitimate, they offer threat actors a way to perform malicious activities while blending into normal business operations, which could evade detection by human analysts or traditional security tools.
One device was also seen making repeated SSL connections to a self-signed endpoint “azure-documents[.]com” (104.168.140[.]84) and further HTTP POSTs to “serv1[.]api[.]9hits[.]com/we/session” (128.199.207[.]131). Although the contents of these connections were encrypted, they were likely additional infrastructure used for C2 in addition to the RMM tools that were used. Self-signed certificates may also be used by an attacker to encrypt C2 communications.
Internal Reconnaissance
Following the exploit, two of the compromised devices then started to conduct internal reconnaissance activity. The following figure shows a spike in the number of internal connections made by one of the compromised devices on the customer’s environment, which typically indicates a network scan.
Figure 2: Advanced Search results of internal connections made an affected device.
Reconnaissance tools such as Advanced Port Scanner (“www[.]advanced-port-scanner[.]com”) and Nmap were also seen being used by one of the devices to conduct scanning activities. Nmap is a network scanning tool commonly used by security teams for legitimate purposes like network diagnostics and vulnerability scanning. However, it can also be abused by threat actors to perform network reconnaissance, a technique known as Living off the Land (LotL). This not only reduces the need for custom or external tools but also reduces the risk of exposure, as the use of a legitimate tool in the network is unlikely to raise suspicion.
Privilege Escalation
In another affected customer network, the threat actor’s attempt to escalate their privileges was also observed, as a FortiClient EMS device was seen with an unusually large number of SMB/NTLM login failures, indicative of brute force activity. This attempt was successful, and the device was later seen authenticating with the credential “administrator”.
Figure 3: Advanced Search results of NTLM (top) and SMB (bottom) login failures.
Lateral Movement
After escalating privileges, attempts to move laterally throughout the same network were seen. One device was seen transferring the file “PSEXESVC.exe” to another device over SMB. This file is associated with PsExec, a command-line tool that allows for remote execution on other systems.
The threat actor was also observed leveraging the DCE-RPC protocol to move laterally within the network. Devices were seen with activity such as an increase in new RPC services, unusual requests to the SVCCTL endpoint, and the execution of WMI commands. The DCE-RPC protocol is typically used to facilitate communication between services on different systems and can allow one system to request services or execute commands on another.
These are further examples of LotL techniques used by threat actors exploiting CVE-2023-48788, as PsExec and the DCE-RPC protocol are often also used for legitimate administrative operations.
Accomplish Mission
In most cases, the threat actor’s end goal was not clearly observed. However, Darktrace did detect one instance where an unusually large volume of data had been uploaded to “put[.]io”, a cloud storage service, indicating that the end goal of the threat actor had been to steal potentially sensitive data.
In a recent investigation of a Medusa ransomware incident that took place in July 2024, Darktrace’s Threat Research team found that initial access to the environment had likely been gained through a FortiClient EMS device. An incoming connection from 209.15.71[.]121 over port 8013 was seen, suggesting that CVE-2023-48788 had been exploited. The device had been compromised almost three weeks before the ransomware was actually deployed, eventually resulting in the encryption of files.
Conclusion
Threat actors have continued to exploit unpatched vulnerabilities in internet-facing systems to gain initial access to a network. This highlights the importance of addressing and patching vulnerabilities as soon as they are disclosed and a fix is released. However, due to the rapid nature of exploitation, this may not always be enough. Furthermore, threat actors may even be exploiting vulnerabilities that are not yet publicly known.
As the end goals for a threat actor can differ – from data exfiltration to deploying ransomware – the post-exploitation behavior can also vary from actor to actor. However, AI security tools such as Darktrace / NETWORK can help identify and alert for post-exploitation behavior based on abnormal activity seen in the network environment.
Despite CVE-2023-48788 having been publicly disclosed and fixed in March, it appears that multiple threat actors, such as the Medusa ransomware group, have continued to exploit the vulnerability on unpatched systems. With new vulnerabilities being disclosed almost every other day, security teams may find it challenging continuously patch their systems.
As such, Darktrace / Proactive Exposure Management could also alleviate the workload of security teams by helping them identify and prioritize the most critical vulnerabilities in their network.
Credit to Emily Megan Lim (Cyber Security Analyst) and Ryan Traill (Threat Content Lead)
Lifting the Fog: Darktrace’s Investigation into Fog Ransomware
Introduction to Fog Ransomware
As ransomware attacks continue to be launched at an alarming rate, Darktrace’s Threat Research team has identified that familiar strains like Akira, LockBit, and BlackBasta remain among the most prevalent threats impacting its customers, as reported in the First 6: Half-Year Threat Report 2024. Despite efforts by law agencies, like dismantling the infrastructure of cybercriminals and shutting down their operations [2], these groups continue to adapt and evolve.
As such, it is unsurprising that new ransomware variants are regularly being created and launched to get round law enforcement agencies and increasingly adept security teams. One recent example of this is Fog ransomware.
What is Fog ransomware?
Fog ransomware is strain that first appeared in the wild in early May 2024 and has been observed actively using compromised virtual private network (VPN) credentials to gain access to organization networks in the education sector in the United States.
Darktrace's detection of Fog Ransomware
In June 2024, Darktrace observed instances of Fog ransomware across multiple customer environments. The shortest time observed from initial access to file encryption in these attacks was just 2 hours, underscoring the alarming speed with which these threat actors can achieve their objectives.
Darktrace identified key activities typical of a ransomware kill chain, including enumeration, lateral movement, encryption, and data exfiltration. In most cases, Darktrace was able to successfully halt the progression Fog attacks in their early stages by applying Autonomous Response actions such as quarantining affected devices and blocking suspicious external connections.
To effectively illustrate the typical kill chain of Fog ransomware, this blog focuses on customer environments that did not have Darktrace’s Autonomous Response enabled. In these cases, the attack progressed unchecked and reached its intended objectives until the customer received Darktrace’s alerts and intervened.
Darktrace’s Coverage of Fog Ransomware
Initial Intrusion
After actors had successfully gained initial access into customer networks by exploiting compromised VPN credentials, Darktrace observed a series of suspicious activities, including file shares, enumeration and extensive scanning. In one case, a compromised domain controller was detected making outgoing NTLM authentication attempts to another internal device, which was subsequently used to establish RDP connections to a Windows server running Hyper-V.
Given that the source was a domain controller, the attacker could potentially relay the NTLM hash to obtain a domain admin Kerberos Ticket Granting Ticket (TGT). Additionally, incoming NTLM authentication attempts could be triggered by tools like Responder, and NTLM hashes used to encrypt challenge response authentication could be abused by offline brute-force attacks.
Darktrace also observed the use of a new administrative credential on one affected device, indicating that malicious actors were likely using compromised privileged credentials to conduct relay attacks.
Establish Command-and-Control Communication (C2)
In many instances of Fog ransomware investigated by Darktrace’s Threat Research team, devices were observed making regular connections to the remote access tool AnyDesk. This was exemplified by consistent communication with the endpoint “download[.]anydesk[.]com” via the URI “/AnyDesk.exe”. In other cases, Darktrace identified the use of another remote management tool, namely SplashTop, on customer servers.
In ransomware attacks, threat actors often use such legitimate remote access tools to establish command-and-control (C2) communication. The use of such services not only complicates the identification of malicious activities but also enables attackers to leverage existing infrastructure, rather than having to implement their own.
Internal Reconnaissance
Affected devices were subsequently observed making an unusual number of failed internal connections to other internal locations over ports such as 80 (HTTP), 3389 (RDP), 139 (NetBIOS) and 445 (SMB). This pattern of activity strongly indicated reconnaissance scanning behavior within affected networks. A further investigation into these HTTP connections revealed the URIs “/nice ports”/Trinity.txt.bak”, commonly associated with the use of the Nmap attack and reconnaissance tool.
Simultaneously, some devices were observed engaging in SMB actions targeting the IPC$ share and the named pipe “srvsvc” on internal devices. Such activity aligns with the typical SMB enumeration tactics, whereby attackers query the list of services running on a remote host using a NULL session, a method often employed to gather information on network resources and vulnerabilities.
Lateral Movement
As attackers attempted to move laterally through affected networks, Darktrace observed suspicious RDP activity between infected devices. Multiple RDP connections were established to new clients, using devices as pivots to propagate deeper into the networks, Following this, devices on multiple networks exhibited a high volume of SMB read and write activity, with internal share drive file names being appended with the “.flocked” extension – a clear sign of ransomware encryption. Around the same time, multiple “readme.txt” files were detected being distributed across affected networks, which were later identified as ransom notes.
Further analysis of the ransom note revealed that it contained an introduction to the Fog ransomware group, a summary of the encryption activity that had been carried out, and detailed instructions on how to communicate with the attackers and pay the ransom.
Figure 1: Packet capture (PCAP) of the ransom note file titled “readme.txt”.
Data Exfiltration
In one of the cases of Fog ransomware, Darktrace’s Threat Research team observed potential data exfiltration involving the transfer of internal files to an unusual endpoint associated with the MEGA file storage service, “gfs302n515[.]userstorage[.]mega[.]co[.]nz”.
This exfiltration attempt suggests the use of double extortion tactics, where threat actors not only encrypt victim’s data but also exfiltrate it to threaten public exposure unless a ransom is paid. This often increases pressure on organizations as they face the risk of both data loss and reputational damage caused by the release of sensitive information.
Darktrace’s Cyber AI Analyst autonomously investigated what initially appeared to be unrelated events, linking them together to build a full picture of the Fog ransomware attack for customers’ security teams. Specifically, on affected networks Cyber AI Analyst identified and correlated unusual scanning activities, SMB writes, and file appendages that ultimately suggested file encryption.
Figure 2: Cyber AI Analyst’s analysis of encryption activity on one customer network.
Figure 3: Cyber AI Analysts breakdown of the investigation process between the linked incident events on one customer network.
Conclusion
As novel and fast-moving ransomware variants like Fog persist across the threat landscape, the time taken for from initial compromise to encryption has significantly decreased due to the enhanced skill craft and advanced malware of threat actors. This trend particularly impacts organizations in the education sector, who often have less robust cyber defenses and significant periods of time during which infrastructure is left unmanned, and are therefore more vulnerable to quick-profit attacks.
Traditional security methods may fall short against these sophisticated attacks, where stealthy actors evade detection by human-managed teams and tools. In these scenarios Darktrace’s AI-driven product suite is able to quickly detect and respond to the initial signs of compromise through autonomous analysis of any unusual emerging activity.
When Darktrace’s Autonomous Response capability was active, it swiftly mitigated emerging Fog ransomware threats by quarantining devices exhibiting malicious behavior to contain the attack and blocking the exfiltration of sensitive data, thus preventing customers from falling victim to double extortion attempts.
Credit to Qing Hong Kwa (Senior Cyber Analyst and Deputy Analyst Team Lead, Singapore) and Ryan Traill (Threat Content Lead)
Appendices
Darktrace Model Detections:
- Anomalous Server Activity::Anomalous External Activity from Critical Network Device
- Anomalous Connection::SMB Enumeration
- Anomalous Connection::Suspicious Read Write Ratio and Unusual SMB
- Anomalous Connection::Uncommon 1 GiB Outbound
- Anomalous File::Internal::Additional Extension Appended to SMB File
![Screenshot of the endpoint 95.181.173[.]172 hosting various files [6].](https://cdn.prod.website-files.com/626ff4d25aca2edf4325ff97/66eb0afcc6a12ac4c8d91f16_66eb0a19eef352705e3e3cf1_Screenshot%25202024-09-18%2520at%252010.12.48%25E2%2580%25AFAM.png)
