Blog

OT

Ransomware

Inside the SOC

How Conti ransomware took down Operational Technology

Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
09
Feb 2022
09
Feb 2022
This blog demonstrates how ransomware can spread throughout converged IT/OT environments, and how Self-Learning AI empowers organizations to contain these threats.

Ransomware has taken the world by storm, and IT is not the only technology affected. Operational Technology (OT), which is increasingly blending with IT, is also susceptible to ransomware tactics, techniques, and procedures (TTPs). And when ransomware strikes OT, the effects have the potential to be devastating.

Here, we will look at a ransomware attack that spread from IT to OT systems. The attack was detected by Darktrace AI.

This threat find demonstrates a use case of Darktrace’s technology that delivers immense value to organizations with OT: spotting and stopping ransomware at its earliest stages, before the damage is done. This is particularly helpful for organizations with interconnected enterprise and industrial environments, as it means:

  1. Emerging attacks can be contained in IT before they spread laterally into OT, and even before they spread from device to device in IT;
  2. Organizations gain granular visibility into their industrial environments, detecting deviations from normal activity, and quick identification of remediating actions.

Threat find: Ransomware and crypto-mining hijack affecting IT and OT systems

Darktrace recently identified an aggressive attack targeting an OT R&D investment firm in EMEA. The attack originally started as a crypto-mining campaign and later evolved into ransomware. This organization deployed Darktrace in a digital estate containing both IT and OT assets that spanned over 3,000 devices.

If the organization had deployed Darktrace’s Autonomous Response technology in active mode, this threat would have been stopped in its earliest stages. Even in the absence of Autonomous Response, however, mere human attention would have stopped this attack’s progression. Darktrace’s Self-Learning AI gave clear indications of an ongoing compromise in the month prior to the detonation of ransomware. In this case, however, the security team was not monitoring Darktrace’s interface, and so the attack was allowed to proceed.

Compromised OT devices

This threat find will focus on the attack techniques used to take over two OT devices, specifically, a HMI (human machine interface), and an ICS Historian used to collect and log industrial data. These OT devices were both VMware virtual machines running Windows OS, and were compromised as part of a wider Conti ransomware infection. Both devices were being used primarily within an industrial control system (ICS), running a popular ICS software package and making regular connections to an industrial cloud platform.

These devices were thus part of an ICSaaS (ICS-as-a-Service) environment, using virtualised and Cloud platforms to run analytics, update threat intelligence, and control the industrial process. As previously highlighted by Darktrace, the convergence of cloud and ICS increases a network’s attack surface and amplifies cyber risk.

Attack lifecycle

Opening stages

The initial infection of the OT devices occurred when a compromised Domain Controller (DC) made unusual Active Directory requests. The devices made subsequent DCE-RPC binds for epmapper, often used by attackers for command execution, and lsarpc, used by attackers to abuse authentication policies and escalate privileges.

The payload was delivered when the OT devices used SMB to connect to the sysvol folder on the DC and read a malicious executable file, called SetupPrep.exe.

Figure 1: Darktrace model breaches across the whole network from initial infection on October 21 to the detonation on November 15.

Figure 2: ICS reads on the HMI in the lead up, during, and following detonation of the ransomware.

Device encryption and lateral spread

The malicious payload remained dormant on the OT devices for three weeks. It seems the attacker used the time to install crypto-mining malware elsewhere on the network and consolidate their foothold.

On the day the ransomware detonated, the attacker used remote management tools to initiate encryption. The PSEXEC tool was used on an infected server (separate from the original DC) to remotely execute malicious .dll files on the compromised OT devices.

The devices then attempted to make command and control (C2) connections to rare external endpoints using suspicious ports. Like in many ICS networks, sufficient network segregation had been implemented to prevent the HMI device from making successful connections to the Internet and the C2 communications failed. But worryingly, the failed C2 did not prevent the attack from proceeding or the ransomware from detonating.

The Historian device made successful C2 connections to around 40 unique external endpoints. Darktrace detected beaconing-type behavior over suspicious TCP/SSL ports including 465, 995, 2078, and 2222. The connections were made to rare destination IP addresses that did not specify the Server Name Indication (SNI) extension hostname and used self-signed and/or expired SSL certificates.

Both devices enumerated network SMB shares and wrote suspicious shell scripts to network servers. Finally, the devices used SMB to encrypt files stored in network shares, adding a file extension which is likely to be unique to this victim and which will be called ABCXX for the purpose of this blog. Most encrypted files were uploaded to the folder in which the file was originally located, but in some instances were moved to the images folder.

During the encryption, the device was using the machine account to authenticate SMB sessions. This is in contrast to other ransomware incidents that Darktrace has observed, in which admin or service accounts are compromised and abused by the attacker. It is possible that in this instance the attacker was able to use ‘Living off the Land’ techniques (for example the use of lsarpc pipe) to give the machine account admin privileges.

Examples of files being encrypted and moved:

  • SMB move success
  • File: new\spbr0007\0000006A.bak
  • Renamed: new\spbr0007\0000006A.bak.ABCXX
  • SMB move success
  • File: ActiveMQ\readme.txt
  • Renamed: Images\10j0076kS1UA8U975GC2e6IY.488431411265952821382.png.ABCXX

Detonation of ransomware

Upon detonation, the ransomware note readme.txt was written by the ICS to targeted devices as part of the encryption activity.

The final model breached by the device was “Unresponsive ICS Device” as the device either stopped working due to the effects of the ransomware, or was removed from the network.

Figure 3: abc-histdev — external connections filtered on destination port 995 shows C2 connections starting around one hour before encryption began.

How the attack bypassed the rest of the security stack

In this threat find, there were a number of factors which resulted in the OT devices becoming compromised.

The first is IT/OT convergence. The ICS network was insufficiently segregated from the corporate network. This means that devices could be accessed by the compromised DC during the lateral movement stage of the attack. As OT becomes more reliant on IT, ensuring sufficient segregation is in place, or that an attacker can not circumvent such segregation, is becoming an ever increasing challenge for security teams.

Another reason is that the attacker used attack methods which leverage Living off the Land techniques to compromise devices with no discrimination as to whether they were part of an IT or OT network. Many of the machines used to operate ICS networks, including the devices highlighted here, rely on operating systems vulnerable to the kinds of TTPs observed here and that are regularly employed by ransomware groups.

Darktrace insights

Darktrace’s Cyber AI Analyst was able to stitch together many disparate forms of unusual activity across the compromised devices to give a clear security narrative containing details of the attack. The incident report for the Historian server is shown below. This provides a clear illustration of how Cyber AI Analyst can close any skills or communication gap between IT and OT specialists.

Figure 4: Cyber AI Analyst of the Historian server (abc-histdev). It investigated and reported the C2 communication (step 2) that started just before network reconnaissance using TCP scanning (step 3) and the subsequent file encryption over SMB (step 4).

In total, the attacker’s dwell time within the digital estate was 25 days. Unfortunately, it lead to disruption to operational technology, file encryption and financial loss. Altogether, 36 devices were crypto-mining for over 20 days – followed by nearly 100 devices (IT and OT) becoming encrypted following the detonation of the ransomware.

If it were active, Autonomous Response would have neutralized this activity, containing the damage before it could escalate into crisis. Darktrace’s Self-Learning AI gave clear indications of an ongoing compromise in the month prior to the detonation of ransomware, and so any degree of human attention toward Darktrace’s revelations would have stopped the attack.

Autonomous Response is highly configurable, and so, in industrial environments — whether air-gapped OT or converged IT/OT ecosystems — Antigena can be deployed in a variety of manners. In human confirmation mode, human operators need to give the green light before the AI takes action. Antigena can also be deployed only in the higher levels of the Purdue model, or the “IT in OT,” protecting the core assets from fast-moving attacks like ransomware.

Ransomware and interconnected IT/OT systems

ICS networks are often operated by machines that rely on operating systems which can be affected by TTPs regularly employed by ransomware groups — that is, TTPs such as Living off the Land, which do not discriminate between IT and OT.

The threat that ransomware poses to organizations with OT, including critical infrastructure, is so severe that the Cyber Infrastructure and Security Agency (CISA) released a fact sheet concerning these threats in the summer of 2021, noting the risk that IT attacks pose to OT networks:

“OT components are often connected to information technology (IT) networks, providing a path for cyber actors to pivot from IT to OT networks… As demonstrated by recent cyber incidents, intrusions affecting IT networks can also affect critical operational processes even if the intrusion does not directly impact an OT network.”

Major ransomware attacks against the Colonial Pipeline and JBS Foods demonstrate the potential for ransomware affecting OT to cause severe economic disruption on a national and international scale. And ransomware can wreak havoc on OT systems regardless of whether they directly target OT systems.

As industrial environments continue to converge and evolve — be they IT/OT, ICSaaS, or simply poorly segregated legacy systems — Darktrace stands ready to contain attacks before the damage is done. It is time for organizations with industrial environments to take the quantum leap forward that Darktrace’s Self-Learning AI is uniquely positioned to provide.

Thanks to Darktrace analysts Ash Brice and Andras Balogh for their insights on the above threat find.

Discover more on how Darktrace protects OT environments from ransomware

Darktrace model detections

HMI in chronological order at time of detonation:

  • Anomalous Connection / SMB Enumeration
  • Anomalous File / Internal / Unusual SMB Script Write
  • Anomalous File / Internal / Additional Extension Appended to SMB File
  • Compromise / Ransomware / Suspicious SMB Activity [Enhanced Monitoring]
  • ICS / Unusual Data Transfer By OT Device
  • ICS / Unusual Unresponsive ICS Device

Historian

  • ICS / Rare External from OT Device
  • Anomalous Connection / Anomalous SSL without SNI to New External
  • Anomalous Connection / Multiple Connections to New External TCP Port
  • ICS / Unusual Activity From OT Device
  • Anomalous Connection / SMB Enumeration
  • Anomalous Connection / Suspicious Activity On High Risk Device
  • Unusual Activity / SMB Access Failures
  • Device / Large Number of Model Breaches
  • ICS / Unusual Data Transfer By OT Device
  • Anomalous File / Internal / Additional Extension Appended to SMB File
  • Device / SMB Lateral Movement
  • Compromise / Ransomware / Suspicious SMB Activity [Enhanced Monitoring]
  • Device / Multiple Lateral Movement Model Breaches [Enhanced Monitoring]

INSIDE THE SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
AUTHOR
ABOUT ThE AUTHOR
Oakley Cox
Analyst Technical Director, APAC

Oakley is a technical expert with 5 years’ experience as a Cyber Analyst. After leading a team of Cyber Analysts at the Cambridge headquarters, he relocated to New Zealand and now oversees the defense of critical infrastructure and industrial control systems across the APAC region. His research into cyber-physical security has been published by Cyber Security journals and CISA. Oakley is GIAC certified in Response and Industrial Defense (GRID), and has a Doctorate (PhD) from the University of Oxford.

Book a 1-1 meeting with one of our experts
share this article
COre coverage

More in this series

No items found.

Blog

No items found.

Elevating Network Security: Confronting Trust, Ransomware, & Novel Attacks

Default blog imageDefault blog image
21
Jun 2024

Understanding the Network Security Market

Old tools blind to new threats

With the rise of GenAI and novel attacks, organizations can no longer rely solely on traditional network security solutions that depend on historical attack data, such as signatures and detection rules, to identify threats. However, in many cases network security vendors and traditional solutions like IDS/IPS focus on detecting known attacks using historical data. What happens is organizations are left vulnerable to unknown and novel threats, as these approaches only detect known malicious behavior and cannot keep up with unknown threats or zero-day attacks.

Advanced threats

Darktrace's End of Year Threat Report for 2023 highlights significant changes in the cyber threat landscape, particularly due to advancements in technology such as generative AI. The report notes a substantial increase in sophisticated attacks, including those utilizing generative AI, which have made it more challenging for traditional security measures to keep up. The report also details the rise of multi-functional malware, like Black Basta ransomware, which not only encrypts data for ransom but also spreads other types of malware such as the Qbot banking trojan. These complex attacks are increasingly being deployed by advanced cybercriminal groups, underscoring the need for organizations to adopt advanced security measures that can detect and respond to novel threats in real-time.

Defenders need a solution that can level the playing field, especially when they are operating with limited resources and getting overloaded with endless alerts. Most network security tools on the market have a siloed approach and do not integrate with the rest of an organization’s digital estate, but attackers don’t operate in a single domain.

Disparate workforce

With so many organizations continuing to support a remote or hybrid working environment, the need to secure devices that are outside the corporate network or off-VPN is increasingly important. While endpoint protection or endpoint detection and response (EDR) tools are a fundamental part of any security stack, it’s not possible to install an agent on every device, which can leave blind spots in an organization’s attack surface. Managing trust and access policies is also necessary to protect identities, however this comes with its own set of challenges in terms of implementation and minimizing business disruption.

This blog will dive into these challenges and show examples of how Darktrace has helped mitigate risk and stop novel and never-before-seen threats.

Network Security Challenge 1: Managing trust

What is trust in cybersecurity?

Trust in cybersecurity means that an entity can be relied upon. This can involve a person, organization, or system to be authorized or authenticated by proving their identity is legitimate and can be trusted to have access to the network or sensitive information.

Why is trust important in cybersecurity?

Granting access and privileges to your workforce and select affiliates has profound implications for cybersecurity, brand reputation, regulatory compliance, and financial liability. In a traditional network security model, traffic gets divided into two categories — trusted and untrusted — with some entities and segments of the network deemed more creditable than others.

How do you manage trust in cybersecurity?

Zero trust is too little, but any is too much.

Modern network security challenges point to an urgent need for organizations to review and update their approaches to managing trust. External pressure to adopt zero trust security postures literally suggests trusting no one, but that impedes your freedom
to do business. IT leaders need a proven but practical process for deciding who should be allowed to use your network and how.

Questions to ask in updating Trusted User policies include:

  • What process should you follow to place trust in third
    parties and applications?
  • Do you subject trusted entities to testing and other due
    diligence first?
  • How often do you review this process — and trusted
    relationships themselves — after making initial decisions?
  • How do you tell when trusted users should no longer be
    trusted?

Once trust has been established, security teams need new and better ways to autonomously verify that those transacting within your network are indeed those trusted users that they claim to be, taking only the authorized actions you’ve allowed them to take.

Exploiting trust in the network

Insider threats have a major head start. The opposite of attacks launched by nameless, faceless strangers, insider threats originate through parties once deemed trustworthy. That might mean a current or former member of your workforce or a partner, vendor, investor, or service provider authorized by IT to access corporate systems and data. Threats also arise when a “pawn” gets unwittingly tricked into disclosing credentials or downloading malware.

Common motives for insider attacks include revenge, stealing or leaking sensitive data, taking down IT systems, stealing assets or IP, compromising your organization’s credibility, and simply harassing your workforce. Put simply, rules and signatures based security solutions won’t flag insider threats because an insider does not immediately present themselves as an intruder. Insider threats can only be stopped by an evolving understanding of ‘normal’ for every user that immediately alerts your team when trusted users do something strange.

“By 2026, 10% of large enterprises will have a comprehensive, mature and measurable zero-trust program in place, up from less than 1% today.” [1]

Use Case: Darktrace spots an insider threat

Darktrace/OT detected a subtle deviation from normal behavior when a reprogram command was sent by an engineering workstation to a PLC controlling a pump, an action an insider threat with legitimized access to OT systems would take to alter the physical process without any malware involved. In this instance, AI Analyst, Darktrace’s investigation tool that triages events to reveal the full security incident, detected the event as unusual based on multiple metrics including the source of the command, the destination device, the time of the activity, and the command itself.  

As a result, AI Analyst created a complete security incident, with a natural language summary, the technical details of the activity, and an investigation process explaining how it came to its conclusion. By leveraging Explainable AI, a security team can quickly triage and escalate Darktrace incidents in real time before it becomes disruptive, and even when performed by a trusted insider.

Read more about insider threats here

Network Security Challenge 2: Stopping Ransomware at every stage    

What is Ransomware?

Ransomware is a type of malware that encrypts valuable files on a victim’s device, denying the account holder access, and demanding money in exchange for the encryption key. Ransomware has been increasingly difficult to deal with, especially with ransom payments being made in crypto currency which is untraceable. Ransomware can enter a system by clicking a link dangerous or downloading malicious files.

Avoiding ransomware attacks ranks at the top of most CISOs’ and risk managers’ priority lists, and with good reason. Extortion was involved in 25% of all breaches in 2022, with front-page attacks wreaking havoc across healthcare, gas pipelines, food processing plants, and other global supply chains. [2]

What else is new?

The availability of “DIY” toolkits and subscription-based ransom- ware-as-a-service (RaaS) on the dark web equips novice threat actors to launch highly sophisticated attacks at machine speed. For less than $500, virtually anyone can acquire and tweak RaaS offerings such as Philadelphia that come with accessible customer interfaces, reviews, discounts, and feature updates — all the signature features of commercial SaaS offerings.                  

Darktrace Cyber AI breaks the ransomware cycle

The preeminence of ransomware keeps security teams on high alert for indicators of attack but hypervigilance — and too many tools churning out too many alerts — quickly exhausts analysts’ bandwidth. To reverse this trend, AI needs to help prioritize and resolve versus merely detect risk.

Darktrace uses AI to recognize and contextualize possible signs of ransomware attacks as they appear in your network and across multiple domains. Viewing behaviors in the context of your organization’s normal ‘pattern of life’ updates and enhances detection that watches for a repeat of previous techniques.

Darktrace's AI brings the added advantage of continuously analyzing behavior in your environment at machine speed.

Darktrace AI also performs Autonomous Response, shutting down attacks at every stage of the ransomware cycle, including the first telltale signs of exfiltration and encryption of data for extortion purposes.

Use Case: Stopping Hive Ransomware attack

Hive is distributed via a RaaS model where its developers update and maintain the code, in return for a percentage of the eventual ransom payment, while users (or affiliates) are given the tools to carry out attacks using a highly sophisticated and complex malware they would otherwise be unable to use.

In early 2022, Darktrace/Network identified several instances of Hive ransomware on the networks of multiple customers. Using its anomaly-based detection, Darktrace was able to successfully detect the attacks and multiple stages of the kill chain, including command and control (C2) activity, lateral movement, data exfiltration, and ultimately data encryption and the writing of ransom notes.

Darktrace’s AI understands customer networks and learns the expected patterns of behavior across an organization’s digital estate. Using its anomaly-based detection Darktrace is able to identify emerging threats through the detection of unusual or unexpected behavior, without relying on rules and signatures, or known IoCs.

Read the full story here

Network Security Challenge 3: Spotting Novel Attacks

You can’t predict tomorrow’s weather by reading yesterday’s forecast, yet that’s essentially what happens when network security tools only look for known attacks.

What are novel attacks?

“Novel attacks” include unknown or previously unseen exploits such as zero-days, or new variations of known threats that evade existing detection rules.

Depending on how threats get executed, the term “novel” can refer to brand new tactics, techniques, and procedures (TTPs), or to subtle new twists on perennial threats like DoS, DDoS, and Domain Name Server (DNS) attacks.

Old tools may be blind to new threats

Stopping novel threats is less about deciding whom to trust than it is about learning to spot something brand new. As we’ve seen with ransomware, the growing “aaS” attack market creates a profound paradigm shift by allowing non-technical perpetrators to tweak, customize, and coin never-before-seen threats that elude traditional network, email, VPN, and cloud security.

Tools based on traditional rules and signatures lack a frame of reference. This is where AI’s ability to spot and analyze abnormalities in the context of normal patterns of life comes into play.                        

Darktrace AI spots what other tools miss                                      

Instead of training in cloud data lakes that pool data from unrelated attacks worldwide, Darktrace AI learns about your unique environment from your environment. By flagging and analyzing everything unusual — instead of only known signs of compromise — Darktrace’s Self-Learning AI keeps security stacks from missing less obvious but potentially more dangerous events.

The real challenge here is achieving faster “time to meaning” and contextualizing behavior that might — or might not — be part of a novel attack. Darktrace/Network does not require a “patient zero” to identify a novel attack, or one exploiting a zero-day vulnerability.

Use Case: Stopping Novel Ransomware Attack

In late May 2023, Darktrace observed multiple instances of Akira ransomware affecting networks across its customer base. Thanks to its anomaly-based approach to threat detection Darktrace successfully identified the novel ransomware attacks and provided full visibility over the cyber kill chain, from the initial compromise to the eventual file encryptions and ransom notes. Darktrace identified Akira ransomware on multiple customer networks, even when threat actors were utilizing seemingly legitimate services (or spoofed versions of them) to carry out malicious activity. While this may have gone unnoticed by traditional security tools, Darktrace’s anomaly-based detection enabled it to recognize malicious activity for what it was. In cases where Darktrace’s autonomous response was enabled these attacks were mitigated in their early stages, thus minimizing any disruption or damage to customer networks.

Read the full story here

References

[1] Gartner, “Gartner Unveils Top Eight Cybersecurity Predictions for 2023-2024,” 28 March 2023.                    

[2] TechTarget, “Ransomware trends, statistics and facts in 2023,” Sean Michael Kerner, 26 January 2023.

Continue reading
About the author
Mikey Anderson
Product Manager, Network Detection & Response

Blog

Inside the SOC

Post-Exploitation Activities on PAN-OS Devices: A Network-Based Analysis

Default blog imageDefault blog image
20
Jun 2024

Introduction

Perimeter devices such as firewalls, virtual private networks (VPNs), and intrusion prevention systems (IPS), have long been the target of adversarial actors attempting to gain access to internal networks. However, recent publications and public service announcements by leading public institutions underscore the increased emphasis threat actors are putting on leveraging such products to initiate compromises.

A blog post by the UK National Cyber Security Center (NCSC) released in early 2024 notes that as improvements are made in the detection of phishing email payloads, threat actors have again begun re-focusing efforts to exploiting network edge devices, many of which are not secure by design, as a means of breach initiation.[i] As such, it comes as no surprise that new Common Vulnerabilities and Exposures (CVEs) are constantly discovered that exploit such internet-exposed systems.

Darktrace analysts frequently observe the impacts of such CVEs first through their investigations via Darktrace’s Security Operations Center (SOC), sometimes even before the public disclosure of proof of concepts for such exploits.  Beginning April 2024, Darktrace’s SOC began handling alerts and customer requests for potential incidents involving Palo Alto Networks firewall devices.  It was during this time that external researchers publicly disclosed what would later be classified as PAN-OS CVE-2024-3400, a form of remote command execution vulnerability that affects several versions of Palo Alto Networks’ firewall operating System (PAN-OS), namely PAN-OS 11.1, 11.0 and 10.2.

The increase in observed SOC activity for Palo Alto firewall devices, coupled with the public announcement of the new CVE, prompted Darktrace researchers to look for evidence of PAN-OS exploitation on customer networks. Researchers also focused on documenting post-exploitation activity from threat actors leveraging the recently disclosed vulnerability.

As such, this blog highlights the network-based behaviors involved in the CVE-2024-3400 attack chains investigated by Darktrace’s SOC and Threat Research teams. Moreover, this investigation also provides a deeper insight into the post-compromise activities of threat actors leveraging the novel CVE.  Such insights will not only prove relevant for cybersecurity teams looking to inhibit compromises in this specific instance, but also highlights general patterns of behavior by threat actors utilizing such CVEs to target internet-facing systems.

CVE-2024-3400

In April 2024, the Darktrace SOC observed an uptick in activity involving recurring patterns of malicious activity from Palo Alto firewall appliances. In response to this trend, Darktrace initiated a Threat Research investigation into such activity to try and identify common factors and indicators across seemingly parallel events. As the Threat Research team opened their investigation, external researchers concurrently provided public details of CVE-2024-3400, a form of remote command execution vulnerability in the GlobalProtect feature on Palo Alto Network firewall devices running PAN-OS versions: 10.2, 11.0, and 11.1.[ii]

In their proof of concept, security researchers at watchTowr demonstrated how an attacker can pass session ID (SESSID) values to these PAN-OS devices to request files that do not exist. In response, the system creates a zero-byte file with root privileges with the same name.[iii] Log data is passed on devices running telemetry services to external servers through command line functionality.[iv] Given this functionality, external actors could then request non-existent files in the SESSID containing command parameters which then be interpreted by the command line functionality.[v] Although researchers first believed the exploit could only be used against devices running telemetry services, this was later discovered to be untrue.[vi]

As details of CVE-2024-3400 began to surface, Darktrace’s Threat Research analysts quickly identified distinct overlaps in the observed activity on specific customer deployments and the post-exploitation behavior reported by external researchers. Given the parallels, Darktrace correlated the patterns of activity observed by the SOC team to exploitation of the newly discovered vulnerability in PAN-OS firewall appliances.

Campaign Analysis

Between the April and May 2024, Darktrace identified four main themes of post-exploitation activity involving Palo Alto Network firewall devices likely targeted via CVE-2024-3400: exploitation validation, shell command and tool retrieval, configuration data exfiltration, and ongoing command and control through encrypted channels and application protocols.

1. Exploit Validation and Further Vulnerability Enumeration

Many of the investigated attack chains began with malicious actors using out-of-band application security testing (OAST) services such as Interactsh to validate exploits against Palo Alto firewall appliances. This exploit validation activity typically resulted in devices attempting to contact unusual external endpoints (namely, subdomains of ‘oast[.]pro’, ‘oast[.]live’, ‘oast[.]site’, ‘oast[.]online’, ‘oast[.]fun’, ‘oast[.]me’, and ‘g3n[.]in’) associated with OAST services such as Interactsh. These services can be used by developers to inspect and debug internet traffic, but also have been easily abused by threat actors.

While attempted connections to OAST services do not alone indicate CVE-2024-3400 exploitation, the prevalence of such activities in observed Palo Alto firewall attack chains suggests widespread usage of these OAST services to validate initial access methods and possibly further enumerate systems for additional vulnerabilities.

Figure 1: Model alert log details showcasing a PAN-OS device making DNS queries for Interactsh domain names in what could be exploit validation, and/or further host enumeration.

2. Command and Payload Transmission

The most common feature across analyzed incidents was HTTP GET requests for shell scripts and Linux executable files (ELF) from external IPs associated with exploitation of the CVE. These HTTP requests were frequently initiated using the utilities, cURL and wget. On nearly every device likely targeted by threat actors leveraging the CVE, Darktrace analysts highlighted the retrieval of shell scripts that either featured enumeration commands, the removal of evidence of compromise activity, or commands to retrieve and start binaries on the destination device.

a) Shell Script Retrieval

Investigated devices commonly performed HTTP GET requests to retrieve shell command scripts. Despite this commonality, there was some degree of variety amongst the retrieved payloads and their affiliation with certain command tools. Several distinct types of shell commands and files were identified during the analyzed breaches. For example, some firewall devices were seen requesting .txt files associated with both Sliver C2, whose malicious use has previously been investigated by Darktrace, and Cobalt Strike. The target URIs of devices’ HTTP requests for these files included, “36shr.txt”, “2.txt”, “bin.txt”, and “data.txt”.

More interestingly, though, was the frequency with which analyzed systems requested bash scripts from rare external IP addresses, sometimes over non-standard ports for the HTTP protocol. These bash scripts would feature commands usually for the recipient system to check for certain existing files and or running processes. If the file did not exist, the system would then use cURL or wget to obtain content from external sites, change the permissions of the file, and then execute, sending output to dev/null as a means of likely defense evasion. In some scripts, the system would first make a new folder, and change directories prior to acquiring external content. Additionally, some samples highlighted multiple attempts at enumeration of the host system.

Figure 2: Packet capture (PCAP) data highlighting the incoming shell scripts providing instructions to use cURL to obtain external content, change the permissions of the file to execute, and then run the binary using the credentials and details provided.
Figure 3: PCAP data highlighting a variation of a shell script seen in an HTTP response processed by compromised devices. The script provides instructions to make a directory, retrieve and execute external content, and to hide the output.

Not every retrieved file that was not explicitly a binary featured bash scripts. Model alerts on some deployments also included file masquerading attempts by threat actors, whereby the Palo Alto firewall device would request content with a misleading extension in the URI. In one such instance, the requested URI, and HTTP response header suggests the returned content is an image/png, but the actual body response featured configuration parameters for a new daemon service to be run on the system.

Figure 4: PCAP data indicating configuration details likely for a new daemon on an investigated host. Such HTTP body content differs from the image/png extension within the request URI and declared content type in the HTTP response header.

Bash scripts analyzed across customer deployments also mirrored those identified by external security teams. External researchers previously reported on a series of identifiable shell commands in some cases of CVE-2024-3400 exploitation analyzed by their teams. Commands frequently involved a persistence mechanism they later labeled as the “UPSTYLE” backdoor.[vii]  This python-based program operates by reading commands hidden in error logs generated by 404 requests to the compromised server. The backdoor interprets the requests and writes the output to CSS files on the device. In many cases, Darktrace’s Threat Research team noted clear parallels between shell commands retrieved via HTTP GET request with those directly involving UPSTYLE. There were also matches with some URI patterns identified with the backdoor and requests observed on Darktrace deployments.

Figure 5: HTTP response data containing shell commands potentially relating to the UPSTYLE backdoor.

The presence of these UPSTYLE-related shell commands in response to Palo Alto firewall devices’ HTTP requests provides further evidence for initial exploitation of the CVE. Many bash scripts in examined cases interacted with folders and files likely related to CVE-2024-3400 exploitation. These scripts frequently sought to delete contents of certain folders, such as “/opt/panlogs/tmp/device_telemetry/minute/*” where evidence of exploitation would likely reside. Moreover, recursive removal and copy commands were frequently seen targeting CSS files within the GlobalProtect folder, already noted as the vulnerable element within PAN-OS versions. This evidence is further corroborated by host-based forensic analysis conducted by external researchers.[viii]

Figure 6: PCAP data from investigated system indicating likely defense evasion by removing content on folders where CVE exploitation occurred.

b) Executable File Retrieval

Typically, following command processing, compromised Palo Alto firewall devices proceeded to make web requests for several unusual and potentially malicious files. Many such executables would be retrieved via processed scripts. While there a fair amount of variety in specific executables and binaries obtained, overall, these executables involved either further command tooling such as Sliver C2 or Cobalt Strike payloads, or unknown executables. Affected systems would also employ uncommon ports for HTTP connections, in a likely attempt to evade detection. Extensions featured within the URI, when visible, frequently noted ‘.elf’ (Linux executable) or ‘.exe’ payloads. While most derived hashes did not feature identifiable open-source intelligence (OSINT) details, some samples did have external information tying the sample to specific malware. For example, one such investigation featured a compromised system requesting a file with a hash identified as the Spark malware (backdoor) while another investigated case included a host requesting a known crypto-miner.

Figure 7: PCAP data highlighting compromised system retrieving ELF content from a rare external server running a simple Python HTTP server.
Figure 8: Darktrace model alert logs highlighting a device labeled “Palo Alto” making a HTTP request on an uncommon port for an executable file following likely CVE exploitation.

3. Configuration Data Exfiltration and Unusual HTTP POST Activity

During Darktrace’s investigations, there were also several instances of sensitive data exfiltration from PAN-OS firewall devices. Specifically, targeted systems were observed making HTTP POST requests via destination port 80 to rare external endpoints that OSINT sources associate with CVE-2024-3400 exploitation and activity. PCAP analysis of such HTTP requests revealed that they often contained sensitive configuration details of the targeted Palo Alto firewall devices, including the IP address, default gateway, domain, users, superusers, and password hashes, to name only a few. Threat actors frequently utilized Target URIs such as “/upload” in their HTTP POST requests of this multi-part boundary form data. Again, the User-Agent headers of these HTTP requests largely involved versions of cURL, typically 7.6.1, and wget.

Figure 9: PCAP datahighlighting Palo Alto Firewall device running vulnerable version of PAN-OSposting configuration details to rare external services via HTTP.
Figure 10: Model alert logs highlighting a Palo Alto firewall device performing HTTP POSTs to a rare external IP, without a prior hostname lookup, on an uncommon port using a URI associated with configuration data exfiltration across analyzed incidents
Figure 11: Examples of TargetURIs of HTTP POST requests involving base64 encoded IPs and potential dataegress.

4. Ongoing C2 and Miscellaneous Activity

Lastly, a smaller number of affected Palo Alto firewall devices were seen engaging in repeated beaconing and/or C2 communication via both encrypted and unencrypted protocols during and following the initial series of kill chain events. Such encrypted channels typically involved protocols such as TLS/SSL and SSH. This activity likely represented ongoing communication of targeted systems with attacker infrastructure. Model alerts typically highlighted unusual levels of repeated external connectivity to rare external IP addresses over varying lengths of time. In some investigated incidents, beaconing activity consisted of hundreds of thousands of connections over several days.

Figure 12:  Advanced search details highlighting high levels of ongoing external communication to endpoints associated with C2 infrastructure exploiting CVE-2024-3400.

Some beaconing activity appears to have involved the use of the WebSocket protocol, as indicated by the appearance of “/ws” URIs and validated within packet captures. Such connections were then upgraded to an encrypted connection.

Figure 13:  PCAP highlighting use of WebSocket protocol to engage in ongoing external connectivity to likely C2 infrastructure following CVE-2024-3400 compromise.

While not directly visible in all the deployments, some investigations also yielded evidence of attempts at further post-exploitation activity. For example, a handful of the analyzed binaries that were downloaded by examined devices had OSINT information suggesting a relation to crypto-mining malware strains. However, crypto-mining activity was not directly observed at this time. Furthermore, several devices also triggered model alerts relating to brute-forcing activity via several authentication protocols (namely, Keberos and RADIUS) during the time of compromise. This brute-force activity likely represented attempts to move laterally from the affected firewall system to deeper parts of the network.

Figure 14: Model alert logs noting repeated SSL connectivity to a Sliver C2-affiliated endpoint in what likely constitutes C2 connectivity.
Figure 15: Model alert logs featuring repeated RADIUS login failures from a compromised PAN-OS device using generic usernames, suggesting brute-force activity.

Conclusion

Between April and late May 2024, Darktrace’s SOC and Threat Research teams identified several instances of likely PAN-OS CVE-2024-3400 exploitation across the Darktrace customer base. The subsequent investigation yielded four major themes that categorize the observed network-based post-exploitation activity. These major themes were exploit validation activity, retrieval of binaries and shell scripts, data exfiltration via HTTP POST activity, and ongoing C2 communication with rare external endpoints. The insights shared in this article will hopefully contribute to the ongoing discussion within the cybersecurity community about how to handle the likely continued exploitation of this vulnerability. Moreover, this article may also help cybersecurity professionals better respond to future exploitation of not only Palo Alto PAN-OS firewall devices, but also of edge devices more broadly.

Threat actors will continue to discover and leverage new CVEs impacting edge infrastructure. Since it is not yet known which CVEs threat actors will exploit next, relying on rules and signatures for the detection of exploitation of such CVEs is not a viable approach. Darktrace’s anomaly-based approach to threat detection, however, is well positioned to robustly adapt to threat actors’ changing methods, since although threat actors can change the CVEs they exploit, they cannot change the fact that their exploitation of CVEs results in highly unusual patterns of activity.

Credit to Adam Potter, Cyber Analyst, Sam Lister, Senior Cyber Analyst

Appendices

Indicators of Compromise

Indicator – Type – Description

94.131.120[.]80              IP             C2 Endpoint

94.131.120[.]80:53/?src=[REDACTED]=hour=root                  URL        C2/Exfiltration Endpoint

134.213.29[.]14/?src=[REDACTED]min=root             URL        C2/Exfiltration Endpoint

134.213.29[.]14/grep[.]mips64            URL        Payload

134.213.29[.]14/grep[.]x86_64             URL        Payload

134.213.29[.]14/?deer               URL        Payload

134.213.29[.]14/?host=IDS   URL        Payload

134.213.29[.]14/ldr[.]sh           URL        Payload

91ebcea4e6d34fd6e22f99713eaf67571b51ab01  SHA1 File Hash               Payload

185.243.115[.]250/snmpd2[.]elf        URL        Payload

23.163.0[.]111/com   URL        Payload

80.92.205[.]239/upload            URL        C2/Exfiltration Endpoint

194.36.171[.]43/upload            URL        C2/Exfiltration Endpoint

update.gl-protect[.]com          Hostname         C2 Endpoint

update.gl-protect[.]com:63869/snmpgp      URL        Payload

146.70.87[.]237              IP address         C2 Endpoint

146.70.87[.]237:63867/snmpdd         URL        Payload

393c41b3ceab4beecf365285e8bdf0546f41efad   SHA1 File Hash               Payload

138.68.44[.]59/app/r URL        Payload

138.68.44[.]59/app/clientr     URL        Payload

138.68.44[.]59/manage            URL        Payload

72.5.43[.]90/patch      URL        Payload

217.69.3[.]218                 IP             C2 Endpoint

5e8387c24b75c778c920f8aa38e4d3882cc6d306                  SHA1 File Hash               Payload

217.69.3[.]218/snmpd[.]elf   URL        Payload

958f13da6ccf98fcaa270a6e24f83b1a4832938a    SHA1 File Hash               Payload

6708dc41b15b892279af2947f143af95fb9efe6e     SHA1 File Hash               Payload

dc50c0de7f24baf03d4f4c6fdf6c366d2fcfbe6c       SHA1 File Hash               Payload

109.120.178[.]253:10000/data[.]txt                  URL        Payload

109.120.178[.]253:10000/bin[.]txt   URL        Payload

bc9dc2e42654e2179210d98f77822723740a5ba6                 SHA1 File Hash               Payload

109.120.178[.]253:10000/123              URL        Payload

65283921da4e8b5eabb926e60ca9ad3d087e67fa                 SHA1 File Hash               Payload

img.dxyjg[.]com/6hiryXjZN0Mx[.]sh                  URL        Payload

149.56.18[.]189/IC4nzNvf7w/2[.]txt                 URL        Payload

228d05fd92ec4d19659d71693198564ae6f6b117 SHA1 File Hash               Payload

54b892b8fdab7c07e1e123340d800e7ed0386600                 SHA1 File Hash               Payload

165.232.121[.]217/rules          URL        Payload

165.232.121[.]217/app/request          URL        Payload

938faec77ebdac758587bba999e470785253edaf SHA1 File Hash               Payload

165.232.121[.]217/app/request63   URL        Payload

165.232.121[.]217:4443/termite/165.232.121[.]217             URL        Payload

92.118.112[.]60/snmpd2[.]elf               URL        Payload

2a90d481a7134d66e8b7886cdfe98d9c1264a386                 SHA1 File Hash               Payload

92.118.112[.]60/36shr[.]txt   URL        Payload

d6a33673cedb12811dde03a705e1302464d8227f                 SHA1 File Hash               Payload

c712712a563fe09fa525dfc01ce13564e3d98d67  SHA1 File Hash               Payload

091b3b33e0d1b55852167c3069afcdb0af5e5e79 SHA1 File Hash               Payload

5eebf7518325e6d3a0fd7da2c53e7d229d7b74b6                  SHA1 File Hash               Payload

183be7a0c958f5ed4816c781a2d7d5aa8a0bca9f SHA1 File Hash               Payload

e7d2f1224546b17d805617d02ade91a9a20e783e                 SHA1 File Hash               Payload

e6137a15df66054e4c97e1f4b8181798985b480d SHA1 File Hash               Payload

95.164.7[.]33:53/sea[.]png    URL        Payload

95.164.7[.]33/rules     URL        Payload

95.164.7[.]33:53/lb64                URL        Payload

c2bc9a7657bea17792048902ccf2d77a2f50d2d7 SHA1 File Hash               Payload

923369bbb86b9a9ccf42ba6f0d022b1cd4f33e9d SHA1 File Hash               Payload

52972a971a05b842c6b90c581b5c697f740cb5b9                 SHA1 File Hash               Payload

95d45b455cf62186c272c03d6253fef65227f63a    SHA1 File Hash               Payload

322ec0942cef33b4c55e5e939407cd02e295973e                  SHA1 File Hash               Payload

6335e08873b4ca3d0eac1ea265f89a9ef29023f2  SHA1 File Hash               Payload

134.213.29[.]14              IP             C2 Endpoint

185.243.115[.]250       IP             C2 Endpoint

80.92.205[.]239              IP             C2 Endpoint

194.36.171[.]43              IP             C2 Endpoint

92.118.112[.]60              IP             C2 Endpoint

109.120.178[.]253       IP             C2 Endpoint

23.163.0[.]111                 IP             C2 Endpoint

72.5.43[.]90     IP             C2 Endpoint

165.232.121[.]217       IP             C2 Endpoint

8.210.242[.]112              IP             C2 Endpoint

149.56.18[.]189              IP             C2 Endpoint

95.164.7[.]33  IP             C2 Endpoint

138.68.44[.]59                 IP             C2 Endpoint

Img[.]dxyjg[.]com         Hostname         C2 Endpoint

Darktrace Model Alert Coverage

·      Anomalous File / Masqueraded File Transfer

·      Anomalous File / EXE from Rare External Location

·      Anomalous File / Multiple EXE from Rare External Locations

·      Anomalous File / Script from Rare External Location

·      Anomalous File / Script and EXE from Rare External

·      Anomalous File / Suspicious Octet Stream Download

·      Anomalous File / Numeric File Download

·      Anomalous Connection / Application Protocol on Uncommon Port

·      Anomalous Connection / New User Agent to IP Without Hostname

·      Anomalous Connection / Posting HTTP to IP Without Hostname

·      Anomalous Connection / Multiple Failed Connections to Rare Endpoint

·      Anomalous Connection / Suspicious Self-Signed SSL

·      Anomalous Connection / Anomalous SSL without SNI to New External

·      Anomalous Connection / Multiple Connections to New External TCP Port

·      Anomalous Connection / Rare External SSL Self-Signed

·      Anomalous Server Activity / Outgoing from Server

·      Anomalous Server Activity / Rare External from Server

·      Compromise / SSH Beacon

·      Compromise / Beacon for 4 Days

·      Compromise / Sustained TCP Beaconing Activity To Rare Endpoint

·      Compromise / High Priority Tunnelling to Bin Services

·      Compromise / Sustained SSL or HTTP Increase

·      Compromise / Connection to Suspicious SSL Server

·      Compromise / Suspicious File and C2

·      Compromise / Large Number of Suspicious Successful Connections

·      Compromise / Slow Beaconing Activity To External Rare

·      Compromise / HTTP Beaconing to New Endpoint

·      Compromise / SSL or HTTP Beacon

·      Compromise / Suspicious HTTP and Anomalous Activity

·      Compromise / Beacon to Young Endpoint

·      Compromise / High Volume of Connections with Beacon Score

·      Compromise / Suspicious Beaconing Behaviour

·      Compliance / SSH to Rare External Destination

·      Compromise / HTTP Beaconing to Rare Destination

·      Compromise / Beaconing Activity To External Rare

·      Device::New User Agent

·      Device / Initial Breach Chain Compromise

·      Device / Multiple C2 Model Breaches

MITRE ATTACK Mapping

Tactic – Technique

Initial Access  T1190 – Exploiting Public-Facing Application

Execution           T1059.004 – Command and Scripting Interpreter: Unix Shell

Persistence      T1543.002 – Create or Modify System Processes: Systemd Service

Defense Evasion           T1070.004 – Indicator Removal: File Deletion

Credential Access       T1110.001 – Brute Force: Password Guessing

Discovery           T1083 – File and System Discovery

T1057 – Process Discovery

Collection         T1005 – Data From Local System

Command and Control             T1071.001 – Application Layer Protocol:  Web Protocols

T1573.002 – Encrypted Channel: Asymmetric Cryptography

T1571 – Non-Standard Port

T1105 – Ingress Tool Transfer

Exfiltration         T1041 – Exfiltration over C2 Protocol

T1048.002 - Exfiltration Over Alternative Protocol: Exfiltration Over Asymmetric Encrypted Non-C2 Protocol

References

[i]  https://www.ncsc.gov.uk/blog-post/products-on-your-perimeter

[ii] https://security.paloaltonetworks.com/CVE-2024-3400

[iii] https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/

[iv] https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/

[v] https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/

[vi] https://security.paloaltonetworks.com/CVE-2024-3400

[vii] https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/

[viii] https://www.volexity.com/blog/2024/05/15/detecting-compromise-of-cve-2024-3400-on-palo-alto-networks-globalprotect-devices/

Continue reading
About the author
Adam Potter
Cyber Analyst
Our ai. Your data.

Elevate your cyber defenses with Darktrace AI

Start your free trial
Darktrace AI protecting a business from cyber threats.