What is Citrix Bleed?
Since August 2023, cyber threat actors have been actively exploiting one of the most significant critical vulnerabilities disclosed in recent years: Citrix Bleed. Citrix Bleed, also known as CVE-2023-4966, remained undiscovered and even unpatched for several months, resulting in a wide range of security incidents across business and government sectors [1].
How does Citrix Bleed vulnerability work?
The vulnerability, which impacts the Citrix Netscaler Gateway and Netscaler ADC products, allows for outside parties to hijack legitimate user sessions, thereby bypassing password and multifactor authentication (MFA) requirements.
When used as a means of initial network access, the vulnerability has resulted in the exfiltration of sensitive data, as in the case of Xfinity, and even the deployment of ransomware variants including Lockbit [2]. Although Citrix has released a patch to address the vulnerability, slow patching procedures and the widespread use of these products has resulted in the continuing exploitation of Citrix Bleed into 2024 [3].
How Does Darktrace Handle Citrix Bleed?
Darktrace has demonstrated its proficiency in handling the exploitation of Citrix Bleed since it was disclosed back in 2023; its anomaly-based approach allows it to efficiently identify and inhibit post-exploitation activity as soon as it surfaces. Rather than relying upon traditional rules and signatures, Darktrace’s Self-Learning AI enables it to understand the subtle deviations in a device’s behavior that would indicate an emerging compromise, thus allowing it to detect anomalous activity related to the exploitation of Citrix Bleed.
In late 2023, Darktrace identified an instance of Citrix Bleed exploitation on a customer network. As this customer had subscribed to the Proactive Threat Notification (PTN) service, the suspicious network activity surrounding the compromise was escalated to Darktrace’s Security Operation Center (SOC) for triage and investigation by Darktrace Analysts, who then alerted the customer’s security team to the incident.
Darktrace’s Coverage
Initial Access and Beaconing of Citrix Bleed
Darktrace’s initial detection of indicators of compromise (IoCs) associated with the exploitation of Citrix Bleed actually came a few days prior to the SOC alert, with unusual external connectivity observed from a critical server. The suspicious connection in question, a SSH connection to the rare external IP 168.100.9[.]137, lasted several hours and utilized the Windows PuTTY client. Darktrace also identified an additional suspicious IP, namely 45.134.26[.]2, attempting to contact the server. Both rare endpoints had been linked with the exploitation of the Citrix Bleed vulnerability by multiple open-source intelligence (OSINT) vendors [4] [5].
![Darktrace model alert highlighting an affected device making an unusual SSH connection to 168.100.9[.]137 via port 22.](https://cdn.prod.website-files.com/626ff4d25aca2edf4325ff97/665644c9d902374aff832499_Screenshot%202024-05-28%20at%201.55.25%20PM.avif)
As Darktrace is designed to identify network-level anomalies, rather than monitor edge infrastructure, the initial exploitation via the typical HTTP buffer overflow associated with this vulnerability fell outside the scope of Darktrace’s visibility. However, the aforementioned suspicious connectivity likely constituted initial access and beaconing activity following the successful exploitation of Citrix Bleed.
Command and Control (C2) and Payload Download
Around the same time, Darktrace also detected other devices on the customer’s network conducting external connectivity to various endpoints associated with remote management and IT services, including Action1, ScreenConnect and Fixme IT. Additionally, Darktrace observed devices downloading suspicious executable files, including “tniwinagent.exe”, which is associated with the tool Total Network Inventory. While this tool is typically used for auditing and inventory management purposes, it could also be leveraged by attackers for the purpose of lateral movement.
Defense Evasion
In the days surrounding this compromise, Darktrace observed multiple devices engaging in potential defense evasion tactics using the ScreenConnect and Fixme IT services. Although ScreenConnect is a legitimate remote management tool, it has also been used by threat actors to carry out C2 communication [6]. ScreenConnect itself was the subject of a separate critical vulnerability which Darktrace investigated in early 2024. Meanwhile, CISA observed that domains associated with Fixme It (“fixme[.]it”) have been used by threat actors attempting to exploit the Citrix Bleed vulnerability [7].
Reconnaissance and Lateral Movement
A few days after the detection of the initial beaconing communication, Darktrace identified several devices on the customer’s network carrying out reconnaissance and lateral movement activity. This included SMB writes of “PSEXESVC.exe”, network scanning, DCE-RPC binds of numerous internal devices to IPC$ shares and the transfer of compromise-related tools. It was at this point that Darktrace’s Self-Learning AI deemed the activity to be likely indicative of an ongoing compromise and several Enhanced Monitoring models alerted, triggering the aforementioned PTNs and investigation by Darktrace’s SOC.
Darktrace observed a server on the network initiating a wide range of connections to more than 600 internal IPs across several critical ports, suggesting port scanning, as well as conducting unexpected DCE-RPC service control (svcctl) activity on multiple internal devices, amongst them domain controllers. Additionally, several binds to server service (srvsvc) and security account manager (samr) endpoints via IPC$ shares on destination devices were detected, indicating further reconnaissance activity. The querying of these endpoints was also observed through RPC commands to enumerate services running on the device, as well as Security Account Manager (SAM) accounts.
Darktrace also identified devices performing SMB writes of the WinRAR data compression tool, in what likely represented preparation for the compression of data prior to data exfiltration. Further SMB file writes were observed around this time including PSEXESVC.exe, which was ultimately used by attackers to conduct remote code execution, and one device was observed making widespread failed NTLM authentication attempts on the network, indicating NTLM brute-forcing. Darktrace observed several devices using administrative credentials to carry out the above activity.
In addition to the transfer of tools and executables via SMB, Darktrace also identified numerous devices deleting files through SMB around this time. In one example, an MSI file associated with the patch management and remediation service, Action1, was deleted by an attacker. This legitimate security tool, if leveraged by attackers, could be used to uncover additional vulnerabilities on target networks.
A server on the customer’s network was also observed writing the file “m.exe” to multiple internal devices. OSINT investigation into the executable indicated that it could be a malicious tool used to prevent antivirus programs from launching or running on a network [8].
Impact and Data Exfiltration
Following the initial steps of the breach chain, Darktrace observed numerous devices on the customer’s network engaging in data exfiltration and impact events, resulting in additional PTN alerts and a SOC investigation into data egress. Specifically, two servers on the network proceeded to read and download large volumes of data via SMB from multiple internal devices over the course of a few hours. These hosts sent large outbound volumes of data to MEGA file storage sites using TLS/SSL over port 443. Darktrace also identified the use of additional file storage services during this exfiltration event, including 4sync, file[.]io, and easyupload[.]io. In total the threat actor exfiltrated over 8.5 GB of data from the customer’s network.
Finally, Darktrace detected a user account within the customer’s Software-as-a-Service (SaaS) environment conducting several suspicious Office365 and AzureAD actions from a rare IP for the network, including uncommon file reads, creations and the deletion of a large number of files.
Unfortunately for the customer in this case, Darktrace RESPOND™ was not enabled on the network and the post-exploitation activity was able to progress until the customer was made aware of the attack by Darktrace’s SOC team. Had RESPOND been active and configured in autonomous response mode at the time of the attack, it would have been able to promptly contain the post-exploitation activity by blocking external connections, shutting down any C2 activity and preventing the download of suspicious files, blocking incoming traffic, and enforcing a learned ‘pattern of life’ on offending devices.
Conclusion
Given the widespread use of Netscaler Gateway and Netscaler ADC, Citrix Bleed remains an impactful and potentially disruptive vulnerability that will likely continue to affect organizations who fail to address affected assets. In this instance, Darktrace demonstrated its ability to track and inhibit malicious activity stemming from Citrix Bleed exploitation, enabling the customer to identify affected devices and enact their own remediation.
Darktrace’s anomaly-based approach to threat detection allows it to identify such post-exploitation activity resulting from the exploitation of a vulnerability, regardless of whether it is a known CVE or a zero-day threat. Unlike traditional security tools that rely on existing threat intelligence and rules and signatures, Darktrace’s ability to identify the subtle deviations in a compromised device’s behavior gives it a unique advantage when it comes to identifying emerging threats.
Credit to Vivek Rajan, Cyber Analyst, Adam Potter, Cyber Analyst
Appendices
Darktrace Model Coverage
Device / Suspicious SMB Scanning Activity
Device / ICMP Address Scan
Device / Possible SMB/NTLM Reconnaissance
Device / Network Scan
Device / SMB Lateral Movement
Device / Possible SMB/NTLM Brute Force
Device / Suspicious Network Scan Activity
User / New Admin Credentials on Server
Anomalous File / Internal::Unusual Internal EXE File Transfer
Compliance / SMB Drive Write
Device / New or Unusual Remote Command Execution
Anomalous Connection / New or Uncommon Service Control
Anomalous Connection / Rare WinRM Incoming
Anomalous Connection / Unusual Admin SMB Session
Device / Unauthorised Device
User / New Admin Credentials on Server
Anomalous Server Activity / Outgoing from Server
Device / Long Agent Connection to New Endpoint
Anomalous Connection / Multiple Connections to New External TCP Port
Device / New or Uncommon SMB Named Pipe
Device / Multiple Lateral Movement Model Breaches
Device / Large Number of Model Breaches
Compliance / Remote Management Tool On Server
Device / Anomalous RDP Followed By Multiple Model Breaches
Device / SMB Session Brute Force (Admin)
Device / New User Agent
Compromise / Large Number of Suspicious Failed Connections
Unusual Activity / Unusual External Data Transfer
Unusual Activity / Enhanced Unusual External Data Transfer
Device / Increased External Connectivity
Unusual Activity / Unusual External Data to New Endpoints
Anomalous Connection / Data Sent to Rare Domain
Anomalous Connection / Uncommon 1 GiB Outbound
Anomalous Connection / Active Remote Desktop Tunnel
Anomalous Server Activity / Anomalous External Activity from Critical Network Device
Compliance / Possible Unencrypted Password File On Server
Anomalous Connection / Suspicious Read Write Ratio and Rare External
Device / Reverse DNS Sweep]
Unusual Activity / Possible RPC Recon Activity
Anomalous File / Internal::Executable Uploaded to DC
Compliance / SMB Version 1 Usage
Darktrace AI Analyst Incidents
Scanning of Multiple Devices
Suspicious Remote Service Control Activity
SMB Writes of Suspicious Files to Multiple Devices
Possible SSL Command and Control to Multiple Devices
Extensive Suspicious DCE-RPC Activity
Suspicious DCE-RPC Activity
Internal Downloads and External Uploads
Unusual External Data Transfer
Unusual External Data Transfer to Multiple Related Endpoints
MITRE ATT&CK Mapping
Technique – Tactic – ID – Sub technique of
Network Scanning – Reconnaissance - T1595 - T1595.002
Valid Accounts – Defense Evasion, Persistence, Privilege Escalation, Initial Access – T1078 – N/A
Remote Access Software – Command and Control – T1219 – N/A
Lateral Tool Transfer – Lateral Movement – T1570 – N/A
Data Transfers – Exfiltration – T1567 – T1567.002
Compressed Data – Exfiltration – T1030 – N/A
NTLM Brute Force – Brute Force – T1110 - T1110.001
AntiVirus Deflection – T1553 - NA
Ingress Tool Transfer - COMMAND AND CONTROL - T1105 - NA
Indicators of Compromise (IoCs)
204.155.149[.]37 – IP – Possible Malicious Endpoint
199.80.53[.]177 – IP – Possible Malicious Endpoint
168.100.9[.]137 – IP – Malicious Endpoint
45.134.26[.]2 – IP – Malicious Endpoint
13.35.147[.]18 – IP – Likely Malicious Endpoint
13.248.193[.]251 – IP – Possible Malicious Endpoint
76.223.1[.]166 – IP – Possible Malicious Endpoint
179.60.147[.]10 – IP – Likely Malicious Endpoint
185.220.101[.]25 – IP – Likely Malicious Endpoint
141.255.167[.]250 – IP – Malicious Endpoint
106.71.177[.]68 – IP – Possible Malicious Endpoint
cat2.hbwrapper[.]com – Hostname – Likely Malicious Endpoint
aj1090[.]online – Hostname – Likely Malicious Endpoint
dc535[.]4sync[.]com – Hostname – Likely Malicious Endpoint
204.155.149[.]140 – IP - Likely Malicious Endpoint
204.155.149[.]132 – IP - Likely Malicious Endpoint
204.155.145[.]52 – IP - Likely Malicious Endpoint
204.155.145[.]49 – IP - Likely Malicious Endpoint
References
- https://www.axios.com/2024/01/02/citrix-bleed-security-hacks-impact
- https://www.csoonline.com/article/1267774/hackers-steal-data-from-millions-of-xfinity-customers-via-citrix-bleed-vulnerability.html
- https://www.cybersecuritydive.com/news/citrixbleed-security-critical-vulnerability/702505/
- https://www.virustotal.com/gui/ip-address/168.100.9.137
- https://www.virustotal.com/gui/ip-address/45.134.26.2
- https://www.trendmicro.com/en_us/research/24/b/threat-actor-groups-including-black-basta-are-exploiting-recent-.html
- https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-325a
- https://www.file.net/process/m.exe.html
.avif)