Darktrace has discovered a significant number of cases involving a successful exploit of GitLab servers — a common open source software used by developers. The vulnerability, tracked as CVE-2021-22205, allows an unauthenticated, remote attacker to execute arbitrary commands as the ‘git’ user, giving them full access to the repository, including deleting, modifying, and exfiltrating source code.
In each case discovered by Darktrace AI, attackers successfully exploited servers and ran crypto-mining malware. However, this vulnerability opens the door into a wider range of possibilities, including data exfiltration, ransomware, and supply chain attacks.
The flaw was fixed on April 14, 2021, but recent research has revealed that this vulnerability is still exploitable with over 30,000 GitLab servers remaining unpatched.
The vulnerability has affected customers in every corner of the world, with Darktrace customers in the US, EMEA and APAC all targeted. Affected industries include technology, transportation, and education.
Attack details
The cases detailed below generally follow the same pattern. First, user accounts with admin privileges are registered on a publicly accessible GitLab server belonging to an unnamed customer. This is followed by a remote execution of commands that grant the rogue accounts elevated permissions.
Figure 1: Multiple model breaches firing on an unusual data egress event on October 30, which resulted in a Proactive Threat Notification model breach.
After multiple model breaches on malicious EXE downloads and command and control (C2) activities with the TOR network, the organization received a Proactive Threat Notification (PTN) from Darktrace that immediately alerted them to the issue. This enabled the customer to remove the compromised device from the network.
The next day, Darktrace discovered cryptocurrency mining occurring on a compromised server that was communicating on a non-standard port. This triggered alerts to the customer through Darktrace’s Proactive Threat Notification service, immediately escalating the threat to their security team.
Figure 2: Multiple cryptocurrency mining model breaches from the same server firing on November 3.
The related breaches include scripts from rare external locations and rare endpoints (endpoints that have never been contacted by the breach devices in the past). Not surprisingly, the endpoints in question are crypto-mining pools.
It is important to note that this GitLab vulnerability represents only the initial attack vector, which could result in a number of scenarios. In the customer environment detailed above, crypto-mining has occurred; however, exploitation of this vulnerability could serve as the first stage of a more destructive ransomware attack, or result in stolen intellectual property.
Lastly, throughout the compromises identified across Darktrace’s customer base, it appears that the Interactsh tool was leveraged by the threat actors in the attack. Interactsh is an open-source tool for out of band data transfers and validation of security flaws, and it is commonly used by both researchers and hackers. Darktrace was easily able to identify this tool as part of the larger threat.
Cyber AI Analyst investigates
Darktrace’s Cyber AI Analyst launched an immediate investigation, stitching together different events across a five-day period and revealing four stages of the attack. This presented the security team with all the information they needed to perform effective investigation and clean up, including isolating the infected devices.
Figure 3: Cyber AI Analyst automatically investigates, piecing together the events into a single narrative.
In another customer environment, Cyber AI Analyst was again able to piece together multiple security events to present a coherent security narrative, determining that the suspicious file downloads likely contained malicious software, and recommending immediate attention from security staff.
Figure 4: In a different case, Cyber AI Analyst surfaces a summary and key metrics around the suspicious file downloads.
Cyber AI Analyst made stellar detections and Proactive Threat Notification alerted affected clients ASAP. Clients were then supported through Ask the Expert (ATE) services. There has been no evidence of ransomware thus far, but these types of attacks typically gain a foothold on Internet-exposed servers and then pivot internally to deploy ransomware.
In a third example with a separate customer, Cyber AI Analyst stitched together six different security events into a single security narrative. Here, Darktrace’s technology was able to connect the dots between C2 behavior, suspicious file downloads, unusual connections, and Tor activity, eventually leading to its discovery of cryptocurrency mining.
Cyber AI Analyst specifically identified GitLab in the suspicious file downloads from a rare external endpoint. The fact that Darktrace was able to identify this in the context of a holistic view of threatening activity across this organization’s digital ecosystem — stretching from suspicious SSL connections to the eventual crypto-mining activity — presents a remarkable picture of Cyber AI Analyst in action.
Figure 5: Cyber AI Analyst identifying the GitLab activity in the context of the wider security narrative.
Concluding thoughts
Though the patch was released in April, over 50% of deployments remain unpatched. There are potential reasons why they remain unpatched — overworked security staff, or simply negligence.
Even when CVEs are mapped and patched promptly, however, novel and never-before-seen attacks can still slip through the cracks. Before the Gitlab flaw was publicly disclosed and fixed, this vulnerability was a zero-day.
And so, rather than wait for CVEs to be publicly disclosed, organizations would be prudent to adopt technologies that can detect and respond to emerging attacks at their earliest stages — regardless of whether they are exploiting known or unknown vulnerabilities.
At Darktrace we talk a lot about the problems novel and unknown threats pose for traditional security solutions. This case shows that even when a threat is known for over six months, difficulties in implementing and rolling out patching mean it can still cause issues.
Thanks to Darktrace’s AI continuously monitoring the behavior of our customer’s devices, they were able to identify the threat at its earliest stages, before it could develop into something more disruptive like ransomware. And had the customers had Darktrace Antigena configured, the technology would have responded autonomously to contain the malicious behavior before the attackers could get past stage one.
Thanks to Darktrace analyst Waseem Akhter for his insights on the above threat find.
Learn more about Darktrace’s Self-Learning AI
Technical details
Proactive Threat Notification model detections:
- Compromise / Anomalous File then Tor
- Compromise / High Priority Crypto Currency Mining
- Device / Initial Breach Chain Compromise
- Device / Large Number of Model Breaches from Critical Network Device
- Unusual Activity / Enhanced Unusual External Data Transfer
Other Darktrace model detections:
- Anomalous Connection / Anomalous SSL without SNI to New External
- Anomalous Connection / Application Protocol on Uncommon Port
- Anomalous Connection / Callback on Web Facing Device
- Anomalous Connection / Data Sent to Rare Domain
- Anomalous Connection / New User Agent to IP Without Hostname
- Anomalous Connection / Posting HTTP to IP Without Hostname
- Anomalous File / Multiple EXE from Rare External Locations
- Anomalous File / Internet Facing System File Download
- Anomalous File / Script from Rare Location
- Anomalous Server Activity / Outgoing from Serve
- Compromise / Beaconing Activity To External Rare
- Compliance / Crypto Currency Mining Activity
- Compromise / High Volume of Connections with Beacon Score
- Compromise / Large DNS Volume for Suspicious Domain
- Compromise / Monero Mining
- Compliance / Possible Tor Usage
- Device / Internet Facing Device with High Priority Alert
- Device / Large Number of Model Breaches
- Device / Large Number of Connections to New Endpoints
- Device / Suspicious Domain
- Unusual Activity / Unusual External Data to New IPs
