The State of AI in Cybersecurity: Understanding AI Technologies
24
Jul 2024
Part 4: This blog explores the findings from Darktrace’s State of AI Cybersecurity Report on security professionals' understanding of the different types of AI used in security programs. Get the latest insights into the evolving challenges, growing demand for skilled professionals, and the need for integrated security solutions by downloading the full report.
About the State of AI Cybersecurity Report
Darktrace surveyed 1,800 CISOs, security leaders, administrators, and practitioners from industries around the globe. Our research was conducted to understand how the adoption of new AI-powered offensive and defensive cybersecurity technologies are being managed by organizations.
How familiar are security professionals with supervised machine learning
Just 31% of security professionals report that they are “very familiar” with supervised machine learning.
Many participants admitted unfamiliarity with various AI types. Less than one-third felt "very familiar" with the technologies surveyed: only 31% with supervised machine learning and 28% with natural language processing (NLP).
Most participants were "somewhat" familiar, ranging from 46% for supervised machine learning to 36% for generative adversarial networks (GANs). Executives and those in larger organizations reported the highest familiarity.
Combining "very" and "somewhat" familiar responses, 77% had familiarity with supervised machine learning, 74% generative AI, and 73% NLP. With generative AI getting so much media attention, and NLP being the broader area of AI that encompasses generative AI, these results may indicate that stakeholders are understanding the topic on the basis of buzz, not hands-on work with the technologies.
If defenders hope to get ahead of attackers, they will need to go beyond supervised learning algorithms trained on known attack patterns and generative AI. Instead, they’ll need to adopt a comprehensive toolkit comprised of multiple, varied AI approaches—including unsupervised algorithms that continuously learn from an organization’s specific data rather than relying on big data generalizations.
Different types of AI
Different types of AI have different strengths and use cases in cyber security. It’s important to choose the right technique for what you’re trying to achieve.
Supervised machine learning: Applied more often than any other type of AI in cyber security. Trained on human attack patterns and historical threat intelligence.
Large language models (LLMs): Applies deep learning models trained on extremely large data sets to understand, summarize, and generate new content. Used in generative AI tools.
Natural language processing (NLP): Applies computational techniques to process and understand human language.
Unsupervised machine learning: Continuously learns from raw, unstructured data to identify deviations that represent true anomalies.
What impact will generative AI have on the cybersecurity field?
More than half of security professionals (57%) believe that generative AI will have a bigger impact on their field over the next few years than other types of AI.
Security stakeholders are highly aware of generative AI and LLMs, viewing them as pivotal to the field's future. Generative AI excels at abstracting information, automating tasks, and facilitating human-computer interaction. However, LLMs can "hallucinate" due to training data errors and are vulnerable to prompt injection attacks. Despite improvements in securing LLMs, the best cyber defenses use a mix of AI types for enhanced accuracy and capability.
AI education is crucial as industry expectations for generative AI grow. Leaders and practitioners need to understand where and how to use AI while managing risks. As they learn more, there will be a shift from generative AI to broader AI applications.
Do security professionals fully understand the different types of AI in security products?
Only 26% of security professionals report a full understanding of the different types of AI in use within security products.
Confusion is prevalent in today’s marketplace. Our survey found that only 26% of respondents fully understand the AI types in their security stack, while 31% are unsure or confused by vendor claims. Nearly 65% believe generative AI is mainly used in cybersecurity, though it’s only useful for identifying phishing emails. This highlights a gap between user expectations and vendor delivery, with too much focus on generative AI.
Key findings include:
Executives and managers report higher understanding than practitioners.
Larger organizations have better understanding due to greater specialization.
As AI evolves, vendors are rapidly introducing new solutions faster than practitioners can learn to use them. There's a strong need for greater vendor transparency and more education for users to maximize the technology's value.
To help ease confusion around AI technologies in cybersecurity, Darktrace has released the CISO’s Guide to Cyber AI. A comprehensive white paper that categorizes the different applications of AI in cybersecurity. Download the White Paper here.
Do security professionals believe generative AI alone is enough to stop zero-day threats?
No! 86% of survey participants believe generative AI alone is NOT enough to stop zero-day threats
This consensus spans all geographies, organization sizes, and roles, though executives are slightly less likely to agree. Asia-Pacific participants agree more, while U.S. participants agree less.
Despite expecting generative AI to have the most impact, respondents recognize its limited security use cases and its need to work alongside other AI types. This highlights the necessity for vendor transparency and varied AI approaches for effective security across threat prevention, detection, and response.
Stakeholders must understand how AI solutions work to ensure they offer advanced, rather than outdated, threat detection methods. The survey shows awareness that old methods are insufficient.
Oops! Something went wrong while submitting the form.
Newsletter
Stay ahead of threats with the Darktrace blog newsletter
Get the latest insights from the cybersecurity landscape, including threat trends, incident analysis, and the latest Darktrace product developments – delivered directly to your inbox, monthly.
Thanks, your request has been received
A member of our team will be in touch with you shortly.
Oops! Something went wrong while submitting the form.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Phishing and Persistence: Darktrace’s Role in Defending Against a Sophisticated Account Takeover
The exploitation of SaaS platforms
As businesses continue to grow and evolve, the need for sharing ideas through productivity and cloud Software-as-a-Service (SaaS) platforms is becoming increasingly crucial. However, these platforms have also become prime targets for cyber attackers.
Threat actors often exploit these widely-used services to gain unauthorized access, steal sensitive information, and disrupt business operations. The growing reliance on SaaS platforms makes them attractive entry points for cybercriminals, who use sophisticated techniques such as phishing, social engineering, and malware to compromise these systems.
Services like Microsoft 365 are regularly targeted by threat actors looking for an entry point into an organization’s environment to carry out malicious activities. Securing these platforms is crucial to protect business data and ensure operational continuity.
Darktrace / EMAIL detection of the phishing attack
In a recent case, Darktrace observed a customer in the manufacturing sector receiving a phishing email that led to a threat actor logging in and creating an email rule. Threat actors often create email rules to move emails to their inbox, avoiding detection. Additionally, Darktrace detected a spoofed domain registered by the threat actor. Despite already having access to the customer’s SaaS account, the actor seemingly registered this domain to maintain persistence on the network, allowing them to communicate with the spoofed domain and conduct further malicious activity.
Darktrace / EMAIL can help prevent compromises like this one by blocking suspicious emails as soon as they are identified. Darktrace’s AI-driven email detection and response recognizes anomalies that might indicate phishing attempts and applies mitigative actions autonomously to prevent the escalation of an attack.
Unfortunately, in this case, Darktrace was not configured in Autonomous Response mode at the time of the attack, meaning actions had to be manually applied by the customer’s security team. Had it been fully enabled, it would have held the emails, preventing them from reaching the intended recipient and stopping the attack at its inception.
However, Darktrace’s Managed Threat Detection alerted the Security Operations Center (SOC) team to the compromise, enabling them to thoroughly investigate the incident and notify the customer before further damage could occur.
The Managed Threat Detection service continuously monitors customer networks for suspicious activities that may indicate an emerging threat. When such activities are detected, alerts are sent to Darktrace’s expert Cyber Analysts for triage, significantly speeding up the remediation process.
Attack Overview
On May 2, 2024, Darktrace detected a threat actor targeting a customer in the manufacturing sector then an unusual login to their SaaS environment was observed prior to the creation of a new email rule.
Darktrace immediately identified the login as suspicious due to the rarity of the source IP (31.222.254[.]27) and ASN, coupled with the absence of multi-factor authentication (MFA), which was typically required for this account.
The new email rule was intended to mark emails as read and moved to the ‘Conversation History’ folder for inbound emails from a specific domain. The rule was named “….,,,”, likely the attacker attempting to setup their new rule with an unnoteworthy name to ensure it would not be noticed by the account’s legitimate owner. Likewise, by moving emails from a specific domain to ‘Conversation History’, a folder that is rarely used by most users, any phishing emails sent by that domain would remain undetected by the user.
The domain in question was identified as being newly registered and an example of a typosquat domain. Typosquatting involves registering new domains with intentional misspelling designed to convince users to visit fake, and often malicious, websites. This technique is often used in phishing campaigns to create a sense of legitimacy and trust and deceive users into providing sensitive information. In this case, the suspicious domain closely resembled several of the customer’s internal domains, indicating an attempt to impersonate the organization’s legitimate internal sites to gain the target’s trust. Furthermore, the creation of this lookalike domain suggests that the attack was highly targeted at this specific customer.
Interestingly, the threat actor registered this spoofed domain despite already having account access. This was likely intended to ensure persistence on the network without having to launch additional phishing attacks. Such use of spoofed domain could allow an attacker to maintain a foothold in their target network and escalate their malicious activities without having to regain access to the account. This persistence can be used for various purposes, including data exfiltration, spreading malware, or launching further attacks.
Following this, Darktrace detected a highly anomalous email being sent to the customer’s account from the same location as the initial unsual SaaS login. Darktrace’s anomaly-based detection is able to identify threats that human security teams and traditional signature-based methods might miss. By analyzing the expected behavior of network users, Darktrace can recognize the subtle deviations from the norm that may indicate malicious activity. Unfortunately, in this instance, without Darktrace’s Autonomous Response capability enabled, the phishing email was able to successfully reach the recipient. While Darktrace / EMAIL did suggest that the email should be held from the recipients inbox, the customer was required to manually approve it.
Despite this, the Darktrace SOC team were still able to support the customer as they were subscribed to the Managed Threat Detection service. Following the detection of the highlight anomalous activity surrounding this compromise, namely the unusual SaaS login followed by a new email rule, an alert was sent to the Darktrace SOC for immediate triage, who then contacted the customer directly urging immediate action.
Conclusion
This case underscores the need to secure SaaS platforms like Microsoft 365 against sophisticated cyber threats. As businesses increasingly rely on these platforms, they become prime targets for attackers seeking unauthorized access and disruption.
Darktrace’s anomaly-based detection and response capabilities are crucial in identifying and mitigating such threats. In this instance, Darktrace detected a phishing email that led to a threat actor logging in and creating a suspicious email rule. The actor also registered a spoofed domain to maintain persistence on the network.
Darktrace / EMAIL, with its AI-driven detection and analysis, can block suspicious emails before they reach the intended recipient, preventing attacks at their inception. Meanwhile, Darktrace’s SOC team promptly investigated the activity and alerted the customer to the compromise, enabling them to take immediate action to remediate the issue and prevent any further damage.
Credit to Vivek Rajan (Cyber Security Analyst) and Ryan Traill (Threat Content Lead).
Lifting the Fog: Darktrace’s Investigation into Fog Ransomware
Introduction to Fog Ransomware
As ransomware attacks continue to be launched at an alarming rate, Darktrace’s Threat Research team has identified that familiar strains like Akira, LockBit, and BlackBasta remain among the most prevalent threats impacting its customers, as reported in the First 6: Half-Year Threat Report 2024. Despite efforts by law agencies, like dismantling the infrastructure of cybercriminals and shutting down their operations [2], these groups continue to adapt and evolve.
As such, it is unsurprising that new ransomware variants are regularly being created and launched to get round law enforcement agencies and increasingly adept security teams. One recent example of this is Fog ransomware.
What is Fog ransomware?
Fog ransomware is strain that first appeared in the wild in early May 2024 and has been observed actively using compromised virtual private network (VPN) credentials to gain access to organization networks in the education sector in the United States.
Darktrace's detection of Fog Ransomware
In June 2024, Darktrace observed instances of Fog ransomware across multiple customer environments. The shortest time observed from initial access to file encryption in these attacks was just 2 hours, underscoring the alarming speed with which these threat actors can achieve their objectives.
Darktrace identified key activities typical of a ransomware kill chain, including enumeration, lateral movement, encryption, and data exfiltration. In most cases, Darktrace was able to successfully halt the progression Fog attacks in their early stages by applying Autonomous Response actions such as quarantining affected devices and blocking suspicious external connections.
To effectively illustrate the typical kill chain of Fog ransomware, this blog focuses on customer environments that did not have Darktrace’s Autonomous Response enabled. In these cases, the attack progressed unchecked and reached its intended objectives until the customer received Darktrace’s alerts and intervened.
Darktrace’s Coverage of Fog Ransomware
Initial Intrusion
After actors had successfully gained initial access into customer networks by exploiting compromised VPN credentials, Darktrace observed a series of suspicious activities, including file shares, enumeration and extensive scanning. In one case, a compromised domain controller was detected making outgoing NTLM authentication attempts to another internal device, which was subsequently used to establish RDP connections to a Windows server running Hyper-V.
Given that the source was a domain controller, the attacker could potentially relay the NTLM hash to obtain a domain admin Kerberos Ticket Granting Ticket (TGT). Additionally, incoming NTLM authentication attempts could be triggered by tools like Responder, and NTLM hashes used to encrypt challenge response authentication could be abused by offline brute-force attacks.
Darktrace also observed the use of a new administrative credential on one affected device, indicating that malicious actors were likely using compromised privileged credentials to conduct relay attacks.
Establish Command-and-Control Communication (C2)
In many instances of Fog ransomware investigated by Darktrace’s Threat Research team, devices were observed making regular connections to the remote access tool AnyDesk. This was exemplified by consistent communication with the endpoint “download[.]anydesk[.]com” via the URI “/AnyDesk.exe”. In other cases, Darktrace identified the use of another remote management tool, namely SplashTop, on customer servers.
In ransomware attacks, threat actors often use such legitimate remote access tools to establish command-and-control (C2) communication. The use of such services not only complicates the identification of malicious activities but also enables attackers to leverage existing infrastructure, rather than having to implement their own.
Internal Reconnaissance
Affected devices were subsequently observed making an unusual number of failed internal connections to other internal locations over ports such as 80 (HTTP), 3389 (RDP), 139 (NetBIOS) and 445 (SMB). This pattern of activity strongly indicated reconnaissance scanning behavior within affected networks. A further investigation into these HTTP connections revealed the URIs “/nice ports”/Trinity.txt.bak”, commonly associated with the use of the Nmap attack and reconnaissance tool.
Simultaneously, some devices were observed engaging in SMB actions targeting the IPC$ share and the named pipe “srvsvc” on internal devices. Such activity aligns with the typical SMB enumeration tactics, whereby attackers query the list of services running on a remote host using a NULL session, a method often employed to gather information on network resources and vulnerabilities.
Lateral Movement
As attackers attempted to move laterally through affected networks, Darktrace observed suspicious RDP activity between infected devices. Multiple RDP connections were established to new clients, using devices as pivots to propagate deeper into the networks, Following this, devices on multiple networks exhibited a high volume of SMB read and write activity, with internal share drive file names being appended with the “.flocked” extension – a clear sign of ransomware encryption. Around the same time, multiple “readme.txt” files were detected being distributed across affected networks, which were later identified as ransom notes.
Further analysis of the ransom note revealed that it contained an introduction to the Fog ransomware group, a summary of the encryption activity that had been carried out, and detailed instructions on how to communicate with the attackers and pay the ransom.
Data Exfiltration
In one of the cases of Fog ransomware, Darktrace’s Threat Research team observed potential data exfiltration involving the transfer of internal files to an unusual endpoint associated with the MEGA file storage service, “gfs302n515[.]userstorage[.]mega[.]co[.]nz”.
This exfiltration attempt suggests the use of double extortion tactics, where threat actors not only encrypt victim’s data but also exfiltrate it to threaten public exposure unless a ransom is paid. This often increases pressure on organizations as they face the risk of both data loss and reputational damage caused by the release of sensitive information.
Darktrace’s Cyber AI Analyst autonomously investigated what initially appeared to be unrelated events, linking them together to build a full picture of the Fog ransomware attack for customers’ security teams. Specifically, on affected networks Cyber AI Analyst identified and correlated unusual scanning activities, SMB writes, and file appendages that ultimately suggested file encryption.
Safeguarding vulnerable sectors with real-time ransomware mitigation
As novel and fast-moving ransomware variants like Fog persist across the threat landscape, the time taken for from initial compromise to encryption has significantly decreased due to the enhanced skill craft and advanced malware of threat actors. This trend particularly impacts organizations in the education sector, who often have less robust cyber defenses and significant periods of time during which infrastructure is left unmanned, and are therefore more vulnerable to quick-profit attacks.
Traditional security methods may fall short against these sophisticated attacks, where stealthy actors evade detection by human-managed teams and tools. In these scenarios Darktrace’s AI-driven product suite is able to quickly detect and respond to the initial signs of compromise through autonomous analysis of any unusual emerging activity.
When Darktrace’s Autonomous Response capability was active, it swiftly mitigated emerging Fog ransomware threats by quarantining devices exhibiting malicious behavior to contain the attack and blocking the exfiltration of sensitive data, thus preventing customers from falling victim to double extortion attempts.
Darktrace’s First 6: Half-Year Threat Report 2024 highlights the latest attack trends and key threats observed by the Darktrace Threat Research team in the first six months of 2024.
Focuses on anomaly detection and behavioral analysis to identify threats
Maps mitigated cases to known, publicly attributed threats for deeper context
Offers guidance on improving security posture to defend against persistent threats
Appendices
Credit to Qing Hong Kwa (Senior Cyber Analyst and Deputy Analyst Team Lead, Singapore) and Ryan Traill (Threat Content Lead)
Darktrace Model Detections:
- Anomalous Server Activity::Anomalous External Activity from Critical Network Device
- Anomalous Connection::SMB Enumeration
- Anomalous Connection::Suspicious Read Write Ratio and Unusual SMB
- Anomalous Connection::Uncommon 1 GiB Outbound
- Anomalous File::Internal::Additional Extension Appended to SMB File