Learn how Darktrace leveraged generative AI tools to detect and combat phishing email campaigns. Discover how AI is reshaping cybersecurity strategies.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Germaine Tan
VP, Security & AI Strategy, Field CISO
Share
26
Sep 2023
Stopping the bad while allowing the good
Since its inception, email has been regarded as one of the most important tools for businesses, revolutionizing communication and allowing global teams to become even more connected. But besides organizations heavily relying on email for their daily operations, threat actors have also recognized that the inbox is one of the easiest ways to establish an initial foothold on the network.
Today, not only are phishing campaigns and social engineering attacks becoming more prevalent, but the level of sophistication of these attacks are also increasing with the help of generative AI tools that allow for the creation of hyper-realistic emails with minimal errors, effectively lowering the barrier to entry for threat actors. These diverse and stealthy types of attacks evade traditional email security tools based on rules and signatures, because they are less likely to contain the low-sophistication markers of a typical phishing attack.
In a situation where the sky is the limit for attackers and security teams are lean, how can teams equip themselves to tackle these threats? How can they accurately detect increasingly realistic malicious emails and neutralize these threats before it is too late? And importantly, how can email security block these threats while allowing legitimate emails to flow freely?
Instead of relying on past attack data, Darktrace’s Self-Learning AI detects the slightest deviation from a user’s pattern of life and responds autonomously to contain potential threats, stopping novel attacks in their tracks before damage is caused. It doesn’t define ‘good’ and ‘bad’ like traditional email tools, rather it understands each user and what is normal for them – and what’s not.
This blog outlines how Darktrace / EMAIL™ used its understanding of ‘normal’ to accurately detect and respond to a sustained phishing campaign targeting a real-life company.
Responding to a sustained phishing attack
Over the course of 24 hours, Darktrace detected multiple emails containing different subjects, all from different senders to different recipients in one organization. These emails were sent from different IP addresses, but all came from the same autonomous system number (ASN).
Figure 1: The sender freemail addresses and subject lines all followed a certain format. The subject lines followed the format of “<First name> <Last name>”, possibly to induce curiosity. The senders were all freemail accounts and contained first names, last names and some numbers, showing the attempts to make these email addresses appear legitimate.
The emails themselves had many suspicious indicators. All senders had no prior association with the recipient, and the emails generated a high general inducement score. This score is generated by structural and non-specific content analysis of the email – a high score indicates that the email is trying to induce the recipient into taking a particular action, which may lead to account compromise.
Additionally, each email contained a visually prominent link to a file storage service, hidden behind a shortened bit.ly link. The similarities across all these emails pointed to a sustained campaign targeting the organization by a single threat actor.
Figure 2: One of the emails is shown above. Like all the other emails, it contained a highly suspicious and shortened link.
Figure 3: In another one of the emails, the link observed had similar characteristics. But this email stands out from the rest. The sender's name seems to be randomly set – the 3 alphabets are close to each other on the keyboard.
With all these suspicious indicators, many models were breached. This drove up the anomaly score, causing Darktrace/Email to hold all suspicious emails from the recipients’ inboxes, safeguarding the recipients from potential account compromise and disallowing the threats from taking hold in the network.
Imagining a phishing attack without Darktrace/Email
So what could have happened if Darktrace had not withheld these emails, and the recipients had clicked on the links? File storage sites have a wide variety of uses that allow attackers to be creative in their attack strategy. If the user had clicked on the shortened link, the possible consequences are numerous. The link could have led to a login page for unsuspecting victims to input their credentials, or it could have hosted malware that would automatically download if the link was clicked. With the compromised credentials, threat actors could even bypass MFA, change email rules, or gain privileged access to a network. The downloaded malware might also be a keylogger, leading to cryptojacking, or could open a back door for threat actors to return to at a later time.
Figure 4: Darktrace / EMAIL highlights suspicious link characteristics and provides an option to preview the pages.
Figure 5: At the point of writing, both links could not be reached. This could be because they were one-time unique links created specifically for the user, and can no longer be accessed once the campaign has ceased.
The limits of traditional email security tools
Secure email gateways (SEGs) and static AI security tools may have found it challenging to detect this phishing campaign as malicious. While Darktrace was able to correlate these emails to determine that a sustained phishing campaign was taking place, the pattern among these emails is far too generic for specific rules as set in traditional security tools. If we take the characteristic of the freemail account sender as an example, setting a rule to block all emails from freemail accounts may lead to more legitimate emails being withheld, since these addresses have a variety of uses.
With these factors in mind, these emails could have easily slipped through traditional security filters and led to a devastating impact on the organization.
Conclusion
As threat actors step up their attacks in sophistication, prioritizing email security is more crucial than ever to preserving a safe digital environment. In response to these challenges, Darktrace / EMAIL offers a set-and-forget solution that continuously learns and adapts to changes in the organization.
Through an evolving understanding of every environment in which it is deployed, its threat response becomes increasingly precise in neutralizing only the bad, while allowing the good – delivering email security that doesn’t come at the expense of business growth.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Journey of a Threat: How Multi-Layered AI Works in Darktrace / EMAIL
Follow a malicious email as it moves through Darktrace / EMAIL’s multi-layered AI system, from raw data to final decision. Each layer works together to detect threats, understand intent, and take autonomous action.
How email-delivered prompt injection attacks can target enterprise AI – and why it matters
Prompt injection is a newly emerging threat, with only a handful of confirmed victims so far – targeting how AI systems use data rather than exploiting traditional software vulnerabilities. As agentic AI becomes embedded across enterprise environments, attackers may attempt to manipulate these systems through hidden instructions in everyday email content.
Security After Signatures: Operating in a World of Pre‑CVE Disclosure Exploitation, Collapsed Trust Boundaries, and Autonomous Systems
Three shifts have reshaped what it means to defend an enterprise securely.
First, exploitation often begins before defenders have a Common Vulnerabilities and Exposures (CVE) identifier, a security advisory, or an entry in the Cybersecurity and Infrastructure Security Agency's (CISA) Known Exploited Vulnerabilities (KEV) catalog.
Secondly, the trust boundary has moved beyond the network edge into identities, tokens, APIs, and Software-as-a-Service (SaaS) workflows.
Third, an increasing share of business activity is executed through automation, integrations, and AI agent-like systems that can act faster than teams can verify intent.
If your security model still relies on detecting known bad artefacts, triaging isolated alerts, and waiting for confirmation before acting, you are already behind the threat.
This is not a failure of security teams; it’s a failure of the operating model to keep pace with how the environment has changed.
A SOC built around alerts and signatures assumes that malicious activity will eventually surface as an event. In real incidents, however, the decisive evidence is rarely a single event. Instead, it is a chain of individually explainable actions that only appears malicious once you connect the dots across identity, non-human identity, cloud, email, SaaS, operational technology (OT), and network telemetry.
The defenders succeeding today observe behaviors, link them into sequences, understand what those sequences mean, and contain impact before the full story unfolds. That is the operating model the current threat environment demands.
In one example, Darktrace observed a sequence of subtle but strategically significant anomalies within a customer environment that later aligned with exploitation of CVE‑2025‑0994 in Trimble Cityworks by likely Chinese-nexus threat actors. Behavioral indicators were visible at least 18 days before public disclosure, with related anomalies emerging 40 to 50 days earlier during the intrusion window.
This case illustrates a familiar pattern: clusters of weak‑signal anomalies combing to form an actionable picture of intrusion long before a CVE is published. Such activity reflects long‑horizon, option‑preserving operator models often associated with mature state‑linked activity.
Figure 1: Darktrace’s detection of malicious exploitation of CVE 2025-0994, later tied to Chinese-nexus threat actors targeting critical national infrastructure (CNI) in the US, weeks before public disclosure.
Throughout 2025 and 2026, Darktrace has continued to observe the value of anomaly-based detections across a range of incidents.
CVE
CVE Public Disclosure Date
Darktrace Detection Date
Days Between Detection of Exploitation and CVE Public Disclosure
CVE 2025 0994 (Trimble City Works)
2025-02-06
2025-01-19
18 Days
CVE 2025-24183 (Apache)
2025-03-10
2025-02-18
20 days
CVE 2025-10035 (Fortra GoAnywhere)
2025-09-18
2025-09-11
7 days
Identity is the real control plane
The second shift is that identity has replaced perimeter as the primary control plane. As Darktrace’s Annual Threat Report 2026 illustrated, identity remains the main challenge in defending against modern intrusions. A clear example is the Adversary-in-the-Middle (AiTM) case published by Darktrace in December 2025. A phishing email led to the compromise of an Office 365 account. Session hijacking bypassed multi-factor authentication (MFA), and the compromised account was used for follow-on phishing and persistence activities including the creation of malicious email rules.
Every step in that sequence mattered. A successful login alone does not prove legitimacy. An inbox rule, on its own, may not appear catastrophic. Mail activity, viewed in isolation, may seem operationally normal. But the behavioral chain tells a different story: credential theft, token abuse, persistence, and onward compromise through a trusted identity.
This is why the question is no longer “Did the user authenticate successfully”. The more important question is, “Does this identity action make sense right now, in this context, given what came before it?” The AiTM case shows how identity can be compromised. In practice, however, attacks rarely remained confined to identity alone.
In another Darktrace case, a compromised SaaS account triggered activity across the email, SaaS, and network layers, including inbox rule changes, phishing propagation, and connections to suspicious infrastructure. Viewed in isolation, none of these events were decisive. Together, however, they formed a behavioral sequence that revealed the intrusion, with the full attack story automatically correlated and surfaced to defenders by Darktrace’s Cyber AI Analyst.
Figure 2: Cyber AI Analyst correlated and appended additional events to the incident, including other users who connected to the suspicious redirect link after outbound phishing emails were sent.
AI accelerates the threat
The third shift is the one many teams still underestimate: trusted tooling, integrations, and AI agent-like systems can create actions that appear legitimate but are strategically dangerous.
The shift becomes clearer when examining how governments are now framing AI risk. In 2026, guidance published by CISA, UK’s National Cyber Security Centre (NCSC) and Five Eyes partners warned that agentic systems expand attack surfaces, accumulate privilege, and can behave in ways that are difficult to predict or explain [1]. The advice is simple: assume unexpected behavior and design controls around it.
The real risk is not AI usage. It is unknown autonomy: systems with credentials, data access, and action paths that can execute workflow steps without sufficient behavioral validation, traceability, or human oversight. Darktrace’s Model Context Protocol (MCP) risk analysis provides a useful framework for understanding this challenge. Over-privileged agents, content injection, and tool abuse become high-consequence risks when connected systems can dynamically retrieve data, execute actions, and communicate externally.
Whether security teams like it or not, AI is already in the enterprise. It will help drive innovation, but it will also be abused, whether accidentally or maliciously. In each of the cases below, AI either scaled the attacker, built the tooling, or existed within the environment as something to exploit or misuse.
1. AI as an Attack Multiplier
In one campaign targeting Mexican government entities, a single operator used commercial AI platforms to generate exploits, automate reconnaissance, and process large volumes of data, compressing work that would traditionally have required an entire team into a single workflow [2].
Attempted AI exploitation is now appearing within customer environments. In one case involving an automation technology manufacturer, a compromised LLM proxy was seemingly used as a stepping stone to access additional AI services. When that attempt failed, the attacker pivoted to cryptomining.
What is clear is that the AI layer has already become an asset worth probing, exploiting, and pivoting through. It is also clear that defenders benefit from rapidly understanding how these activities connect. In this case, Cyber AI Analyst automatically pieced together the intrusion, while Darktrace’s Managed Threat Detection service alerted to the customer, enabling the activity to be contained before it could progress further.
Figure 3: Cyber AI Analyst's investigation into a compromised LLM proxy that was abused for cryptomining activity.
AI as a trusted but dangerous actor
This does not require a cinematic vision of “rogue AI.” The Salesloft incident provides a more grounded example, where AI and automation operate with legitimate access but served malicious intent. In that case, attackers abused compromised OAuth tokens associated with the Drift AI chat agent to export significant volumes of data from Salesforce environments.
The activity resembled legitimate API usage and relied on trusted SaaS integrations rather than malware or other obvious signs of intrusion. That is precisely the challenge. Traditional security controls are good at detecting forced entry, but far less effective when a trusted application integration behaves in a way that is technically permitted yet operationally harmful.
In these scenarios, the security challenge shifts from validating access to validating behavior.
This is what that looks like in practice: AI-linked identities executing legitimate actions that require behavioral validation rather than access validation.
Figure 4: Darktrace / SECURE AI highlights anomalous activity across AI identities, surfacing critical behavior that requires validation and containment.
Early observations from Darktrace / SECURE AI deployments reinforce this reality. Across Darktrace's observed fleet, AI service connections per deployment increased 13% during the first half of 2026, reaching over 16 million connections overall. The typical organisation now interacts with seven different AI providers, evidence that AI is no longer operating at the edges of the enterprise. It is increasingly woven into day-to-day business activity.
The most common risks are not compromised models or advanced AI attacks. Instead, they stem from employees and business functions exposing sensitive information through entirely legitimate-looking interactions. Darktrace has observed repeated submission of personally identifiable information (PII), tax information, identification documents, and medical data into LLM prompts, alongside widespread use of unsanctioned (shadow) AI services and growing AI activity from mobile devices.
For defenders, the challenge is increasingly one of context: understanding when legitimate business use crosses into material risk, while preserving privacy and user trust.
Conclusion
Across all three shifts, the pattern is the same: behavior precedes understanding. Security teams are not losing because adversaries have become invisible. An increasingly outdated security model assumes that malicious activity will reveal itself cleanly and early. It no longer does.
In 2026 and beyond, defenders win by understanding behavioral sequences, continuously validating trust, and acting before certainty becomes hindsight. That is security after signatures. That is security in the AI era.
Credit to: Daniel Levy, Threat Hunting Data Scientist
2026年6月12日、DarktraceはLiteLLM-Proxyという名前のAmazon Web Service (AWS) EC2インスタンスから暗号通貨マイニング発生中とみられるアクティビティを観測しました。このインスタンスはLiteLLMアクティビティをサポートしており、Amazon Bedrockリソースへのアクセス権を有するインスタンスプロファイルと関連付けられていました。
_11zon.avif)
.avif)
