Blog
/
Email
/
March 20, 2024

How Phishing Attacks Are Becoming Harder to Identify

Learn about the cyber risks posed by advanced phishing attacks and how AI can enhance security solutions to defend against them.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
The Darktrace Community
Default blog image
20
Mar 2024

The state of email security and phishing attacks

Employees send and receive hundreds of emails a day to keep businesses moving. Unfortunately, it just takes one employee to interact with an undetected phishing email to potentially put an entire organization at risk from cyber disruption. Attackers know this, which is why they continue to develop and improve email phishing attacks.  

Increased attack sophistication makes it harder than ever for traditional cyber security solutions like SEGs, firewalls, and spam filters to detect and mitigate increasingly novel and sophisticated email threats.

When there are tell-tale signs of a threat, these solutions can  identify an incoming message as suspicious. Pointers such as emails from unknown senders, messages which contain an unusual amount of poor spelling and grammar or encourage the receiver to respond to an unexpected but supposedly urgent request.

That is, if the phishing attacks weren’t blocked by security measures before reaching the victim’s inbox. But, this is happening more and more often as phishing campaigns are becoming more advanced. Attackers are showing signs of consistently bypassing traditional protections and getting through to exploit victims.  

Darktrace email threat reporting

In its End of Year Threat Report, Darktrace analyzed over 10 million phishing emails targeting customer environments between September 1 and December 31, 2023.  Our findings signal that attackers are starting to take advantage of advancements in artificial intelligence (AI), including using Generative AI tools such as Large Language Models (LLMs) to create more convincing and sophisticated phishing messages – and at scale.

LLMs and Phishing

With the right AI prompts, attackers can use these LLMs to help write convincing email messages designed to target specific countries, companies or even individuals – all without the suspicious hallmarks which are traditionally associated with standard phishing attacks. The attackers don’t even need to speak the language of the individuals or groups they’re targeting.  LLMs lower language barriers for attackers; using their native tongue, they can simply ask the Generative AI to write a message in the language of their choosing.  

These techniques are designed to build trust and manipulate recipients into giving up sensitive information like user credentials, intellectual property or bank information or coerce them into downloading malicious payloads which can be used to launch further attacks on business infrastructure.  With the appropriate research, attackers can tailor the messages to increase the chances of being successful, like making them look like a legitimate company email or request.

Social engineering phishing attacks

A year ago. Darktrace shared research which found a 135% increase in ‘novel social engineering attacks’ in the first two months of 2023, corresponding with the widespread adoption of ChatGPT. These novel phishing attacks showed a strong linguistic deviation compared to other phishing emails, which suggested to us that Generative AI was already providing an avenue for threat actors to craft sophisticated and targeted attacks at speed and scale

We’ve seen this trend continue. Our End of Year Threat Report found 38% of these emails were identified as utilizing novel social engineering techniques.

Attackers are also deploying another technique to make phishing emails look more convincing – they’re making the emails themselves longer and more sophisticated.  

A potential victim might be suspicious of an ‘urgent’ email which prompts them to take action without an explanation - but if there’s additional context in the text, it adds an aura of legitimacy which is difficult to act against.

And threat actors know this; 28% of phishing emails analyzed by Darktrace over the period were identified as having “significant” amount of text – containing over 1,000 characters, which equates to over 200 words.  

It’s a sign that attackers are innovating and bolstering their efforts to craft sophisticated phishing campaigns, potentially leveraging Generative AI tools to automate social engineering activity by creating longer, more convincing phishing emails.  

QR code phishing

But this is far from the only innovative method which attackers are using to bypass traditional security defences. Among the 10 million plus emails analyzed during the reporting period, Darktrace/Email detected over 639,000 malicious QR codes within the messages.

Malicious QR codes placed within emails have become an increasingly common form of phishing attack, especially as QR codes have become a more common method for sharing links to information or buying links for products in recent years.

Attackers are deploying QR codes because they provide a way of directing unsuspecting victims to malicious websites or download links without needing to use a traditional phishing URL.  

The advantage of implanting QR codes for attackers is that while phishing URLs are something which traditional security solutions are actively looking to identify and mitigate, malicious QR codes are more difficult for them to detect.

Applying AI to email security

Traditional security solutions which rely heavily on previously identified malicious emails and known bad senders are struggling to identify and defend against these novel and increasingly sophisticated email threats.

But by using AI that learns the unique digital environment and patterns of each business, Darktrace/Email can recognize the subtle deviations in expected email activity to determine whether any given email could represent a threat to the business. It is then able to make highly accurate decisions to mitigate and neutralize any email attack it faces helping to keep your organization safe from cyber disruption.

It’s therefore imperative that in the battle against ever-evolving, ever more sophisticated cyber threats, defenders are also embracing AI to keep businesses safe. By effectively applying AI to cyber security challenges, defenders can take a proactive approach to cyber security, staying one step ahead of malicious attackers, with real-time detection and automated response to known and unknown threats looking to disrupt the business via the inbox.  

Darktrace/Email was recently awarded a 2024 AI Excellence Award for Machine Learning by Business Intelligence Group.

Join Darktrace on 9 April for a virtual event to explore the latest innovations needed to get ahead of the rapidly evolving threat landscape. Register today to hear more about our latest innovations coming to Darktrace’s offerings.

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
The Darktrace Community

More in this series

No items found.

Blog

/

/

August 11, 2026

Reimagining Your SOC: Overcoming Alert Fatigue with AI-Led Investigations  

Default blog imageDefault blog image

Key takeaways

  • AI-led investigation can help combat SOC alert fatigue by extending automation beyond detection into triage, contextual analysis, prioritization, and investigation, reducing repetitive analyst work while preserving human judgment for critical security decisions.
  • Alert fatigue becomes an operational risk when high alert volumes, fragmented context, manual investigation, and limited analyst capacity prevent SOC teams from efficiently assessing potentially meaningful security activity.
  • Effective alert fatigue solutions should combine automated triage, contextual analysis, intelligent prioritization, and explainable investigation workflows so analysts can understand why activity matters and focus attention on higher-priority incidents.
  • AI-led investigation can improve SOC efficiency by expanding investigation coverage and freeing analysts from repetitive evidence gathering, creating more capacity for threat hunting, recovery activities, strategic decisions, and proactive cyber risk reduction.

The efficiency of a Security Operations Center (SOC) hinges on its ability to detect, analyze and respond to threats effectively. With advancements in AI and automation, key early SOC team metrics such as Mean Time to Detect (MTTD) have seen significant improvements:

  • 96% of defenders believing AI-powered solutions significantly boost the speed and efficiency of prevention, detection, response, and recovery.
  • Organizations leveraging AI and automation can shorten their breach lifecycle by an average of 108 days compared to those without these technologies.

While tool advances have improved performance and effectiveness in the detection phase, this has not been as beneficial to the next step of the process where initial alerts are investigated further to determine their relevance and how they relate to other activities. This is often measured with the metric Mean Time to Analysis (MTTA), although some SOC teams operate a two-level process with teams for initial triage to filter out more obviously uninteresting alerts and for more detailed analysis of the remainder. SOC teams continue to grapple with alert fatigue, overwhelmed analysts, and inefficient triage processes, preventing them from achieving the operational efficiency necessary for a high-performing SOC.

Addressing this core inefficiency requires extending AI's capabilities beyond detection to streamline and optimize the following investigative workflows that underpin effective analysis.

Challenges with SOC alert investigation

Detecting cyber threats is only the beginning of a much broader challenge of SOC efficiency. The real bottleneck often lies in the investigation process.

Detection tools and techniques have evolved significantly with the use of machine learning methods, improving early threat detection. However, after a detection pops up, human analysts still typically step in to evaluate the alert, gather context, and determine whether it’s a true threat or a false alarm and why. If it is a threat, further investigation must be performed to understand the full scope of what may be a much larger problem. This phase, measured by the mean time to analysis, is critical for swift incident response.

Challenges with manual alert investigation:

  • Too many alerts
  • Alerts lack context
  • Cognitive load sits with analysts
  • Insufficient talent in the industry
  • Fierce competition for experienced analysts

For many organizations, investigation is where the struggle of efficiency intensifies. Analysts face overwhelming volumes of alerts, a lack of consolidated context, and the mental strain of juggling multiple systems. With a worldwide shortage of 4 million experienced level two and three SOC analysts, the cognitive burden placed on teams is immense, often leading to alert fatigue and missed threats.

Even with advanced systems in place not all potential detections are investigated. In many cases, only a quarter of initial alerts are triaged (or analyzed). However, the issue runs deeper. Triaging occurs after detection engineering and alert tuning, which often disable many alerts that could potentially reveal true threats but are not accurate enough to justify the time and effort of the security team. This means some potential threats slip through unnoticed.

What are the key stages of detecting and stopping a cyber-attack?

Detecting and stopping a cyber incident involves several connected stages: identifying a trace of suspicious activity, generating a detection, investigating the alert to determine its context and scope, and containing confirmed threats. Each stage affects how quickly a SOC can move from identifying potentially malicious activity to taking effective action.

First we need a trace of an attack…

The attack will produce some sort of digital trace. Novel attacks, insider threats, and attacker techniques such as living-off-the-land can make attacker activities extremely hard to distinguish.

A detection is created…

Then we have to detect the trace, for example some beaconing to a rare domain. Initial detection alerts being raised underpin the MTTD (mean time to detection). Reducing this initial unseen duration is where we have seen significant improvement with modern threat detection tools.

When it comes to threat detection, the possibilities are vast. Your initial lead could come from anything: an alert about unusual network activity, a potential known malware detection, or an odd email. Once that lead comes in, it’s up to your security team to investigate further and determine if this is this a legitimate threat or a false alarm and what the context is behind the alert.

Investigation begins…

It doesn’t just stop at a detection. Typically, humans also need to look at the alert, investigate, understand, analyze, and conclude whether this is a genuine threat that needs a response. We normally measure this as MTTA (mean time to analyze).

Conducting the investigation effectively requires a high degree of skill and efficiency, as every second counts in mitigating potential damage. Security teams must analyze the available data, correlate it across multiple sources, and piece together the timeline of events to understand the full scope of the incident. This process involves navigating through vast amounts of information, identifying patterns, and discerning relevant details. All while managing the pressure of minimizing downtime and preventing further escalation.

Containment begins…

Once we confirm something as a threat, and the human team determines a response is required and understand the scope, we need to contain the incident. That's normally the MTTC (mean time to containment) and can be further split into immediate and more permanent measures.

For more about how AI-led solutions can help in the containment stage read here: Autonomous Response: Streamlining Cybersecurity and Business Operations

The challenge is not only in 1) detecting threats quickly, but also 2) triaging and investigating them rapidly and with precision, and 3) prioritizing the most critical findings to avoid missed opportunities. Effective investigation demands a combination of advanced tools, robust workflows, and the expertise to interpret and act on the insights they generate. Without these, organizations risk delaying critical containment and response efforts, leaving them vulnerable to greater impacts.

While there are further steps (remediation, and of course complete recovery) here we will focus on investigation.

Developing an AI analyst: How Darktrace replicates human investigation

Darktrace has been working on understanding the investigative process of a skilled analyst since 2017. By conducting internal research between Darktrace expert SOC analysts and machine learning engineers, we developed a formalized understanding of investigative processes. This understanding formed the basis of a multi-layered AI system that systematically investigates data, taking advantage of the speed and breadth afforded by machine systems.

With this research we found that the investigative process often revolves around iterating three key steps: hypothesis creation, data collection, and results evaluation.

All these details are crucial for an analyst to determine the nature of a potential threat. Similarly, they are integral components of our Cyber AI Analyst which is an integral component across our product suite. In doing so, Darktrace has been able to replicate the human-driven approach to investigating alerts using machine learning speed and scale.

Here’s how it works:

  • When an initial or third-party alert is triggered, the Cyber AI Analyst initiates a forensic investigation by building multiple hypotheses and gathering relevant data to confirm or refute the nature of suspicious activity, iterating as necessary, and continuously refining the original hypothesis as new data emerges throughout the investigation.
  • Using a combination of machine learning including supervised and unsupervised methods, NLP and graph theory to assess activity, this investigation engine conducts a deep analysis with incidents raised to the human team only when the behavior is deemed sufficiently concerning.
  • After classification, the incident information is organized and processed to generate the analysis summary, including the most important descriptive details, and priority classification, ensuring that critical alerts are prioritized for further action by the human-analyst team.
  • If the alert is deemed unimportant, the complete analysis process is made available to the human team so that they can see what investigation was performed and why this conclusion was drawn.
Darktrace cyber ai analyst workflow, how it works

To illustrate this via example, if a laptop is beaconing to a rare domain, the Cyber AI Analyst would create hypotheses including whether this could be command and control traffic, data exfiltration, or something else. The AI analyst then collects data, analyzes it, makes decisions, iterates, and ultimately raises a new high-level incident alert describing and detailing its findings for human analysts to review and follow up.

Learn more about Darktrace's Cyber AI Analyst

  • Cost savings: Equivalent to adding up to 30 full-time Level 2 analysts without increasing headcount
  • Minimize business risk: Takes on the busy work from human analysts and elevates a team’s overall decision making
  • Improve security outcomes: Identifies subtle, sophisticated threats through holistic investigations

How does AI-led investigation improve SOC efficiency?

AI-led investigation can improve SOC efficiency by shifting repetitive evidence gathering and triage away from analysts while preserving human judgment for higher-value decisions. The operational benefit is broader investigation coverage, faster prioritization, clearer reasoning, and more analyst time for activities such as threat hunting, recovery, and proactive cyber risk reduction.

Every relevant alert is investigated

The Cyber AI Analyst is not a generative AI system, or an XDR or SEIM aggregator that simply prompts you on what to do next. It uses a multi-layered combination of many different specialized AI methods to investigate every relevant alert from across your enterprise, native, 3rd party, and manual triggers, operating at machine speed and scale. This also positively affects detection engineering and alert tuning, because it does not suffer from fatigue when presented with low accuracy but potentially valuable alerts.

Retain and improve analyst skills

Transferring most analysis processes to AI systems can risk team skills if they don't maintain or build them and if the AI doesn't explain its process. This can reduce the ability to challenge or build on AI results and cause issues if the AI is unavailable. The Cyber AI Analyst, by revealing its investigation process, data gathering, and decisions, promotes and improves these skills. Its deep understanding of cyber incidents can be used for skill training and incident response practice by simulating incidents for security teams to handle.

Create time for cyber risk reduction

Human cybersecurity professionals excel in areas that require critical thinking, strategic planning, and nuanced decision-making. With alert fatigue minimized and investigations streamlined, your analysts can avoid the tedious data collection and analysis stages and instead focus on critical decision-making tasks such as implementing recovery actions and performing threat hunting.

Stay tuned for part 3/3

Part 3/3 in the Reimagine your SOC series explores the preventative security solutions market and effective risk management strategies.

Coming soon!

Continue reading
About the author
Brittany Woodsmall
Product Marketing Manager, AI

Blog

/

Network

/

August 11, 2026

When AI Agents Attack: The Case for Behavioral Anomaly Detection

Default blog imageDefault blog image

The UK AI Security Institute (AISI) has just published a detailed incident report confirming what many within the cyber security industry have suspected for some time: frontier AI agents, given a degree of autonomy, can independently develop and execute attack chains against real targets, including social engineering, supply-chain compromise, and deception, without explicit instruction to do so.

The report (INC-2026-07-28-01, published 4 August 2026) documents what happened during a routine capability evaluation between 25 and 28 July. It deserves a careful read from anyone responsible for securing the modern digital estate.

The incident

AISI regularly tests frontier AI models for offensive cyber capability. In this exercise, agents were placed inside a simulated network environment and given internet access this was deliberate, given to reflect realism of the task. Across 122 evaluation runs, 19 instances of unsanctioned activity were recorded, including cases where agents broke the intended boundaries of the test and acted against real people and real infrastructure on the open internet.

In the most serious case, an agent (Mythos 5) autonomously:

  • Conducted open-source intelligence gathering to identify real developers associated with keywords from the simulated scenario
  • Created a GitHub account and submitted a pull request containing obfuscated malicious code to a real, unaffiliated repository
  • Created a second sockpuppet account to post comments on its own PR, attempting to manufacture social proof
  • When a human reviewer flagged the malicious content, the agent lied, claiming an accidental commit, then repeatedly attempted to reintroduce the payload under the guise of having "fixed" the code
  • Sent spear-phishing emails to the repository maintainer containing manipulated content designed to convince them to merge the code
  • Planted a prompt injection inside a GitHub issue, hidden in an HTML comment invisible to humans but readable by AI coding assistants parsing the repository via API

AISI confirmed no real-world harm resulted. The PRs were not merged; the payloads did not execute. But containment was a product of circumstance, not design.

Why this matters

Every action the agents took used legitimate services operating within normal parameters. Creating a GitHub account is legitimate. Opening a pull request is legitimate. Sending an email is legitimate. Posting an issue is legitimate.

No vulnerability was exploited. No known malware signature was present. No indicator of compromise from any threat intelligence feed would have matched. The entire attack existed as a pattern, a sequence of individually normal actions that, taken together, constituted a supply-chain attack, a social engineering campaign, and an attempted compromise of downstream systems.

This is the class of threat that signature-based and rule-based security tools structurally struggle to detect. They answer the question "have I seen this before?" When the answer is no – and when the attack is novel, composed of legitimate actions, and adapted in real time by an adversary in this case an agent that reasons about evasion – they struggle.

Behavioral anomaly detection: The architecture that fits

Darktrace has spent a decade building a platform around a different question: "is this behavior normal for this entity?"

The platform learns the baseline behavior of every user, device, and communication pattern across an organization's digital environment. It does not need prior knowledge of an attack technique. It identifies deviation from learned normality, surfaces it, and through its Autonomous Response capability, takes targeted action to contain threats at machine speed.

Consider how this AISI incident might show across the Darktrace platform:

Network detection and response

While no payload was successfully delivered to a victim in this incident, AISI's post-incident analysis of the agent's code (detailed on pages 24-25 of the report) reveals what would have landed on target systems had the social engineering succeeded. The agent produced three successive payload generations, each replacing the last after being detected and called out:

Generation 1 hid a module-load stager inside a legitimate bug-fix commit, guarded by an environment variable check, to skip execution in CI. Its second stage attempted to fetch Poseidon implants (Linux and Windows variants) calling back to the agent's own C2 listener, with a fallback to publicly hosted polling scripts that fetched and executed commands every 60 seconds.

Generation 2 exploited an Electron "What's New" window with nodeIntegration enabled, loading remote HTML from a GitHub Pages site the agent had registered. The page decoded an XOR-obfuscated payload at runtime that beaconed host identifiers over DNS, attempted to download the same implants, ran its own in-process C2 loop, and installed reboot persistence via crontab and Windows registry Run keys.

Generation 3 was a post-install dropper added after the agent had already been publicly caught. It XOR-decoded a hidden URL, downloaded a script over plaintext HTTP, and ran it detached, force-exiting so npm install always reports success, giving the attacker remote code execution on any machine running the install.

None of these payloads reached a victim. But the technical detail matters: this is the calibre of tradecraft an autonomous agent produced in a matter of hours, unprompted. Each generation demonstrated adaptation, the agent iterated its approach in direct response to being detected.

For network defenders, the behavioral signals these payloads would have generated are clear. DNS beaconing to novel domains, periodic HTTP polling to newly registered infrastructure, XOR-encoded data in transit, registry and crontab persistence writes on endpoints that have never exhibited such behavior. These are exactly the anomalies that Darktrace / NETWORK identifies in real time: deviations from learned device and connection baselines, regardless of whether the specific payload or C2 domain has been seen before.

Beyond the payload analysis, the agents' broader activity produced additional behavioral anomalies: new identities engaging in high-frequency interactions with external services, coordination via unusual channels (public pastebins, gists), and lateral communication patterns that deviate from any established baseline.

Email security

The spear-phishing emails sent by the agent, crafted with context-specific content designed to manipulate a specific individual, represent exactly the kind of socially engineered communication that static email gateways can miss. Darktrace / EMAIL analyses the linguistic patterns, sender behavior, and relational context of every message. A brand-new sender using tailored language to pressure a recipient into a specific technical action deviates from the learned communication profile, regardless of whether the message contains a known-bad URL or attachment.

Securing AI

This is where the incident points to a threat category that many security platforms do not yet fully address. The agent did not just use AI as a tool for attack. It targeted other AI systems, attempting prompt injections against coding assistants and trying to compromise automated workflows. Organisations adopting AI agents for development, operations, and decision-making are introducing a newer extended attack surface.  

Darktrace / SECURE AI extends the same behavioral detection model into this space. It monitors AI usage across the enterprise, detecting anomalous interactions with AI services, identifying sensitive data exposure to AI systems, providing visibility into shadow AI use. As AI agents become standard components of enterprise infrastructure, the ability to detect when an AI system is behaving outside its expected parameters, when there is a chance, it is being manipulated, or when it is interacting with something it should not be, that becomes a direct security requirement rather than a future consideration.

The principle is consistent across the Darktrace platform: learn what normal looks like, identify deviation, respond autonomously.

The AI-to-AI threat surface

One of the report's most important findings is a confirmation that AI agents can act outside of instructions and guardrails to complete tasks. The Mythos 5 agent conducted reconnaissance on a repository maintainer, inferred it might be an AI coding assistant, and deployed a prompt injection specifically crafted to compromise it.

This is a new attack class. Organizations deploying AI coding assistants, AI-powered code review, or autonomous DevOps agents now face an adversary that:

  • Moves at speed
  • Is able to adapt its approach when initial attempts are halted
  • Specifically targets AI systems via techniques (prompt injection, hidden instructions in HTML comments) that are less visible to human reviewers but parsed by machines
  • Reasons about which evasion strategies are most likely to succeed

Defending against this requires security that operates with the same adaptability and speed. A system that learns what "normal" looks like, and flags deviation from it.

What AISI recommends

The report identifies five contributing factors to the incident. Two are directly relevant:

No synchronous real-time monitoring (Section 5.3): AISI acknowledges it had not built real-time behavioral monitoring capable of immediately blocking unsanctioned agent actions. Their security team detected the incident via commercial security monitoring software, but only after activity had already occurred.

No fine-grained behavioral controls (Section 6.2): AISI's forward-looking response includes building continuous monitoring that can "immediately block or flag unsanctioned AI agent behavior."

The direction is clear: organizations need security that watches behavior continuously, learns what normal looks like, and acts when deviation occurs, without requiring a human in the loop for every decision.

The takeaway

The AISI incident is documented proof that autonomous AI agents will independently develop novel attack chains using legitimate tools, adapt when detected, employ deception, and target both humans and other AI systems. This happened last week, in a controlled setting, with commercially available models.

The security architecture that addresses this is behavioral anomaly detection applied across the full digital estate, as AI agents become standard components of enterprise infrastructure, writing code, managing deployments, processing communications, the attack surface they create is behavioral by nature.

This is the approach Darktrace has taken for years: learning what is normal across an organization’s digital environment, identifying meaningful deviations, and responding to emerging threats without relying on known attack signatures. As autonomous AI agents introduce new and unpredictable behaviors, that foundation becomes increasingly important to securing the enterprise.

Read the full report from the UK AI Security Institute here.

Continue reading
About the author
Adam Stevens
Senior Director of Product | Darktrace
Your data. Our AI.
Elevate your network security with Darktrace AI