Blog
/
/
March 7, 2024

Defending Against the New Normal in Cybercrime: AI

This blog outlines research & data points on the evolving threat landscape, the impact of malicious AI, and why proactive cyber readiness is essential.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Max Heinemeyer
Global Field CISO
Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
07
Mar 2024

AI in Cyber Security

Over the last 18 months, discussions about artificial intelligence (AI) – specifically generative AI – ranged from excitement and optimism about its transformative potential to fear and uncertainty about the new risks it introduces.  

New research1 commissioned by Darktrace shows that 89 percent of IT security teams polled globally believe AI-augmented cyber threats will have a significant impact on their organization within the next two years, yet 60 percent believe they are currently unprepared to defend against these attacks. Their concerns include increased volume and sophistication of malware that targets known vulnerabilities and increased exposure of sensitive or proprietary information from using generative AI tools.  

At Darktrace, we monitor trends across our global customer base to understand how the challenges facing security teams are evolving alongside industry advancements in AI. We’ve observed that AI, automation, and cybercrime-as-a-service have increased the speed, sophistication and efficacy of cyber security attacks.  

How AI Impacts Phishing Attempts

Darktrace has observed immediate impacts on phishing, which remains one of the most common forms of attack. In April 2023, Darktrace shared research that found a 135 percent increase in ‘novel social engineering attacks’ in the first two months of 2023, corresponding with the widespread adoption of ChatGPT2. These phishing attacks showed a strong linguistic deviation – semantically and syntactically – compared to other phishing emails, which suggested to us that generative AI is providing an avenue for threat actors to craft sophisticated and targeted attacks at speed and scale. A year later, we’ve seen this trend continue. Darktrace customers received approximately 2,867,000 phishing emails in December 2023 alone, a 14 percent increase on what was observed months prior in September3. Between September and December 2023, phishing attacks that used novel social engineering techniques grew by 35 percent on average across the Darktrace customer base4.  

These observations reinforce trends that others in the industry have shared. For example, Microsoft and OpenAI recently published research on tactics, techniques, and procedures (TTPs) augmented by large language models (LLMs) that they have observed nation-state threat actors using. That includes using LLMs to draft and generate social engineering attacks, inform reconnaissance, assist with vulnerability research and more.  

The Rise of Cybercrime-as-as-a-Service

The increasing cyber challenge facing defenders cannot be attributed to AI alone. The rise of cybercrime as-a-service is also changing the dynamic. Darktrace’s 2023 End of Year Threat Report found that cybercrime-as-a-service continue to dominate the threat landscape, with malware-as-a-Service (MaaS) and ransomware-as-a-Service (RaaS) tools making up most malicious tools in use by attackers. The as-a-Service ecosystem can provide attackers with everything from pre-made malware to templates for phishing emails, payment processing systems and even helplines to enable bad actors to mount attacks with limited technical knowledge.  

These trends make it clear that attackers now have a more widely accessible toolbox that reduces their barriers.

AI Enabling Accidental Insider Threats

However, the new risks facing businesses aren’t from external threat actors alone. Use of generative AI tools within the enterprise introduces a new category of accidental insider threats. Employees using generative AI tools now have easier access to more organizational data than ever before. Even the most well-intentioned employee could unintentionally leak or access restricted, sensitive data via these tools. In the second half of 2023, we observed that approximately half of Darktrace customers had employees accessing generative AI services. As this continues to increase, organizations need policies in place to guide the use cases for generative AI tools as well as strong data governance and the ability to enforce these policies to minimize risk.  

It is inevitable that AI will increase the risks and threats facing an organization, but this is not an unsolvable challenge from a defensive perspective. While advancements in generative AI may be worsening issues like novel social engineering and creating new types of accidental insider threats, AI itself offers a strong defense.  

The Shift to Proactive Cyber Readiness

According to the World Economic Forum’s Global Cybersecurity Outlook 2024, the number of organizations that “maintain minimum viable cyber resilience is down 30 percent compared to 2023”, and “while large organizations have demonstrated gains in cyber resilience, small and medium-sized companies showed significant decline.” The importance of cyber resilience cannot be understated in the face of today’s increasingly as-a-service, automated, and AI-augmented threat landscape.  

Historically, organizations wait for incidents to happen and rely on known attack data for threat detection and response, making it nearly impossible to identify never-before-seen threats. The traditional security stack has also relied heavily on point solutions focused on protecting different pieces of the digital environment, with individual tools for endpoint, email, network, on-premises data centers, SaaS applications, cloud, OT and beyond. These point solutions fail to correlate disparate incidents to form a complete picture of an orchestrated attack. Even with the addition of tools that can stitch together events from across the enterprise, they are in a reactive state that focuses heavily on threat detection and response.  

Organizations need to evolve from a reactive posture to a stance of proactive cyber readiness. To do so, they need an approach that proactively identifies internal and external vulnerabilities, identifies gaps in security policy and process before an attack occurs, breaks down silos to investigate all threats (known and unknown) during an attack, and uplifts the human analyst beyond menial tasks to incident validation and recovery after an attack.  

AI can help break down silos within the SOC and provide a more proactive approach to scale up and augment defenders. It provides richer context when it is fed information from multiple systems, data sets, and tools within the stack and can build an in-depth, real-time behavioural understanding of a business that humans alone cannot.

Lessons From AI in the SOC

At Darktrace, we’ve been applying AI to the challenge of cyber security for more than ten years, and we know that proactive cyber readiness requires the right mix of people, process, and technology.  

When the right AI is applied responsibly to the right cyber security challenge, the impact on both the human security team and the business is profound.

AI can bring machine speed and scale to some of the most time-intensive, error-prone, and psychologically draining components of cyber security, helping humans focus on the value-added work that only they can provide. Incident response and continuous monitoring are two areas where AI has already been proven to effectively augment defenders. For example, a civil engineering company used Darktrace’s AI to uplift its SOC team from the repetitive, manual tasks of analyzing and responding to email incidents. The analysts estimated they were each spending 10 hours per week on email incident analysis. With AI autonomously analyzing and responding to email incidents, the analysts could gain approximately 20 percent of their time back to focus on proactive cyber security measures

An effective human-AI partnership is key to proactive cyber readiness and can directly benefit the work-life of defenders. It can help to reduce burnout, support data-driven decision-making, and reduce the reliance on hard-to-find, specialized talent that has created a skills shortage in cyber security for many years. Most importantly, AI can free up team members to focus on more meaningful tasks, such as compliance initiatives, user education, and sophisticated threat hunting.  

Advancements in AI are happening at a rapid pace. As we’ve already observed, attackers will be watching these developments and looking for ways to use it to their advantage. Luckily, AI has already proved to be an asset for defenders, and embracing a proactive approach to cyber resilience can help organizations increase their readiness for this next phase. Prioritizing cyber security will be an enabler of innovation and progress as AI development continues.  

--

Join Darktrace on 9 April for a virtual event to explore the latest innovations needed to get ahead of the rapidly evolving threat landscape. Register today to hear more about our latest innovations coming to Darktrace’s offerings.

References

[1] The survey was undertaken by AimPoint Group & Dynata on behalf Darktrace between December 2023 & January 2024. The research polled 1773 security professionals in positions across the security team from junior roles to CISOs, across 14 countries – Australia, Brazil, France, Germany, Italy, Japan, Mexico, Netherlands, Singapore, Spain, Sweden, UAE, UK, and USA.

[2] Based on the average change in email attacks between January and February 2023 detected across Darktrace/Email deployments with control of outliers.

[3] Average calculated across Darktrace customers from 31st August to 21st December.

[4] Average calculated across Darktrace customers from 31st August to 21st December. Novel social engineering attacks use linguistic techniques that are different to techniques used in the past, as measured by a combination of semantics, phrasing, text volume, punctuation, and sentence length.

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Max Heinemeyer
Global Field CISO

More in this series

No items found.

Blog

/

Email

/

December 18, 2025

Why organizations are moving to label-free, behavioral DLP for outbound email

Man at laptopDefault blog imageDefault blog image

Why outbound email DLP needs reinventing

In 2025, the global average cost of a data breach fell slightly — but remains substantial at USD 4.44 million (IBM Cost of a Data Breach Report 2025). The headline figure hides a painful reality: many of these breaches stem not from sophisticated hacks, but from simple human error: mis-sent emails, accidental forwarding, or replying with the wrong attachment. Because outbound email is a common channel for sensitive data leaving an organization, the risk posed by everyday mistakes is enormous.

In 2025, 53% of data breaches involved customer PII, making it the most commonly compromised asset (IBM Cost of a Data Breach Report 2025). This makes “protection at the moment of send” essential. A single unintended disclosure can trigger compliance violations, regulatory scrutiny, and erosion of customer trust –consequences that are disproportionate to the marginal human errors that cause them.

Traditional DLP has long attempted to mitigate these impacts, but it relies heavily on perfect labelling and rigid pattern-matching. In reality, data loss rarely presents itself as a neat, well-structured pattern waiting to be caught – it looks like everyday communication, just slightly out of context.

How data loss actually happens

Most data loss comes from frustratingly familiar scenarios. A mistyped name in auto-complete sends sensitive data to the wrong “Alex.” A user forwards a document to a personal Gmail account “just this once.” Someone shares an attachment with a new or unknown correspondent without realizing how sensitive it is.

Traditional, content-centric DLP rarely catches these moments. Labels are missing or wrong. Regexes break the moment the data shifts formats. And static rules can’t interpret the context that actually matters – the sender-recipient relationship, the communication history, or whether this behavior is typical for the user.

It’s the everyday mistakes that hurt the most. The classic example: the Friday 5:58 p.m. mis-send, when auto-complete selects Martin, a former contractor, instead of Marta in Finance.

What traditional DLP approaches offer (and where gaps remain)

Most email DLP today follows two patterns, each useful but incomplete.

  • Policy- and label-centric DLP works when labels are correct — but content is often unlabeled or mislabeled, and maintaining classification adds friction. Gaps appear exactly where users move fastest
  • Rule and signature-based approaches catch known patterns but miss nuance: human error, new workflows, and “unknown unknowns” that don’t match a rule

The takeaway: Protection must combine content + behavior + explainability at send time, without depending on perfect labels.

Your technology primer: The three pillars that make outbound DLP effective

1) Label-free (vs. data classification)

Protects all content, not just what’s labeled. Label-free analysis removes classification overhead and closes gaps from missing or incorrect tags. By evaluating content and context at send time, it also catches misdelivery and other payload-free errors.

  • No labeling burden; no regex/rule maintenance
  • Works when tags are missing, wrong, or stale
  • Detects misdirected sends even when labels look right

2) Behavioral (vs. rules, signatures, threat intelligence)

Understands user behavior, not just static patterns. Behavioral analysis learns what’s normal for each person, surfacing human error and subtle exfiltration that rules can’t. It also incorporates account signals and inbound intel, extending across email and Teams.

  • Flags risk without predefined rules or IOCs
  • Catches misdelivery, unusual contacts, personal forwards, odd timing/volume
  • Blends identity and inbound context across channels

3) Proprietary DSLM (vs. generic LLM)

Optimized for precise, fast, explainable on-send decisions. A DSLM understands email/DLP semantics, avoids generative risks, and stays auditable and privacy-controlled, delivering intelligence reliably without slowing mail flow.

  • Low-latency, on-send enforcement
  • Non-generative for predictable, explainable outcomes
  • Governed model with strong privacy and auditability

The Darktrace approach to DLP

Darktrace / EMAIL – DLP stops misdelivery and sensitive data loss at send time using hold/notify/justify/release actions. It blends behavioral insight with content understanding across 35+ PII categories, protecting both labeled and unlabeled data. Every action is paired with clear explainability: AI narratives show exactly why an email was flagged, supporting analysts and helping end-users learn. Deployment aligns cleanly with existing SOC workflows through mail-flow connectors and optional Microsoft Purview label ingestion, without forcing duplicate policy-building.

Deployment is simple: Microsoft 365 routes outbound mail to Darktrace for real-time, inline decisions without regex or rule-heavy setup.

A buyer’s checklist for DLP solutions

When choosing your DLP solution, you want to be sure that it can deliver precise, explainable protection at the moment it matters – on send – without operational drag.  

To finish, we’ve compiled a handy list of questions you can ask before choosing an outbound DLP solution:

  • Can it operate label free when tags are missing or wrong? 
  • Does it truly learn per user behavior (no shortcuts)? 
  • Is there a domain specific model behind the content understanding (not a generic LLM)? 
  • Does it explain decisions to both analysts and end users? 
  • Will it integrate with your label program and SOC workflows rather than duplicate them? 

For a deep dive into Darktrace’s DLP solution, check out the full solution brief.

[related-resource]

Continue reading
About the author
Carlos Gray
Senior Product Marketing Manager, Email

Blog

/

Email

/

December 17, 2025

Beyond MFA: Detecting Adversary-in-the-Middle Attacks and Phishing with Darktrace

Beyond MFA: Detecting Adversary-in-the-Middle Attacks and Phishing with DarktraceDefault blog imageDefault blog image

What is an Adversary-in-the-middle (AiTM) attack?

Adversary-in-the-Middle (AiTM) attacks are a sophisticated technique often paired with phishing campaigns to steal user credentials. Unlike traditional phishing, which multi-factor authentication (MFA) increasingly mitigates, AiTM attacks leverage reverse proxy servers to intercept authentication tokens and session cookies. This allows attackers to bypass MFA entirely and hijack active sessions, stealthily maintaining access without repeated logins.

This blog examines a real-world incident detected during a Darktrace customer trial, highlighting how Darktrace / EMAILTM and Darktrace / IDENTITYTM identified the emerging compromise in a customer’s email and software-as-a-service (SaaS) environment, tracked its progression, and could have intervened at critical moments to contain the threat had Darktrace’s Autonomous Response capability been enabled.

What does an AiTM attack look like?

Inbound phishing email

Attacks typically begin with a phishing email, often originating from the compromised account of a known contact like a vendor or business partner. These emails will often contain malicious links or attachments leading to fake login pages designed to spoof legitimate login platforms, like Microsoft 365, designed to harvest user credentials.

Proxy-based credential theft and session hijacking

When a user clicks on a malicious link, they are redirected through an attacker-controlled proxy that impersonates legitimate services.  This proxy forwards login requests to Microsoft, making the login page appear legitimate. After the user successfully completes MFA, the attacker captures credentials and session tokens, enabling full account takeover without the need for reauthentication.

Follow-on attacks

Once inside, attackers will typically establish persistence through the creation of email rules or registering OAuth applications. From there, they often act on their objectives, exfiltrating sensitive data and launching additional business email compromise (BEC) campaigns. These campaigns can include fraudulent payment requests to external contacts or internal phishing designed to compromise more accounts and enable lateral movement across the organization.

Darktrace’s detection of an AiTM attack

At the end of September 2025, Darktrace detected one such example of an AiTM attack on the network of a customer trialling Darktrace / EMAIL and Darktrace / IDENTITY.

In this instance, the first indicator of compromise observed by Darktrace was the creation of a malicious email rule on one of the customer’s Office 365 accounts, suggesting the account had likely already been compromised before Darktrace was deployed for the trial.

Darktrace / IDENTITY observed the account creating a new email rule with a randomly generated name, likely to hide its presence from the legitimate account owner. The rule marked all inbound emails as read and deleted them, while ignoring any existing mail rules on the account. This rule was likely intended to conceal any replies to malicious emails the attacker had sent from the legitimate account owner and to facilitate further phishing attempts.

Darktrace’s detection of the anomalous email rule creation.
Figure 1: Darktrace’s detection of the anomalous email rule creation.

Internal and external phishing

Following the creation of the email rule, Darktrace / EMAIL observed a surge of suspicious activity on the user’s account. The account sent emails with subject lines referencing payment information to over 9,000 different external recipients within just one hour. Darktrace also identified that these emails contained a link to an unusual Google Drive endpoint, embedded in the text “download order and invoice”.

Darkrace’s detection of an unusual surge in outbound emails containing suspicious content, shortly following the creation of a new email rule.
Figure 2: Darkrace’s detection of an unusual surge in outbound emails containing suspicious content, shortly following the creation of a new email rule.
Darktrace / EMAIL’s detection of the compromised account sending over 9,000 external phishing emails, containing an unusual Google Drive link.
Figure 3: Darktrace / EMAIL’s detection of the compromised account sending over 9,000 external phishing emails, containing an unusual Google Drive link.

As Darktrace / EMAIL flagged the message with the ‘Compromise Indicators’ tag (Figure 2), it would have been held automatically if the customer had enabled default Data Loss Prevention (DLP) Action Flows in their email environment, preventing any external phishing attempts.

Figure 4: Darktrace / EMAIL’s preview of the email sent by the offending account.
Figure 4: Darktrace / EMAIL’s preview of the email sent by the offending account.

Darktrace analysis revealed that, after clicking the malicious link in the email, recipients would be redirected to a convincing landing page that closely mimicked the customer’s legitimate branding, including authentic imagery and logos, where prompted to download with a PDF named “invoice”.

Figure 5: Download and login prompts presented to recipients after following the malicious email link, shown here in safe view.

After clicking the “Download” button, users would be prompted to enter their company credentials on a page that was likely a credential-harvesting tool, designed to steal corporate login details and enable further compromise of SaaS and email accounts.

Darktrace’s Response

In this case, Darktrace’s Autonomous Response was not fully enabled across the customer’s email or SaaS environments, allowing the compromise to progress,  as observed by Darktrace here.

Despite this, Darktrace / EMAIL’s successful detection of the malicious Google Drive link in the internal phishing emails prompted it to suggest ‘Lock Link’, as a recommended action for the customer’s security team to manually apply. This action would have automatically placed the malicious link behind a warning or screening page blocking users from visiting it.

Autonomous Response suggesting locking the malicious Google Drive link sent in internal phishing emails.
Figure 6: Autonomous Response suggesting locking the malicious Google Drive link sent in internal phishing emails.

Furthermore, if active in the customer’s SaaS environment, Darktrace would likely have been able to mitigate the threat even earlier, at the point of the first unusual activity: the creation of a new email rule. Mitigative actions would have included forcing the user to log out, terminating any active sessions, and disabling the account.

Conclusion

AiTM attacks represent a significant evolution in credential theft techniques, enabling attackers to bypass MFA and hijack active sessions through reverse proxy infrastructure. In the real-world case we explored, Darktrace’s AI-driven detection identified multiple stages of the attack, from anomalous email rule creation to suspicious internal email activity, demonstrating how Autonomous Response could have contained the threat before escalation.

MFA is a critical security measure, but it is no longer a silver bullet. Attackers are increasingly targeting session tokens rather than passwords, exploiting trusted SaaS environments and internal communications to remain undetected. Behavioral AI provides a vital layer of defense by spotting subtle anomalies that traditional tools often miss

Security teams must move beyond static defenses and embrace adaptive, AI-driven solutions that can detect and respond in real time. Regularly review SaaS configurations, enforce conditional access policies, and deploy technologies that understand “normal” behavior to stop attackers before they succeed.

Credit to David Ison (Cyber Analyst), Bertille Pierron (Solutions Engineer), Ryan Traill (Analyst Content Lead)

Appendices

Models

SaaS / Anomalous New Email Rule

Tactic – Technique – Sub-Technique  

Phishing - T1566

Adversary-in-the-Middle - T1557

Continue reading
About the author
Your data. Our AI.
Elevate your network security with Darktrace AI