Blog
/
AI
/
April 1, 2023

Cyber Security Threats - Email Compromise With Generative AI

Discover how generative AI is impacting email attacks and what companies can do to prepare for more sophisticated and targeted attacker campaigns.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Max Heinemeyer
Global Field CISO
Default blog image
01
Apr 2023

Email communications in the era of generative AI

Today’s headlines on cyber security attacks are dominated by serious breaches that cripple critical national infrastructure providers, wreak geopolitical havoc through nation state espionage, and create paralyzing financial demands through extortion and ransomware. But how do these breaches occur in the first place, and why do they keep happening?

Social engineering – specifically malicious cyber campaigns delivered via email - remain the primary source of an organization’s vulnerability to attack. A recent survey conducted by Darktrace to 6,700 global cross-industry professionals across the UK, US, Australia, France, Germany, and Netherlands, found almost a third (30%) have fallen victim to a fraudulent email. The risk, as ever, seems to be uncontrollably on the rise – 70% of respondents have noticed an increase in the frequency of malicious emails in the last six months. 

At the end of last year, I wrote about how widespread accessibility to generative AI has compounded the problem, leading to likely efficiency gains for skilled threat actors launching email attacks. Just a few weeks ago, we published research which demonstrated that while the number of email attacks across our own customer base has remained steady since ChatGPT’s release, those that rely on tricking victims into clicking malicious links have declined, while linguistic complexity, including text volume, punctuation, and sentence length among others, have increased. This indicates that cyber-criminals may be redirecting their focus to crafting more sophisticated social engineering scams that exploit user trust delivered over email. 

Our latest findings, conducted by Darktrace Research, today reveal a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email customers from January to February 2023, corresponding with the widespread adoption of ChatGPT. A novel social engineering phishing email is an email attack that shows a strong linguistic deviation – semantically and syntactically – compared to other phishing emails. The trend suggests that generative AI, such as ChatGPT, is providing an avenue for threat actors to craft sophisticated and targeted attacks at speed and scale.

Organizations are already well aware of the danger novel threats pose – globally, 82% of respondents reported are concerned that hackers can use generative AI to create scam emails that are indistinguishable from genuine communication.

Navigating the trust deficit 

Employees are being exploited continuously through their primary means of communication - email. Given that humans, and our psychology, are at the heart of the problem, organizations have scrambled to develop awareness and training programs that are delivered routinely to employees in an attempt to help them defend themselves and their organization. While a defense-in-depth approach is generally a good practice to follow, the value of security awareness training, teaching employees to spot attempts at social engineering delivered in text format, is diminishing rapidly. This problem has become especially evident now that we find ourselves in a world where ‘bad’ emails look all but undistinguishable to the human eye from ‘good’ emails. 

Over emphasizing the responsibility of employees poses a business challenge as well as a security problem: a bloated trust deficit. In times of economic uncertainty, business success is predicated on employee productivity, creativity, and efficiency. All three of these are squashed when employees treat the communication they receive, both internal and external, with forensic suspicion and mistrust. 

As generative AI becomes more mainstream – across images, audio, video, and text – we anticipate that this erosion of trust in digital communication will only continue. What happens to the workplace if an employee is left questioning their own perception when communicating with their manager or a colleague, who they can see and hear over a video conference call, but who may be an entirely fictitious creation?

The way forward

The future lies in a partnership between artificial intelligence and human beings where the algorithms are responsible for determining whether communication is malicious or benign, thereby taking the burden of responsibility off the human. In turn, humans can pursue activities that drive productivity and performance. Current training and awareness programs, coupled with traditional email security technologies which assume that knowledge of past attacks will help predict campaigns, are not sustainable in this rapidly evolving landscape. 

Our research reveals that traditional email security tools that rely on knowledge of past threats, take an average of thirteen days from an attack being launched on a victim to that attack being detected. This leaves defenders vulnerable for almost two weeks if they rely solely on traditional tools. When this statistic is contextualized against the 135% increase we’ve seen in novel social engineering campaigns, the inefficiency of ‘attack centric’ email security is made perfectly evident. If we don’t know how to define entirely new and unique attacks, how can we ever hope to recognize them based on past attacks? 

A deep understanding of ‘you’ is needed in the face of these radical changes to the email threat landscape. Instead of trying to predict attacks, an understanding of your employees’ behavior, based from their email inbox, is needed to create patterns of life for every email user: their relationships, tone and sentiments and hundreds of other data points. By leveraging AI to combat email security threats, we not only reduce risk, but revitalize organizational trusts and contribute to business outcomes.

Ironically, generative AI may be worsening the social engineering challenge, but AI that knows you, could be the parry. 

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Max Heinemeyer
Global Field CISO

More in this series

No items found.

Blog

/

Proactive Security

/

June 2, 2026

Stopping Stealth Attacks with Precision: How Núclea Prevented a Breach Without Disruption

Default blog imageDefault blog image

Núclea is a Brazilian data and technology company that supports the country’s financial system by delivering digital services exclusively to banks and financial institutions. Operating in an environment where trust, availability, and data integrity are critical, the company faces a threat landscape that has evolved rapidly—particularly with the rise of AI-driven cyberattacks.

Brazil has experienced a wave of successful cyber incidents targeting financial institutions, many of them enabled by insiders or compromised credentials. The result was a noticeable shift in attacker strategy: instead of focusing on end customers, threat actors began targeting the institutions and platforms that underpin the financial ecosystem itself.

“Attacks became far more directed and contextual,” explains Guilherme, who leads incident response within Núclea’s security platform engineering team. “They weren’t noisy or obviously malicious—they were precise, patient, and designed to blend into normal operations.”

That precision was on full display in January 2026, when Núclea faced one of the most convincing phishing attacks the team had seen.

A real attack, built on trust and context

The attack began with a seemingly routine email.

It was sent from a real Brazilian government institution, using legitimate infrastructure and valid credentials that were later confirmed to have been compromised. Núclea had an established, ongoing relationship with this organization, and the email’s language, tone, and subject matter aligned perfectly with the type of communication the recipient team handled every day.

Attached to the email was a PDF document containing content that looked entirely legitimate.

The problem? A single URL embedded inside that PDF.

“The message itself was correct. The sender was real. The context was familiar. Even the document content made sense,” Guilherme explains. “There was just one small element that didn’t belong.”

That small detail was enough to initiate a full attack chain.

What the attackers were trying to do

If clicked, the URL would have downloaded a malicious payload designed to:

  • Collect information about the user and device
  • Identify where the system was located within the financial ecosystem
  • Install remote access tools to maintain control
  • Deploy an infostealer to extract sensitive data
  • Execute anti-forensic scripts to erase traces of the intrusion

In other words, it was a carefully engineered operation designed for persistence and stealth, not immediate disruption.

The attack also employed urgency—a classic social engineering technique. When the link didn’t open as expected, employees requested assistance from the security team, insisting the document was important and needed to be accessed quickly.

This is precisely the kind of scenario where traditional security tools struggle: almost everything about the interaction is legitimate.

Where Darktrace made the difference

Instead of blocking the entire message or relying on known indicators of compromise, Darktrace focused on behavioral context.

Darktrace recognized:

  • That the sending organization was normally trusted
  • That the communication pattern matched historical behavior
  • That the PDF content itself was not suspicious

But it also identified that the URL embedded within the document deviated from established behavioral patterns.

Rather than disrupting business operations, Darktrace took precise action: it rewrote the URL, preventing the malicious download while leaving the rest of the email untouched.

“When we analyzed it afterward, it became clear how dangerous the attack would have been,” says Guilherme. “But it never progressed—because Darktrace acted at exactly the right point.”

Subsequent forensic analysis confirmed the payload’s malicious intent. The attack never succeeded.

Precision over disruption

For Núclea, this incident reinforced a critical lesson: modern attacks don’t always look malicious—they hide within normal activity.

“What stands out to me is the precision,” Guilherme says. “Darktrace doesn’t rely on big, obvious signals. It’s effective in situations that fall outside the standard patterns we all know.”

Building resilience in a high trust ecosystem

For Núclea, cybersecurity is not just a defensive measure—it’s a business enabler.

Availability failures or successful breaches in the financial ecosystem can have immediate, large-scale consequences, from financial loss to reputational damage. Preventing those outcomes protects not just Núclea, but its partners and customers as well.

“Cyber resilience means keeping the business running—even under attack,” Guilherme explains. “And that requires people, processes, and technology working together.”

As AI continues to accelerate both attacks and defenses, the role of security is evolving. Precision, behavioral understanding, and intelligent automation are no longer optional—they’re essential.

“The easy days were yesterday,” Guilherme says. “The challenges ahead are bigger. We need to be prepared—internally and with partners that help us build resilience.”

Continue reading
About the author
Mariana Pereira
VP, Field CISO

Blog

/

Proactive Security

/

June 1, 2026

Defend What You Trust: Stories from the Front Lines of Modern Cyber Defense

Default blog imageDefault blog image

Modern attacks don’t always announce themselves, follow obvious patterns, or rely on known malware. Often, they move quietly inside trusted systems, authenticated sessions, and everyday behavior.

They don’t break in. They blend in.

That’s why an AI-powered defense is essential. It turns invisible signals into actionable insights at a scale neither analysts nor traditional tools can achieve alone.

Confidence is creating risk

One of the most dangerous assumptions in cybersecurity today is that strong controls equal strong protection.

Multi-factor authentication (MFA), for example, is widely viewed as a foundational safeguard. But as the CISO for a professional sports organization explains, that confidence can be misplaced. “A lot of organizations assume that once you have MFA, those accounts are safe. That’s not true.”

In one instance, his team identified a sophisticated attack where a threat actor bypassed MFA entirely, not by breaking it, but by going around it. A user’s authenticated session was hijacked and re-used, allowing the attacker to impersonate them without triggering traditional controls.

“Darktrace picked up that a session had been re-injected by the hacker, and we were able to block it right away,” he explains.

Attackers anticipate what we miss

Even well-trained users can become entry points.

“An email bypassed our existing security tools,” shares the VP of IT at a U.S.-based risk management services provider.  “The user missed one signal and entered their credentials into a malicious site. That’s what the bad guys count on.”

The organization responded quickly, but not before damage was done. Crucially, this occurred while Darktrace was in “watch mode,” before autonomous response was fully enabled. “Darktrace would have seen that and shut it down immediately,” he notes.

Mistakes and oversights like misconfigurations, forgotten machines, and missed patches can create serious vulnerabilities.

The CIO of a utility services organization shares an instance when Darktrace detected a breach to a client’s network via their ZTNA VPN due to misconfigured MFA. “Darktrace alerted us and autonomously blocked the scanning, preventing what could have been a ransomware-type incident.”  

The most dangerous threats are already inside

The Head of Security at a global business services provider knows firsthand how blind spots can persist inside environments. His team uncovered evidence of dormant ransomware artifacts sitting unnoticed within a company’s environment ¬¬– long before modern detection was in place.

“During a routine file transfer, Darktrace flagged the suspicious activity, identified the ransomware, and immediately quarantined the server,” he recalls.  While the attack was never executed, the implication was significant: the risk existed long before it was finally detected.

Cyber threats are also successful because they take advantage of normal human behavior, exploiting moments of cognitive overload, urgency, and trust.

The Executive Director of IT and Business Applications at a pharmaceutical lab describes the time Darktrace flagged an employee logging into Microsoft 365 from Singapore, despite him being physically located in the U.S. Darktrace immediately cut off his access and within minutes revealed that the employee’s son was using a VPN to play a video game.

While the threat was benign, it demonstrated the strength of AI to use contextual information to detect threats other tools miss. The information also saved security analysts hours of investigation and minimized downtime for the employee. “That level of precision and speed isn’t just convenient, it’s game changing.”

“Unusual” behavior is the new red flag

Detecting modern threats requires an understanding of what “normal” looks like and recognizing when something subtly deviates.

One security leader  at an AI technology enterprise described a scenario in which an employee connected to a proxy service in China. The service itself was legitimate, and although traditional tools didn’t flag it, the behavior was unusual for that user specifically.

“That’s what Darktrace picked up on. The activity turned out to be benign, but without visibility into behavioral deviations, it could just as easily have been something more serious.”

AI shifts defense from reaction to anticipation

These stories point to a fundamental shift by cyber attackers, both tactically and strategically. Because traditional security tools were built to detect what’s already known, modern attacks are often:

  • Credential-based, not malware-based
  • Behavioral, not signature-based
  • Subtle, not overt

They may operate within the boundaries of what appears normal, exploiting what organizations trust, not what they block:

  • Trusted sessions
  • Legitimate services
  • Human error

This is where AI is changing the equation. Rather than relying on predefined rules or known threat signatures, AI can:

  • Establish a baseline of normal behavior
  • Detect subtle anomalies in real time
  • Act autonomously to contain potential threats

Resilience, not perfection, is the new security standard

As these frontline experiences show, the organizations that lead are those that move beyond reactive defense and embrace AI as a core part of their strategy.

It eliminates the blind spots and uncertainty, says the CISO of a professional sports organization. “If you lack visibility, you’re not managing risk, you’re assuming it. AI gives you the actionable insights needed to turn uncertainty into control.”

And it provides the speed and agility that are vital when seconds matter, says the Executive Director of IT and Business Applications. “When Darktrace alerted us at 3:00 am to a ransomware attack, it had already quarantined the affected systems, blocked the attacker’s access, and provided us with the critical details and time needed to investigate. That action likely saved us hundreds of thousands, if not millions, of dollars.”

The modern SOC has become a cornerstone of enterprise resilience, responsible for protecting data and operational continuity while enabling digital growth and innovation. For today’s security professional, that means success is no longer measured by what they keep out, but by what they protect: revenue, reputation, and trust.

Continue reading
About the author
Your data. Our AI.
Elevate your network security with Darktrace AI