Learn how Darktrace leveraged generative AI tools to detect and combat phishing email campaigns. Discover how AI is reshaping cybersecurity strategies.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Germaine Tan
VP, Security & AI Strategy, Field CISO
Share
26
Sep 2023
Stopping the bad while allowing the good
Since its inception, email has been regarded as one of the most important tools for businesses, revolutionizing communication and allowing global teams to become even more connected. But besides organizations heavily relying on email for their daily operations, threat actors have also recognized that the inbox is one of the easiest ways to establish an initial foothold on the network.
Today, not only are phishing campaigns and social engineering attacks becoming more prevalent, but the level of sophistication of these attacks are also increasing with the help of generative AI tools that allow for the creation of hyper-realistic emails with minimal errors, effectively lowering the barrier to entry for threat actors. These diverse and stealthy types of attacks evade traditional email security tools based on rules and signatures, because they are less likely to contain the low-sophistication markers of a typical phishing attack.
In a situation where the sky is the limit for attackers and security teams are lean, how can teams equip themselves to tackle these threats? How can they accurately detect increasingly realistic malicious emails and neutralize these threats before it is too late? And importantly, how can email security block these threats while allowing legitimate emails to flow freely?
Instead of relying on past attack data, Darktrace’s Self-Learning AI detects the slightest deviation from a user’s pattern of life and responds autonomously to contain potential threats, stopping novel attacks in their tracks before damage is caused. It doesn’t define ‘good’ and ‘bad’ like traditional email tools, rather it understands each user and what is normal for them – and what’s not.
This blog outlines how Darktrace / EMAIL™ used its understanding of ‘normal’ to accurately detect and respond to a sustained phishing campaign targeting a real-life company.
Responding to a sustained phishing attack
Over the course of 24 hours, Darktrace detected multiple emails containing different subjects, all from different senders to different recipients in one organization. These emails were sent from different IP addresses, but all came from the same autonomous system number (ASN).
Figure 1: The sender freemail addresses and subject lines all followed a certain format. The subject lines followed the format of “<First name> <Last name>”, possibly to induce curiosity. The senders were all freemail accounts and contained first names, last names and some numbers, showing the attempts to make these email addresses appear legitimate.
The emails themselves had many suspicious indicators. All senders had no prior association with the recipient, and the emails generated a high general inducement score. This score is generated by structural and non-specific content analysis of the email – a high score indicates that the email is trying to induce the recipient into taking a particular action, which may lead to account compromise.
Additionally, each email contained a visually prominent link to a file storage service, hidden behind a shortened bit.ly link. The similarities across all these emails pointed to a sustained campaign targeting the organization by a single threat actor.
Figure 2: One of the emails is shown above. Like all the other emails, it contained a highly suspicious and shortened link.
Figure 3: In another one of the emails, the link observed had similar characteristics. But this email stands out from the rest. The sender's name seems to be randomly set – the 3 alphabets are close to each other on the keyboard.
With all these suspicious indicators, many models were breached. This drove up the anomaly score, causing Darktrace/Email to hold all suspicious emails from the recipients’ inboxes, safeguarding the recipients from potential account compromise and disallowing the threats from taking hold in the network.
Imagining a phishing attack without Darktrace/Email
So what could have happened if Darktrace had not withheld these emails, and the recipients had clicked on the links? File storage sites have a wide variety of uses that allow attackers to be creative in their attack strategy. If the user had clicked on the shortened link, the possible consequences are numerous. The link could have led to a login page for unsuspecting victims to input their credentials, or it could have hosted malware that would automatically download if the link was clicked. With the compromised credentials, threat actors could even bypass MFA, change email rules, or gain privileged access to a network. The downloaded malware might also be a keylogger, leading to cryptojacking, or could open a back door for threat actors to return to at a later time.
Figure 4: Darktrace / EMAIL highlights suspicious link characteristics and provides an option to preview the pages.
Figure 5: At the point of writing, both links could not be reached. This could be because they were one-time unique links created specifically for the user, and can no longer be accessed once the campaign has ceased.
The limits of traditional email security tools
Secure email gateways (SEGs) and static AI security tools may have found it challenging to detect this phishing campaign as malicious. While Darktrace was able to correlate these emails to determine that a sustained phishing campaign was taking place, the pattern among these emails is far too generic for specific rules as set in traditional security tools. If we take the characteristic of the freemail account sender as an example, setting a rule to block all emails from freemail accounts may lead to more legitimate emails being withheld, since these addresses have a variety of uses.
With these factors in mind, these emails could have easily slipped through traditional security filters and led to a devastating impact on the organization.
Conclusion
As threat actors step up their attacks in sophistication, prioritizing email security is more crucial than ever to preserving a safe digital environment. In response to these challenges, Darktrace / EMAIL offers a set-and-forget solution that continuously learns and adapts to changes in the organization.
Through an evolving understanding of every environment in which it is deployed, its threat response becomes increasingly precise in neutralizing only the bad, while allowing the good – delivering email security that doesn’t come at the expense of business growth.
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Darktrace / EMAIL Expands Behavioral Defense Across Email and Collaboration Workflows
New capabilities help employees understand suspicious content, give administrators more control over sensitive data, and show security teams where risky workflows sit across the organization.
“I wonder if running that tool influenced the connectivity in some way.”
“I might need to target down or reboot, but first, I'll investigate the network using ARP. I wonder if ARP is responding properly. If I see "ip neigh stale," that could indicate a problem. I think arping might help in this situation, but I’ll need to keep an eye on how the network is behaving. It's all about figuring out where the issue lies!”
図10：エージェントの異常なLDAPアクティビティに対してトリガーされたこのLinuxサーバーに対するモデル検知は、Cyber AI Analystによって自律的に調査され、その調査結果にはエージェントの異常なSMBセッションアクティビティに関する内容が含まれ、Cyber AIAnalystインシデントに記録されました。
DarktraceのCyber AI Analystは、エージェントのアクティビティによってトリガーされたモデル検知を自律的に調査し、その調査結果を1つの重要なインシデントレポートにまとめました。インシデント報告書の内容は、エージェントによるネットワークスキャンから始まりコーディング演習内容のSMBベースの上書きで終わる、エージェントの行動の各段階を詳しくカバーしています。
図20：Cyber AI Analystは、モデル検知に対する自律的な調査を通じて得られた所見をつなぎ合わせ、エージェントの進行状況のタイムラインを作成し、アクティビティを視覚的に表したグラフとともに提供しました。
Cyber Verification Program下のAnthropic Opus 4.6モデルなど、他のフロンティアモデルを使用したエージェントも、Darktraceのシミュレートされた環境で同じ設定とタスクを使ってテストされました。反復の度に、エージェントは演習の内容または採点プログラムを変更して不正を行おうとしました。基盤となるモデルに関係なく、エージェントは自分が置かれた環境を侵害することで、100％のスコアを達成する方法を見つけることに成功していました。これらのエージェントによるハッキングは、アクティビティの内容こそ違っていましたが、すべてその環境のベースラインの振る舞いから大きく逸脱した動作を引き起こしていました。これらの逸脱はDarktraceプラットフォームに十分な検知の機会を提供し、エージェントの逃げ場はありませんでした。
エージェンティックハーネスは会話履歴をローカルに保存しますが、ダークトレースの研究者たちは、保存されたAIの応答が本当にモデルによって生成されたものであるかどうかの検証が行われていないことを発見しました。また、この設計上の選択がAnthropic Claude Code、AWS Kiro-CLI、OpenAI Codex、およびオープンソースのPiにわたって共通していることを確認しました。
会話履歴は多くの場合クライアント側に保存されます。例えば、Anthropic Claude Code、OpenAI Codex、AWS Kiro-CLI、Piなどでもそうです。そのため、ユーザーはセッションを自由に再開でき、一部の製品にはその履歴を操作する機能が組み込まれています。たとえば、会話の中の特定の時点に戻り、送信したメッセージを編集して、会話を別の軌道へ進めていくことができます。きわめて重要な点として、私たちが調査したすべてのケースにおいて、保存されているAIの応答が対応するモデルから生成されたものであり改ざんされていないことの検証は行われていませんでした。
_11zon.avif)
.avif)
