Blog
/

Inside the SOC

/
October 30, 2023

Exploring AI Threats: Package Hallucination Attacks

Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
30
Oct 2023
Learn how malicious actors exploit errors in generative AI tools to launch packet attacks. Read how Darktrace products detect and prevent these threats!

AI tools open doors for threat actors

On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.

ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].

However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.

Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].

Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.

What is an AI hallucination?

AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.  

Why are AI hallucinations a problem?

Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.

Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.

Use of generative AI in software development

Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.

Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].

Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.

Malicious packages as attack vectors

Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.

Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.

In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.

An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.

Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.

For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.

How could Darktrace detect an AI Package Hallucination Attack?

In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:

  • Compliance / Anomalous Upload to Generative AI
  • Compliance / Beaconing to Rare Generative AI and Generative AI
  • Compliance / Generative AI

Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.

Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:

  • Device / Anomalous GitHub Download
  • Device / Anomalous Script Download Followed By Additional Packages

Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.

As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.

Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).

Conclusion

Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.

As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.

As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.

It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.

Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst

References

[1] https://seo.ai/blog/chatgpt-user-statistics-facts

[2] https://darktrace.com/news/darktrace-addresses-generative-ai-concerns

[3] https://darktrace.com/news/darktrace-email-defends-organizations-against-evolving-cyber-threat-landscape

[4] https://vulcan.io/blog/ai-hallucinations-package-risk?nab=1&utm_referrer=https%3A%2F%2Fwww.google.com%2F

[5] https://twitter.com/OpenAI/status/1707077710047216095

[6] https://www.reuters.com/technology/openai-says-chatgpt-can-now-browse-internet-2023-09-27/

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Author
Charlotte Thompson
Cyber Analyst
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst
Book a 1-1 meeting with one of our experts
Share this article

More in this series

No items found.

Blog

/

January 14, 2025

/

Ransomware

RansomHub Ransomware: Darktrace’s Investigation of the Newest Tool in ShadowSyndicate's Arsenal

Default blog imageDefault blog image

What is ShadowSyndicate?

ShadowSyndicate, also known as Infra Storm, is a threat actor reportedly active since July 2022, working with various ransomware groups and affiliates of ransomware programs, such as Quantum, Nokoyawa, and ALPHV. This threat actor employs tools like Cobalt Strike, Sliver, IcedID, and Matanbuchus malware in its attacks. ShadowSyndicate utilizes the same SSH fingerprint (1ca4cbac895fc3bd12417b77fc6ed31d) on many of their servers—85 as of September 2023. At least 52 of these servers have been linked to the Cobalt Strike command and control (C2) framework [1].

What is RansomHub?

First observed following the FBI's takedown of ALPHV/BlackCat in December 2023, RansomHub quickly gained notoriety as a Ransomware-as-a-Service (RaaS) operator. RansomHub capitalized on the law enforcement’s disruption of the LockBit group’s operations in February 2024 to market themselves to potential affiliates who had previously relied on LockBit’s encryptors. RansomHub's success can be largely attributed to their aggressive recruitment on underground forums, leading to the absorption of ex-ALPHV and ex-LockBit affiliates. They were one of the most active ransomware operators in 2024, with approximately 500 victims reported since February, according to their Dedicated Leak Site (DLS) [2].

ShadowSyndicate and RansomHub

External researchers have reported that ShadowSyndicate had as many as seven different ransomware families in their arsenal between July 2022, and September 2023. Now, ShadowSyndicate appears to have added RansomHub’s their formidable stockpile, becoming an affiliate of the RaaS provider [1].

Darktrace’s analysis of ShadowSyndicate across its customer base indicates that the group has been leveraging RansomHub ransomware in multiple attacks in September and October 2024. ShadowSyndicate likely shifted to using RansomHub due to the lucrative rates offered by this RaaS provider, with affiliates receiving up to 90% of the ransom—significantly higher than the general market rate of 70-80% [3].

In many instances where encryption was observed, ransom notes with the naming pattern “README_[a-zA-Z0-9]{6}.txt” were written to affected devices. The content of these ransom notes threatened to release stolen confidential data via RansomHub’s DLS unless a ransom was paid. During these attacks, data exfiltration activity to external endpoints using the SSH protocol was observed. The external endpoints to which the data was transferred were found to coincide with servers previously associated with ShadowSyndicate activity.

Darktrace’s coverage of ShadowSyndicate and RansomHub

Darktrace’s Threat Research team identified high-confidence indicators of compromise (IoCs) linked to the ShadowSyndicate group deploying RansomHub. The investigation revealed four separate incidents impacting Darktrace customers across various sectors, including education, manufacturing, and social services. In the investigated cases, multiple stages of the kill chain were observed, starting with initial internal reconnaissance and leading to eventual file encryption and data exfiltration.

Attack Overview

Timeline attack overview of ransomhub ransomware

Internal Reconnaissance

The first observed stage of ShadowSyndicate attacks involved devices making multiple internal connection attempts to other internal devices over key ports, suggesting network scanning and enumeration activity. In this initial phase of the attack, the threat actor gathers critical details and information by scanning the network for open ports that might be potentially exploitable. In cases observed by Darktrace affected devices were typically seen attempting to connect to other internal locations over TCP ports including 22, 445 and 3389.

C2 Communication and Data Exfiltration

In most of the RansomHub cases investigated by Darktrace, unusual connections to endpoints associated with Splashtop, a remote desktop access software, were observed briefly before outbound SSH connections were identified.

Following this, Darktrace detected outbound SSH connections to the external IP address 46.161.27[.]151 using WinSCP, an open-source SSH client for Windows used for secure file transfer. The Cybersecurity and Infrastructure Security Agency (CISA) identified this IP address as malicious and associated it with ShadowSyndicate’s C2 infrastructure [4]. During connections to this IP, multiple gigabytes of data were exfiltrated from customer networks via SSH.

Data exfiltration attempts were consistent across investigated cases; however, the method of egress varied from one attack to another, as one would expect with a RaaS strain being employed by different affiliates. In addition to transfers to ShadowSyndicate’s infrastructure, threat actors were also observed transferring data to the cloud storage and file transfer service, MEGA, via HTTP connections using the ‘rclone’ user agent – a command-line program used to manage files on cloud storage. In another case, data exfiltration activity occurred over port 443, utilizing SSL connections.

Lateral Movement

In investigated incidents, lateral movement activity began shortly after C2 communications were established. In one case, Darktrace identified the unusual use of a new administrative credential which was quickly followed up with multiple suspicious executable file writes to other internal devices on the network.

The filenames for this executable followed the regex naming convention “[a-zA-Z]{6}.exe”, with two observed examples being “bWqQUx.exe” and “sdtMfs.exe”.

Cyber AI Analyst Investigation Process for the SMB Writes of Suspicious Files to Multiple Devices' incident.
Figure 1: Cyber AI Analyst Investigation Process for the SMB Writes of Suspicious Files to Multiple Devices' incident.

Additionally, script files such as “Defeat-Defender2.bat”, “Share.bat”, and “def.bat” were also seen written over SMB, suggesting that threat actors were trying to evade network defenses and detection by antivirus software like Microsoft Defender.

File Encryption

Among the three cases where file encryption activity was observed, file names were changed by adding an extension following the regex format “.[a-zA-Z0-9]{6}”. Ransom notes with a similar naming convention, “README_[a-zA-Z0-9]{6}.txt”, were written to each share. While the content of the ransom notes differed slightly in each case, most contained similar text. Clear indicators in the body of the ransom notes pointed to the use of RansomHub ransomware in these attacks. As is increasingly the case, threat actors employed double extortion tactics, threatening to leak confidential data if the ransom was not paid. Like most ransomware, RansomHub included TOR site links for communication between its "customer service team" and the target.

Figure 2: The graph shows the behavior of a device with encryption activity, using the “SMB Sustained Mimetype Conversion” and “Unusual Activity Events” metrics over three weeks.

Since Darktrace’s Autonomous Response capability was not enabled during the compromise, the ransomware attack succeeded in its objective. However, Darktrace’s Cyber AI Analyst provided comprehensive coverage of the kill chain, enabling the customer to quickly identify affected devices and initiate remediation.

Figure 3: Cyber AI Analyst panel showing the critical incidents of the affected device from one of the cases investigated.

In lieu of Autonomous Response being active on the networks, Darktrace was able to suggest a variety of manual response actions intended to contain the compromise and prevent further malicious activity. Had Autonomous Response been enabled at the time of the attack, these actions would have been quickly applied without any human interaction, potentially halting the ransomware attack earlier in the kill chain.

Figure 4: A list of suggested Autonomous Response actions on the affected devices."

Conclusion

The Darktrace Threat Research team has noted a surge in attacks by the ShadowSyndicate group using RansomHub’s RaaS of late. RaaS has become increasingly popular across the threat landscape due to its ease of access to malware and script execution. As more individual threat actors adopt RaaS, security teams are struggling to defend against the increasing number of opportunistic attacks.

For customers subscribed to Darktrace’s Security Operations Center (SOC) services, the Analyst team promptly investigated detections of the aforementioned unusual and anomalous activities in the initial infection phases. Multiple alerts were raised via Darktrace’s Managed Threat Detection to warn customers of active ransomware incidents. By emphasizing anomaly-based detection and response, Darktrace can effectively identify devices affected by ransomware and take action against emerging activity, minimizing disruption and impact on customer networks.

Credit to Kwa Qing Hong (Senior Cyber Analyst and Deputy Analyst Team Lead, Singapore) and Signe Zahark (Principal Cyber Analyst, Japan)

Appendices

Darktrace Model Detections

Antigena Models / Autonomous Response:

Antigena / Network / Insider Threat / Antigena Network Scan Block

Antigena / Network / Insider Threat / Antigena SMB Enumeration Block

Antigena / Network / Insider Threat / Antigena Internal Anomalous File Activity

Antigena / Network / Insider Threat / Antigena Large Data Volume Outbound Block

Antigena / Network / Significant Anomaly / Antigena Significant Anomaly from Client Block

Antigena / Network / Significant Anomaly / Antigena Breaches Over Time Block

Antigena / Network / Significant Anomaly / Antigena Controlled and Model Breach

Antigena / Network / Significant Anomaly / Antigena Significant Server Anomaly Block

Antigena / Network / Significant Anomaly / Antigena Enhanced Monitoring from Server Block

Antigena / Network / External Threat / Antigena Suspicious Activity Block

Antigena / Network / External Threat / Antigena Suspicious File Pattern of Life Block

Antigena / Network / External Threat / Antigena File then New Outbound Block


Network Reconnaissance:

Device / Network Scan

Device / ICMP Address Scan

Device / RDP Scan
Device / Anomalous LDAP Root Searches
Anomalous Connection / SMB Enumeration
Device / Spike in LDAP Activity

C2:

Enhanced Monitoring - Device / Lateral Movement and C2 Activity

Enhanced Monitoring - Device / Initial Breach Chain Compromise

Enhanced Monitoring - Compromise / Suspicious File and C2

Compliance / Remote Management Tool On Server

Anomalous Connection / Outbound SSH to Unusual Port


External Data Transfer:

Enhanced Monitoring - Unusual Activity / Enhanced Unusual External Data Transfer

Unusual Activity / Unusual External Data Transfer

Anomalous Connection / Data Sent to Rare Domain

Unusual Activity / Unusual External Data to New Endpoint

Compliance / SSH to Rare External Destination

Anomalous Connection / Application Protocol on Uncommon Port

Enhanced Monitoring - Anomalous File / Numeric File Download

Anomalous File / New User Agent Followed By Numeric File Download

Anomalous Server Activity / Outgoing from Server

Device / Large Number of Connections to New Endpoints

Anomalous Connection / Multiple HTTP POSTs to Rare Hostname

Anomalous Connection / Uncommon 1 GiB Outbound

Lateral Movement:

User / New Admin Credentials on Server

Anomalous Connection / New or Uncommon Service Control

Anomalous Connection / High Volume of New or Uncommon Service Control

Anomalous File / Internal / Executable Uploaded to DC

Anomalous Connection / Suspicious Activity On High Risk Device

File Encryption:

Compliance / SMB Drive Write

Anomalous File / Internal / Additional Extension Appended to SMB File

Compromise / Ransomware / Possible Ransom Note Write

Anomalous Connection / Suspicious Read Write Ratio

List of Indicators of Compromise (IoCs)

IoC - Type - Description + Confidence

83.97.73[.]198 - IP - Data exfiltration endpoint

108.181.182[.]143 - IP - Data exfiltration endpoint

46.161.27[.]151 - IP - Data exfiltration endpoint

185.65.212[.]164 - IP - Data exfiltration endpoint

66[.]203.125.21 - IP - MEGA endpoint used for data exfiltration

89[.]44.168.207 - IP - MEGA endpoint used for data exfiltration

185[.]206.24.31 - IP - MEGA endpoint used for data exfiltration

31[.]216.148.33 - IP - MEGA endpoint used for data exfiltration

104.226.39[.]18 - IP - C2 endpoint

103.253.40[.]87 - IP - C2 endpoint

*.relay.splashtop[.]com - Hostname - C2 & data exfiltration endpoint

gfs***n***.userstorage.mega[.]co.nz - Hostname - MEGA endpoint used for data exfiltration

w.api.mega[.]co.nz - Hostname - MEGA endpoint used for data exfiltration

ams-rb9a-ss.ams.efscloud[.]net - Hostname - Data exfiltration endpoint

MITRE ATT&CK Mapping

Tactic - Technqiue

RECONNAISSANCE – T1592.004 Client Configurations

RECONNAISSANCE – T1590.005 IP Addresses

RECONNAISSANCE – T1595.001 Scanning IP Blocks

RECONNAISSANCE – T1595.002 Vulnerability Scanning

DISCOVERY – T1046 Network Service Scanning

DISCOVERY – T1018 Remote System Discovery

DISCOVERY – T1083 File and Directory Discovery
INITIAL ACCESS - T1189 Drive-by Compromise

INITIAL ACCESS - T1190 Exploit Public-Facing Application

COMMAND AND CONTROL - T1001 Data Obfuscation

COMMAND AND CONTROL - T1071 Application Layer Protocol

COMMAND AND CONTROL - T1071.001 Web Protocols

COMMAND AND CONTROL - T1573.001 Symmetric Cryptography

COMMAND AND CONTROL - T1571 Non-Standard Port

DEFENSE EVASION – T1078 Valid Accounts

DEFENSE EVASION – T1550.002 Pass the Hash

LATERAL MOVEMENT - T1021.004 SSH

LATERAL MOVEMENT – T1080 Taint Shared Content

LATERAL MOVEMENT – T1570 Lateral Tool Transfer

LATERAL MOVEMENT – T1021.002 SMB/Windows Admin Shares

COLLECTION - T1185 Man in the Browser

EXFILTRATION - T1041 Exfiltration Over C2 Channel

EXFILTRATION - T1567.002 Exfiltration to Cloud Storage

EXFILTRATION - T1029 Scheduled Transfer

IMPACT – T1486 Data Encrypted for Impact

References

1.     https://www.group-ib.com/blog/shadowsyndicate-raas/

2.     https://www.techtarget.com/searchsecurity/news/366617096/ESET-RansomHub-most-active-ransomware-group-in-H2-2024

3.     https://cyberint.com/blog/research/ransomhub-the-new-kid-on-the-block-to-know/

4.     https://www.cisa.gov/sites/default/files/2024-05/AA24-131A.stix_.xml

Continue reading
About the author
Qing Hong Kwa
Senior Cyber Analyst and Deputy Analyst Team Lead, Singapore

Blog

/

January 14, 2025

/

Email

Why AI-powered Email Protection Became Essential for this Global Financial Services Leader

Default blog imageDefault blog image

When agile cyber-attackers don’t stop, but pivot  

When he first joined this leading financial services provider, it was clear to the CISO that email security needed to be a top priority. The organization provides transfer services to millions of consumers via a network of thousands of agent locations across the US. Those agents are connected to hundreds of thousands of global payers to complete consumer transfers, ranging from leading financial institutions to small local businesses.

With this vast network of agents and payers, the provider relies on email as its primary communications channel. Transmitting billions of dollars every year, the organization is a prime target for cyber criminals looking to steal credentials, financial assets, and sensitive data.

Vulnerable to attacks with gaps in email security and visibility

The CISO discovered that employees were under constant attack by phishing emails impersonating his company’s own executives. The business email compromise (BEC) attacks were designed to deceive employees into sharing credentials or clicking on malicious links.

Upon discovering that their Microsoft 365 tenant lacked secure configuration, the CISO implemented necessary changes to strengthen the service, including enabling authentication controls. While his efforts significantly reduced BEC attacks, cyber criminals changed their tactics, sending employees malicious phishing emails from seemingly valid email accounts from trusted domains like Google and Yahoo. The emails passed through the organization’s native email filters without detection.

The CISO also sought to strengthen defenses against third-party supply chain attacks that could originate with any of the hundreds of thousands of third-party agents and payers the company works with around the world. While the larger institutions typically have sophisticated email security strategies in place, the smaller businesses may lack the cybersecurity expertise needed to effectively secure and manage their data, putting the organization at risk.

While the CISO knew the company was vulnerable to phishing and third-party threats, he didn’t have visibility across the flow of email. Without access to key metrics and valuable data, he couldn’t get the crucial insights needed to quickly identify possible threats and adjust security protocols.  

Skilled analysts bogged down with low-level tasks

Like many enterprise organizations, this leading financial services provider relied on a crew of highly skilled analysts to respond to alerts and analyze and triage emails most of their workday. “That shouldn’t be how we operate,” said the CISO. “My role and the role of my staff should be to focus on more strategic projects, support the business, and work on important new product development.”

Balancing user experience with mitigating threats

Enabling greater email security measures without negatively impacting the business, user experience, and customer satisfaction was a daunting challenge the CISO and his security team faced. Imposing restrictions that are too stringent could restrict communication, delay the delivery of important messages, or block legitimate emails – potentially slowing down money transfers, frustrating customers, affecting employee productivity, and impacting revenue. However, maintaining controls that are too permissive could result in serious outcomes like data theft, financial fraud, operational disruption, compliance penalties, and customer attrition.  

Self-Learning AI is a game changer

After conducting a thorough POC with several modern security solution providers, this global financial services provider chose the Darktrace / EMAIL an AI-driven email security platform. The CISO said they chose the solution for two key reasons:

First, Darktrace / EMAIL offers modern capabilities

  • Self-Learning AI uses business data to recognize anomalies in communication patterns and user behavior to stop known and unknown threats
  • Secures the organization’s entire mailflow across all inbound, outbound, and lateral email
  • Protects against account takeover attacks by identifying subtle anomalies in cloud SaaS
  • Catches sophisticated threats like impersonations, session token misuse, adversary-in-the-middle attacks, credential theft, and data exfiltration

Second, they pointed to Darktrace’s experience, innovation, and expertise

  • Deep cybersecurity and industry knowledge
  • Demonstrated customers successes worldwide
  • At the forefront of innovation and research, establishing new thresholds in cybersecurity, with technology advances backed by over 200 patents and pending applications

Moreover, and most importantly, this organization trusted Darktrace to deliver on its promises.  And according to the CISO, that’s just what happened.

Significantly reduced phishing threats and business risk

Since implementing Darktrace / EMAIL, the threat posed by BEC attacks has dropped sharply. “Phishing is not an issue that concerns me anymore. I estimate we are now identifying and blocking more than 85% of threats our previous solution was missing,” said the CISO. The biggest factor contributing to this success? The power of AI.

With Darktrace / EMAIL, this leadingglobal financial services provider is identifying and blocking more than 85% ofthe phishing email threats its previous solution missed.

AI wasn’t originally on the financial service provider’s list of criteria. But after seeing AI in action and understanding its potential to vastly scale their detection and response capabilities–without adding headcount, the CISO determined AI wasn’t an option but an imperative. “AI is essential when it comes to email security, it’s an absolute necessity,” he said.  

Darktrace / EMAIL’s Self-Learning AI is uniquely powerful because it learns the content and context of every internal and external user and can spot the subtle differences in behavioral patterns that point to possible social engineering attacks. Through patented behavioral anomaly detection, Darktrace / EMAIL continuously learns about the organization’s business and users, based on its own operations and data, adjusting security protocols accordingly.  

For example, when clients are transferring large amounts of money, they are required to send photos of their driver’s licenses and passports via email to the organization for verification – accounting for a large percentage of its’ inbound email. Darktrace / EMAIL recognizes that it’s normal for customers to send this sensitive information, and it also knows that it’s not normal for that same sensitive information to leave the organization via outbound mail. In addition, Darktrace identifies patterns in user behavior, including who employees communicate with and what kind of information they share. When user behavior falls outside of established norms, such as an email sent from the CFO to employees the CEO would not typically communicate with, Darktrace can take the appropriate action to remove the threat.  

“After the implementation, we gave the solution two weeks to ingest our data and learn the specifics of our business. After that, it was perfect, just amazing,” said the CISO.  

Boosted team productivity and elevated value to the business

With Darktrace / EMAIL, the organization has successfully scaled its detection and response efforts without scaling personnel. The security team has reduced the number of emails requiring manual investigation by 90%. And because analysts now have the benefit of Darktrace / EMAIL’s analytics and reporting, the investigation process is much easier and faster. “The impact of this solution on my team has been very positive,” said the CISO. “Darktrace / EMAIL essentially manages itself, freeing up time for our skilled analysts–and for myself–to focus on more important projects.”  

The security team has scaled its detection and response efforts without scaling personnel,reducing the number of emails it manually investigates by 90%

Increased visibility delivers business-critical insights

You can’t control what you can’t see, and with zero visibility into critical data and metrics, this financial services provider was at a serious disadvantage. That has all changed. “Something that I love about Darktrace / EMAIL is the visibility that it provides into key metrics from a single dashboard. We can now understand the behavior of our email flow and data traffic and can make insight-driven decisions to continuously optimize our email security. It’s awesome,” said the CISO.  

An efficient user interface also improves productivity and reduces mean time to action by enabling teams to easily visualize key data points and quickly evaluate what actions need to be taken. Darktrace / EMAIL was developed with that experience in mind, allowing users to access data and take quick action without having to constantly log into the solution.

Keeping the business focused on cybersecurity

The leadership of this global organization takes information security very seriously, understanding that cyber-attacks aren’t just an IT problem but a business problem. When it came to evaluating Darktrace, the CISO said numerous stakeholders were involved including C-level executives, infrastructure, and IT, which operates separately from information security. The CISO initially identified the need, conducted the market research, engaged the target vendors, and then brought the other decision makers into the process for the solution evaluation and final decision. “Our IT group, infrastructure team, CTO and CEO are all involved when it comes to making major cybersecurity investments. We always try to make these decisions jointly to ensure we are taking everything into consideration.”

The organization has reached a higher level of maturity when it comes to email cybersecurity. The ability to automate routine email detection and investigation tasks has both strengthened the organization’s cyber resilience and enabled the CISO and his team to contribute more to the business. His advice for other IT leaders facing the same email security and visibility challenges he once experienced: “For those companies that need greater insight and control over their email but have limited resources and people, AI is the answer.”  

Darktrace / Email solution brief screenshot

Secure Your Inbox with Cutting-Edge AI Email Protection

Discover the most advanced cloud-native AI email security solution to protect your domain and brand while preventing phishing, novel social engineering, business email compromise, account takeover, and data loss.

  • Gain up to 13 days of earlier threat detection and maximize ROI on your current email security
  • Experience 20-25% more threat blocking power with Darktrace / EMAIL
  • Stop the 58% of threats bypassing traditional email security

Continue reading
About the author
The Darktrace Community
Your data. Our AI.
Elevate your network security with Darktrace AI