Blog
/
AI
/
October 30, 2023

Exploring AI Threats: Package Hallucination Attacks

Learn how malicious actors exploit errors in generative AI tools to launch packet attacks. Read how Darktrace products detect and prevent these threats!
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst
Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
30
Oct 2023

AI tools open doors for threat actors

On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.

ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].

However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.

Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].

Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.

What is an AI hallucination?

AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.  

Why are AI hallucinations a problem?

Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.

Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.

Use of generative AI in software development

Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.

Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].

Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.

Malicious packages as attack vectors

Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.

Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.

In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.

An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.

Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.

For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.

How could Darktrace detect an AI Package Hallucination Attack?

In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:

  • Compliance / Anomalous Upload to Generative AI
  • Compliance / Beaconing to Rare Generative AI and Generative AI
  • Compliance / Generative AI

Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.

Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:

  • Device / Anomalous GitHub Download
  • Device / Anomalous Script Download Followed By Additional Packages

Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.

As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.

Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).

Conclusion

Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.

As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.

As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.

It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.

Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst

References

[1] https://seo.ai/blog/chatgpt-user-statistics-facts

[2] https://darktrace.com/news/darktrace-addresses-generative-ai-concerns

[3] https://darktrace.com/news/darktrace-email-defends-organizations-against-evolving-cyber-threat-landscape

[4] https://vulcan.io/blog/ai-hallucinations-package-risk?nab=1&utm_referrer=https%3A%2F%2Fwww.google.com%2F

[5] https://twitter.com/OpenAI/status/1707077710047216095

[6] https://www.reuters.com/technology/openai-says-chatgpt-can-now-browse-internet-2023-09-27/

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Charlotte Thompson
Cyber Analyst
Written by
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst

More in this series

No items found.

Blog

/

Email

/

April 14, 2025

Email bombing exposed: Darktrace’s email defense in action

picture of a computer screen showing a password loginDefault blog imageDefault blog image

What is email bombing?

An email bomb attack, also known as a "spam bomb," is a cyberattack where a large volume of emails—ranging from as few as 100 to as many as several thousand—are sent to victims within a short period.

How does email bombing work?

Email bombing is a tactic that typically aims to disrupt operations and conceal malicious emails, potentially setting the stage for further social engineering attacks. Parallels can be drawn to the use of Domain Generation Algorithm (DGA) endpoints in Command-and-Control (C2) communications, where an attacker generates new and seemingly random domains in order to mask their malicious connections and evade detection.

In an email bomb attack, threat actors typically sign up their targeted recipients to a large number of email subscription services, flooding their inboxes with indirectly subscribed content [1].

Multiple threat actors have been observed utilizing this tactic, including the Ransomware-as-a-Service (RaaS) group Black Basta, also known as Storm-1811 [1] [2].

Darktrace detection of email bombing attack

In early 2025, Darktrace detected an email bomb attack where malicious actors flooded a customer's inbox while also employing social engineering techniques, specifically voice phishing (vishing). The end goal appeared to be infiltrating the customer's network by exploiting legitimate administrative tools for malicious purposes.

The emails in these attacks often bypass traditional email security tools because they are not technically classified as spam, due to the assumption that the recipient has subscribed to the service. Darktrace / EMAIL's behavioral analysis identified the mass of unusual, albeit not inherently malicious, emails that were sent to this user as part of this email bombing attack.

Email bombing attack overview

In February 2025, Darktrace observed an email bombing attack where a user received over 150 emails from 107 unique domains in under five minutes. Each of these emails bypassed a widely used and reputable Security Email Gateway (SEG) but were detected by Darktrace / EMAIL.

Graph showing the unusual spike in unusual emails observed by Darktrace / EMAIL.
Figure 1: Graph showing the unusual spike in unusual emails observed by Darktrace / EMAIL.

The emails varied in senders, topics, and even languages, with several identified as being in German and Spanish. The most common theme in the subject line of these emails was account registration, indicating that the attacker used the victim’s address to sign up to various newsletters and subscriptions, prompting confirmation emails. Such confirmation emails are generally considered both important and low risk by email filters, meaning most traditional security tools would allow them without hesitation.

Additionally, many of the emails were sent using reputable marketing tools, such as Mailchimp’s Mandrill platform, which was used to send almost half of the observed emails, further adding to their legitimacy.

 Darktrace / EMAIL’s detection of an email being sent using the Mandrill platform.
Figure 2: Darktrace / EMAIL’s detection of an email being sent using the Mandrill platform.
Darktrace / EMAIL’s detection of a large number of unusual emails sent during a short period of time.
Figure 3: Darktrace / EMAIL’s detection of a large number of unusual emails sent during a short period of time.

While the individual emails detected were typically benign, such as the newsletter from a legitimate UK airport shown in Figure 3, the harmful aspect was the swarm effect caused by receiving many emails within a short period of time.

Traditional security tools, which analyze emails individually, often struggle to identify email bombing incidents. However, Darktrace / EMAIL recognized the unusual volume of new domain communication as suspicious. Had Darktrace / EMAIL been enabled in Autonomous Response mode, it would have automatically held any suspicious emails, preventing them from landing in the recipient’s inbox.

Example of Darktrace / EMAIL’s response to an email bombing attack taken from another customer environment.
Figure 4: Example of Darktrace / EMAIL’s response to an email bombing attack taken from another customer environment.

Following the initial email bombing, the malicious actor made multiple attempts to engage the recipient in a call using Microsoft Teams, while spoofing the organizations IT department in order to establish a sense of trust and urgency – following the spike in unusual emails the user accepted the Teams call. It was later confirmed by the customer that the attacker had also targeted over 10 additional internal users with email bombing attacks and fake IT calls.

The customer also confirmed that malicious actor successfully convinced the user to divulge their credentials with them using the Microsoft Quick Assist remote management tool. While such remote management tools are typically used for legitimate administrative purposes, malicious actors can exploit them to move laterally between systems or maintain access on target networks. When these tools have been previously observed in the network, attackers may use them to pursue their goals while evading detection, commonly known as Living-off-the-Land (LOTL).

Subsequent investigation by Darktrace’s Security Operations Centre (SOC) revealed that the recipient's device began scanning and performing reconnaissance activities shortly following the Teams call, suggesting that the user inadvertently exposed their credentials, leading to the device's compromise.

Darktrace’s Cyber AI Analyst was able to identify these activities and group them together into one incident, while also highlighting the most important stages of the attack.

Figure 5: Cyber AI Analyst investigation showing the initiation of the reconnaissance/scanning activities.

The first network-level activity observed on this device was unusual LDAP reconnaissance of the wider network environment, seemingly attempting to bind to the local directory services. Following successful authentication, the device began querying the LDAP directory for information about user and root entries. Darktrace then observed the attacker performing network reconnaissance, initiating a scan of the customer’s environment and attempting to connect to other internal devices. Finally, the malicious actor proceeded to make several SMB sessions and NTLM authentication attempts to internal devices, all of which failed.

Device event log in Darktrace / NETWORK, showing the large volume of connections attempts over port 445.
Figure 6: Device event log in Darktrace / NETWORK, showing the large volume of connections attempts over port 445.
Darktrace / NETWORK’s detection of the number of the login attempts via SMB/NTLM.
Figure 7: Darktrace / NETWORK’s detection of the number of the login attempts via SMB/NTLM.

While Darktrace’s Autonomous Response capability suggested actions to shut down this suspicious internal connectivity, the deployment was configured in Human Confirmation Mode. This meant any actions required human approval, allowing the activities to continue until the customer’s security team intervened. If Darktrace had been set to respond autonomously, it would have blocked connections to port 445 and enforced a “pattern of life” to prevent the device from deviating from expected activities, thus shutting down the suspicious scanning.

Conclusion

Email bombing attacks can pose a serious threat to individuals and organizations by overwhelming inboxes with emails in an attempt to obfuscate potentially malicious activities, like account takeovers or credential theft. While many traditional gateways struggle to keep pace with the volume of these attacks—analyzing individual emails rather than connecting them and often failing to distinguish between legitimate and malicious activity—Darktrace is able to identify and stop these sophisticated attacks without latency.

Thanks to its Self-Learning AI and Autonomous Response capabilities, Darktrace ensures that even seemingly benign email activity is not lost in the noise.

Credit to Maria Geronikolou (Cyber Analyst and SOC Shift Supervisor) and Cameron Boyd (Cyber Security Analyst), Steven Haworth (Senior Director of Threat Modeling), Ryan Traill (Analyst Content Lead)

Appendices

[1] https://www.microsoft.com/en-us/security/blog/2024/05/15/threat-actors-misusing-quick-assist-in-social-engineering-attacks-leading-to-ransomware/

[2] https://thehackernews.com/2024/12/black-basta-ransomware-evolves-with.html

Darktrace Models Alerts

Internal Reconnaissance

·      Device / Suspicious SMB Scanning Activity

·      Device / Anonymous NTLM Logins

·      Device / Network Scan

·      Device / Network Range Scan

·      Device / Suspicious Network Scan Activity

·      Device / ICMP Address Scan

·      Anomalous Connection / Large Volume of LDAP Download

·      Device / Suspicious LDAP Search Operation

·      Device / Large Number of Model Alerts

Continue reading
About the author
Maria Geronikolou
Cyber Analyst

Blog

/

Email

/

April 11, 2025

FedRAMP High-compliant email security protects federal agencies from nation-state attacks

U.S. government building with flag against blue skyDefault blog imageDefault blog image

What is FedRAMP High Authority to Operate (ATO)?

Federal Risk and Authorization Management Program (FedRAMP®) High is a government-wide program that promotes the adoption of secure cloud services across the federal government by providing a standardized approach to security and risk assessment for cloud technologies and federal agencies, ensuring the protection of federal information.  

Cybersecurity is paramount in the Defense Industrial Base (DIB), where protecting sensitive information and ensuring operational resilience from the most sophisticated adversaries has national security implications. Organizations within the DIB must comply with strict security standards to work with the U.S. federal government, and FedRAMP High is one of those standards.

Darktrace achieves FedRAMP High ATO across IT, OT, and email

Last week, Darktrace Federal shared that we achieved FedRAMP® High ATO, a significant milestone that recognizes our ability to serve federal customers across IT, OT, and email via secure cloud-native deployments.  

Achieving the FedRAMP High ATO indicates that Darktrace Federal has achieved the highest standard for cloud security controls and can handle the U.S. federal government’s most sensitive, unclassified data in cloud environments.

Azure Government email security with FedRAMP High ATO

Darktrace has now released Darktrace Commercial Government Cloud High/Email (DCGC High/Email). This applies our email coverage to systems hosted in Microsoft's Azure Government, which adheres to NIST SP 800-53 controls and other federal standards. DCGC High/Email both meets and exceeds the compliance requirements of the Department of Defense’s Cybersecurity Maturity Model Certification (CMMC), providing organizations with a much-needed email security solution that delivers unparalleled, AI-driven protection against sophisticated cyber threats.

In these ways, DCGC High/Email enhances compliance, security, and operational resilience for government and federally-affiliated customers. Notably, it is crucial for securing contractors and suppliers within DIB, helping those organizations implement necessary cybersecurity practices to protect Controlled Unclassified Information (CUI) and Federal Contract Information (FCI).

Adopting DCGC High/Email ensures organizations within the DIB can work with the government without needing to invest extensive time and money into meeting the strict compliance standards.

Building DCGC High/Email to ease DIB work with the government

DCGC High/Email was built to achieve FedRAMP High standards and meet the most rigorous security standards required of our customers. This level of compliance not only allows more organizations than ever to leverage our AI-driven technology, but also ensures that customer data is protected by the highest security measures available.

The DIB has never been more critical to national security, which means they are under constant threats from nation state and cyber criminals. We built DCGC High/Email to FedRAMP High controls to ensure sensitive company and federal government communications are secured at the highest level possible.” – Marcus Fowler, CEO of Darktrace Federal

Evolving threats now necessitate DCGC High/Email

According to Darktrace’s 2025 State of AI Cybersecurity report, more than half (54%) of global government cybersecurity professionals report seeing a significant impact from AI-powered cyber threats.  

These aren’t the only types of sophisticated threats. Advanced Persistent Threats (APTs) are launched by nation-states or cyber-criminal groups with the resources to coordinate and achieve long-term objectives.  

These attacks are carefully tailored to specific targets, using techniques like social engineering and spear phishing to gain initial access via the inbox. Once inside, attackers move laterally through networks, often remaining undetected for months or even years, silently gathering intelligence or preparing for a decisive strike.  

However, the barrier for entry for these threat actors has been lowered immensely, likely related to the observed impact of AI-powered cyber threats. Securing email environments is more important than ever.  

Darktrace’s 2025 State of AI Cybersecurity report also found that 89% of government cybersecurity professionals believe AI can help significantly improve their defensive capabilities.  

Darktrace's AI-powered defensive tools are uniquely capable of detecting and neutralizing APTs and other sophisticated threats, including ones that enter via the inbox. Our Self-Learning AI continuously adapts to evolving threats, providing real-time protection.

Darktrace builds to secure the DIB to the highest degree

In summary, Darktrace Federal's achievement of FedRAMP High ATO and the introduction of DCGC High/Email mark significant advancements in our ability to protect defense contractors and federal customers against sophisticated threats that other solutions miss.

For a technical review of Darktrace Federal’s Cyber AI Mission Defense™ solution, download an independent evaluation from the Technology Advancement Center here.

[related-resource]

Continue reading
About the author
Marcus Fowler
CEO of Darktrace Federal and SVP of Strategic Engagements and Threats
Your data. Our AI.
Elevate your network security with Darktrace AI