ブログ
/
Network
/
September 4, 2022

Steps of a BumbleBee Intrusion to a Cobalt Strike

Discover the steps of a Bumblebee intrusion, from initial detection to Cobalt Strike deployment. Learn how Darktrace defends against evolving threats with AI.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Sam Lister
Specialist Security Researcher
Default blog image
04
Sep 2022

Introduction

Throughout April 2022, Darktrace observed several cases in which threat actors used the loader known as ‘BumbleBee’ to install Cobalt Strike Beacon onto victim systems. The threat actors then leveraged Cobalt Strike Beacon to conduct network reconnaissance, obtain account password data, and write malicious payloads across the network. In this article, we will provide details of the actions threat actors took during their intrusions, as well as details of the network-based behaviours which served as evidence of the actors’ activities.  

BumbleBee 

In March 2022, Google’s Threat Analysis Group (TAG) provided details of the activities of an Initial Access Broker (IAB) group dubbed ‘Exotic Lily’ [1]. Before March 2022, Google’s TAG observed Exotic Lily leveraging sophisticated impersonation techniques to trick employees of targeted organisations into downloading ISO disc image files from legitimate file storage services such as WeTransfer. These ISO files contained a Windows shortcut LNK file and a BazarLoader Dynamic Link Library (i.e, DLL). BazarLoader is a member of the Bazar family — a family of malware (including both BazarLoader and BazarBackdoor) with strong ties to the Trickbot malware, the Anchor malware family, and Conti ransomware. BazarLoader, which is typically distributed via email campaigns or via fraudulent call campaigns, has been known to drop Cobalt Strike as a precursor to Conti ransomware deployment [2]. 

In March 2022, Google’s TAG observed Exotic Lily leveraging file storage services to distribute an ISO file containing a DLL which, when executed, caused the victim machine to make HTTP requests with the user-agent string ‘bumblebee’. Google’s TAG consequently called this DLL payload ‘BumbleBee’. Since Google’s discovery of BumbleBee back in March, several threat research teams have reported BumbleBee samples dropping Cobalt Strike [1]/[3]/[4]/[5]. It has also been reported by Proofpoint [3] that other threat actors such as TA578 and TA579 transitioned to BumbleBee in March 2022.  

Interestingly, BazarLoader’s replacement with BumbleBee seems to coincide with the leaking of the Conti ransomware gang’s Jabber chat logs at the end of February 2022. On February 25th, 2022, the Conti gang published a blog post announcing their full support for the Russian state’s invasion of Ukraine [6]. 

Figure 1: The Conti gang's public declaration of their support for Russia's invasion of Ukraine

Within days of sharing their support for Russia, logs from a server hosting the group’s Jabber communications began to be leaked on Twitter by @ContiLeaks [7]. The leaked logs included records of conversations among nearly 500 threat actors between Jan 2020 and March 2022 [8]. The Jabber logs were supposedly stolen and leaked by a Ukrainian security researcher [3]/[6].

Affiliates of the Conti ransomware group were known to use BazarLoader to deliver Conti ransomware [9]. BumbleBee has now also been linked to the Conti ransomware group by several threat research teams [1]/[10]/[11]. The fact that threat actors’ transition from BazarLoader to BumbleBee coincides with the leak of Conti’s Jabber chat logs may indicate that the transition occurred as a result of the leaks [3]. Since the transition, BumbleBee has become a significant tool in the cyber-crime ecosystem, with links to several ransomware operations such as Conti, Quantum, and Mountlocker [11]. The rising use of BumbleBee by threat actors, and particularly ransomware actors, makes the early detection of BumbleBee key to identifying the preparatory stages of ransomware attacks.  

Intrusion Kill Chain 

In April 2022, Darktrace observed the following pattern of threat actor activity within the networks of several Darktrace clients: 

1.     Threat actor socially engineers user via email into running a BumbleBee payload on their device

2.     BumbleBee establishes HTTPS communication with a BumbleBee C2 server

3.     Threat actor instructs BumbleBee to download and execute Cobalt Strike Beacon

4.     Cobalt Strike Beacon establishes HTTPS communication with a Cobalt Strike C2 server

5.     Threat actor instructs Cobalt Strike Beacon to scan for open ports and to enumerate network shares

6.     Threat actor instructs Cobalt Strike Beacon to use the DCSync technique to obtain password account data from an internal domain controller

7.     Threat actor instructs Cobalt Strike Beacon to distribute malicious payloads to other internal systems 

With limited visibility over affected clients’ email environments, Darktrace was unable to determine how the threat actors interacted with users to initiate the BumbleBee infection. However, based on open-source reporting on BumbleBee [3]/[4]/[10]/[11]/[12]/[13]/[14]/[15]/[16]/[17], it is likely that the actors tricked target users into running BumbleBee by sending them emails containing either a malicious zipped ISO file or a link to a file storage service hosting the malicious zipped ISO file. These ISO files typically contain a LNK file and a BumbleBee DLL payload. The properties of these LNK files are set in such a way that opening them causes the corresponding DLL payload to run. 

In several cases observed by Darktrace, devices contacted a file storage service such as Microsoft OneDrive or Google Cloud Storage immediately before they displayed signs of BumbleBee infection. In these cases, it is likely that BumbleBee was executed on the users’ devices as a result of the users interacting with an ISO file which they were tricked into downloading from a file storage service. 

Figure 2: The above figure, taken from the event log for an infected device, shows that the device contacted a OneDrive endpoint immediately before making HTTPS connections to the BumbleBee C2 server, 45.140.146[.]244
Figure 3: The above figure, taken from the event log for an infected device, shows that the device contacted a Google Cloud Storage endpoint and then the malicious endpoint ‘marebust[.]com’ before making HTTPS connections to the  BumbleBee C2 servers, 108.62.118[.]61 and 23.227.198[.]217

After users ran a BumbleBee payload, their devices immediately initiated communications with BumbleBee C2 servers. The BumbleBee samples used HTTPS for their C2 communication, and all presented a common JA3 client fingerprint, ‘0c9457ab6f0d6a14fc8a3d1d149547fb’. All analysed samples excluded domain names in their ‘client hello’ messages to the C2 servers, which is unusual for legitimate HTTPS communication. External SSL connections which do not specify a destination domain name and whose JA3 client fingerprint is ‘0c9457ab6f0d6a14fc8a3d1d149547fb’ are potential indicators of BumbleBee infection. 

Figure 4:The above figure, taken from Darktrace's Advanced Search interface, depicts an infected device's spike in HTTPS connections with the JA3 client fingerprint ‘0c9457ab6f0d6a14fc8a3d1d149547fb’

Once the threat actors had established HTTPS communication with the BumbleBee-infected systems, they instructed BumbleBee to download and execute Cobalt Strike Beacon. This behaviour resulted in the infected systems making HTTPS connections to Cobalt Strike C2 servers. The Cobalt Strike Beacon samples all had the same JA3 client fingerprint ‘a0e9f5d64349fb13191bc781f81f42e1’ — a fingerprint associated with previously seen Cobalt Strike samples [18]. The domain names ‘fuvataren[.]com’ and ‘cuhirito[.]com’ were observed in the samples’ HTTPS communications. 

Figure 5:The above figure, taken from Darktrace's Advanced Search interface, depicts the Cobalt Strike C2 communications which immediately followed a device's BumbleBee C2 activity

Cobalt Strike Beacon payloads call home to C2 servers for instructions. In the cases observed, threat actors first instructed the Beacon payloads to perform reconnaissance tasks, such as SMB port scanning and SMB enumeration. It is likely that the threat actors performed these steps to inform the next stages of their operations.  The SMB enumeration activity was evidenced by the infected devices making NetrShareEnum and NetrShareGetInfo requests to the srvsvc RPC interface on internal systems.

Figure 6: The above figure, taken from Darktrace’s Advanced Search interface, depicts a spike in srvsvc requests coinciding with the infected device's Cobalt Strike C2 activity

After providing Cobalt Strike Beacon with reconnaissance tasks, the threat actors set out to obtain account password data in preparation for the lateral movement phase of their operation. To obtain account password data, the actors instructed Cobalt Strike Beacon to use the DCSync technique to replicate account password data from an internal domain controller. This activity was evidenced by the infected devices making DRSGetNCChanges requests to the drsuapi RPC interface on internal domain controllers. 

Figure 7: The above figure, taken from Darktrace’s Advanced Search interface, depicts a spike in DRSGetNCChanges requests coinciding with the infected device’s Cobalt Strike C2 activity

After leveraging the DCSync technique, the threat actors sought to broaden their presence within the targeted networks.  To achieve this, they instructed Cobalt Strike Beacon to get several specially selected internal systems to run a suspiciously named DLL (‘f.dll’). Cobalt Strike first established SMB sessions with target systems using compromised account credentials. During these sessions, Cobalt Strike uploaded the malicious DLL to a hidden network share. To execute the DLL, Cobalt Strike abused the Windows Service Control Manager (SCM) to remotely control and manipulate running services on the targeted internal hosts. Cobalt Strike first opened a binding handle to the svcctl interface on the targeted destination systems. It then went on to make an OpenSCManagerW request, a CreateServiceA request, and a StartServiceA request to the svcctl interface on the targeted hosts: 

·      Bind request – opens a binding handle to the relevant RPC interface (in this case, the svcctl interface) on the destination device

·      OpenSCManagerW request – establishes a connection to the Service Control Manager (SCM) on the destination device and opens a specified SCM database

·      CreateServiceA request – creates a service object and adds it to the specified SCM database 

·      StartServiceA request – starts a specified service

Figure 8: The above figure, taken from Darktrace’s Advanced Search interface, outlines an infected system’s lateral movement activities. After writing a file named ‘f.dll’ to the C$ share on an internal server, the infected device made several RPC requests to the svcctl interface on the targeted server

It is likely that the DLL file which the threat actors distributed was a Cobalt Strike payload. In one case, however, the threat actor was also seen distributing and executing a payload named ‘procdump64.exe’. This may suggest that the threat actor was seeking to use ProcDump to obtain authentication material stored in the process memory of the Local Security Authority Subsystem Service (LSASS). Given that ProcDump is a legitimate Windows Sysinternals tool primarily used for diagnostics and troubleshooting, it is likely that threat actors leveraged it in order to evade detection. 

In all the cases which Darktrace observed, threat actors’ attempts to conduct follow-up activities after moving laterally were thwarted with the help of Darktrace’s SOC team. It is likely that the threat actors responsible for the reported activities were seeking to deploy ransomware within the targeted networks. The steps which the threat actors took to make progress towards achieving this objective resulted in highly unusual patterns of network traffic. Darktrace’s detection of these unusual network activities allowed security teams to prevent these threat actors from achieving their disruptive objectives. 

Darktrace Coverage

Once threat actors succeeded in tricking users into running BumbleBee on their devices, Darktrace’s Self-Learning AI immediately detected the command-and-control (C2) activity generated by the loader. BumbleBee’s C2 activity caused the following Darktrace models to breach:

·      Anomalous Connection / Anomalous SSL without SNI to New External

·      Anomalous Connection / Suspicious Self-Signed SSL

·      Anomalous Connection / Rare External SSL Self-Signed

·      Compromise / Suspicious TLS Beaconing To Rare External

·      Compromise / Beacon to Young Endpoint

·      Compromise / Beaconing Activity To External Rare

·      Compromise / Sustained SSL or HTTP Increase

·      Compromise / Suspicious TLS Beaconing To Rare External

·      Compromise / SSL Beaconing to Rare Destination

·      Compromise / Large Number of Suspicious Successful Connections

·      Device / Multiple C2 Model Breaches 

BumbleBee’s delivery of Cobalt Strike Beacon onto target systems resulted in those systems communicating with Cobalt Strike C2 servers. Cobalt Strike Beacon’s C2 communications resulted in breaches of the following models: 

·      Compromise / Beaconing Activity To External Rare

·      Compromise / High Volume of Connections with Beacon Score

·      Compromise / Large Number of Suspicious Successful Connections

·      Compromise / Sustained SSL or HTTP Increase

·      Compromise / SSL or HTTP Beacon

·      Compromise / Slow Beaconing Activity To External Rare

·      Compromise / SSL Beaconing to Rare Destination 

The threat actors’ subsequent port scanning and SMB enumeration activities caused the following models to breach:

·      Device / Network Scan

·      Anomalous Connection / SMB Enumeration

·      Device / Possible SMB/NTLM Reconnaissance

·      Device / Suspicious Network Scan Activity  

The threat actors’ attempts to obtain account password data from domain controllers using the DCSync technique resulted in breaches of the following models: 

·      Compromise / Unusual SMB Session and DRS

·      Anomalous Connection / Anomalous DRSGetNCChanges Operation

Finally, the threat actors’ attempts to internally distribute and execute payloads resulted in breaches of the following models:

·      Compliance / SMB Drive Write

·      Device / Lateral Movement and C2 Activity

·      Device / SMB Lateral Movement

·      Device / Multiple Lateral Movement Model Breaches

·      Anomalous File / Internal / Unusual SMB Script Write

·      Anomalous File / Internal / Unusual Internal EXE File Transfer

·      Anomalous Connection / High Volume of New or Uncommon Service Control

If Darktrace/Network had been configured in the targeted environments, then it would have blocked BumbleBee’s C2 communications, which would have likely prevented the threat actors from delivering Cobalt Strike Beacon into the target networks. 

Figure 9: Attack timeline

Conclusion

Threat actors use loaders to smuggle more harmful payloads into target networks. Prior to March 2022, it was common to see threat actors using the BazarLoader loader to transfer their payloads into target environments. However, since the public disclosure of the Conti gang’s Jabber chat logs at the end of February, the cybersecurity world has witnessed a shift in tradecraft. Threat actors have seemingly transitioned from using BazarLoader to using a novel loader known as ‘BumbleBee’. Since BumbleBee first made an appearance in March 2022, a growing number of threat actors, in particular ransomware actors, have been observed using it.

It is likely that this trend will continue, which makes the detection of BumbleBee activity vital for the prevention of ransomware deployment within organisations’ networks. During April, Darktrace’s SOC team observed a particular pattern of threat actor activity involving the BumbleBee loader. After tricking users into running BumbleBee on their devices, threat actors were seen instructing BumbleBee to drop Cobalt Strike Beacon. Threat actors then leveraged Cobalt Strike Beacon to conduct network reconnaissance, obtain account password data from internal domain controllers, and distribute malicious payloads internally.  Darktrace’s detection of these activities prevented the threat actors from achieving their likely harmful objectives.  

Thanks to Ross Ellis for his contributions to this blog.

Appendices 

References 

[1] https://blog.google/threat-analysis-group/exposing-initial-access-broker-ties-conti/ 

[2] https://securityintelligence.com/posts/trickbot-gang-doubles-down-enterprise-infection/ 

[3] https://www.proofpoint.com/us/blog/threat-insight/bumblebee-is-still-transforming

[4] https://www.cynet.com/orion-threat-alert-flight-of-the-bumblebee/ 

[5] https://research.nccgroup.com/2022/04/29/adventures-in-the-land-of-bumblebee-a-new-malicious-loader/ 

[6] https://www.bleepingcomputer.com/news/security/conti-ransomwares-internal-chats-leaked-after-siding-with-russia/ 

[7] https://therecord.media/conti-leaks-the-panama-papers-of-ransomware/ 

[8] https://www.secureworks.com/blog/gold-ulrick-leaks-reveal-organizational-structure-and-relationships 

[9] https://www.prodaft.com/m/reports/Conti_TLPWHITE_v1.6_WVcSEtc.pdf 

[10] https://www.kroll.com/en/insights/publications/cyber/bumblebee-loader-linked-conti-used-in-quantum-locker-attacks 

[11] https://symantec-enterprise-blogs.security.com/blogs/threat-intelligence/bumblebee-loader-cybercrime 

[12] https://isc.sans.edu/diary/TA578+using+thread-hijacked+emails+to+push+ISO+files+for+Bumblebee+malware/28636 

[13] https://isc.sans.edu/diary/rss/28664 

[14] https://www.logpoint.com/wp-content/uploads/2022/05/buzz-of-the-bumblebee-a-new-malicious-loader-threat-report-no-3.pdf 

[15] https://ghoulsec.medium.com/mal-series-23-malware-loader-bumblebee-6ab3cf69d601 

[16]  https://blog.cyble.com/2022/06/07/bumblebee-loader-on-the-rise/  

[17]  https://asec.ahnlab.com/en/35460/ 

[18] https://thedfirreport.com/2021/07/19/icedid-and-cobalt-strike-vs-antivirus/

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Sam Lister
Specialist Security Researcher

More in this series

No items found.

Blog

/

AI

/

May 28, 2026

From Efficiency to Exposure: How AI Adoption Is Creating Unseen Vulnerabilities on the Factory Floor

Default blog imageDefault blog image

How AI agents impact the manufacturing industry

Security teams and IT personnel across the manufacturing industry are under constant pressure to protect production, maintain uptime, and safeguard critical assets but the rise of AI is bringing huge new opportunities alongside new cyber risks. Across manufacturing, AI is embedded into workflows, decision-making, and increasingly, autonomous AI agents are acting on behalf of employees and systems.  

Agentic systems are powerful because they can act independently, but that same autonomy also creates cyber and operational risk. Agents have extensive permissions and are capable of carrying out complex tasks, making decisions, and interacting with tools or external systems with little to no human intervention.

Unlike traditional AI models that perform predefined tasks, AI agents use advanced techniques to mimic human decision-making processes, dynamically adapting to new challenges, making decision and taking action based on their own judgement. They look like employees operationally but lack judgment, ethics, or fear of consequences like humans do. This means they can be easily manipulated by cybercriminals, and an AI agent embedded across an OT network creates threats that extend well beyond data exposure. For example, at BMW, AI identifies faults in welding processes as they occur. At its Spartanburg plant, AI monitors the weld of 300-400 metal studs onto every SUV frame to detect misplaced or faulty studs and correct them instantly. Corruption of BMW’s AI system could lead to catastrophic quality control errors.

Adopting agentic AI systems across manufacturing raises some concerns across security teams. New data from our State of AI Cybersecurity survey shows that 78% of manufacturing security professionals are worried about employee use of AI agents – their top concern. That’s followed by employee use of generative AI tools like CoPilot and ChatGPT, a worry for 76% of security professionals at manufacturing organizations. As these tools gain more access to business data and processes, and more autonomy within organizations, security teams, who today have minimal visibility of agent activity in their environments, increasingly have sensitive data exposure (a worry for 60%) and accidental policy and regulatory violations (59%) on their minds.

External AI-powered threats are evolving just as quickly

The same capabilities transforming manufacturing are also reshaping cyberattacks.

AI is enabling attackers to automate reconnaissance, refine targeting, and adapt in real time. What once required time and manual effort can now be executed continuously and at scale. Manufacturers are already seeing the impact. According to manufacturing security professionals we surveyed, 76% are already being impacted by AI-powered threats and 90% see AI increasing the success of social engineering attacks.

And the techniques themselves are evolving. Concerns across the manufacturing sector show growing anxiety about the range of AI-powered attack routes, most pressingly of adaptive malware that evolves in real-time – a prospect half (49%) of manufacturing security professionals we surveyed are worried by, a full 9% more than the average across industries. AI adaptive malware is followed by:

  • Automated vulnerability scanning and exploit chaining (48%) which has become even more pressing as Anthropic’s new Mythos AI Model supercharges vulnerability discovery
  • Hyper-personalized phishing campaigns (46%), which remain a mainstay in hackers’ arsenals, and AI has amplified their effectiveness by making phishing emails more convincing and harder to detect.

This is not just an increase in volume, it is a shift toward threats that evolve as they unfold - often faster than static defenses can respond.

Despite rising awareness, many manufacturers are not yet equipped to manage this shift. More than half (51%) say they are not adequately prepared for AI-driven threats, and only 37% have formal policies governing AI deployment.  

Securing AI through visibility, context, and guardrails

Addressing this challenge does not require manufacturers to slow innovation. It requires a different approach to security, one that can operate at the same speed and scale as AI. Three specific priorities are emerging for manufacturers looking to take advantage of the power of AI.

Visibility is foundational.  

Organizations need to understand where AI is being used, what it can access, and how it behaves across both IT and OT environments. Without that, risk cannot be measured or managed. It is no surprise that Darktrace’s research found that 91% of manufacturing security professionals said that they need to understand how AI makes decisions before trusting it. This is even more critical in operational settings where disruption has safety, environmental, financial, and reputational impacts.

Context is what turns visibility into action.  

In environments shaped by AI, normal behavior is constantly shifting. Detecting threats requires a behavioral approach; understanding patterns of life across the organization and identifying subtle deviations in real time – a step change in organizations’ traditional approach to security and risk management.

Guardrails ensure that agency does not become exposure  

As AI systems take on greater responsibility, organizations need clear boundaries around what they can do and when they can act independently. These controls must be embedded into systems themselves, not applied after the fact.  

Securing AI Agents Across Manufacturing IT and OT

The rise of agentic AI is transforming manufacturing - powering next-generation operations while reshaping the security landscape. This is not just an increase in threats, but a shift to autonomous systems, continuously evolving behaviors, and risks moving at machine speed. For organizations trying to grapple with the challenge of enabling AI while managing the risk, visibility, context and guardrails should be foundational.

Darktrace helps manufacturers build secure AI approaches by making those foundations possible. It provides visibility and real-time detection and response to unusual activity across IT and OT environments and allows organizations to understand AI activity from the prompts employees use and the agents they build to how those agents are behaving across the environment. For manufacturers scaling AI, this delivers a foundation for innovation without sacrificing control.

Continue reading
About the author
Oakley Cox
Director of Product

Blog

/

AI

/

May 28, 2026

How to Evaluate AI Vendors: 5 Key categories for AI Adoption

Default blog imageDefault blog image

Understanding the AI buyers’ market

AI adoption has become a central topic of discussion in boardrooms, drawing growing interest from business leaders. Ultimately, organizations hope that an investment in AI technology will have tremendous returns. However, the process of buying an AI solution is not as straight forward as it appears on the surface.  

While business leaders may be eager to improve productivity across their operations, practitioners responsible for evaluating and selecting AI solutions may not always have the visibility or technical understanding needed to make the right decisions for their business. What is typically marketed as a holistic solution to their most critical problems is usually followed by uncertainty when AI tools are finally operationalized in real environments.

This guide is intended to support security leaders who are under growing pressure to adopt AI tools while navigating complex terminology, vendor claims, and increasingly crowded buying cycles. Ultimately, the goal is to help organizations evaluate and adopt AI in a safe, effective, and well-governed way. To support this, we’ve structured the evaluation framework across five key categories:

  1. Governance, safety, and data controls
  1. Data gathering and training
  1. Model and technique choice
  1. Performance and accuracy validation    
  1. Interpretability, adjustability, and transparency    

What buying AI looks like in cybersecurity

While investing in AI can bring immense benefits to your security team, first-time buyers of AI cybersecurity solutions may not know where to start. They will have to determine the type of tool they want, know the options available, and evaluate vendors. Research and understanding are critical to ensure purchases are worth the investment.  

With acceleration in AI adoption, accompanied by the recent boom in agentic AI and autonomous agents, CISOs must look “beneath the hood" of these tools to understand how they work, how they are governed, and to ensure the system is secure and compliant with internal policies.

Challenges in the AI buyers’ marketplace  

The AI security software market is buzzing with hype and flashy promises, which, understandably, needs to be addressed with due diligence. Potential buyers, especially in the cybersecurity space, are hesitant when it comes to allowing AI autonomous capabilities across their workflows, and a lack of vendor transparency can exacerbate those feelings.  

Reinforcing this sentiment, research from this year's Darktrace’s State of AI Cybersecurity report shows where confidence and hesitancy emerge amongst potential buyers. On the one hand, security professionals agree that they have good visibility into the logic and reasoning processes their AI solutions use. However, they lack the explainability and trust to allow AI to take independent remedial action.

  • 89% say they have good visibility into the reasoning behind the outputs generated by AI solutions
  • 92% say they need to understand how a defensive AI tool makes decisions before they can trust it
  • Only 14% say they allow AI to act independently, performing autonomous actions without human approval
  • 74% say they are limiting the autonomy of AI taking action in their SOC until explainability improves

Given the desire for trust and explainability we are seeing from buyers, it's important for them to be equipped with the right questions to ask vendors during an assessment or POV of AI tools in order to demystify marketing hype from real operational outcomes.

Below is a list of categories in which buyers can assess AI vendors or AI Service Providers (AISPs) to help reach safe adoption and maximize their ROI.  

5 categories of AI vendor assessment

Darktrace groups these AI-related questions into 5 categories: governance, data and training, model and technique choice, performance validation, and interpretability and adjustability. By asking questions regarding each of these 5 categories, buyers can gain a deeper understanding of how an AISP’s systems work and whether they suit their business requirements.

Governance, safety, and data controls

Governance of AI systems is critical for all AISPs. Whether their platform is based around a single model, or is a more complex, composite AI solution, strong governance is essential to ensure the system is safe, robust, and reliable.

A simple question you could ask is:

What AI governance policies and frameworks do you follow, and/or certifications do you currently maintain?

For more questions you can ask vendors, download the full guide here.

Darktrace is certified to the ISO/IEC 42001 standard, the world’s first AI Management System (AIMS) standard. ISO/IEC 42001 addresses the unique ethical and technical challenges AI poses by setting out a structured way to manage risks such as transparency, accuracy, and misuse. This includes a commitment to ethical AI development, and effective management and monitoring of AI systems both prior to and continually after release.

Data gathering and training

Accurate, meaningful, and unbiased data gathering is the first important step in producing any AI system. An AI model trained using inaccurate, unbalanced, or poor-quality training data will fail to perform optimally.

To alleviate concerns regarding training data quality, a question you could ask is:

What steps do you take to prevent bias in your AI models and training data?

For more questions, download the full guide here.

AISPs should be able to provide information about the steps taken, workflows followed, and auditing performed to reduce AI bias where appropriate. While it’s sometimes impossible to fully remove bias from an AI model, appropriate actions should be taken to mitigate or reduce bias where relevant.

Model and technique choice

Different AI techniques are optimal for different tasks. For example, research from Gartner suggests that relying on a single “one-size-fits-all" model can lead to data gaps, especially in highly specialized domains.

To achieve more accurate and robust AI solutions, AI leaders should move beyond using just one model or technique, embrace composite AI practices, and adopt a holistic AI system perspective.

A straightforward question you could ask is simply:

What type(s) of AI model(s) do you utilize in your solution?

For more questions, download the full guide here.

While specific detailed information about custom systems used by AISPs is likely proprietary, buyers should expect vendors to be able to provide an overview of the broad techniques used. This will allow you as a buyer to determine if the type of model is appropriate for your use case.

Performance and accuracy validation  

Testing and evaluation of performance is essential for all AI systems. Performance analysis should be performed both before release and continually after release to identify potential data or model drift.  

A question you could ask to understand an AISPs testing workflow is:

How do you audit, test, evaluate, verify, and validate your AI model outputs?

For more questions, download the full guide here.

Testing workflows will likely vary depending on the type of model – measurements relevant to one system may not always be relevant to others. Assessment of systems should also extend beyond these standard accuracy and robustness tests, and should also feature physical performance, such as latency and resource consumption.  

Interpretability, adjustability, and transparency  

AI systems are typically a black box, simply providing an output without an explanation of how that output was attained. Interpretability and transparency are critical to ensure that both SOC teams and end-users trust the outputs of a system to be accurate and meaningful.

A question you could ask is:

How do you promote a trust relationship between human analysts and AI outputs?

For more questions, download the full guide here.

In the context of cybersecurity, trust and interpretability are even more essential. This is particularly relevant for generative AI-based systems (including most AI Agents), where the risk of hallucination can reduce trust in responses.

Cybersecurity systems often need to perform autonomous actions to block incoming threats – an email filtering system may hold potentially dangerous emails; a firewall may block malicious inbound connections. If SOC teams can’t trust these systems to perform accurately, these systems may be limited or disabled, critically reducing their defensive power.

Darktrace as an AI-native cybersecurity vendor

Darktrace has been building and applying AI in cybersecurity for over a decade, developing its capabilities alongside an increasingly complex and fast‑moving threat landscape. This experience has resulted in a mature, multi-layered approach to AI, which continuously learns the normal patterns of each organization to understand behavior, interpret context, and identify meaningful deviations — without relying on predefined rules or known attack signatures. Over time, this has enabled a proven behavioral understanding that helps uncover subtle signals of risk that may otherwise be missed.

With the backing of our ISO/IEC 42001 certification, stakeholders, customers, and partners can be confident that Darktrace is responsibly, ethically, and safely developing its AI systems, and managing the use of AI in day-to-day operations in a compliant and secure manner.  

Explore the principles behind Darktrace’s responsible AI approach, informed by collaboration with global experts in academia and governments, detailing how accountability, explainability, and continuous validation are built into its cybersecurity technology.

How Darktrace secures AI systems

Darktrace now brings these capabilities to monitor and respond to risk generated from AI systems across organizations with Darktrace / SECURE AI. This solution analyzes how prompts, agents, and systems are used within the context of each organization, bringing every AI interaction into a single view. This unique approach helps teams understand intent, assess risk, protect sensitive data, and enforce policy across both human and AI agent activity.

Stay up to date

Sign up for the Secure AI Readiness Program here: This gives you exclusive access to the latest news on the latest AI threats, updates on emerging approaches shaping AI security, and insights into the latest innovations, including Darktrace’s ongoing work in this area.

Ready to talk with a Darktrace expert on securing AI? Register here to receive practical guidance on the AI risks that matter most to your business, paired with clarity on where to focus first across governance, visibility, risk reduction, and long-term readiness.  

Further Reading on AI in cybersecurity

When deciding to invest in an AI solution, it’s important to understand what this means for you and your organization. The questions presented here are only a starting point in understanding an AI solution and whether it is appropriate for your use case.  

Gain deeper knowledge on applications of AI in cybersecurity and Darktrace’s multi-layered AI in the AI Arsenal White Paper.

[related-resource]

Continue reading
About the author
Jamie Bali
Technical Author (AI) Developer
あなたのデータ × DarktraceのAI
唯一無二のDarktrace AIで、ネットワークセキュリティを次の次元へ