Blog
/
Network
/
July 4, 2024

A Busy Agenda: Darktrace's Detection of Qilin Ransomware as a Service Operator

This blog breaks down how Darktrace detected and analyzed Qilin, a Ransomware-as-a-Service group behind recent high-impact attacks. You’ll see how Qilin affiliates customize attacks with flexible encryption, process termination, and double-extortion techniques, as well as why its cross-platform builds in Rust and Golang make it especially evasive. Darktrace highlights three real-world cases where its AI identified likely Qilin activity across customer environments, offering insights into how behavioral detection can spot novel ransomware before disruption occurs. Readers will gain a clear view of Qilin’s toolkit, tactics, and how self-learning defense adapts to these evolving threats.
Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Alexandra Sentenac
Cyber Analyst
Default blog image
04
Jul 2024

Qilin ransomware has recently dominated discussions across the cyber security landscape following its deployment in an attack on Synnovis, a UK-based medical laboratory company. The ransomware attack ultimately affected patient services at multiple National Health Service (NHS) hospitals that rely on Synnovis diagnostic and pathology services. Qilin’s origins, however, date back further to October 2022 when the group was observed seemingly posting leaked data from its first known victim on its Dedicated Leak Site (DLS) under the name Agenda[1].

The Darktrace Threat Research team investigated network artifacts related to Qilin and identified three probable cases of the ransomware across the Darktrace customer base between June 2022 and May 2024.

Qilin Ransomware-as-a-Service Operator

Qilin operates as a Ransomware-as-a-Service (RaaS) that employs double extortion tactics, whereby harvested data is exfiltrated and threatened of publication on the group's DLS, which is hosted on Tor. Qilin ransomware has samples written in both the Golang and Rust programming languages, making it compilable with various operating systems, and is highly customizable. When building Qilin ransomware variants to be used on their target(s), affiliates can configure settings such as the encryption mode (i.e., skip-step, percent, and speed), the file extension being appended, files, extensions and directories to be skipped during the encryption, and the processes and services to be terminated, among others[1] [2].  

Trend Micro analysts, who were the first to discover Qilin samples in August 2022, when the name "Agenda" was still used in ransom notes, found that each analyzed sample was customized for the intended victims and that "unique company IDs were used as extensions of encrypted files" [3]. This information is configurable from within the Qilin's affiliate panel's 'Targets' section, shown below. The panel's background image features the eponym Chinese legendary chimerical creature Qilin (pronounced “Ke Lin”). Despite this Chinese mythology reference, Russian language was observed being used by a Qilin operator in an underground forum post aimed at hiring affiliates and advertising their RaaS operation[2].

Figure 1: Qilin ransomware’s affiliate panel.

Qilin's RaaS program purportedly has an attractive affiliates' payment structure, with affiliates allegedly able to earn 80% of ransom payments of USD 3m or less and 85% for payments above that figure[2], making it a possibly appealing option in the RaaS ecosystem.  Publication of stolen data and ransom payment negotiations are purportedly handled by Qilin operators. Qilin affiliates have been known to target companies located around the world and within a variety of industries, including critical sectors such as healthcare and energy.

As Qilin is a RaaS operation, the choice of targets does not necessarily reflect Qilin operators' intentions, but rather that of its affiliates.  Similarly, the tactics, techniques, procedures (TTPs) and indicators of compromise (IoC) identified by Darktrace are associated with the given affiliate deploying Qilin ransomware for their own purpose, rather than TTPs and IoCs of the Qilin group. Likewise, initial vectors of infection may vary from affiliate to affiliate. Previous studies show that initial access to networks were gained via spear phishing emails or by leveraging exposed applications and interfaces.

Differences have been observed in terms of data exfiltration and potential C2 external endpoints, suggesting the below investigations are not all related to the same group or actor(s).

Darktrace’s Threat Research Investigation

June 2022

Darktrace first detected an instance of Qilin ransomware back in June 2022, when an attacker was observed successfully accessing a customer’s Virtual Private Network (VPN) and compromising an administrative account, before using RDP to gain access to the customer’s Microsoft System Center Configuration Manager (SCCM) server

From there, an attack against the customer's VMware ESXi hosts was launched. Fortunately, a reboot of their virtual machines (VM) caught the attention of the security team who further uncovered that custom profiles had been created and remote scripts executed to change root passwords on their VM hosts. Three accounts were found to have been compromised and three systems encrypted by ransomware.  

Unfortunately, Darktrace was not configured to monitor the affected subnets at the time of the attack. Despite this, the customer was able to work directly with Darktrace analysts via the Ask the Expert (ATE) service to add the subnets in question to Darktrace’s visibility, allowing it to monitor for any further unusual behavior.

Once visibility over the compromised SCCM server was established, Darktrace observed a series of unusual network scanning activities and the use of Kali (a Linux distribution designed for digital forensics and penetration testing). Furthermore, the server was observed making connections to multiple rare external hosts, many using the “[.]ru” Top Level Domain (TLD). One of the external destinations the server was attempting to connect was found to be related to SystemBC, a malware that turns infected hosts into SOCKS5 proxy bots and provides command-and-control (C2) functionality.

Additionally, the server was observed making external connections over ports 993 and 143 (typically associated with the use of the Interactive Message Access Protocol (IMAP) to multiple rare external endpoints. This was likely due to the presence of Tofsee malware on the device.

After the compromise had been contained, Darktrace identified several ransom notes following the naming convention “README-RECOVER-<extension/company_id>.txt”” on the network. This naming convention, as well as the similar “<company_id>-RECOVER-README.txt” have been referenced by open-source intelligence (OSINT) providers as associated with Qilin ransom notes[5] [6] [7].

April 2023

The next case of Qilin ransomware observed by Darktrace took place in April 2023 on the network of a customer in the manufacturing sector in APAC. Unfortunately for the customer in this instance, Darktrace RESPOND™ was not active on their environment and no autonomous response actions were taken to contain the compromise.

Over the course of two days, Darktrace identified a wide range of malicious activity ranging from extensive initial scanning and lateral movement attempts to the writing of ransom notes that followed the aforementioned naming convention (i.e., “README-RECOVER-<extension/company_id>.txt”).

Darktrace observed two affected devices attempting to move laterally through the SMB, DCE-RPC and RDP network protocols. Default credentials (e.g., UserName, admin, administrator) were also observed in the large volumes of SMB sessions initiated by these devices. One of the target devices of these SMB connections was a domain controller, which was subsequently seen making suspicious WMI requests to multiple devices over DCE-RPC and enumerating SMB shares by binding to the ‘server service’ (srvsvc) named pipe to a high number of internal devices within a short time frame. The domain controller was further detected establishing an anomalously high number of connections to several internal devices, notably using the RDP administrative protocol via a default admin cookie.  

Repeated connections over the HTTP and SSL protocol to multiple newly observed IPs located in the 184.168.123.0/24 range were observed, indicating C2 connectivity.  WebDAV user agent and a JA3 fingerprint potentially associated with Cobalt Strike were notably observed in these connections. A few hours later, Darktrace detected additional suspicious external connections, this time to IPs associated with the MEGA cloud storage solution. Storage solutions such as MEGA are often abused by attackers to host stolen data post exfiltration. In this case, the endpoints were all rare for the network, suggesting this solution was not commonly used by legitimate users. Around 30 GB of data was exfiltrated over the SSL protocol.

Darktrace did not observe any encryption-related activity on this customer’s network, suggesting that encryption may have taken place locally or within network segments not monitored by Darktrace.

May 2024

The most recent instance of Qilin observed by Darktrace took place in May 2024 and involved a customer in the US. In this case, Darktrace initially detected affected devices using unusual administrative and default credentials, before additional internal systems were observed making extensive suspicious DCE-RPC requests to a range of internal locations, performing network scanning, making unusual internal RDP connections, and transferring suspicious executable files like 'a157496.exe' and '83b87b2.exe'.  SMB writes of the file "LSM_API_service" were also observed, activity which was considered 100% unusual by Darktrace; this is an RPC service that can be abused to enumerate logged-in users and steal their tokens. Various repeated connections likely representative of C2 communications were detected via both HTTP and SSL to rare external endpoints linked in OSINT to Cobalt Strike use. During these connections, HTTP GET requests for the following URIs were observed:

/asdffHTTPS

/asdfgdf

/asdfgHTTP

/download/sihost64.dll

Notably, this included a GET request a DLL file named "sihost64.dll" from a domain controller using PowerShell.  

Over 102 GB of data may have been transferred to another previously unseen endpoint, 194.165.16[.]13, via the unencrypted File Transfer Protocol (FTP). Additionally, many non-FTP connections to the endpoint could be observed, over which more than 783 GB of data was exfiltrated. Regarding file encryption activity, a wide range of destination devices and shares were targeted.

Figure 2: Advanced Search graph displaying the total volume of data transferred over FTP to a malicious IP.

During investigations, Darktrace’s Threat Research team identified an additional customer, also based in the United States, where similar data exfiltration activity was observed in April 2024. Although no indications of ransomware encryption were detected on the network, multiple similarities were observed with the case discussed just prior. Notably, the same exfiltration IP and protocol (194.165.16[.]13 and FTP, respectively) were identified in both cases. Additional HTTP connectivity was further observed to another IP using a self-signed certificate (i.e., CN=ne[.]com,OU=key operations,O=1000,L=,ST=,C=KM) located within the same ASN (i.e., AS48721 Flyservers S.A.). Some of the URIs seen in the GET requests made to this endpoint were the same as identified in that same previous case.

Information regarding another device also making repeated connections to the same IP was described in the second event of the same Cyber AI Analyst incident. Following this C2 connectivity, network scanning was observed from a compromised domain controller, followed by additional reconnaissance and lateral movement over the DCE-RPC and SMB protocols. Darktrace again observed SMB writes of the file "LSM_API_service", as in the previous case, activity which was also considered 100% unusual for the network. These similarities suggest the same actor or affiliate may have been responsible for activity observed, even though no encryption was observed in the latter case.

Figure 3: First event of the Cyber AI Analyst investigation following the compromise activity.

According to researchers at Microsoft, some of the IoCs observed on both affected accounts are associated with Pistachio Tempest, a threat actor reportedly associated with ransomware distribution. The Microsoft threat actor naming convention uses the term "tempest" to reference criminal organizations with motivations of financial gain that are not associated with high confidence to a known non-nation state or commercial entity. While Pistachio Tempest’s TTPs have changed over time, their key elements still involve ransomware, exfiltration, and extortion. Once they've gained access to an environment, Pistachio Tempest typically utilizes additional tools to complement their use of Cobalt Strike; this includes the use of the SystemBC RAT and the SliverC2 framework, respectively. It has also been reported that Pistacho Tempest has experimented with various RaaS offerings, which recently included Qilin ransomware[4].

Conclusion

Qilin is a RaaS group that has gained notoriety recently due to high-profile attacks perpetrated by its affiliates. Despite this, the group likely includes affiliates and actors who were previously associated with other ransomware groups. These individuals bring their own modus operandi and utilize both known and novel TTPs and IoCs that differ from one attack to another.

Darktrace’s anomaly-based technology is inherently threat-agnostic, treating all RaaS variants equally regardless of the attackers’ tools and infrastructure. Deviations from a device’s ‘learned’ pattern of behavior during an attack enable Darktrace to detect and contain potentially disruptive ransomware attacks.

Credit to: Alexandra Sentenac, Emma Foulger, Justin Torres, Min Kim, Signe Zaharka for their contributions.

References

[1] https://www.sentinelone.com/anthology/agenda-qilin/  

[2] https://www.group-ib.com/blog/qilin-ransomware/

[3] https://www.trendmicro.com/en_us/research/22/h/new-golang-ransomware-agenda-customizes-attacks.html

[4] https://www.microsoft.com/en-us/security/security-insider/pistachio-tempest

[5] https://www.trendmicro.com/en_us/research/22/h/new-golang-ransomware-agenda-customizes-attacks.html

[6] https://www.bleepingcomputer.com/forums/t/790240/agenda-qilin-ransomware-id-random-10-char;-recover-readmetxt-support/

[7] https://github.com/threatlabz/ransomware_notes/tree/main/qilin

Darktrace Model Detections

Internal Reconnaissance

Device / Suspicious SMB Scanning Activity

Device / Network Scan

Device / RDP Scan

Device / ICMP Address Scan

Device / Suspicious Network Scan Activity

Anomalous Connection / SMB Enumeration

Device / New or Uncommon WMI Activity

Device / Attack and Recon Tools

Lateral Movement

Device / SMB Session Brute Force (Admin)

Device / Large Number of Model Breaches from Critical Network Device

Device / Multiple Lateral Movement Model Breaches

Anomalous Connection / Unusual Admin RDP Session

Device / SMB Lateral Movement

Compliance / SMB Drive Write

Anomalous Connection / New or Uncommon Service Control

Anomalous Connection / Anomalous DRSGetNCChanges Operation

Anomalous Server Activity / Domain Controller Initiated to Client

User / New Admin Credentials on Client

C2 Communication

Anomalous Server Activity / Outgoing from Server

Anomalous Connection / Multiple Connections to New External TCP Port

Anomalous Connection / Anomalous SSL without SNI to New External

Anomalous Connection / Rare External SSL Self-Signed

Device / Increased External Connectivity

Unusual Activity / Unusual External Activity

Compromise / New or Repeated to Unusual SSL Port

Anomalous Connection / Multiple Failed Connections to Rare Endpoint

Device / Suspicious Domain

Device / Increased External Connectivity

Compromise / Sustained SSL or HTTP Increase

Compromise / Botnet C2 Behaviour

Anomalous Connection / POST to PHP on New External Host

Anomalous Connection / Multiple HTTP POSTs to Rare Hostname

Anomalous File / EXE from Rare External Location

Exfiltration

Unusual Activity / Enhanced Unusual External Data Transfer

Anomalous Connection / Data Sent to Rare Domain

Unusual Activity / Unusual External Data Transfer

Anomalous Connection / Uncommon 1 GiB Outbound

Unusual Activity / Unusual External Data to New Endpoint

Compliance / FTP / Unusual Outbound FTP

File Encryption

Compromise / Ransomware / Suspicious SMB Activity

Anomalous Connection / Sustained MIME Type Conversion

Anomalous File / Internal / Additional Extension Appended to SMB File

Compromise / Ransomware / Possible Ransom Note Write

Compromise / Ransomware / Possible Ransom Note Read

Anomalous Connection / Suspicious Read Write Ratio

IoC List

IoC – Type – Description + Confidence

93.115.25[.]139 IP C2 Server, likely associated with SystemBC

194.165.16[.]13 IP Probable Exfiltration Server

91.238.181[.]230 IP C2 Server, likely associated with Cobalt Strike

ikea0[.]com Hostname C2 Server, likely associated with Cobalt Strike

lebondogicoin[.]com Hostname C2 Server, likely associated with Cobalt Strike

184.168.123[.]220 IP Possible C2 Infrastructure

184.168.123[.]219 IP Possible C2 Infrastructure

184.168.123[.]236 IP Possible C2 Infrastructure

184.168.123[.]241 IP Possible C2 Infrastructure

184.168.123[.]247 IP Possible C2 Infrastructure

184.168.123[.]251 IP Possible C2 Infrastructure

184.168.123[.]252 IP Possible C2 Infrastructure

184.168.123[.]229 IP Possible C2 Infrastructure

184.168.123[.]246 IP Possible C2 Infrastructure

184.168.123[.]230 IP Possible C2 Infrastructure

gfs440n010.userstorage.me ga.co[.]nz Hostname Possible Exfiltration Server. Not inherently malicious; associated with MEGA file storage.

gfs440n010.userstorage.me ga.co[.]nz Hostname Possible Exfiltration Server. Not inherently malicious; associated with MEGA file storage.

Get the latest insights on emerging cyber threats

This report explores the latest trends shaping the cybersecurity landscape and what defenders need to know in 2025

Inside the SOC
Darktrace cyber analysts are world-class experts in threat intelligence, threat hunting and incident response, and provide 24/7 SOC support to thousands of Darktrace customers around the globe. Inside the SOC is exclusively authored by these experts, providing analysis of cyber incidents and threat trends, based on real-world experience in the field.
Written by
Alexandra Sentenac
Cyber Analyst

More in this series

No items found.

Blog

/

/

September 22, 2026

Darktrace / SECURE AI: Extending Behavioral Security for the Age of Agentic AI

Default blog imageDefault blog image

AI is moving faster than the controls built to secure it

Over 80% of Darktrace’s customers now use generative AI services, and the average organization interacted with five different AI provides in August 20261. That level of adoption reflects how quickly AI has become embedded in day-to-day business operations. But as adoption accelerates, so do the opportunities for new forms of exposure that often operate outside traditional security controls. Prompts can expose sensitive information, an over-permissioned agent can turn a routine task into a serious security concern, and an employee using an unsanctioned AI tool to get work done can quietly introduce risk long before the security team is aware of its existence.

Securing AI isn’t a nice-to-have. It's an urgent, board-level requirement for any business that wants AI adoption to be an advantage rather than a liability.

Why traditional security controls fall short

The challenge isn’t a lack of AI security tools, it’s that most security solutions weren’t built for the way AI behaves.  

AI systems are adaptive and, by nature, unpredictable. A prompt can be entirely benign in one context and a serious risk in another. An agent's permissions can look reasonable in isolation and dangerous the moment they're combined with what that agent is actually doing. Rule-based controls, built to catch known signatures and static policy violations, simply aren't designed to catch this kind of subtlety. They might be able to tell you what happened, but they can’t tell you if it mattered.  

Effective AI security requires something different: a deep understanding of what normal looks like across every human, system, and agent in an environment, so the earliest signs of drift, misuse, or compromise stand out as they emerge.

Behavioral understanding is our foundation

Darktrace isn’t reinventing itself to secure AI. We’re extending what we already do. For over a decade, Darktrace has been built on a single premise – that every organization has its own unique, evolving way of operating, shaped by how its systems, users, and devices behave. Understanding that is the only reliable way to catch what rules and signatures miss. Our Adaptive AI™ continuously learns the distinct behaviors, relationships, and operational patterns of every enterprise it protects, building a Unique Behavioral Profile that no other vendor can replicate. It's how we've spent years interpreting ambiguity, uncovering subtle intent, and spotting drift before it becomes dangerous across networks, cloud, identities, email, OT, and endpoints. AI is a new domain, but behavioral security isn't a new discipline for us.

Introducing Darktrace / SECURE AI

Darktrace / SECURE AI helps organizations embrace AI innovation without losing control of how it is used. Delivered through the Darktrace Behavioral Defense Platform™, it provides visibility into AI activity across employees, agents, and AI systems, helping security teams understand how AI is being used throughout their business.

Darktrace then applies behavioral understanding to that activity, adding the context needed to distinguish routine usage from genuine risk. By understanding the relationships between users, agents, systems, and data, / SECURE AI helps organizations move beyond simply observing AI usage to understanding risk, governing behavior against policy, and enabling AI adoption with confidence. The result is greater oversight, stronger governance, and the ability to confidently accelerate AI innovation without creating unmanaged risk.

Securing your AI ecosystem

AI risk doesn't originate from a single source, and effective AI security can't focus on just one layer of the problem. Organizations need visibility, understanding, and governance across the entire AI ecosystem, from the prompts users submit and the agents they create, to the development environments where AI is built and the unsanctioned tools operating outside approved channels. Here's how Darktrace / SECURE AI helps organizations secure each of these areas.

Shadow AI management

Darktrace / SECURE AI helps security teams discover unsanctioned AI services and track usage trends over time using Darktrace telemetry and supported SASE integrations such as Microsoft Entra Global Secure Access. It also extends visibility to Model Context Protocol (MCP) usage, helping teams understand where AI tools and agents are connecting to external services. Through Darktrace platform integrations, organizations can block unsanctioned AI usage or quarantine affected devices when needed, helping them reduce exposure and guide users toward approved options.

AI prompt analysis

Prompts reveal what users are asking AI to do, what information they are sharing, and the outcomes they are trying to produce. Darktrace / SECURE AI provides visibility into prompts, sessions, and responses across supported platforms, including Microsoft Copilot, Copilot Studio, ChatGPT Enterprise, Claude, Salesforce, and AWS Bedrock2. Behavioral analysis detects activity such as attempted jailbreaks, sensitive data exposure, and potential indirect prompt injection, while risk scoring helps analysts prioritize the sessions that need attention. Policy Manager maps organizational AI policies against prompt activity and surfaces potential violations, helping teams govern AI use with direct evidence instead of relying on fragmented logs or inferred intent.

AI agent identities and actions

AI agents have their own permissions, roles, relationships, and access to systems and data. Darktrace / SECURE AI brings agent identities together with the users interacting with them, giving security teams visibility into access, sessions, and connections across supported environments. A real-time audit trail helps teams continuously evaluate whether an agent’s activity remains aligned with its intended purpose, so they can identify excessive access or behavioral drift before it becomes a security issue.

AI agent development risk management

Many AI risks are introduced during development, when agents are created, permissions are assigned, and connections to data sources are established. Darktrace / SECURE AI provides visibility across both low-code and high-code environments. In platforms such as AWS Bedrock, teams can examine AI architecture and the agent identities involved in development and deployment. In low-code environments, such as Copilot Studio, they can observe agents as they are created, monitor whether behavior stays aligned with their intended purpose, and connect prompt activity during development with behavior in production. This helps organizations address misconfigurations and excessive permissions before they reach production.

The promise of AI, secured

The question isn't whether to adopt AI, it's whether security teams can enable secure adoption that allows every business to benefit from AI’s potential.

Darktrace / SECURE AI is built to solve this problem – extending a decade of behavioral understanding to the newest, fastest-moving part of the enterprise, using the same principles which already protect people and hybrid infrastructure.

Darktrace / SECURE AI is generally available now. Discover the product, or get a demo today.

[related-resource]

[1] Based on aggregated Darktrace product telemetry across a fleet of ~8,200 observed deployments, measuring generative AI service usage across accounts monitored in August 2026. The 80%+ figure reflects the share of monitored accounts with any generative AI service usage during the same period.

[2] Microsoft, Copilot, ChatGPT, Claude, Salesforce, and Amazon Bedrock are trademarks of their respective owners.

Continue reading
About the author
Brittany Woodsmall
Product Marketing Manager, AI

Blog

/

/

September 17, 2026

The Problem of Re-defining Human Value in the Agentic Age

Default blog imageDefault blog image

Newsfeeds are constantly informing us about the rapid escalation of agentic AI systems. These systems move far beyond simple machine-based computations, and  the focused, defined and bounded assistance that most AI systems started out as.

The next evolution of AI will harness the agentic properties of orchestration, automation, and heightened value-chains in IT, taking on the burden of workflow management, not just workflow delivery.

In nearly all of these instances, promises are made such as ‘this will free up human time’ or ‘this will allow people to focus on higher-order strategy’. However that message is delivered, one thing is clear: in ceding the orchestration and management of work to increasingly sophisticated AI agents and agentic systems, human value will be elevated to a particular and specific layer: the ability to judge its outputs.  As AI takes on more tasks, humans should be able to focus on a higher level of governance; making sure the decisions that AI offers us are ethical, responsible and worthwhile.  

But there are two major problems with that approach.

This blog discusses the problem of how Agentic systems are re-shaping how we review information, where we fit in, and when we make decisions.  It also discusses the problem of how the increased flow of confident, generated information affects the way we make judgement.  This blog considers how human judgement needs to adapt, and how a behavioral defense approach – using techniques pioneered by Darktrace – can help us do that.

The challenge of knowing where human judgement belongs

As we confer more automated decision-making to agentic systems, it might look increasingly less like ‘granting permissions’, and more like ‘surrendering authority’.  

The judgement layer for AI-generated work is not a fixed boundary. We have become used to the idea of a ‘human in the loop’ (HITL) and, historically, relationships between humans and IT systems were reasonably clear and bounded.  Computer and software systems were programmed to carry out certain tasks or automated functions, and humans could control the gates and decision points where actions were undertaken. Even across highly complex computational workflows, human interaction was a controllable node within the process; we were able to configure and regulate. But in the agentic age, where that human interaction sits, and what it can influence shifts every time AI systems are granted autonomy.  

This leads us to the first problem: if humans are moving themselves (or are being moved) into the ‘judgement’ part of the value chain, exactly where and when do we exercise that judgement?  

Humans are no longer the sole shepherds of computer-based or software-controlled outputs.  We are at times at least one step further (and slower) behind the new agentic shepherds.  We might also be blind to what they are doing.  Not only might we be removed and blind to the actions of our AI shepherds, but with the challenge of unknown, unapproved AI systems operating beyond our control, humans might not even know that our work is being shepherded by an AI at all.  Simply put, with the advent of greater levels of autonomy and orchestration, humans are at risk of not even knowing where to apply our newly-extended powers of strategic judgement.

Shadow AI – the use of unapproved AI systems or processes – is a growing threat to the role of effective governance and oversight. Shadow AI isn't just the AI you can't see. Its the AI you already know about being used in an unapproved way. The ability to generate effective oversight of the AI systems you use (or that are used on your behalf) will be increasingly important to ensure that human judgement in the AI value chain is effective, and deliberately placed.

The problem of what makes good judgement

The second problem lies in how flawed human judgement can be.  Humans are historically, notoriously, and, sometimes dangerously, unreliable when it comes to exercising judgement.  Humans are prone to the worst kinds of bias, the seduction of malign influence, and the sometimes-overwhelming urge to succeed. AI has long had a known flaw of operating with sycophancy, providing outputs that tend to agree with or flatter the human user.  But as AI grows ever more effective, there is a risk of both hyper-enablement (where humans increasingly and knowingly enable AI despite potential harm), as well as the greater risk of suggestion. Both of these aspects could skew the newly-elevated input of human judgement.

Imagine a highly competent AI system that has just orchestrated and managed a dizzying array of processes and workflows.  The AI is designed to present the human decision-maker with recommendations; based on analysis, comparison and other programmed factors.  This is where the human judgement layer is enabled.  But what if that judgement is summarily diffused by an AI-based recommendation that emulates the decision, provides plausible but unattractive alternatives, then suggests (or, worse, directs) the human end-user to take a particular course of action.

The risk here is that you are given a recommendation, tailored to your preferences (which the AI has learned, or which you have divulged), and which appears to make perfect sense.  It appears to be a well-weighted recommendation, with sound arguments that tap into our inherent biases or inclinations so that a specific decision-path is followed. With the growth of agentic systems specifically designed to match user profiles (from Cowork agents to ‘digital twin’ models), the likelihood of agentic influence could badly skew human judgement or, at the least, devalue the proposition that humans are taking a higher-layer of strategic control over AI-based decisions.

If AI convincingly recommends something that may be problematic, it can be difficult to discern both accurate data, and the context required to make the right judgement.

Given the two problems described above, the job of exercising valuable human judgement in the agentic age can draw down to these two questions:

  • When should humans intervene in the agentic process?  
  • How can we make the best possible judgement calls?

What humans contribute that AI cannot

For all the flaws that make human judgement unreliable, people have the edge over even the most sophisticated and powerful AI systems when it comes to issues such as ethics and social context.  An AI system can, with startling granularity, rank the value of adopting a new business proposal: offering predictive metrics on costs, returns, market value, time-to-deliver operations, conformance with legal registers, etc.  But it can’t tell if the business proposal is ethically sound, or if the business venture will potentially affect groups outside of the analyzed proposal. It can’t tell you if the CEO has a ‘bad feeling’ about this effort.  It can’t tell you if this is the right thing to do.  

The ability to add social context, balance complex interpersonal dynamics, understand nuance, and to go beyond what seems economically reasonable is where human judgement can add value.  

Human judgement is difficult to encapsulate in metrics. And the way we train our development may need to adapt too. Rather than building up a gradual, experiential knowledge base, we should think about training the skill of judgement itself; especially for an agentic age.

How behavioral security strengthens AI governance

If this all feels like a vicious circle (‘I need AI help to make good judgements’ / ‘AI can twist what I need to judge’) it needn’t be. The key to this is having a defense-in-depth approach, with tools that can actually help.

This is precisely where behavioral security becomes important. The complex and nuanced way that humans exercise judgement is often rooted in our ability to recognize behavior that doesn't look right. We may not always be able to articulate it immediately, but we can often identify when an action, recommendation, or outcome feels inconsistent with the context around it. As AI systems take on more responsibility across the decision chain, preserving that ability to recognize meaningful deviations becomes increasingly important.

Darktrace’s / SECURE AI is designed to do exactly that. It applies behavioral security to AI ecosystems, helping organizations understand how people, AI tools, identities, and agents interact across the business. By learning the patterns of normal AI usage and surfacing activity that deviates from those patterns, it provides security teams with the context needed to investigate risk, understand unusual behavior, and make informed governance decisions. Rather than relying solely on predefined rules or assumptions, this behavioral understanding helps organizations distinguish between expected AI activity and behavior that warrants closer scrutiny.

This matters because we are already in an era of information overload. If humans are expected to elevate their value through strategic judgement, the ability to do this without being overwhelmed by data (good or bad) will be critical.  

We need the ability to discern when we're being misled by AI, and whether our judgement calls are being made on the basis of accurate, contextual information. Darktrace / SECURE AI provides that additional layer of defensive security for activity we cannot easily see. Whether it is suspected Shadow AI or skewed recommendations, the net result is a protected organization, where users can more effectively use AI to make positive judgements.

For those where that judgement is a critical skill (both individuals, as well as those working in security teams), improving our metacognition - the ability to understand information in a broader context - will supercharge the value of human judgement. When those judgements are grounded in context rather than assumptions we have better information to make sound decisions.

Conclusion

Human judgement is a skill that is honed over time and experience.  Darktrace’s / SECURE AI employs the same principles, but at machine-speed. Rather than influencing or directing, Darktrace / SECURE AI offers AI-enabled assurance; providing human-based judgement with the right context to make a balanced decision.  

What we judge can be valued by the legitimacy of its outputs. For AI, those outputs are valued on the speed and accuracy of the information provided.  Increasingly for humans, the value of our outputs will be based on the validity of our judgement, and how we justify our decisions in ways that engineer confidence.  

Humans often know more than we can express, while AI is prone to expressing more than it truly understands. Humans can bridge the context AI often fails to appreciate. When that judgement is supported by relevant, impartial AI systems, this is the future space where good AI governance will be exercised.

Discover Darktrace / SECURE AI.

[related-resource]

Continue reading
About the author
Jason Lusted
AI Governance Advisor
Your data. Our AI.
Elevate your network security with Darktrace AI