5.7 Threat modeling AI systems
A lightweight method to threat-model an AI workflow before enabling it, covering surface enumeration, trust boundaries, data flows, tools, and failure modes.
Last reviewed: August 18, 2026
What security teams need to know about threat modeling AI systems
Threat modeling an AI system should be lightweight enough to use before pilot approval. The output is a data-flow diagram, a control list, and a few abuse cases that matter.
You do not need a new methodology. Use STRIDE to enumerate threats at each trust boundary, MITRE ATLAS to enumerate adversary techniques against AI-enabled systems, and the OWASP LLM Top 10 to seed abuse cases such as prompt injection and excessive agency. If your organization uses the NIST AI Risk Management Framework, the resulting risks and owners feed its GOVERN and MANAGE activities.
The model should include human users, agents, tools, connectors, model endpoints, memory, retrieval, files, logs, and external destinations.
Common threat modeling AI systems security failures
The diagram stops at the model API and ignores tools.
Prompt injection is listed as a risk but not tied to a data flow.
The threat model excludes logs, evals, and artifacts.
Agent identity is missing from the diagram.
Controls are listed without owners or evidence.
Threat modeling AI systems security controls checklist
Map users, agents, models, tools, memory, retrieval stores, files, browsers, endpoints, queues, networks, logs, and outputs.
Mark identity, tenant, workspace, project, device, runtime, region, and downstream trust boundaries.
List what every principal can read, write, execute, publish, send, spend, and administer.
Model local, hosted, remote, scheduled, and fallback execution separately.
Write abuse cases for injection, exfiltration, excessive agency, confused deputy, tool-result spoofing, memory and data poisoning, SSRF, cross-tenant access, replay, and denial-of-wallet.
Record assumptions about provider controls and unsupported evidence.
Map each threat to prevention, detection, containment, recovery, owner, and test evidence.
Revisit the model after changes to models, tools, identity, data, memory, connectors, runtime, autonomy, or publishing.
Model state, cost, and evidence as assets
Traditional diagrams often stop at the model endpoint. Extend the diagram to persistent memory, embeddings and vector stores, evaluation and fine-tuning data, caches, traces, compliance exports, generated artifacts, queues, browser sessions, and connected-system records.
Treat budget and evidence as security assets. An attacker may aim to exhaust spend or destroy attribution rather than steal data. Mark where token, rate, duration, concurrency, storage, and third-party costs are enforced, and where logs can be disabled, sampled, delayed, or attributed to a shared identity.
For each consequential action, trace the chain from initiator to agent identity, model, tool, credential, target, approval, result, and evidence. Any missing link is a design gap that should be accepted, compensated for, or blocked before production.
Anthropic
Overview
Anthropic threat models should map Claude web, Desktop, Mobile, Cowork remote and local execution, Claude Tag, Office add-ins, Code, Console/API, connectors, MCP, Agent Skills, and Managed Agents. Draw the trust boundary around tool execution, source-system identity, prompt files, skill scripts, desktop access, Slack channels and memory, cross-app transfers, and external actions.
Draw two boundaries accurately. Managed Agents sessions persist state server-side and are not currently eligible for zero data retention or HIPAA BAA coverage, and the product is in beta. A Managed Agents deployment therefore places a stateful, Anthropic-hosted component in the diagram, with data-class limits regulated workflows must respect. Managed Agents can run in Anthropic-managed cloud sandboxes, isolated Linux containers with network access disabled by default, or in self-hosted sandboxes, which moves the execution boundary. Connectors act with the person's source-system permissions, so the source system's boundary, not Claude's, defines what a connector can reach.
Draw Claude Tag channel work and direct messages as separate paths. Channel work crosses Slack, an Anthropic-hosted per-thread sandbox, Agent Proxy, a credential store, and service accounts in connected systems. Direct messages run under the user's Claude account and personal connectors. Those paths have different authority and attribution even though both begin with @Claude.
Anthropic documentation
Applicable Harmonic guides for Anthropic
OpenAI
Overview
OpenAI threat models should map Chat, Work, and Codex in the combined desktop app; cloud Work on web and mobile; plugins and apps; Sites; Office-native agents; Workspace Agents; Codex CLI, IDE, and cloud; API tools; and Agents SDK or Codex SDK services. Keep ChatGPT workspace controls, desktop or Codex managed configuration, and API project controls as separate control planes.
Each plane has its own enforcement. ChatGPT RBAC covers workspace permissions such as apps, Codex, GPTs, projects, and canvas networking, with effective permissions taking the maximum across a user's roles, a detail that matters when modeling who can do what. Codex managed configuration controls approval policy, sandbox behavior, and MCP-related settings. Workspace Agent access tokens are distinct from platform and Compliance API credentials, so agent identities cross a boundary of their own.
OpenAI documentation
Applicable Harmonic guides for OpenAI
Frequently asked questions about threat modeling AI systems
What is an AI threat model?
It is a map of an AI workflow that identifies data flows, trust boundaries, tools, identities, and abuse cases. It differs from a classic application threat model mainly in what counts as an actor: the model can be steered by content it reads, so untrusted data becomes an active participant. The output should be a diagram, a control list, and a handful of abuse cases that matter.
What should be in an AI data-flow diagram?
Include prompts, files, retrieval, connectors, tools, model calls, outputs, logs, memory, and external destinations. Draw the arrows for data returning from tools, not just data going in, because tool results are an instruction channel. Mark which identity each arrow uses: the user's, the agent's, or a service account's.
Which threats should be considered first?
Start with prompt injection, data exfiltration, excessive agency, confused deputy behavior, and audit gaps. These map directly to OWASP LLM Top 10 categories and recur across vendors and deployment models. STRIDE and MITRE ATLAS can then broaden the list once the top risks have owners.
Who should join the threat-modeling session?
Include security, product, platform, identity, data owner, and the team building or approving the workflow. The builder knows what the agent actually calls, the data owner knows what a leak would cost, and identity knows whose credentials each hop uses. A small session with those roles beats a large review without them.
When should threat modeling happen?
Do it before pilot, before production expansion, and after major changes to tools, data, or autonomy. A new connector, a jump in autonomy, or a switch to unattended runs each redraw trust boundaries and warrant a revisit. Keep the model lightweight enough that revisiting it is cheap.
Applicable regulations and frameworks
G.1 Map your controls to NIST AI RMF and CSF 2.0
This article supplies implementation evidence for the NIST AI RMF and matching NIST CSF 2.0 outcomes.
G.2 EU AI Act obligations for deployers
Conditional: for an in-scope high-risk system, this supports technical input to a fundamental-rights impact assessment.
G.3 DORA and AI resilience in financial services
Conditional: for a DORA-regulated workflow, this supports resilience testing, vulnerability handling, and incident response.
G.5 SANS Critical AI Security Guidelines mapping
This article implements relevant SANS Inference Security, Monitoring, and GRC guidance.
G.7 Ownership and RACI for AI security
This control depends on the ownership and evidence responsibilities defined in the RACI.
G.8 ISO/IEC 42001 AI management system
This article supports ISO/IEC 42001 AIMS preparation through AI risk identification and treatment planning.
G.9 HIPAA controls for AI systems handling PHI
Conditional: for a workflow handling ePHI, this supports HIPAA risk analysis, safeguard testing, and security-incident response.
G.2, G.3, G.4, and G.9 are conditional mappings. They apply only when the deployment is within the legal or regulatory scope described on the linked governance page.
Related handbook guidance
Was this helpful?