> For the complete documentation index, see [llms.txt](https://handbook.harmonic.security/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://handbook.harmonic.security/handbook/5.-threats-and-adversarial/5.7-threat-modeling-ai-systems.md). # 5.7 Threat modeling AI systems A lightweight method to threat-model an AI workflow before enabling it, covering surface enumeration, trust boundaries, data flows, tools, and failure modes. *Last reviewed: August 18, 2026* {% hint style="info" %} An AI threat model is a map of an AI workflow that identifies data flows, trust boundaries, tools, identities, and abuse cases. Build it before pilot approval, then keep asking what the agent can read, decide, and do. {% endhint %} ## What security teams need to know about threat modeling AI systems Threat modeling an AI system should be lightweight enough to use before pilot approval. The output is a data-flow diagram, a control list, and a few abuse cases that matter. You do not need a new methodology. Use STRIDE to enumerate threats at each trust boundary, MITRE ATLAS to enumerate adversary techniques against AI-enabled systems, and the OWASP LLM Top 10 to seed abuse cases such as prompt injection and excessive agency. If your organization uses the NIST AI Risk Management Framework, the resulting risks and owners feed its GOVERN and MANAGE activities. The model should include human users, agents, tools, connectors, model endpoints, memory, retrieval, files, logs, and external destinations. ## Common threat modeling AI systems security failures * The diagram stops at the model API and ignores tools. * Prompt injection is listed as a risk but not tied to a data flow. * The threat model excludes logs, evals, and artifacts. * Agent identity is missing from the diagram. * Controls are listed without owners or evidence. ## Threat modeling AI systems security controls checklist * Map users, agents, models, tools, memory, retrieval stores, files, browsers, endpoints, queues, networks, logs, and outputs. * Mark identity, tenant, workspace, project, device, runtime, region, and downstream trust boundaries. * List what every principal can read, write, execute, publish, send, spend, and administer. * Model local, hosted, remote, scheduled, and fallback execution separately. * Write abuse cases for injection, exfiltration, excessive agency, confused deputy, tool-result spoofing, memory and data poisoning, SSRF, cross-tenant access, replay, and denial-of-wallet. * Record assumptions about provider controls and unsupported evidence. * Map each threat to prevention, detection, containment, recovery, owner, and test evidence. * Revisit the model after changes to models, tools, identity, data, memory, connectors, runtime, autonomy, or publishing. ## Model state, cost, and evidence as assets Traditional diagrams often stop at the model endpoint. Extend the diagram to persistent memory, embeddings and vector stores, evaluation and fine-tuning data, caches, traces, compliance exports, generated artifacts, queues, browser sessions, and connected-system records. Treat budget and evidence as security assets. An attacker may aim to exhaust spend or destroy attribution rather than steal data. Mark where token, rate, duration, concurrency, storage, and third-party costs are enforced, and where logs can be disabled, sampled, delayed, or attributed to a shared identity. For each consequential action, trace the chain from initiator to agent identity, model, tool, credential, target, approval, result, and evidence. Any missing link is a design gap that should be accepted, compensated for, or blocked before production. ## Anthropic ### Overview Anthropic threat models should map Claude web, Desktop, Mobile, Cowork remote and local execution, Claude Tag, Office add-ins, Code, Console/API, connectors, MCP, Agent Skills, and Managed Agents. Draw the trust boundary around tool execution, source-system identity, prompt files, skill scripts, desktop access, Slack channels and memory, cross-app transfers, and external actions. Draw two boundaries accurately. Managed Agents sessions persist state server-side and are not currently eligible for zero data retention or HIPAA BAA coverage, and the product is in beta. A Managed Agents deployment therefore places a stateful, Anthropic-hosted component in the diagram, with data-class limits regulated workflows must respect. Managed Agents can run in Anthropic-managed cloud sandboxes, isolated Linux containers with network access disabled by default, or in self-hosted sandboxes, which moves the execution boundary. Connectors act with the person's source-system permissions, so the source system's boundary, not Claude's, defines what a connector can reach. Draw Claude Tag channel work and direct messages as separate paths. Channel work crosses Slack, an Anthropic-hosted per-thread sandbox, Agent Proxy, a credential store, and service accounts in connected systems. Direct messages run under the user's Claude account and personal connectors. Those paths have different authority and attribution even though both begin with `@Claude`. ### Anthropic documentation * [Use connectors to extend Claude's capabilities](https://support.claude.com/en/articles/11176164-use-connectors-to-extend-claude-s-capabilities) * [Claude Managed Agents overview](https://platform.claude.com/docs/en/managed-agents/overview) * [Cloud sandbox reference](https://platform.claude.com/docs/en/managed-agents/cloud-sandboxes-reference) * [Claude Cowork architecture overview](https://support.claude.com/en/articles/14479288-claude-cowork-architecture-overview) * [Work across Microsoft 365 apps](https://support.claude.com/en/articles/13892150-work-across-microsoft-365-apps) * [How Claude Tag agent identity works](https://claude.com/docs/claude-tag/concepts/agent-identity) * [Claude Tag security and data handling](https://claude.com/docs/claude-tag/concepts/security-and-data) ### Applicable Harmonic guides for Anthropic * [Securing Claude Cowork: A Security Practitioner's Guide](https://www.harmonic.security/resources/securing-claude-cowork-a-security-practitioners-guide) ## OpenAI ### Overview OpenAI threat models should map Chat, Work, and Codex in the combined desktop app; cloud Work on web and mobile; plugins and apps; Sites; Office-native agents; Workspace Agents; Codex CLI, IDE, and cloud; API tools; and Agents SDK or Codex SDK services. Keep ChatGPT workspace controls, desktop or Codex managed configuration, and API project controls as separate control planes. Each plane has its own enforcement. ChatGPT RBAC covers workspace permissions such as apps, Codex, GPTs, projects, and canvas networking, with effective permissions taking the maximum across a user's roles, a detail that matters when modeling who can do what. Codex managed configuration controls approval policy, sandbox behavior, and MCP-related settings. Workspace Agent access tokens are distinct from platform and Compliance API credentials, so agent identities cross a boundary of their own. ### OpenAI documentation * [RBAC](https://help.openai.com/en/articles/11750701-rbac) * [Managed configuration](https://developers.openai.com/codex/enterprise/managed-configuration) * [Workspace Agents](https://developers.openai.com/workspace-agents) * [ChatGPT Work Admin FAQ](https://learn.chatgpt.com/docs/enterprise/work-admin-faq) * [ChatGPT desktop app](https://learn.chatgpt.com/docs/app) * [Creating and managing ChatGPT Sites](https://help.openai.com/en/articles/20001339) ### Applicable Harmonic guides for OpenAI * [Securing Codex Best Practice](https://www.harmonic.security/resources/securing-codex-best-practice) ## Frequently asked questions about threat modeling AI systems ### What is an AI threat model? It is a map of an AI workflow that identifies data flows, trust boundaries, tools, identities, and abuse cases. It differs from a classic application threat model mainly in what counts as an actor: the model can be steered by content it reads, so untrusted data becomes an active participant. The output should be a diagram, a control list, and a handful of abuse cases that matter. ### What should be in an AI data-flow diagram? Include prompts, files, retrieval, connectors, tools, model calls, outputs, logs, memory, and external destinations. Draw the arrows for data returning from tools, not just data going in, because tool results are an instruction channel. Mark which identity each arrow uses: the user's, the agent's, or a service account's. ### Which threats should be considered first? Start with prompt injection, data exfiltration, excessive agency, confused deputy behavior, and audit gaps. These map directly to OWASP LLM Top 10 categories and recur across vendors and deployment models. STRIDE and MITRE ATLAS can then broaden the list once the top risks have owners. ### Who should join the threat-modeling session? Include security, product, platform, identity, data owner, and the team building or approving the workflow. The builder knows what the agent actually calls, the data owner knows what a leak would cost, and identity knows whose credentials each hop uses. A small session with those roles beats a large review without them. ### When should threat modeling happen? Do it before pilot, before production expansion, and after major changes to tools, data, or autonomy. A new connector, a jump in autonomy, or a switch to unattended runs each redraw trust boundaries and warrant a revisit. Keep the model lightweight enough that revisiting it is cheap. ## Applicable regulations and frameworks | Governance page | Relationship to this article | | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | G.1 Map your controls to NIST AI RMF and CSF 2.0 | This article supplies implementation evidence for the NIST AI RMF and matching NIST CSF 2.0 outcomes. | | G.2 EU AI Act obligations for deployers | Conditional: for an in-scope high-risk system, this supports technical input to a fundamental-rights impact assessment. | | G.3 DORA and AI resilience in financial services | Conditional: for a DORA-regulated workflow, this supports resilience testing, vulnerability handling, and incident response. | | G.5 SANS Critical AI Security Guidelines mapping | This article implements relevant SANS Inference Security, Monitoring, and GRC guidance. | | G.7 Ownership and RACI for AI security | This control depends on the ownership and evidence responsibilities defined in the RACI. | | G.8 ISO/IEC 42001 AI management system | This article supports ISO/IEC 42001 AIMS preparation through AI risk identification and treatment planning. | | G.9 HIPAA controls for AI systems handling PHI | Conditional: for a workflow handling ePHI, this supports HIPAA risk analysis, safeguard testing, and security-incident response. | *G.2, G.3, G.4, and G.9 are conditional mappings. They apply only when the deployment is within the legal or regulatory scope described on the linked governance page.* ## Related handbook guidance * [5. Threats & Adversarial](/handbook/5.-threats-and-adversarial.md) * [5.1 Prompt injection: the connective risk](/handbook/5.-threats-and-adversarial/5.1-prompt-injection-the-connective-risk.md) * [5.4 Agent-specific threats: tool poisoning and confused deputy](/handbook/5.-threats-and-adversarial/5.4-agent-specific-threats-tool-poisoning-and-confused-deputy.md) * [7.1 Roll out by risk: the phased plan](/handbook/7.-rollout-and-operations/7.1-roll-out-by-risk-the-phased-plan.md) * [G.1 Map your controls to NIST AI RMF and CSF 2.0](/reference/governance-and-frameworks/g.1-map-your-controls-to-nist-ai-rmf-and-csf-2.0.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://handbook.harmonic.security/handbook/5.-threats-and-adversarial/5.7-threat-modeling-ai-systems.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.