> For the complete documentation index, see [llms.txt](https://handbook.harmonic.security/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://handbook.harmonic.security/handbook/5.-threats-and-adversarial/5.5-red-teaming-ai-systems.md). # 5.5 Red-teaming AI systems A repeatable program to probe injection, exfiltration, destructive actions, permission abuse, tool misuse, and unsafe agent autonomy across your AI deployment. *Last reviewed: August 18, 2026* {% hint style="info" %} Red-team AI workflows by attacking the whole system: prompts, tools, data, identity, network, approvals, and logs. {% endhint %} ## What security teams need to know about red-teaming AI systems AI red teaming should move beyond jailbreak prompts. The test target is the workflow: what content the agent reads, which tools it can call, what data it can reach, and where outputs can go. Anchor the test catalog to public frameworks rather than inventing categories: MITRE ATLAS enumerates adversary techniques against AI-enabled systems, and the OWASP LLM Top 10 names the risk categories (prompt injection, sensitive information disclosure, supply chain, insecure plugin design, excessive agency) that findings should map to. A good program produces findings that map to controls. Each finding should say which chain worked, which control failed, and which owner can fix it. ## Common red-teaming AI systems security failures * Tests focus on model refusal instead of tool abuse. * The red team lacks realistic connectors or data classes. * Findings do not map to owners or controls. * The same test set is reused after the product changes. * No one retests after a mitigation. ## Red-teaming AI systems security controls checklist * Maintain a scenario catalog indexed to the system's threat model, OWASP LLM Top 10, MITRE ATLAS, and known product boundaries. * Test direct and indirect injection through documents, pages, email, code, retrieval, memory, tool results, and inter-agent messages. * Test exfiltration through connectors, URLs, browser actions, generated files, messages, publishing, and covert or encoded output. * Test poisoned tools, skills, plugins, hooks, templates, packages, models, retrieval data, and updates. * Test confused-deputy, cross-tenant, cross-workspace, shared-service-account, and permission-inheritance paths. * Test SSRF, internal network pivoting, metadata access, redirect handling, and local service access. * Test denial-of-wallet, recursion, runaway tool use, concurrency, storage, and timeout limits. * Test replay, duplicate action, stale approval, stop failure, and recovery behavior. * Run tests before pilot expansion and after material model, tool, policy, data, or architecture changes. * Use synthetic sensitive data and traceable markers. * Map each finding to an owner, blocking decision, remediation, evidence, and retest date. ## Minimum scenario catalog Cover at least one scenario for each of these paths: 1. untrusted content changes the agent's goal; 2. the agent retrieves data the requester should not receive; 3. a legitimate credential is used for the wrong principal or task; 4. a tool description or result changes after approval; 5. persistent memory or retrieval data affects a later run; 6. an agent reaches an internal or prohibited network destination; 7. a retry, replay, or race duplicates a consequential action; 8. work exceeds token, spend, duration, concurrency, or storage limits; 9. the user stops the task but downstream work continues; 10. required evidence is absent, delayed, redacted incorrectly, or attributed to the wrong identity. Record the expected control and evidence before executing the test. A red-team finding is incomplete if the team cannot state which production gate it blocks. ## Anthropic ### Overview Anthropic red-team plans should test Claude chat and remote Cowork, connectors, Desktop extensions, computer use, Office add-ins, Claude Code, API tools, Agent Skills, Managed Agents, and multi-agent messages. Include prompt leakage, cross-app context transfer, indirect injection, malicious tool output, unsafe file edits, connector abuse, and unattended-run drift. Anthropic's own guidance supplies test targets. The prompt-leak guardrail guidance treats prompt, tool, and system instruction exposure as adversarial-review material, so include leak cases alongside injection cases. The skill-review guidance calls out scripts, adversarial instructions, MCP references, network access, and tool invocations: a ready-made checklist for poisoned-skill test cases. Managed Agents sessions persist state server-side and run unattended, so give long-running sessions their own scenarios rather than reusing interactive-chat cases. For Claude Tag, test both identity modes. In channels, probe shared Access bundles, channel memory, watched channels, scheduled routines, allowed domains, and service-account attribution. In direct messages, test the user's personal connectors and attribution. Include a case where an attacker plants instructions before the scheduled routine runs and a case where a broad channel member tries to use a credential intended for a narrower team. ### Anthropic documentation * [Reduce prompt leak](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-prompt-leak) * [Skills for enterprise](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/enterprise) * [Use connectors to extend Claude's capabilities](https://support.claude.com/en/articles/11176164-use-connectors-to-extend-claude-s-capabilities) * [Claude Managed Agents overview](https://platform.claude.com/docs/en/managed-agents/overview) * [Use Claude Cowork safely](https://support.claude.com/en/articles/13364135-use-claude-cowork-safely) * [Work across Microsoft 365 apps](https://support.claude.com/en/articles/13892150-work-across-microsoft-365-apps) * [How Claude Tag agent identity works](https://claude.com/docs/claude-tag/concepts/agent-identity) * [Set up Claude Tag routines](https://claude.com/docs/claude-tag/users/proactivity) ### Applicable Harmonic guides for Anthropic * [Securing Claude Cowork: A Security Practitioner's Guide](https://www.harmonic.security/resources/securing-claude-cowork-a-security-practitioners-guide) ## OpenAI ### Overview OpenAI red-team plans should test Chat, Work, and Codex in the combined desktop app; Work on web and mobile; plugins and apps; Sites; Office-native agents; Workspace Agents; hosted agent mode; local browser and computer use; API tools; and Agents SDK workflows. Product-specific controls such as Work RBAC, Site publishing, app actions, and Codex approval profiles define the boundaries the red team should try to cross. OpenAI publishes two distinct resources that are easy to conflate. The red-teaming guide covers testing itself: prompts, tools, model behavior, and system-level safeguards. The safety-checks guidance is about gates: checks that can block deployment or expansion until they pass. Use the first to generate findings and the second to decide what a finding blocks, and keep the two feeding each other. ### OpenAI documentation * [Red teaming](https://developers.openai.com/api/docs/guides/red-teaming) * [Safety checks](https://developers.openai.com/api/docs/guides/safety-checks) * [Safety best practices](https://developers.openai.com/api/docs/guides/safety-best-practices) * [Agent approvals and security](https://developers.openai.com/codex/agent-approvals-security) * [ChatGPT Work Admin FAQ](https://learn.chatgpt.com/docs/enterprise/work-admin-faq) * [Creating and managing ChatGPT Sites](https://help.openai.com/en/articles/20001339) ### Applicable Harmonic guides for OpenAI * [Securing Codex Best Practice](https://www.harmonic.security/resources/securing-codex-best-practice) ## Frequently asked questions about red-teaming AI systems ### What is AI red teaming? It is structured testing of AI workflows to find ways prompts, tools, data, and controls can fail. Unlike model-only jailbreak testing, it attacks the whole system: what the agent reads, what it can call, what it can reach, and where outputs go. The output is a set of findings mapped to controls and owners, not a list of clever prompts. ### What should AI red teams test? Test prompt injection, tool abuse, exfiltration, excessive agency, data leakage, and audit gaps. Index the catalog to MITRE ATLAS techniques and OWASP LLM Top 10 categories so coverage is checkable rather than anecdotal. Include the unglamorous cases, such as logging gaps and approval fatigue, alongside the flashy ones. ### How often should AI red-team tests run? Quarterly is a recommended starting point for high-risk workflows, adjusted to release tempo and risk appetite. Also test before major rollouts and after significant vendor or product changes, since a new tool, connector, or model can invalidate earlier results. Retesting after each mitigation is part of the cadence, not an extra. ### Should tests use real sensitive data? Use synthetic data that looks realistic. Real secrets or regulated records can leak through test logs, model context, or findings documents, turning the exercise into an incident. Seed synthetic records with markers so any later appearance is traceable to the test. ### What makes a good finding? A good finding includes the chain that worked, impact, evidence, the control that failed, an owner, the fix, and a retest date. Findings phrased as model behavior ("the model complied") age poorly; findings phrased as control gaps ("the egress allowlist did not cover this destination") get fixed. ## Applicable regulations and frameworks | Governance page | Relationship to this article | | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | G.1 Map your controls to NIST AI RMF and CSF 2.0 | This article supplies implementation evidence for the NIST AI RMF and matching NIST CSF 2.0 outcomes. | | G.3 DORA and AI resilience in financial services | Conditional: for a DORA-regulated workflow, this supports resilience testing, vulnerability handling, and incident response. | | G.5 SANS Critical AI Security Guidelines mapping | This article implements relevant SANS Inference Security, Monitoring, and GRC guidance. | | G.7 Ownership and RACI for AI security | This control depends on the ownership and evidence responsibilities defined in the RACI. | | G.8 ISO/IEC 42001 AI management system | This article supports ISO/IEC 42001 AIMS preparation through verification, validation, and evaluation evidence. | *G.2, G.3, G.4, and G.9 are conditional mappings. They apply only when the deployment is within the legal or regulatory scope described on the linked governance page.* ## Related handbook guidance * [5. Threats & Adversarial](/handbook/5.-threats-and-adversarial.md) * [5.7 Threat modeling AI systems](/handbook/5.-threats-and-adversarial/5.7-threat-modeling-ai-systems.md) * [5.1 Prompt injection: the connective risk](/handbook/5.-threats-and-adversarial/5.1-prompt-injection-the-connective-risk.md) * [5.6 Incident response for AI system](/handbook/5.-threats-and-adversarial/5.6-incident-response-for-ai-system.md) * [7.2 Pilot design and success metrics](/handbook/7.-rollout-and-operations/7.2-pilot-design-and-success-metrics.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://handbook.harmonic.security/handbook/5.-threats-and-adversarial/5.5-red-teaming-ai-systems.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.