> For the complete documentation index, see [llms.txt](https://handbook.harmonic.security/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://handbook.harmonic.security/handbook/7.-rollout-and-operations/7.2-pilot-design-and-success-metrics.md). # 7.2 Pilot design and success metrics What a good AI security pilot looks like, how to select cohorts, and how to know whether controls worked before widening rollout. *Last reviewed: August 18, 2026* {% hint style="info" %} A good pilot tests controls as well as usefulness. Measure productivity, policy fit, support load, and evidence quality before expansion. {% endhint %} ## What security teams need to know about pilot design and success metrics Pilots often measure whether users like the tool. That is useful, but security needs more. A pilot should test whether the approved workflows stay inside the intended controls. Choose users who represent the eventual rollout, but keep the first cohort small enough to support. Include at least one workflow that touches real operational constraints, such as connectors, code, or sensitive internal data. ## Common pilot design and success metrics security failures * The pilot cohort is too friendly to reveal misuse. * Success is measured only by user satisfaction. * No one checks whether DLP and audit logs captured the work. * Support tickets are ignored as rollout signals. * Go/no-go criteria are decided after the pilot ends. ## Pilot design and success metrics security controls checklist * Define pilot use cases, users, data classes, allowed actions, prohibited uses, and execution paths in advance. * Train users and publish a clear support and incident-reporting route. * Measure productivity, control hits, cost, support load, incidents, exceptions, telemetry quality, and audit completeness. * Test joiner, mover, leaver, device-loss, revocation, exception, stop, rollback, and exit paths. * Collect feedback on confusing approvals, blocked actions, output review, and workaround behavior. * Include skeptical and non-expert users, not only enthusiasts. * Set quantitative go, hold, and stop criteria before launch. * Assign owners for every defect and require closure or explicit risk acceptance before expansion. ## Pilot metrics Agree a small set of metrics before launch. Each metric needs a definition, a data source, and a go/no-go threshold. The data sources map to the observability pages: compliance APIs (see 6.3, Compliance APIs by platform), analytics and usage APIs (see 6.4, Analytics and usage APIs), and SIEM routing (see 6.5, Routing AI telemetry to your SIEM). Where runtime tools emit OpenTelemetry signals, route them through your collector so denials and usage land beside the rest of the telemetry. | Metric | Definition | Data source | Example go/no-go threshold | | -------------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Active adoption | Share of the pilot cohort using the tool in a given week | Workspace analytics and usage APIs (see 6.4) | At least 60% weekly active by week 4, or the cohort was wrong | | Blocked sensitive prompts | DLP alerts raised on pilot traffic | DLP alerts routed to the SIEM (see 6.5; 4.1, DLP for GenAI) | 100% of alerts triaged within 2 business days; repeat-offender rate falling | | Denied tool and permission calls | Approval or permission denials in agent runtimes | Runtime telemetry and logs in the SIEM (see 6.5) | Every denial reviewed; no unexplained week-over-week spike | | Connector approval coverage | Share of connectors and apps in use that have a logged intake approval | Intake register reconciled against compliance API records (see 6.3; 2.7, The supply-chain review workflow) | 100%; any unapproved connector in use is a no-go | | Evidence coverage | Share of planned event types with a tested evidence path, including documented gaps | Compliance API, OpenTelemetry, endpoint, app, and target-system records reconciled against scripted pilot actions (see 6.3 and 6.6) | 100% of required event types either captured and attributable or blocked from production | | Support load | Tickets per pilot user per week | Ticket queue tagged for the pilot | Falling trend by week 3; no unresolved control-confusion themes | | Incidents and exceptions | Security incidents and policy exceptions opened during the pilot | IR tracker and exception register (see 5.6, Incident response for AI systems) | Zero unresolved high-severity incidents; every exception has an owner and expiry date | | Telemetry delivery | Required event types arrive within the defined latency and completeness objective | Source-to-collector-to-SIEM reconciliation (see 6.2 and 6.6) | No missing high-value events; unexplained loss or attribution failure is a no-go | | Exit readiness | Active work, identities, connections, artifacts, and data can be stopped or removed | Technical exit exercise and deletion evidence (see 7.1 and 5.6) | Stop and revocation complete within the agreed objective; known exceptions accepted | | User readiness | Users understand permitted data, approvals, output review, and incident reporting | Training completion plus scenario check | Required cohort completes training and passes the scenario check | | Output quality | Sampled task outputs reviewed against acceptance criteria you define | Your own evaluation harness plus reviewer sampling | Acceptance rate meets the bar agreed before launch | Build quality measurement on your own evaluation harness rather than a vendor-hosted one. Vendor note (OpenAI): OpenAI documents its hosted Evals platform as deprecated, with the platform going read-only on October 31, 2026, so a 2026 pilot should not build its quality gate on it. ## Anthropic ### Overview Anthropic pilots need a surface-specific instrument panel. Claude Code analytics dashboards provide adoption and contribution measures, while Claude Code monitoring provides runtime telemetry. Cowork requires a separate OpenTelemetry test because Anthropic does not include Cowork activity in audit logs, the Compliance API, or data exports. Script remote and local Cowork actions, then confirm that prompts, tool calls, file access, approvals, and downstream changes can be reconstructed without relying on endpoint tools to inspect the remote sandbox. Scope the cohort with identity. Assign the pilot population a group whose custom role enables only the surfaces under test, whether that is chat, Claude Code, Cowork, Office add-ins, or connectors. If the pilot uses Zero Data Retention, use a productivity measure other than Claude Code contribution metrics, which are unavailable under ZDR. For a Claude Tag pilot, measure active channels, requesters, task completion, per-channel spend, routines created, blocked requests, and connected-system actions. Validate that the Slack thread, Audit view, optional network events, and service-account logs can reconstruct representative tasks. Claude Tag retains memory and transcripts and is unavailable under ZDR, so it needs its own eligibility check rather than inheriting the Claude Code pilot decision. ### Anthropic documentation * [Track team usage with analytics](https://code.claude.com/docs/en/analytics) * [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage) * [Manage custom roles on Enterprise plans](https://support.claude.com/en/articles/13930452-manage-custom-roles-on-enterprise-plans) * [Monitor Claude Cowork activity with OpenTelemetry](https://support.claude.com/en/articles/14477985-monitor-claude-cowork-activity-with-opentelemetry) * [Claude Cowork architecture overview](https://support.claude.com/en/articles/14479288-claude-cowork-architecture-overview) * [What is Claude Tag?](https://support.claude.com/en/articles/15594475-what-is-claude-tag) * [Review what Claude Tag has done](https://claude.com/docs/claude-tag/admins/audit) ### Applicable Harmonic guides for Anthropic * [Securing Claude Cowork: A Security Practitioner's Guide](https://www.harmonic.security/resources/securing-claude-cowork-a-security-practitioners-guide) ## OpenAI ### Overview OpenAI pilots should separate Chat, Work cloud, desktop Work, local Codex clients, and API-key automation. ChatGPT Desktop (Work / Codex) puts Chat, Work, and Codex in one interface, but web and mobile Work run in OpenAI's cloud while desktop Work can use local files and applications. OpenAI's Compliance API covers messages and responses across Chat, Work, and Codex, but not Work files, actions, or tool calls. Script those events and prove the endpoint, plugin, connected-app, and target-system evidence paths. Codex Enterprise adds analytics dashboards, an Analytics API, and a Compliance API for ChatGPT-authenticated activity. API-key-authenticated automation still needs API organization records and endpoint evidence. Keep Chat convenience metrics separate from Work, Codex, and API autonomy metrics. For output quality, use your own evaluation harness because the hosted Evals platform is deprecated. ### OpenAI documentation * [Compliance APIs for Enterprise Customers](https://help.openai.com/en/articles/9261474-compliance-apis-for-enterprise-customers) * [Codex governance](https://developers.openai.com/codex/enterprise/governance) * [ChatGPT Work and Codex](https://help.openai.com/en/articles/20001275) * [Work Admin FAQ](https://learn.chatgpt.com/docs/enterprise/work-admin-faq) * [Working with evals](https://developers.openai.com/api/docs/guides/evals) — deprecated; OpenAI is retiring the hosted Evals platform, so treat this page as transition-period reference only ### Applicable Harmonic guides for OpenAI * [Securing ChatGPT Enterprise Guide](https://www.harmonic.security/resources/securing-chatgpt-enterprise-guide) * [Securing Codex Best Practice](https://www.harmonic.security/resources/securing-codex-best-practice) ## Frequently asked questions about pilot design and success metrics ### What should an AI pilot measure? Measure six things: productivity and adoption, control effectiveness such as DLP hits and denied calls, connector approval coverage, support load, audit completeness, and incidents or exceptions. Define each metric and its threshold before launch, because criteria decided after the pilot ends always pass. The Pilot metrics table above is a starting set to adapt. ### Who should be in the pilot? Choose users from the real target groups, across different job functions, risk levels, and technical comfort. Avoid a cohort made only of enthusiasts: friendly pilots hide misuse, confusing approvals, and workaround behavior that will surface at scale. Include at least one user whose workflow touches a sensitive data class, so the DLP and approval paths actually get exercised. ### How long should a pilot run? Run long enough to see repeat work, edge cases, and one full operational cycle — typically several weeks for workforce tools. A useful endpoint test: the pilot has covered at least one joiner-mover-leaver cycle, one access review, and one exercised incident-response path. If none of those happened, the pilot measured enthusiasm, not operations. ### What are guardrail metrics? Guardrail metrics show whether controls worked, as distinct from whether users were productive. Examples: blocked sensitive prompts triaged, denied tool calls reviewed, connector approvals logged, and audit records complete for the cohort. A pilot that reports only satisfaction and time saved has not tested security. ### What should block expansion? Block expansion if controls fail for the pilot cohort, support volume is unmanageable, audit logs are incomplete or cannot be attributed, or users rely on prohibited workarounds to get work done. Workarounds are the strongest signal: they mean the approved path does not fit the workflow, and scaling the rollout will scale the workaround with it. ## Applicable regulations and frameworks | Governance page | Relationship to this article | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | G.1 Map your controls to NIST AI RMF and CSF 2.0 | This article supplies implementation evidence for the NIST AI RMF and matching NIST CSF 2.0 outcomes. | | G.3 DORA and AI resilience in financial services | Conditional: for a DORA-regulated workflow, this supports risk-based rollout, control validation, and recovery readiness. | | G.5 SANS Critical AI Security Guidelines mapping | This article implements relevant SANS Deployment Strategies and GRC guidance. | | G.8 ISO/IEC 42001 AI management system | This article supports ISO/IEC 42001 AIMS preparation through objectives, validation criteria, and measurement during controlled deployment. | *G.2, G.3, G.4, and G.9 are conditional mappings. They apply only when the deployment is within the legal or regulatory scope described on the linked governance page.* ## Related handbook guidance * [7. Rollout & Operations](/handbook/7.-rollout-and-operations.md) * [7.1 Roll out by risk: the phased plan](/handbook/7.-rollout-and-operations/7.1-roll-out-by-risk-the-phased-plan.md) * [6.7 Continuous review cadence](/handbook/6.-observability-audit-and-evidence/6.7-continuous-review-cadence.md) * [5.5 Red-teaming AI systems](/handbook/5.-threats-and-adversarial/5.5-red-teaming-ai-systems.md) * [7.4 The vendor-neutral control matrix](/handbook/7.-rollout-and-operations/7.4-the-vendor-neutral-control-matrix.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://handbook.harmonic.security/handbook/7.-rollout-and-operations/7.2-pilot-design-and-success-metrics.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.