7.2 Pilot design and success metrics
What a good AI security pilot looks like, how to select cohorts, and how to know whether controls worked before widening rollout.
Last reviewed: August 18, 2026
What security teams need to know about pilot design and success metrics
Pilots often measure whether users like the tool. That is useful, but security needs more. A pilot should test whether the approved workflows stay inside the intended controls.
Choose users who represent the eventual rollout, but keep the first cohort small enough to support. Include at least one workflow that touches real operational constraints, such as connectors, code, or sensitive internal data.
Common pilot design and success metrics security failures
The pilot cohort is too friendly to reveal misuse.
Success is measured only by user satisfaction.
No one checks whether DLP and audit logs captured the work.
Support tickets are ignored as rollout signals.
Go/no-go criteria are decided after the pilot ends.
Pilot design and success metrics security controls checklist
Define pilot use cases, users, data classes, allowed actions, prohibited uses, and execution paths in advance.
Train users and publish a clear support and incident-reporting route.
Measure productivity, control hits, cost, support load, incidents, exceptions, telemetry quality, and audit completeness.
Test joiner, mover, leaver, device-loss, revocation, exception, stop, rollback, and exit paths.
Collect feedback on confusing approvals, blocked actions, output review, and workaround behavior.
Include skeptical and non-expert users, not only enthusiasts.
Set quantitative go, hold, and stop criteria before launch.
Assign owners for every defect and require closure or explicit risk acceptance before expansion.
Pilot metrics
Agree a small set of metrics before launch. Each metric needs a definition, a data source, and a go/no-go threshold. The data sources map to the observability pages: compliance APIs (see 6.3, Compliance APIs by platform), analytics and usage APIs (see 6.4, Analytics and usage APIs), and SIEM routing (see 6.5, Routing AI telemetry to your SIEM). Where runtime tools emit OpenTelemetry signals, route them through your collector so denials and usage land beside the rest of the telemetry.
Active adoption
Share of the pilot cohort using the tool in a given week
Workspace analytics and usage APIs (see 6.4)
At least 60% weekly active by week 4, or the cohort was wrong
Blocked sensitive prompts
DLP alerts raised on pilot traffic
DLP alerts routed to the SIEM (see 6.5; 4.1, DLP for GenAI)
100% of alerts triaged within 2 business days; repeat-offender rate falling
Denied tool and permission calls
Approval or permission denials in agent runtimes
Runtime telemetry and logs in the SIEM (see 6.5)
Every denial reviewed; no unexplained week-over-week spike
Connector approval coverage
Share of connectors and apps in use that have a logged intake approval
Intake register reconciled against compliance API records (see 6.3; 2.7, The supply-chain review workflow)
100%; any unapproved connector in use is a no-go
Evidence coverage
Share of planned event types with a tested evidence path, including documented gaps
Compliance API, OpenTelemetry, endpoint, app, and target-system records reconciled against scripted pilot actions (see 6.3 and 6.6)
100% of required event types either captured and attributable or blocked from production
Support load
Tickets per pilot user per week
Ticket queue tagged for the pilot
Falling trend by week 3; no unresolved control-confusion themes
Incidents and exceptions
Security incidents and policy exceptions opened during the pilot
IR tracker and exception register (see 5.6, Incident response for AI systems)
Zero unresolved high-severity incidents; every exception has an owner and expiry date
Telemetry delivery
Required event types arrive within the defined latency and completeness objective
Source-to-collector-to-SIEM reconciliation (see 6.2 and 6.6)
No missing high-value events; unexplained loss or attribution failure is a no-go
Exit readiness
Active work, identities, connections, artifacts, and data can be stopped or removed
Technical exit exercise and deletion evidence (see 7.1 and 5.6)
Stop and revocation complete within the agreed objective; known exceptions accepted
User readiness
Users understand permitted data, approvals, output review, and incident reporting
Training completion plus scenario check
Required cohort completes training and passes the scenario check
Output quality
Sampled task outputs reviewed against acceptance criteria you define
Your own evaluation harness plus reviewer sampling
Acceptance rate meets the bar agreed before launch
Build quality measurement on your own evaluation harness rather than a vendor-hosted one. Vendor note (OpenAI): OpenAI documents its hosted Evals platform as deprecated, with the platform going read-only on October 31, 2026, so a 2026 pilot should not build its quality gate on it.
Anthropic
Overview
Anthropic pilots need a surface-specific instrument panel. Claude Code analytics dashboards provide adoption and contribution measures, while Claude Code monitoring provides runtime telemetry. Cowork requires a separate OpenTelemetry test because Anthropic does not include Cowork activity in audit logs, the Compliance API, or data exports. Script remote and local Cowork actions, then confirm that prompts, tool calls, file access, approvals, and downstream changes can be reconstructed without relying on endpoint tools to inspect the remote sandbox.
Scope the cohort with identity. Assign the pilot population a group whose custom role enables only the surfaces under test, whether that is chat, Claude Code, Cowork, Office add-ins, or connectors. If the pilot uses Zero Data Retention, use a productivity measure other than Claude Code contribution metrics, which are unavailable under ZDR.
For a Claude Tag pilot, measure active channels, requesters, task completion, per-channel spend, routines created, blocked requests, and connected-system actions. Validate that the Slack thread, Audit view, optional network events, and service-account logs can reconstruct representative tasks. Claude Tag retains memory and transcripts and is unavailable under ZDR, so it needs its own eligibility check rather than inheriting the Claude Code pilot decision.
Anthropic documentation
Applicable Harmonic guides for Anthropic
OpenAI
Overview
OpenAI pilots should separate Chat, Work cloud, desktop Work, local Codex clients, and API-key automation. ChatGPT Desktop (Work / Codex) puts Chat, Work, and Codex in one interface, but web and mobile Work run in OpenAI's cloud while desktop Work can use local files and applications. OpenAI's Compliance API covers messages and responses across Chat, Work, and Codex, but not Work files, actions, or tool calls. Script those events and prove the endpoint, plugin, connected-app, and target-system evidence paths.
Codex Enterprise adds analytics dashboards, an Analytics API, and a Compliance API for ChatGPT-authenticated activity. API-key-authenticated automation still needs API organization records and endpoint evidence. Keep Chat convenience metrics separate from Work, Codex, and API autonomy metrics. For output quality, use your own evaluation harness because the hosted Evals platform is deprecated.
OpenAI documentation
Working with evals — deprecated; OpenAI is retiring the hosted Evals platform, so treat this page as transition-period reference only
Applicable Harmonic guides for OpenAI
Frequently asked questions about pilot design and success metrics
What should an AI pilot measure?
Measure six things: productivity and adoption, control effectiveness such as DLP hits and denied calls, connector approval coverage, support load, audit completeness, and incidents or exceptions. Define each metric and its threshold before launch, because criteria decided after the pilot ends always pass. The Pilot metrics table above is a starting set to adapt.
Who should be in the pilot?
Choose users from the real target groups, across different job functions, risk levels, and technical comfort. Avoid a cohort made only of enthusiasts: friendly pilots hide misuse, confusing approvals, and workaround behavior that will surface at scale. Include at least one user whose workflow touches a sensitive data class, so the DLP and approval paths actually get exercised.
How long should a pilot run?
Run long enough to see repeat work, edge cases, and one full operational cycle — typically several weeks for workforce tools. A useful endpoint test: the pilot has covered at least one joiner-mover-leaver cycle, one access review, and one exercised incident-response path. If none of those happened, the pilot measured enthusiasm, not operations.
What are guardrail metrics?
Guardrail metrics show whether controls worked, as distinct from whether users were productive. Examples: blocked sensitive prompts triaged, denied tool calls reviewed, connector approvals logged, and audit records complete for the cohort. A pilot that reports only satisfaction and time saved has not tested security.
What should block expansion?
Block expansion if controls fail for the pilot cohort, support volume is unmanageable, audit logs are incomplete or cannot be attributed, or users rely on prohibited workarounds to get work done. Workarounds are the strongest signal: they mean the approved path does not fit the workflow, and scaling the rollout will scale the workaround with it.
Applicable regulations and frameworks
G.1 Map your controls to NIST AI RMF and CSF 2.0
This article supplies implementation evidence for the NIST AI RMF and matching NIST CSF 2.0 outcomes.
G.3 DORA and AI resilience in financial services
Conditional: for a DORA-regulated workflow, this supports risk-based rollout, control validation, and recovery readiness.
G.5 SANS Critical AI Security Guidelines mapping
This article implements relevant SANS Deployment Strategies and GRC guidance.
G.8 ISO/IEC 42001 AI management system
This article supports ISO/IEC 42001 AIMS preparation through objectives, validation criteria, and measurement during controlled deployment.
G.2, G.3, G.4, and G.9 are conditional mappings. They apply only when the deployment is within the legal or regulatory scope described on the linked governance page.
Related handbook guidance
Was this helpful?