Key takeaways
- Treat evals as release gates, not a post-incident review checklist.
- Require kill switch, observability, and sandbox before write autonomy.
- Score task success and incident reconstructability, not only model latency.
- Promote production failures into CI evals every week so the same bug cannot return quietly.
Why demos are not evaluation
Evaluating an AI agent is different from unit-testing deterministic code. The same prompt can yield different tool sequences, and a polite final message can hide a bad write two steps earlier. If your only check is "the demo looked fine," you are shipping nondeterministic behavior without a stop button.
Teams often treat the agent like a service that either returns 200 or does not. Agents fail in structure: loops, wrong tools, silent policy denies, and drift against a baseline that nobody recorded. Evaluation has to catch those modes before autonomy rises, not after stakeholders lose trust.
Related control planes matter here. Without containment you cannot halt a bad run (agent containment). Without traces you cannot explain it (agent observability). Without a sandbox you cannot rehearse it safely (sandbox and egress).
Operator scenario: latency is green and the chat transcript is calm. A customer still got a duplicate refund. The missing story is a retry without an idempotency key and no eval that scored task success. That is an evaluation gap, not a model mystery.
What goes wrong when you skip AI-specific evals
Hidden drift: scores look stable in a one-off notebook while production prompts and tool schemas quietly change. Without a fixed reference distribution, nobody notices until support volume rises.
Surprise edge actions: happy-path demos never exercise the messy tickets. Scenario stress is where agents invent tool calls, skip approvals, or loop on the same args.
No abort path: if there is no human-in-the-loop override or kill switch, operators chase processes instead of stopping them. Compute burns while someone digs for the deploy owner.
Unreconstructable incidents: logs show tokens and latency. They do not show plan, tools, and outcome under one run ID. Root cause becomes a debate, not a replay.
Mitigation is not "more prompts." It is a deliberate eval framework: statistical checks against a baseline, scenario stress for high-risk paths, and an override that works when the agent disagrees with you.
How to prove the agent stays in bounds
Prove bounds in three layers, not with a slide titled "guardrails."
First, pre-release statistical benchmarks against a reference set. Track deviation from baseline on task success and known failure cases, not only fluency.
Second, a runtime sandbox with resource caps and egress checks. The agent should practice under the same network and write limits it will face in production (sandbox and egress).
Third, continuous telemetry into a health view that can trigger a kill switch when thresholds break. Confidence comes from rehearsal: operators should halt a staged runaway once before anyone grants write tools.
Together, those layers answer a simple question: if this agent misbehaves at 2 a.m., can we stop it, see what it did, and show that evals would have caught a similar case?
Controls that must exist before autonomy
Three controls are non-negotiable before write autonomy.
Kill switch: a hard stop outside the agent's process tree that on-call can invoke without reading the original author's notes. Soft "please stop" prompts are not a control.
Observability: metrics, logs, and traces that reconstruct plan, tool calls, cost, and outcome under one run ID. Chat transcripts alone are not enough (agent observability).
Sandboxing: isolate the agent from critical production resources while still allowing realistic interaction. Expand blast radius only after shadow and limited-write stages pass.
With those in place, autonomy can grow on purpose. Without them, every new tool is an unscoped trust grant. Containment design belongs beside eval design (agent containment).
How kill-switch monitoring changes operator load
Real-time kill-switch monitoring moves on-call from chasing runaway loops to deciding whether a threshold was right. When the system aborts on clear signals, mean time to resolve drops and fatigue drops with it.
That only works if detection is fast and alerts are sparse. A noisy kill path trains people to ignore it. A silent path trains them to discover damage in support tickets.
Practical shape: a small rule set around loop detection, tool error spikes, and egress denies, wired to the same halt path you rehearse in staging. Measure alert precision as carefully as you measure agent task success.
Signals that your evals are already insufficient
Watch for rising incident count after releases, longer mean time to detect, and compute wasted on failed or looped runs. If on-call still manually kills agents by hunting PIDs, coverage is thin.
Correlate incident cost by month with recent prompt, tool, or index changes. Spikes after "small" prompt tweaks usually mean the change never hit a scenario gate.
When those signals cross a threshold you set in advance, tighten statistical and scenario suites before adding more autonomy. Expanding tools while MTTD climbs is how demos become outages.
Metrics that translate into release confidence
Four metrics carry more weight than a polished demo:
- Statistical benchmark deviation from baseline
- Scenario stress pass rate on high-risk paths
- Kill-switch activation frequency in shadow or limited write
- Telemetry-derived health score for complete, reconstructable runs
Low deviation, high stress pass, rare kill triggers in shadow, and healthy telemetry together say the agent is ready for a wider rollout. Wire them into CI as merge gates, not as a dashboard someone checks after the fact.
Prefer task success and reconstructability over raw model latency. A fast wrong write is still a failed release.
Scaling sandbox tests without stalling velocity
Sandbox coverage scales with automation and risk order, not with a separate QA week. Spin containerized sandboxes on demand. Cover irreversible and customer-facing paths first. Parallelize cheap nodes. Cache deterministic fixtures so flaky waits do not become the bottleneck.
Treat sandbox runs as part of CI, not a ceremony after merge. Feature velocity stays high when high-risk scenarios block bad changes early and low-risk paths stay lighter.
Loading diagram…
Diagnose → Model → Build → Harden fits agents cleanly. Diagnose failure modes (drift, edge cases, missing abort). Model safe behavior with baselines and scenarios. Build inside a sandbox and run the suite. Harden by wiring telemetry, alerts, and the kill switch into production.
This week: add one evaluation gate to CI that blocks merge without a recent statistical run on the touched workflow, and rehearse the kill path once in staging. That is an operable barrier before autonomy expands.
FAQ
- What breaks first when teams skip AI agent evaluation?
- Drift and edge-case actions reach users before operators notice. Without statistical baselines, scenario stress, and a halt path, on-call discovers failures after compute is wasted and trust is already damaged.
- Which controls must exist before granting agent autonomy?
- A real kill switch operators can find without the author, reconstructable traces for each run, and a sandbox with egress limits. Expand write access only after those three are rehearsed under load.
- Which metrics show an AI agent is ready to release?
- Benchmark deviation from baseline, scenario stress pass rate, kill-switch trigger rate in shadow, and a telemetry health score. Gate merges on those signals in CI, not on a polished demo transcript.