Multi-agent harness deployment:
from notebook demos to gated production runners

Where notebook multi-agent demos collapse

Happy paths hide shared tool contention, partial failures, and ambiguous ownership between agents. In a notebook, the author restarts. In production, a half-completed purchase or ticket update is an incident.

Retries without idempotency duplicate side effects. Handoffs without typed state lose context. Tool lists without scopes let a worker call APIs the planner never intended.

If you cannot answer who can call which tool under which approval, you are not ready to deploy. You are ready to demo. The gap is harness design, not model quality. Pair role scoping with the same write-gate discipline covered in human-in-the-loop write gates.

Role boundaries and tool registries

Define roles explicitly: planner, researcher, executor, reviewer. Each role gets a tool allowlist and a maximum blast radius. Planners propose; workers execute only what their registry entry allows.

Register tools with schemas, side-effect class (read, write, irreversible), and required approval level. Do not pass raw SDKs into every agent prompt. Prefer capability tokens or scoped credentials over a shared god key. When an agent is compromised or confused, the registry should limit damage.

Version registry entries with the harness, not with prompt text. A widened URL parameter or a new HTTP helper is a control-plane change. Review it like an ACL diff. Task-scoped access patterns in task-based access control for agents fit the same model.

Durable run state and failure taxonomies

Persist run state outside the model context. Steps, tool results, decisions, and pending approvals need a store you can resume after a crash. Context windows are not durable storage.

Classify failures: transient tool error, policy block, human timeout, model loop, missing input. Each class needs a different recovery path. A retry that helps a 429 will worsen a policy deny.

Detect loops. If an agent retries the same tool with the same arguments without progress, stop and escalate instead of burning tokens. Persist the loop signature so the next eval suite can catch it in CI.

Human approval gates for irreversible work

Anything that spends money, deletes data, messages customers, or changes production config should require a gate. Draft the action, show the diff, wait for approval. Encode the gate in policy and the tool registry, not in prompt instructions that drift between releases.

Gates shape the UX of the harness and the audit trail compliance teams will ask for later. Timeouts matter. Decide whether pending approvals expire, auto-reject, or park the run for a human queue. For high-blast-radius classes, timeout should hold, not auto-apply.

Promote write tools from draft to limited auto only after shadow scoring on a frozen eval set. Expand one irreversible action type at a time. Latency charts alone are not permission to skip the gate.

Deployment paths: local, cloud, and gated runners

Local runners help developers reproduce incidents. Cloud runners need identity, secrets, concurrency limits, and isolation between tenants or teams. Shared god credentials across environments turn a lab mistake into a production event.

Version the harness separately from prompts. A prompt change should not silently rewrite permission rules. Ship kill switches: pause all runs, disable a tool, or freeze a role. Observability without control leaves operators watching damage they cannot stop.

Gateway and MCP auth patterns in MCP enterprise gateway auth belong on the same deploy path as the runner. Identity, tool scope, and kill must share one run identity across local and cloud.

Traces operators can actually use

A useful trace shows the plan, each tool call, latency, errors, and the final outcome. Nested spans beat a flat chat log. Carry durable run ID, principal, tool ID, side-effect class, and outcome code on every span.

Correlate traces with eval results. When quality drops, you need to know whether orchestration or a single tool regression caused it. Incomplete span trees are a defect, same class as missing payment logs.

Autonomous cloud agents amplify risk. Higher autonomy requires stronger tracing, tighter scopes, and clearer ownership of who restarts a failed fleet. Measure task success, time-to-halt, and reconstructability of a bad run before you raise concurrency.

Roll out in layers: local reproduce, shadow or draft with no irreversible writes, limited writes behind gates, then concurrency only after kill and traces hold under load. Document who may widen a tool allowlist and how temporary break-glass appears in audit. The harness is production infrastructure. Treat prompt polish as secondary until roles, registries, state, gates, and kill paths are operable.