Key takeaways
- Trace the plan, each tool call, handoffs, and final outcome under one run ID.
- Separate performance metrics from quality evals; both are required.
- Attribute cost to workflows and roles, not only to model providers.
- Feed production failures back into CI eval sets every week.
Why chat logs are not observability
Agents fail in structure, not only in wording. A wrong tool, a silent policy deny, a loop of identical calls, or a bad handoff will not look like a classic HTTP 500. If your only view is the final assistant message, you are debugging with a transcript of symptoms.
Observability for agents means answering four questions after every incident: what did it plan, what did it call, what did it cost, and was the outcome any good. Latency alone answers none of those.
Harness design and observability travel together. Without durable run IDs and tool registries, traces stay shallow. See multi-agent harness deployment.
Operator scenario: p95 latency is green. A customer still got a refund twice. The chat log shows a polite confirmation. The missing story is three tool spans, one retry without an idempotency key, and a planner that treated a timeout as failure and called again. Until those spans exist, the postmortem is guesswork.
Failure modes when instrumentation is thin
Flat logs: one blob per run with no span boundaries around tools. Engineers paste the blob into a model and ask it to explain itself. That is not a control plane.
Orphan costs: model invoices arrive monthly while workflow owners cannot see spend per feature. Finance asks which product burned the budget; nobody can map tokens and tool calls to a run type.
Quality blindness: dashboards are green while citation accuracy or task success drifts down. Ops celebrates uptime while support files the same failure for the third week.
No feedback loop: production failures never become eval cases, so the same bug returns after the next prompt tweak. Traces exist as screenshots in tickets, not as fixtures in CI.
Incomplete trees: LLM spans are present; MCP and handoff spans are missing because those SDKs bypassed the instrumented client. The run looks short and cheap; the damage happened off-trace.
PII in cleartext spans: teams log full prompts and tool payloads "for debugging" and create a second sensitive data store. Observability that cannot redact becomes a compliance incident waiting for a search.
The trace and eval model that holds
Emit a root span per run. Child spans cover planner steps, model calls, retrievals, MCP tools, human gates, and peer handoffs. Carry identity, tool name, schema version, and side-effect class on tool spans. Carry run ID, tenant, and workflow name on every child so joins stay trivial.
Record outcomes explicitly: success, policy deny, tool error, human reject, loop abort. Do not infer everything from HTTP status codes. A 200 from a tool that wrote the wrong entity is still a failed task.
Pair traces with evals. Online scores and sampled human review answer "was it good?" Tracing answers "what happened?" You need both. Sample by risk: irreversible writes and customer-facing answers get higher review rates than FAQ traffic.
Loading diagram…
Operable minimum fields per tool span: tool ID, schema version, principal, arg fingerprint, latency, outcome code, and side-effect class. Without those, "we have OpenTelemetry" is not agent observability.
Cost, quality, and gates in one loop
Attribute tokens and tool calls to team, workflow, and environment. Cost-aware routing only works when attribution is honest; see cost-aware model routing. Include human-gate wait time as a first-class metric when approvals are part of the path.
Sample production failures into the eval suite weekly. Gate prompt, index, and harness changes on that suite (eval gates in CI). Prefer cases with attached span trees so regressions show which step broke, not only which score fell.
Expose a run inspector to operators: plan, spans, denials, final answer. Without it, every incident becomes a prompt debate. Restrict who can expand redacted fields. Log those expansions.
Detect loops and fan-out in near real time: identical tool args repeating, hop counts above budget, or A2A bounce patterns. Alert on structure, not only on error rate. When traffic crosses LLM, MCP, and A2A, align instrumentation with the triple-gate so traces and policy share the same run identity.
Rollout sequence and metrics
Instrument one high-value workflow end to end before fleets of agents. Prove you can reconstruct a bad run in minutes with the run inspector alone. Only then expand to sibling workflows that share the same harness libraries.
Measure time-to-diagnose, eval regression catch rate, cost per successful task, percentage of runs with complete span trees, and weekly count of production failures promoted into CI. Incomplete trees are a first-class defect; treat them like missing logs on a payment service.
Document retention, PII redaction in spans, and who may view traces. Observability that leaks secrets is not a win. Publish a short contract: what is always logged, what is hashed, what requires break-glass, and how long raw payloads survive. That contract is what lets security approve deep tracing instead of blocking it.
Train on-call on one reconstructed incident per week using the inspector. If the team cannot narrate plan, tools, cost, and outcome from spans alone, instrumentation is still a backlog item, not a finished dashboard.
FAQ
- What is agent observability?
- Agent observability is the ability to inspect plans, model calls, tool invocations, handoffs, costs, and outcome quality for an agent run, with correlated traces and evals, not only uptime or token latency for a single LLM request.
- How is agent tracing different from logging chat messages?
- Chat logs show text. Traces show structured spans: which tool ran, with which schema version, how long it took, whether policy denied it, and how the planner reacted. That structure is what makes incidents diagnosable.
- What should teams measure for production agents?
- Measure success rate, tool error rate, loop detections, latency by span, cost per workflow, eval score trends, and human-gate wait times. Quality and safety regressions matter as much as p95 latency.