Eval gates in CI:
shipping prompt and index changes like code

Why prompt review is not a release process

Engineering already refuses to merge untested code. LLM systems often still ship prompt text, temperature, and index rebuilds on trust. The change looks small. The blast radius is not.

A new system prompt can improve three stakeholder demos while reducing retrieval discipline, weakening refusals, or encouraging tool calls the harness never intended. An index rebuild can shift chunk boundaries and silently destroy recall on the failure set.

If the only gate is "someone liked the new answer," you do not have LLMOps. You have taste-driven deploy. Ground quality contracts in production RAG and feed live misses into suites via agent observability.

Operator scenario: a Friday afternoon prompt tweak makes the bot sound sharper on the exec walkthrough. Monday, support sees a spike of answers that cite the wrong policy version. Nobody can point to a failing test because there was no suite. The revert is a git blame debate instead of a red CI job.

Failure modes without eval gates

Demo overfitting: suites are five happy questions that always pass. Real tickets involve messy phrasing, missing entities, and adversarial users. Those never enter the suite, so CI stays green while production burns.

Coupled scores: one "answer quality" number hides a retrieval miss under fluent prose. Judges reward tone; customers need the right source. Separating dimensions is how you catch that.

Index blindness: embedding model upgrades ship without re-scoring the corpus-specific set. Average benchmarks look fine; your failure set collapses because chunking and synonyms changed.

Flaky judges: LLM-as-judge runs without seeds, rubrics, or baselines, so CI becomes noise and teams ignore it. Once ignored, the gate is dead even if the YAML still runs.

Shadow deploys: someone edits the prompt in a console or feature flag outside the repo. CI never sees it. The next merge looks unrelated when production regresses.

Cost regressions ignored: a prompt change doubles tool calls or pushes traffic to a frontier model. Quality holds; the budget does not. Gates that omit cost will ship expensive "improvements." Tie this to cost-aware model routing budgets.

The control model: artifacts, suites, thresholds

Version prompts, model IDs, index builds, and tool schemas. Every pull request that touches them runs the suite. Store prompts as files with owners, not as anonymous strings in a dashboard.

Structure cases: input, expected sources or actions, allowed refusals, and risk tags. Score retrieval (recall, citation correctness) separately from generation (faithfulness, format) and from safety (injection, policy). Add a cost dimension: tokens, tool calls, or dollars per case against a budget.

Set thresholds against main. Absolute scores matter less than regressions. Allow small noise bands; block clear drops. Publish the threshold policy in the repo so waivers are exceptional, not cultural.

Operable steps: pin a baseline commit, run the suite on the PR, emit a per-dimension delta report, fail on any dimension past budget, and attach the report to the PR so review is evidence-based.

Building suites that catch real regressions

Mine support tickets, denied tool calls, and bad citations. Each week, add a handful of new cases from production. Retire stale cases when the product no longer supports that path. Prefer cases with linked span trees so authors know which hop or tool failed.

Use deterministic checks first: exact IDs, required citation URIs, JSON schema, forbidden tool names. Add LLM-as-judge only where human rubrics are clear and calibrated. Lock judge model, temperature, and rubric version. Re-run calibration when any of those change.

Keep a smoke suite fast enough for every PR and a nightly deep suite for expensive re-rankers or large corpora. Speed matters; ignored CI helps nobody. Put index rebuilds behind the deep suite before the new index becomes the default for live traffic.

Separate safety and refusal cases from helpfulness cases so a more chatty prompt cannot trade refusals for style points. Include "ignore previous instructions" and "search as admin" style prompts even for non-agent RAG, because prompts still drift toward compliance theater without them.

Rollout sequence and metrics

Start with one workflow and twenty failure-derived cases. Wire the gate to the repo that owns prompts and index jobs. Expand coverage before expanding autonomy. Do not unblock write tools or multi-hop agents until the smoke suite covers their failure modes.

Measure gate catch rate, false-positive rate, time-to-fix failures into the suite, median PR eval runtime, and production incidents with no matching case. The last metric is your coverage debt. Track waivers: count, age, and owner. Waivers without expiry are how gates die.

Document who may change thresholds and how to waive a gate with a written risk accept and an expiry date. Require a follow-up case when a waiver ships. Eval gates turn LLM changes into engineering changes teams already know how to review. When traces feed the suite weekly, observability and CI become one loop instead of two disconnected dashboards.

Treat index jobs like migrations: the rebuild PR carries the deep-suite report, a rollback pointer to the prior index build ID, and an owner who watches recall on the failure set for the first day after cutover.