Updated 2026-09-11
How to detect retrieval drift
Retrieval drift is the quiet decline in RAG quality after corpus, chunking, embedding, or filter changes. Dashboards may stay green while users get the wrong version of the truth.
RAG systems retrieve the wrong version of the truth long before anyone notices fluency dropped.
Symptoms of drift
Rising “I do not know” or wrong-version answers, citation mismatches, and support tickets that mention documents the system used to find.
Instrument the retrieval path
Log retrieved IDs, scores, filters, and index versions per query. Without that, you cannot tell generation failure from retrieval failure.
Canary and regression sets
Keep a fixed set of known queries with expected document IDs. Fail deploys when hit rates drop past a threshold.
Ownership of indexes
Treat index rebuilds like schema migrations: owners, changelogs, and rollback. Unowned indexes drift by default.
Related Knackline capability
After the diagnosis, see how Knackline hardens this class of system in production.
Open production rag capabilityRelated reports
Prompt and index change management: owners, versioning, and rollback contracts
Unowned prompt edits ship Friday and become Monday incidents
Production RAG for enterprise knowledge: why demos fail and what to measure instead
Assistants that answer well in demos often fail the first week of real tickets.
Eval gates in CI: shipping prompt and index changes like code
A Friday prompt edit that improves three demos can tank the failure set that actually runs in support.
FAQ
- Can embedding model upgrades cause drift?
- Yes. New embeddings without re-indexing or without eval gates change neighbor sets. Treat embedding upgrades as migrations.
- What is a practical alert?
- Alert when known-query hit@k drops, citation mismatch rises, or empty retrieval rate spikes for a traffic segment.
- How does Knackline approach this?
- We bind chunking and index contracts to owners, add retrieval eval gates, and give operators traces that show which version was retrieved.