Production RAG for enterprise knowledge:
why demos fail and what to measure instead

Why RAG demos lie about production readiness

Demo corpora are curated. They are short, consistent, and free of contradictory policies. Enterprise knowledge is the opposite: versioned PDFs, overlapping wikis, stale Confluence pages, and tickets that quote obsolete answers.

When teams judge quality only by a handful of happy-path questions, they optimize for fluency. Fluency hides retrieval misses. The assistant still sounds confident when it cites the wrong source or invents a bridge between two outdated policies.

Production readiness starts when you can explain a wrong answer. If nobody can inspect which chunks were retrieved, ranked, and filtered, you do not have an operable system. You have a conversation toy.

Failure modes that show up after the pilot

Stale chunks: documents update, embeddings do not. Teams keep publishing while ingestion freezes, so the assistant answers from last quarter's truth.

Conflicting sources: two policies disagree. Without ranking rules, authority tags, or recency bias you control, the model picks whichever chunk is nearest in vector space.

Opaque misses: the right document exists, but chunk boundaries split the answer across windows, or metadata filters drop the only relevant page.

Unverifiable answers: operators cannot open the cited passage. Trust collapses even when the answer is occasionally correct.

Chunking contracts that survive messy corpora

Chunking is not a preprocessing detail. It is a contract between writers, the indexer, and the answer path. Define maximum size, overlap, heading inheritance, and how tables or lists are preserved.

Prefer structure-aware splits over blind token windows when the corpus has headings. Carry section titles into chunk metadata so ranking can prefer complete procedural steps.

Version every document. Store source URI, updatedAt, owner, and sensitivity. When a source updates, invalidate or re-embed affected chunks instead of hoping a nightly job catches up.

Hybrid retrieval under a latency budget

Dense retrieval alone struggles with IDs, error codes, and exact policy names. Lexical search alone struggles with paraphrase. Hybrid pipelines combine both, then re-rank.

Decide the budget first. A support assistant may tolerate 800ms of retrieval. An internal analyst tool may not. That number decides how many candidates you fetch and whether a cross-encoder re-ranker is affordable.

Log recall@k on your failure set every deploy. If you only watch answer ratings, you will never know whether quality dropped because of generation or because retrieval stopped finding the right page.

Eval harnesses built from real tickets

Collect questions where the assistant already failed. Label the expected source, the acceptable answer shape, and whether a refusal is correct.

Score retrieval separately from generation. A beautiful answer with the wrong citation is still a failure. A grounded refusal on an out-of-corpus question is a success.

Gate releases on eval deltas. Prompt tweaks that improve a handful of demos while tanking the failure set are regressions, even if stakeholders liked the new tone.

What operable handoff looks like

Operators need a retrieval inspector: query, candidates, filters, scores, and final context window. Without it, every incident becomes a prompt debate.

Human gates matter when answers affect customers or compliance. Draft freely, publish with review where risk is high.

Document the contracts. The next team should know how to add a corpus, how to mark authority, and how to extend the eval set without restarting the project.