Production RAG applications
Retrieval quality looks excellent on demo questions and collapses when users ask messy, multi-hop, or stale-data questions. Hallucinations often start as retrieval misses, not model creativity.
Why common approaches fail
Teams optimize embedding similarity on clean corpora and skip chunking contracts, citation paths, ACL-aware retrieval, freshness, and evals that mirror production query distribution.
Activation insight
Retrieval quality can look excellent until users stop asking clean demo questions.
How Knackline solves it
We treat RAG as a production data path: chunking and index ownership, hybrid search, grounded citations, tenant isolation, and evaluation gates before raising autonomy.
System components
Chunking and index contracts
Owners, versioning, and rollback for prompts and indexes.
Hybrid retrieval
Sparse plus dense search with filters that respect ACLs.
Citation and groundedness
Answers that point to sources operators can audit.
RAG evaluation sets
Failure-derived cases, not only happy-path FAQ pairs.
Operational hardening
Drift detection on retrieval quality, multi-tenant ACL isolation that survives multi-hop retrieval, and change management when documents or models move.
Evidence from reports
Production RAG for enterprise knowledge: why demos fail and what to measure instead
Assistants that answer well in demos often fail the first week of real tickets.
Tenant isolation for multi-tenant RAG: ACLs that survive multi-hop retrieval
Chunk filters applied once still leak across hops and summaries
GraphRAG when vector search is not enough: relational questions need structure
Nearest neighbors find similar paragraphs; they do not reliably answer who owns what across a changing org graph.
Agentic RAG ACLs: why single-query filters fail on multi-hop retrieval
Each hop looks authorized until the agent stitches three fragments into a document nobody should see whole.
FAQ
- When is vector search not enough?
- When relationships, ACL boundaries, or freshness matter more than nearest-neighbor similarity. GraphRAG and hybrid retrieval exist for those failure modes.
- How do you evaluate RAG?
- With sets drawn from real user failures: wrong version retrieved, missing citations, ACL leaks, and stale indexes, not only demo Q&A accuracy.