The Sands of Time:
OB Rag Build Decisions

The demo impressed the execs. The slides were clean, the latency was low, and the answers looked right. But the quiet cost is the unowned maintenance burden that lands on your team after the hype fades. You are now responsible for a system that writes to production without a clear path to stop it if it breaks.

You must decide whether to build a shadow-write pipeline now or defer full write access until you can prove the system handles edge cases safely. This decision determines if you own the incident response or just the initial launch.

The desired outcome is a RAG system that updates the knowledge base with high confidence. The actual outcome is often a patchwork of manual fixes because the system lacks a defined halt path when retrieval quality drops.

The failure mode is silent degradation. The system continues to write low-quality data because no metric triggers a stop, and the team only notices when users report broken answers.

How do you prove the write path is safe?

You do not prove it by running it. You prove it by not running it. The first build decision is to construct a shadow-write loop. The system proposes changes to the knowledge base, but it does not commit them. You compare these proposed writes against a gold standard set of known-correct data.

This is not a test environment. It is a production-parallel environment that processes live traffic but discards the output. The mechanism is simple: intercept the write call, log the proposed payload, and score it against your baseline. If the match rate exceeds your threshold, you have evidence. If it does not, you have a bug to fix before you ever touch the live database.

This phase protects you from the "silent degradation" failure mode. Without it, you are flying blind. With it, you have a dashboard that tells you exactly when the model starts hallucinating updates. You can see the drift before it corrupts your index.

What happens when retrieval latency spikes?

Retrieval latency spikes cause write timeouts. When the vector store takes too long to respond, the write operation times out. In a naive implementation, this leaves the database in a partial state. Some chunks are written, others are not. The transaction is incomplete, but the system assumes it succeeded.

You must build idempotency into the write path. Every write operation needs a unique run_id. If a timeout occurs, the retry mechanism checks for the run_id. If it exists, the system knows the write already happened or is in progress. It does not create a duplicate. It does not leave an orphaned record.

This control prevents the "indexing jobs collide" failure mode. Duplicates confuse subsequent queries. They dilute the relevance scores and introduce noise into the retrieval context. By enforcing idempotency, you ensure that a latency spike degrades performance but not data integrity. The system slows down, but it does not break.

When should you enable limited writes?

You enable limited writes only when the shadow phase has proven stability. This is not a date on the calendar. It is a metric. You need a match rate against your gold standard that exceeds your threshold for a continuous period. Two weeks of stable traffic is a reasonable baseline. One week is not enough to catch seasonal drift or slow decay.

The mechanism is a canary deployment. You route a small percentage of traffic, perhaps 5 percent, to the live write path. The rest continues to shadow. You monitor the canary group for discrepancies. If the live writes start to diverge from the shadow predictions, you halt the canary. You revert to shadow-only mode.

This protects you from the "query volume surges" failure mode. If the system is overwhelmed, the canary group will show increased error rates or latency spikes before the full population is affected. You have a small blast radius. You can fix the issue without rolling back a week of corrupted data.

Why do batch jobs leave orphaned records?

Batch processing fails mid-run. The job starts, processes a thousand documents, and crashes on the thousand and first. The first thousand are written to the database. The rest are lost. There is no cleanup job to remove the partial batch. These orphaned records sit in the index, waiting to be retrieved by a future query.

You must treat batch jobs as transactions. The entire batch is either committed or rolled back. If the job fails, you need a cleanup routine that identifies records associated with the failed run_id and removes them. This is not an afterthought. It is a core part of the build.

This control prevents the "source document updates are not detected" failure mode. If you do not clean up partial batches, stale data from a failed run can overwrite fresh data in a subsequent run. The system becomes a graveyard of half-finished updates. Users see answers that are a mix of old and new information.

How do you detect stale data overwrites?

Source document updates are not detected by the RAG pipeline by default. The system sees a new version of a document and writes it to the index. It does not know that an older version exists. It does not check if the older version is still relevant. It just adds the new chunk to the pool.

You need a versioning check in the write path. Before committing a new chunk, the system queries the index for existing chunks with the same source_id. If an older version exists, it is marked for deletion. The new version is then written. This ensures that the index always contains the latest state of the source document.

This control prevents the "stale data being written over fresh data" failure mode. Without it, the index accumulates duplicates. Retrieval scores become unstable. The system retrieves the same fact from multiple versions, confusing the LLM. The answers become inconsistent and unreliable.

What is the cost of silent degradation?

The cost is not just technical. It is operational. When the system silently degrades, the team spends hours debugging a problem that is not a bug in the code. It is a bug in the data. The model is working as intended. The data is wrong.

You cannot debug data quality issues by looking at logs. You need a feedback loop. You need a way to score the quality of the retrieved chunks. You need a metric that tells you when the retrieval quality drops below an acceptable level. This metric triggers a halt. It stops the writes. It alerts the team.

This control prevents the "team only notices when users report broken answers" failure mode. You catch the issue before it reaches the user. You fix it before it becomes a support ticket. You protect the trust of the users who rely on the system.

How do you structure the build sequence?

The build sequence is not linear. It is iterative. You build the shadow pipeline first. You run it for two weeks. You fix the bugs you find. You build the idempotency controls. You test the retry logic. You build the versioning check. You test the cleanup routine.

You do not build the live write path until the shadow pipeline is stable. You do not enable the canary until the idempotency controls are tested. You do not enable full writes until the versioning check is verified. This is the Diagnose, Model, Build, Harden method.

Diagnose the failure modes. Model the control logic. Build the shadow pipeline. Harden the live write path. This sequence ensures that you are not building on sand. You are building on a foundation of proven stability.

What should you do this week?

You should not enable any writes. You should build the shadow pipeline. You should create a gold standard set of 100 known-correct updates. You should run the pipeline against this set and log the discrepancies.

You should identify the top three failure modes in your logs. Latency spikes, duplicate writes, stale data. You should build the control for the most frequent failure mode. You should test it in the shadow environment.

You should not wait for the first incident. You should not wait for the user report. You should prove the system is safe before you give it the power to break. The sands of time are shifting. The demo is over. The work is just beginning.