Durable Agent State Sidecar:
Surviving Pod Eviction in Long-Running Workflows

The demo worked because the session stayed in RAM. In production, a pod restart during a multi-step workflow wipes the intermediate state. The agent restarts from step one, re-calling APIs and potentially corrupting downstream data. You feel this pain when a long-running task fails at step four of ten, and the system cannot tell you why. It simply starts over.

The desired outcome is a workflow that resumes exactly where it left off after a crash. The actual outcome with in-memory stores is a hallucination loop where the agent repeats calls because it cannot recall completed steps. This is not a minor inconvenience. It is a silent data corruption risk.

You must decide between an in-memory session store and an external durable state sidecar. This is not a feature request; it is an architectural commitment to transactional guarantees. You need to prove that atomic state transitions prevent silent data corruption before scaling volume.

How do you prevent the stale read race condition?

The core failure mode is the "stale read." When a long-running step updates state asynchronously, the next step may read the pre-update value. This causes the agent to act on outdated context, leading to duplicate side effects. For example, an agent might send an email twice because it read the "sent" flag before the write committed.

The fix is optimistic locking. Every state object carries a version number. When the agent attempts to write a new checkpoint, it includes the version it read. The database rejects the write if the version has changed. This forces the agent to re-read the state and reconcile before proceeding.

This mechanism eliminates the need for complex in-memory locking logic. It shifts the consistency burden to the storage layer. The agent becomes a stateless executor that trusts the external store as the single source of truth. If a conflict occurs, the agent aborts the current step and retries with fresh data.

When should you externalize state to a sidecar?

You should externalize state the moment your workflow exceeds a single LLM call. If the agent performs a sequence of actions (fetch data, transform, send request, verify response), each intermediate result is a candidate for loss. In-memory storage is only safe for trivial, single-turn interactions.

The trigger for building a sidecar is the introduction of external side effects. If the agent can write to a database, send an API call, or modify a file, the state must be durable. Without durability, a network partition or pod eviction creates a split-brain scenario. The agent thinks it completed a step, but the external system never received the confirmation.

Start with a simple key-value store if your state is small. As the complexity grows, move to a relational database or a document store with transactional support. The key is not the technology, but the guarantee. The store must support atomic commits. If it cannot guarantee that a write either fully succeeds or fully fails, it is not a durable state store.

What is the cost of silent data corruption?

The cost is not just wasted tokens. It is the trust erosion that occurs when operators cannot reconstruct what happened. When an agent corrupts downstream data, the incident response time explodes. Engineers spend hours tracing logs to find where the state diverged from reality.

In a high-volume system, this leads to a halt in operations. Operators disable the agent to prevent further damage. The time-to-halt metric becomes critical. If you cannot quickly identify and isolate the corrupted state, you are forced to stop the entire pipeline.

The financial impact includes redundant API calls. If the agent repeats steps because it lost context, you pay for every token twice. More importantly, you pay for the manual labor required to fix the data. The sidecar pattern reduces this cost by providing an audit trail of every state transition.

Why does versioning matter for schema evolution?

Lack of versioning on state objects means a schema change breaks resumption of old sessions. If you add a new field to your state object, existing sessions in the store do not have that field. When the agent tries to resume, it may crash or behave unpredictably.

The solution is to embed a version number in every state object. On resume, the agent checks the version. If it matches the current handler, the agent proceeds. If it does not match, the agent triggers a migration function. This function transforms the old state into the new format.

This approach allows you to evolve your agent logic without breaking active workflows. You can deploy new versions of the agent that understand both old and new state formats. The migration logic is isolated and testable. It prevents the "ghost session" problem where old data causes new code to fail.

How do you handle pod eviction during inference?

Pod eviction during a long LLM inference call loses the partial response buffer. The agent is halfway through a step when the pod dies. The partial response is gone. The agent restarts and begins the step from scratch.

To mitigate this, you must checkpoint before the LLM call and after. The "before" checkpoint records the input context. The "after" checkpoint records the output. If the pod dies during the call, the agent resumes from the "before" checkpoint. It re-executes the LLM call.

This is not free. You pay for the redundant LLM call. However, it is cheaper than the cost of data corruption. The alternative is to lose the entire workflow. By checkpointing around the expensive operation, you limit the blast radius of a failure. You accept the cost of retrying the LLM call to guarantee the integrity of the workflow.

What controls prevent split-brain divergence?

A network partition between the agent and the memory store causes a split-brain state divergence. The agent writes a checkpoint, but the write never reaches the store. The agent assumes the state is updated. It proceeds to the next step. When it tries to read the state, it gets the old value.

The control here is idempotency. Every state transition must be idempotent. If the agent retries a step, the result must be the same. This prevents duplicate side effects. For example, if the agent sends an email, it must check if the email was already sent before sending it again.

You also need a heartbeat mechanism. The agent periodically pings the state store. If the ping fails, the agent assumes the store is unavailable. It pauses execution and waits for the store to recover. This prevents the agent from proceeding with stale state during a network outage.

How do you prove the system is safe to scale?

You prove safety by simulating failures. You inject pod evictions, network partitions, and store outages into your test environment. You verify that the agent resumes correctly in each case. You check that no data is corrupted and no side effects are duplicated.

The proof is not in the code review. It is in the chaos test. You need a test harness that can kill pods mid-execution. You need a way to verify the final state of the workflow. If the workflow completes successfully after a simulated crash, you have proof of durability.

This proof unlocks more autonomy. You can allow the agent to run longer, more complex workflows. You can increase the volume of concurrent agents. The confidence comes from knowing that the system will fail safely. It will not corrupt data. It will not lose state. It will resume and continue.

The path to a reliable agent system is a practitioner method: Diagnose, Model, Build, Harden. Diagnose the failure modes in your current system. Identify where state is lost. Model the state transitions. Define what constitutes a valid state. Build the sidecar. Implement the checkpoint logic. Harden the system by injecting failures.

This is not a one-time project. It is an ongoing discipline. As your agent logic evolves, you must update the state model. You must test the new transitions. You must verify that the sidecar can handle the new schema.

The goal is an operable system. One that you can trust to run unattended. One that you can debug when it fails. One that you can scale without fear of silent corruption.

This week, pick one long-running workflow. Add a checkpoint before and after each external API call. Simulate a pod kill. Verify that the workflow resumes correctly. This single check will reveal the gaps in your current state management. It will give you the confidence to build the full sidecar.