Agent Transactional Rollback Semantics

The demo looked flawless until the third step failed. Now your ops team spends four hours manually reconciling invoices and inventory records that are out of sync. You built an agent to handle end-to-end task completion. You wanted zero manual intervention. Instead, you got partial executions that leave external systems in inconsistent states. The agent did its job, then it broke the world.

You must decide whether to build a global transaction log with compensating actions or rely on per-step idempotent retries. This choice determines if your team scales or drowns in manual cleanup tickets. Most teams start with retries because they are easy. Retries work for pure computation. They fail for anything that touches the outside world.

What is the compensation gap

The core failure is the "compensation gap." When a step fails, the undo action for previous steps may not exist or may fail itself. There is no unified path to restore consistency. The agent moves forward, hits a wall, and stops. The state before the wall remains. The state after the wall never happened. The system is stuck in a limbo that no single component owns.

This is not a bug in the code. It is a gap in the architecture. You treated the workflow as a sequence of independent tasks. You did not treat it as a single unit of work. The agent cannot know that step two depends on step one being reversible. It only knows that step two returned an error.

The result is manual remediation. Your engineers become the rollback mechanism. They read the logs, identify which steps succeeded, and manually call the undo APIs. This is slow, error-prone, and scales linearly with incident count. You are paying for autonomy with your on-call time.

How do you choose between logs and retries

Build a global transaction log with compensating actions first. Defer per-step idempotent retries for internal, stateless computations. The proof that unlocks more autonomy is the ability to reverse a completed multi-step workflow without human intervention.

Retries are cheap. They are also dangerous when side effects exist. If you retry a "charge card" step, you might charge the card twice. If you retry a "send email" step, the customer gets two emails. Idempotency keys help, but they require every downstream system to support them. You cannot force your vendors to change their APIs.

A global log records every step and its corresponding undo action. When a failure occurs, the orchestrator walks the log backward. It executes the undo actions in reverse order. This is the Saga pattern. It does not require atomicity across systems. It requires coordination. You build the coordination layer once. You reuse it for every workflow.

Why partial execution breaks trust

Trust is binary. If the agent fails, you want to know it failed cleanly. Partial execution is the worst outcome. It looks like success until someone checks the database. The invoice is created. The inventory is reserved. The payment fails. The agent logs an error. The user sees a failure message. But the inventory is still reserved. The invoice is still in the system.

This creates a hidden debt. Every partial execution is a ticket waiting to be filed. Your support team sees the user complaint. They dig into the logs. They find the inconsistency. They spend twenty minutes fixing it. Multiply that by a hundred incidents a week. Your team is not building new features. They are cleaning up the agent's mess.

The cost of waiting is high. The longer you run with partial executions, the more inconsistent your data becomes. You start to distrust the agent. You add manual approvals for every step. You lose the autonomy you wanted. You end up with a system that is slower than the manual process it replaced.

What happens when the undo action fails

The compensating action is not magic. It is an API call. It can fail. It can time out. It can hit a rate limit. If the undo action fails, the workflow is stranded. You have a half-completed transaction with no way to finish the rollback.

This is the critical failure mode. You must handle it explicitly. Do not assume the undo will work. Treat a failed compensation as a critical incident. The system must halt. It must alert a human. It must not attempt further steps.

This prevents silent data corruption. If the undo fails, the data is in an unknown state. Only a human can determine the correct action. They might need to contact the vendor. They might need to manually update the database. The system's job is to stop and flag the problem. It is not to guess.

How to handle timeouts and stale snapshots

Timeouts are common in distributed systems. A long-running step might take longer than the orchestrator expects. The orchestrator assumes a failure. It starts the rollback. But the step was actually still running. The undo action executes against a stale snapshot. It corrupts data that changed meanwhile.

This is a race condition. You cannot prevent it entirely. You can mitigate it. Use idempotency keys for the undo actions. If the undo action is called twice, it should have the same effect. If the step was actually successful, the undo action should detect that and skip the reversal.

You also need a timeout policy. If a step takes too long, the orchestrator should not immediately roll back. It should wait. It should check the status of the step. Only if the step is confirmed dead should it proceed with the rollback. This adds latency, but it prevents corruption. Latency is cheaper than data loss.

Why the orchestrator crash is a build blocker

The orchestrator is the brain of the workflow. If it crashes mid-rollback, the context is lost. The undo sequence is incomplete. The system is left in a partial state. The next orchestrator instance starts up. It does not know what happened. It does not know which steps were undone. It does not know which steps remain.

This is a build blocker. You must persist the rollback state. The transaction log must record the progress of the undo sequence. If the orchestrator crashes, the next instance reads the log. It sees that the rollback is in progress. It resumes from the last completed undo step.

This requires durable storage. You cannot keep the rollback state in memory. You must write it to a database. You must ensure that the write is atomic with the step execution. This is complex. It is also necessary. Without it, you are gambling on the reliability of your infrastructure.

What to do this week

Run the rollback logic in shadow mode. Do not execute real compensations. Log what would happen. Verify that the undo sequence correctly identifies and reverses all prior steps. Check for gaps. Check for stale snapshots. Check for missing undo actions.

This is your proof. If the shadow mode reveals a gap, you know where to build. If it reveals a stale snapshot, you know where to add the idempotency check. You are not guessing. You are testing. You are building confidence.

The method is simple. Diagnose the current failure modes. Model the desired state. Build the transaction log. Harden the rollback mechanism with idempotency and durable storage. Do this before you add more workflows. Do this before you scale. The cost of fixing it later is higher than the cost of fixing it now.

Start with one workflow. Pick the one with the most external side effects. Build the transaction log for it. Run it in shadow mode. Fix the gaps. Then move to the next workflow. This is how you build trust. Not with promises. With proofs.