Catching Semantic Drift Before Step Four

The agent does not crash. It does not throw an exception. It simply stops understanding the point.

You built a five-step workflow. Step one retrieves the data. Step two filters it. Step three formats it. Step four sends it. Step five archives it.

In step one, the intent is clear. The model knows exactly what to do. By step four, the context window is full of intermediate JSON blobs and tool outputs. The original instruction is buried under tokens. The model sees the recent data and acts on it, ignoring the initial constraint.

The result is plausible. The syntax is valid. The action executes. But the intent is gone.

This is semantic drift. It is not a bug in the code. It is a failure of attention.

How do you catch drift before it compounds

You need a checkpoint between steps, not just at the end.

End-to-end trajectory monitoring is too late. By the time you see the final output, step four has already executed. You have to undo the side effects. That is expensive.

Build a lightweight semantic consistency check. Compare the output of step N against the original goal vector. Not against step N-1. Against the start.

If the divergence exceeds a threshold, halt the workflow. Do not retry. Do not log and move on. Stop.

This check is cheap. It is a vector comparison, not a full inference. It adds milliseconds, not seconds.

The proof is simple. Run your existing workflows. Inject a subtle constraint in step one. Watch where it breaks. You will see it break at step three or four. That is your drift point.

What is the actual failure mechanism

Context rot is the primary driver.

As the context window fills, the model’s attention to early tokens dilutes. The initial instruction becomes noise. The recent tokens become signal.

This is not a linear decline. It is a cliff. At a certain context length, the early constraints become invisible. The model acts on the most recent data, assuming it is the current goal.

Lost-in-the-middle effects make this worse. Early constraints are often ignored in favor of recent context. The model does not "forget" in a human sense. It prioritizes.

Cross-agent contamination adds another layer. If agent A drifts, its output becomes agent B’s input. Agent B treats the corrupted data as ground truth. The error propagates.

This is not a single point of failure. It is a cascade.

Why end-to-end monitoring is not enough

You can monitor the entire trajectory. You can log every step. You can alert on anomalies.

But you are still reacting.

By the time the alert fires, step four has executed. The email is sent. The refund is processed. The data is migrated.

Undoing that is harder than preventing it.

End-to-end monitoring tells you what happened. It does not stop it.

Intermediate checks stop it. They halt the workflow before the damage is done.

The tradeoff is latency. But the latency is small. The cost of a wrong action is large.

You are not choosing between speed and safety. You are choosing between cheap prevention and expensive cleanup.

When should you halt the workflow

Not on every drift. Only on significant drift.

If the divergence is small, the workflow can continue. The model is still on track.

If the divergence is large, halt. Do not guess. Do not try to correct it automatically.

Automatic correction is risky. The model might "fix" the drift by introducing a new error.

Halt for human review. A human can see the context. They can decide if the drift is acceptable or if the workflow should be stopped.

This reduces the load on operators. You are not reviewing every step. You are only reviewing the ones that drift.

The goal is not zero drift. The goal is zero silent drift.

How to measure intent fidelity

You need a baseline.

The original instruction is the baseline. Convert it to a vector. This is your goal vector.

At each step, convert the output to a vector. Compare it to the goal vector.

Use cosine similarity. If the similarity drops below a threshold, flag it.

The threshold is not fixed. It depends on the workflow. A financial workflow needs a higher threshold than a data retrieval workflow.

Start with a conservative threshold. Tighten it as you learn what normal drift looks like.

Log the similarity score for every step. This gives you a trend. You can see if drift is increasing over time.

This is not a one-time check. It is a continuous metric.

What to defer until you have proof

Do not build a full semantic validator yet.

Do not build a custom model for drift detection.

Do not build a complex rule engine.

Start with the vector comparison. It is simple. It is fast. It works.

Once you have data on where drift happens, you can add complexity.

Maybe you need a secondary check for specific step types. Maybe you need a human-in-the-loop for high-risk actions.

But do not build it until you know you need it.

The first version is a check. The second version is a system.

Build the check first. Prove it catches drift. Then build the system.

Why this matters for release confidence

You cannot release an autonomous workflow if you do not know where it will fail.

Semantic drift is a known failure mode. It is not random. It is predictable.

If you can predict where drift happens, you can design around it.

You can add checkpoints at the drift points. You can shorten the context window. You can re-inject the original instruction.

This gives you confidence. You know the system will not fail silently.

You know that if it does drift, it will halt. You know that a human will review it.

This is release confidence. It is not about perfection. It is about predictability.

The mermaid diagram below shows the flow.

The diagram shows the check at each step. If the similarity is high, the workflow continues. If it is low, it halts.

This is not a complex system. It is a simple loop.

Diagnose, Model, Build, Harden

This is the method.

Diagnose: Find where your current workflows drift. Run them. Log the outputs. Compare them to the goal.

Model: Understand the mechanism. Is it context rot? Is it lost-in-the-middle? Is it cross-agent contamination?

Build: Add the vector check. Set the threshold. Add the halt logic.

Harden: Tighten the threshold. Add human review. Add logging. Add alerting.

This is not a one-time project. It is a continuous process.

You will find new drift points. You will adjust the threshold. You will improve the check.

The goal is not a perfect system. The goal is an operable system.

What to do this week

Run your most critical workflow.

Add a vector comparison at step two.

Log the similarity score.

Run it ten times.

Look at the scores.

If they are stable, you are safe.

If they are dropping, you have drift.

Fix the drift.

Then move to step three.

This is how you build trust. Not with a big announcement. With a small, verifiable check.

Start small. Prove it works. Then scale.

The drift is real. The fix is simple. Do it now.