Verifying LLM Reasoning Fidelity in Multi-Step Planning

The demo looked flawless. The agent planned a complex refund workflow, cited the correct policy clauses, and executed every step in order. Then the first real-world ticket came in. The model skipped a prerequisite check for customer eligibility. It did not crash. It did not error out. It simply proceeded, generating a refund that violated a negative constraint. You are paying for compute on plans that look plausible but violate basic logical constraints.

You must decide whether to invest in formal verification of planning steps or rely on heuristic spot-checks. This choice determines if you can trust the system to execute complex workflows without human review. Most teams default to spot-checks because formal verification feels heavy. But spot-checks are a sampling problem. They miss the edge cases where the logic drifts.

You want reliable autonomous execution where every step follows from the previous one. In reality, the model often hallucinates intermediate states that do not exist in the data. This leads to cascading errors. The model believes it has a coupon code because it saw one in a previous turn, even if that code was expired or invalid for the current user.

The core failure is silent logic drift in planning. The model does not break loudly. It simply omits a constraint or misinterprets a dependency. The result is a plan that is syntactically valid but logically unsound. The JSON parses. The API calls succeed. The business logic fails.

How do you catch logic drift before execution?

You cannot catch it by reading the model's natural language explanation. The model is a persuasive storyteller, not a rigorous accountant. You must force the model to declare its assumptions in a structured format. This is the first gate.

Require the agent to output a JSON object containing assumptions, constraints_checked, and next_action. The assumptions field must list every fact the model is relying on that is not explicitly in the prompt. For example, "Customer has active subscription" or "Refund window is open."

This structured output is the input for your deterministic rule engine. The rule engine does not understand language. It checks facts. It queries your database. If the model assumes the subscription is active, the rule engine checks the billing system. If the status is cancelled, the plan is rejected. This is not a heuristic. It is a hard stop.

The outcome is a plan that has been verified against ground truth before any side effects occur. You are not trusting the model's confidence. You are trusting your database.

What is the cost of formal verification vs. heuristic checks?

Formal verification, such as using a theorem prover or SMT solver, is too slow for most real-time agent loops. It requires translating the entire planning context into logical predicates. This takes seconds to minutes. Your users are waiting.

Heuristic checks, like running a second LLM to critique the first, are faster but unreliable. The critic LLM has the same biases as the planner. If the planner hallucinates a state, the critic often hallucinates it too. They agree on the error.

The middle path is a deterministic rule engine. It is fast. It is cheap. It is deterministic. You define the rules once. You do not need to re-train or re-prompt. The rule engine checks the declared assumptions against your business logic.

This approach reduces the need for human review. You only escalate to humans when the rule engine finds a contradiction. This is a significant reduction in operator load. You are not reviewing every plan. You are reviewing only the failures.

Why does the model skip prerequisite checks?

Models are trained to complete tasks, not to verify preconditions. In training data, the "correct" answer is usually the final result. The intermediate steps are often implicit. The model learns to jump to the conclusion.

This is a training bias, not a bug. You cannot fix it by prompting harder. You must architect around it. The model will always try to take the shortest path to the goal. If the shortest path skips a check, it will skip it.

Your architecture must make the check non-optional. By requiring the assumptions field in the output schema, you force the model to acknowledge the prerequisite. If it does not list the assumption, the schema validation fails. The plan is rejected.

This turns a probabilistic behavior into a deterministic requirement. The model must prove it considered the constraint. This is the foundation of reliable autonomous execution.

How do you audit the divergence between plan and execution?

The plan is a promise. The execution is the reality. They often differ. The model plans to check inventory, but the execution engine skips the check because the API call was optional. Or the model plans to use a specific tool, but the router sends it to a different one.

You need an audit trail that logs both the declared plan and the actual execution path. This is not standard logging. Standard logs show what happened. An audit trail shows what was intended vs. what happened.

Create a run_id that ties the plan generation to the execution. Store the assumptions and constraints_checked from the plan phase. Then, as the agent executes, log every action taken. At the end of the run, compare the two.

If the plan said "Check inventory" but the execution log shows no inventory check, flag the run. This is a divergence. It means the system did not do what it said it would do. This is a critical failure mode. It breaks the trust model.

When should you defer full formal verification?

Defer it until your workflow involves financial transactions or legal compliance. For most operational tasks, the rule engine is sufficient. Formal verification is a hammer. You do not need a hammer to hang a picture.

The cost of formal verification is not just latency. It is complexity. You must maintain the logical predicates. You must translate natural language business rules into formal logic. This is a specialized skill. Your team likely does not have it.

Start with the rule engine. It covers 90% of the failure modes. It is easy to test. It is easy to debug. You can see exactly which rule failed.

If you move to formal verification later, you can do so incrementally. Start with the most critical constraints. But do not let the pursuit of perfect verification block your rollout. You need to ship. You need to learn from real-world data.

What does the audit trail prove?

It proves that the logic held. It is not just a log of actions. It is a record of intent vs. reality. This is crucial for post-incident analysis.

When an incident occurs, you can look at the audit trail. Did the model declare the assumption? Did the rule engine verify it? Did the execution follow the plan?

If the model did not declare the assumption, the failure is in the planning phase. You need to improve the prompt or the schema. If the rule engine verified it but the execution skipped it, the failure is in the execution engine. You need to fix the code.

This separation of concerns is key. It tells you where to fix the problem. Without the audit trail, you are guessing. With it, you are diagnosing.

The audit trail also provides a basis for trust. You can show stakeholders that the system is not just "trying." It is being verified. Every action is checked. This is a defensible position.

How do you build this without stalling the team?

Use the Diagnose, Model, Build, Harden method.

Diagnose: Pick one workflow that has failed silently. Look at the logs. What was the model supposed to do? What did it actually do? What was the missing check?

Model: Define the assumptions for that workflow. What facts must be true for the plan to be valid? Write these down. This is your rule set.

Build: Implement the structured output schema. Force the model to output these assumptions. Build the rule engine to check them against your database.

Harden: Add the audit comparator. Log the divergence. Set up alerts for divergences. Monitor the rate of failure.

This is a 2-week sprint. It is not a 6-month project. It is a focused effort on one workflow. Once you have it working, you can replicate it to other workflows.

The key is to start small. Do not try to verify every agent in your system. Pick the one that hurts the most. Fix that first.

What to do this week

Pick one agent workflow that has caused a silent failure.

1. Identify the specific prerequisite check that was skipped.
2. Write a JSON schema that requires the model to declare this assumption.
3. Write a simple script that checks this assumption against your database.
4. Run 10 test cases through this new pipeline.

If the script catches the failure, you have a working audit. If it does not, refine the schema. This is your first proof. It is not a full system. It is a single, verifiable check.

This is how you build trust. Not by promising reliability. By proving it, one check at a time.