The Cost Ledger:
Proving Savings Before You Route

The demo looked cheap. Then the invoice arrived. You are paying premium rates for simple tasks because the system defaults to the most expensive option. This is the quiet cost of static configuration.

You must decide what to prove in shadow mode before granting any write access. The decision is not about buying a new router. It is about defining the specific cost-per-task threshold that justifies the complexity of dynamic selection.

You want lower spend per unit of work. You are currently paying for maximum capability on every request. The gap is the margin you are losing on low-complexity tasks.

The failure mode is "capability inflation." Teams assume every task needs the largest model. They do not measure the actual complexity of the input. They pay for a sledgehammer when a screwdriver would work.

What is the actual cost of a simple task?

You cannot optimize what you cannot see. Most teams look at the monthly total and feel a vague anxiety, but they cannot allocate that cost to specific features or workflows. The first step is not routing; it is accounting.

You need a cost ledger. Every request must log the model used, the token count, and the estimated cost. This data must be queryable by task type. If you cannot answer "how much did we spend on summarization last month?" you are not ready to route.

This ledger becomes your baseline. It is the control against which you measure any new routing logic. Without it, every claim of savings is anecdotal. With it, savings are a line item in your P&L.

How do you define the threshold for dynamic selection?

A router is only worth building if the savings exceed the engineering overhead. That overhead is real: latency added by the selection step, complexity in debugging, and the risk of misclassification.

Define a hard number. For example, if a small model saves $0.005 per request but adds 200ms of latency, is that trade-off worth it? For a background batch job, yes. For a user-facing chat, maybe not. The threshold depends on the user's tolerance for delay versus your tolerance for margin erosion.

Start with a conservative threshold. You want to be sure the savings are real before you add complexity. If the gap between the large and small model cost is less than your engineering time cost per feature, do not build the router.

What must be proven in shadow mode?

Shadow mode is where you run the dynamic selection logic without using the result. You log what the router would have chosen. You compare that choice against the actual model used.

You are looking for two specific proofs. First, accuracy parity. Does the smaller model produce acceptable output for the tasks the router classifies as "low complexity"? You need a metric. It might be a human review of a sample, or an automated eval score. If the small model fails 20% of the time, the cost savings are offset by the cost of rework.

Second, volume distribution. How many tasks are actually "low complexity"? If only 10% of traffic qualifies, the total savings might be negligible. You need to see that the small model can handle a significant portion of the load before you commit to the infrastructure.

Why does latency mask the cost problem?

Users perceive speed, not price. If a response takes 2 seconds instead of 1 second, they might complain about slowness. They do not see that you saved them money by using a cheaper model.

This creates a feedback loop where engineers optimize for latency, often by choosing the fastest, most expensive model. They do not see the cost impact because it is not in the user's face.

To break this, you must decouple latency from cost in your monitoring. Track them separately. If a smaller model is slower but cheaper, you need a policy decision: do you value speed or margin? The answer depends on the product context. For a real-time trading tool, speed wins. For a document processing pipeline, margin wins.

How does vendor lock-in affect the build decision?

If you build custom routing logic tied to a specific provider's API, you are creating a migration cost. If you switch providers, you have to rewrite the routing logic.

This is a significant risk. It reduces your negotiating power and increases the cost of switching. To mitigate this, abstract the routing logic from the provider. The router should output a "model class" (e.g., "small," "medium," "large") rather than a specific model ID. The mapping from class to provider-specific model should be a configuration file, not hard-coded logic.

This abstraction adds a layer of indirection, but it is worth it. It allows you to swap providers without rewriting the core routing logic. It also allows you to test open-source models against commercial ones in the same pipeline.

What is the role of open-source models in the ledger?

Open-source models are not just a cost-saving measure; they are a capability option. They allow you to run tasks on your own infrastructure, which can be cheaper at scale and offers more control over data privacy.

However, they come with infrastructure costs. You need to account for GPU hours, maintenance, and scaling. The cost ledger must include these overheads. A model that is free per token but requires a dedicated GPU cluster may be more expensive than a commercial API for low-volume tasks.

The decision to use open-source models should be based on the total cost of ownership. For high-volume, low-complexity tasks, self-hosting often wins. For sporadic, high-complexity tasks, API pricing is usually lower. The ledger should reflect this trade-off.

How do you roll out dynamic selection safely?

You do not flip a switch. You roll out in stages.

Shadow: Log costs for all tasks using both small and large models. Prove the small model handles 60% of tasks at 80% accuracy.

Limited: Enable dynamic selection for internal tools only. Prove the cost savings exceed the engineering overhead.

Full: Roll out to customer-facing features. Monitor for regressions in quality and latency.

At each stage, you have a halt path. If the cost savings disappear, or if quality drops, you revert to the static configuration. The cost ledger is your tripwire. If the spend per task increases, you stop.

This staged approach reduces risk. It allows you to validate the economics before you expose the complexity to your users. It also gives you time to tune the thresholds.

What is the practitioner method for this build?

The method is Diagnose, Model, Build, Harden.

Diagnose: Look at your current spend. Identify the tasks that are high-volume and low-complexity. Calculate the potential savings.

Model: Build the cost ledger. Run shadow mode. Measure the accuracy and latency of the smaller model. Define the threshold.

Build: Implement the router. Abstract the model selection. Connect the ledger to your monitoring.

Harden: Roll out in stages. Monitor for regressions. Tune the thresholds. Document the halt paths.

This method is not about buying a tool. It is about building a capability. The tool is the router. The capability is the ability to make cost-effective decisions about model selection.

The goal is not to use the smallest model possible. The goal is to use the right model for the task. The cost ledger is the mechanism that ensures you are making that decision based on data, not intuition.

What should you do this week?

Pick one high-volume, low-complexity task in your system. Instrument it to log the model used and the cost. Run it for a week.

Calculate the total cost. Then, manually test a smaller model on a sample of those tasks. Measure the accuracy. If the smaller model is acceptable, you have a candidate for dynamic routing.

If the accuracy is not acceptable, you have saved yourself the effort of building a router that will not work. You have also identified a gap in your model capabilities.

This is the first step. It is small, but it is concrete. It moves you from guessing to measuring. And measurement is the foundation of better AI economics.