Cost-aware model routing:
SLMs for the 80%, frontier models for the 20%

Why one frontier default fails economically

Frontier models are strong. They are also expensive and often slower than needed for classification, extraction, FAQ, and boilerplate drafting. Teams that wire one SDK call everywhere pay frontier prices for work a small language model could finish.

Finance notices first as a line item. Product notices next as p95 latency that never improves because every ticket, every rewrite, and every trivial extraction shares the same heavy path. Engineering notices last when a Friday model string change ships without an eval gate and Monday's workflows look different.

Cost-aware routing is not "always use the cheap model." It is matching model capacity to task difficulty under an explicit quality floor. Without classification and measurement, teams either overspend or ship silent quality drops that users blame on "the AI" instead of on an ungated router.

Routing belongs in the gateway control plane described in AI gateway triple-gate, with costs visible in agent observability. Putting routing in app code as scattered if statements recreates the same problem the gateway was meant to solve.

Failure modes of naive routing

Static cheap default. Everything goes to an SLM. Hard reasoning, multi-step planning, and ambiguous customer replies quietly degrade. Satisfaction surveys stay flat while defect tickets rise on the hardest 20% of work.

Vendor lock in code. Each service hardcodes a model string. Finance cannot steer spend. Platform cannot pin a safer model for one task class without a multi-repo change. Incidents become "find every SDK call."

Cascade loops. Escalation rules bounce between models without a stop condition or a max escalations-per-run. Latency and cost both explode while the user waits on a loop that never converges.

Ungated experiments. Someone changes router weights on Friday. Monday's evals tank. Nobody can prove whether the drop came from prompts, indexes, or the new route table.

Cache blindness. Identical FAQ and extraction jobs re-hit the frontier path because nobody keyed a cache on workflow plus inputs. Routing alone cannot fix spend if repeat work is never memoized where freshness allows.

The routing control model

Define task classes: FAQ, extraction, planning, code repair, customer write prep. Pin a primary model and a fallback per class. Name an owner for each pin so changes have a person, not only a config file.

Classify at the edge with cheap signals: intent rules, lightweight classifiers, or a tiny model. Prefer deterministic routes when the API already knows the workflow (a dedicated "summarize ticket" endpoint should not re-discover intent every time).

Cascade when needed: SLM first, structured validation, escalate to frontier on failure. Cap escalations per run. Fail closed to a documented fallback when providers error, instead of retrying into an unbounded spend spiral.

Operable steps:

1. Inventory call sites and replace hardcoded model strings with class IDs at the gateway.
2. Ship classifiers or rules for the top traffic classes first (often FAQ and extraction).
3. Add structured checks (JSON schema, required fields, citation presence) before accepting an SLM answer.
4. Escalate only on check failure or hard class; log the reason code.
5. Attach budgets and cache keys per class before enforcing routes in production.

Quality floors and budget guards

Attach budgets per team and workflow. Reject or queue when spend ceilings hit, instead of surprising finance at month end. Budgets without user-visible behavior (queue, degrade, or refuse) become accounting theater.

Cache idempotent prompts and retrieval packs where freshness allows. Caching is often the largest cost win after routing. Key caches on workflow ID plus normalized inputs, and document TTL owners the same way you document model pins.

Gate router changes with the same discipline as prompts: cost per success and eval scores must both stay in band (eval gates in CI). A cheaper route that fails structured checks is not a win. A more expensive route that does not move quality is not a win either.

Keep per-class quality floors in the same place as the route table. When an SLM starts failing checks above a threshold, auto-bias that class toward frontier until a human updates the pin. Do not wait for a quarterly cost review to notice.

Shadow routing and ownership

Shadow-route production traffic before enforcing. Compare SLM vs frontier outcomes on the same cases: exact match rates for extraction, rubric scores for planning, and human spot checks for customer-facing drafts.

Document owners for each task class model pin. Routing without ownership becomes another invisible microservice. Require a short change note when pins move: expected cost delta, expected quality delta, and the eval suite that must stay green.

Rollout and metrics

Measure cost per successful task, escalation rate, latency by class, and quality deltas after each router change. Break out cache hit rate so you can tell routing wins from memoization wins.

Roll out one class at a time. FAQ and extraction usually pay back first. Keep planning and irreversible write prep on stronger models until evals say otherwise. Pair spend dashboards with quality dashboards in the same weekly review; looking at cost alone invites silent degradation.

Done well, most traffic stays cheap, hard work stays capable, and spend becomes a controllable engineering input instead of a surprise invoice. The gateway owns the route. The eval suite owns the floor. Teams stop treating "default frontier" as architecture.