LLM cost discipline, designed in rather than retrofitted.

Cost is an architectural property

Inference spend is not an operations problem discovered after launch; it is a consequence of decisions made during design. Token budgets belong in the system specification next to latency and accuracy requirements. Teams that retrofit cost control after deployment consistently spend more engineering time on it than teams that designed for it, and they do it under pressure.

The three levers, in order of leverage

Routing first. Most steps in an agent workflow do not need a frontier model — classification, extraction, routing and formatting are frequently handled by a much cheaper model at indistinguishable quality. The price gap between tiers is large enough that sending everything to the most capable model is usually the single biggest line item and the easiest to fix.

Caching second. Prefix and prompt caching cut the cost of repeated context, and semantic caching deflects near-duplicate requests entirely. Both are structural: they change what you pay for the same behaviour.

Context discipline third. Reducing before you generate — passing a compact structured summary rather than raw rows — bounds input size by something you control rather than by data volume.

Budgets should degrade, not explode

A token budget that throws a hard error at the ceiling turns a cost control into an outage. Budgets should degrade gracefully: fall back to a cheaper model, reduce context, queue the request, or return a partial result — and emit telemetry so the ceiling being hit is visible rather than merely survived.