Cost is decided at design time
Inference spend is not an operations problem to be discovered after launch. It is a consequence of architectural decisions, and it should be specified alongside latency and accuracy rather than found on an invoice.
Teams that retrofit cost control consistently spend more engineering effort on it than teams that designed for it, and they do that work under pressure, which is when unfortunate shortcuts get taken.
Routing: the largest lever
An agent workflow is a sequence of steps with very different requirements. Classification, extraction, routing, reformatting and summarising a short input are handled well by small models. Genuinely hard reasoning is not. Sending every step to the most capable available model is the default, and it is almost always the biggest line item.
The price gap between tiers is not marginal — it spans orders of magnitude between the cheapest capable models and frontier ones. That gap is large enough that routing decisions dominate almost every other optimisation you could make.
The practical method is to enumerate the steps, ask of each what capability it actually requires, and route accordingly. Then evaluate: the eval suite tells you whether the cheaper model was in fact adequate, which converts an anxious guess into a measured decision.
Caching: same behaviour, lower price
Prompt and prefix caching reduce the cost of repeated context — valuable in agent systems specifically because the system prompt and tool definitions are resent constantly. Semantic caching goes further and deflects near-duplicate requests entirely.
Caching is attractive because it is structural rather than behavioural: the output is unchanged, only the price moves. It also improves latency, which means it is usually an easy sell to whoever owns the user experience.
Context discipline
The cheapest token is the one not sent. Reducing before generating — passing a compact structured summary rather than raw rows, trimming conversation history to what is load-bearing, keeping tool definitions tight — bounds input size by something you control rather than by data volume.
This has a second benefit that is easy to miss: a smaller, better-shaped context usually produces better output, because the model is not sifting for the relevant part. Cost work and quality work point the same direction more often than people expect.
Budgets should degrade, not explode
A token budget that throws a hard error at its ceiling has converted a cost control into an outage. Budgets should degrade: fall back to a cheaper model, reduce context, queue, or return a partial result with an honest indication that it is partial.
And every ceiling event should emit telemetry. A budget that is being hit constantly and silently absorbed is not protecting you; it is hiding a capacity problem.
Instrument per feature, not per account
A single monthly provider total tells you almost nothing actionable. Attributing spend to feature, workflow and step is what turns cost into an engineering signal — it shows which part of the product is expensive, which is the only way to decide whether that part is worth what it costs.
It also makes regressions visible. A change that quietly triples the cost of one workflow is invisible in an aggregate and obvious in a breakdown.