Skip to content
← Insights///Engineering///Field note

Latency budgets: what users tolerate when they know a machine is thinking

Agent systems are slow by construction. Tolerance depends far more on framing and feedback than on raw seconds.

By Matheus MelloFounder & Principal AI Engineer

Multi-step means multi-latency

Every hop adds up: model call, tool call, another model call to interpret the result. A workflow that looks tidy in a diagram is a serial accumulation of network round trips, and the total is frequently an order of magnitude beyond what the same interface would have cost before.

Perceived and actual are different problems

The strongest lever is usually not making it faster but showing progress. Streaming output, naming the current step, and revealing intermediate results change tolerance substantially — the same duration reads as working rather than as broken.

A spinner is the worst option, because it conveys duration without conveying progress. Users interpret an undifferentiated wait as a hang far sooner than they interpret a narrated one.

Set the budget before building

Latency belongs in the specification alongside cost and accuracy. Deciding afterwards that a workflow must complete in five seconds, when it was architected as six sequential model calls, means rearchitecting rather than optimising.

Concretely: which steps can run in parallel, which can be precomputed, which can be cached, and which genuinely must be sequential. That analysis is cheap during design and expensive afterwards.

Not everything must be synchronous

A meaningful fraction of agent workflows do not need to happen while a user waits. Reports, enrichment, analysis and summarisation can run asynchronously and notify on completion, which removes the latency constraint entirely and often permits a better, more thorough system.

Asking whether the user actually needs to wait is worth doing explicitly. The answer is no more often than the default interface pattern suggests.

Watch the tail

Average latency hides the problem. A workflow that usually takes four seconds and occasionally takes ninety — because a tool retried three times — will be judged on the ninety. Multi-step systems have long tails by construction, since each step contributes its own worst case.

Budget and monitor the tail explicitly, and make sure something times out cleanly rather than allowing the worst case to be unbounded.

Related

Agentic systems — how this shows up in the work.