Agent latency is two different metrics wearing one name: time to first useful feedback (does the user know something is happening?) and total task time (when is it actually done?). Users forgive a slow task that talks; they abandon a fast one that goes silent for forty seconds. Optimize both, but never confuse them — streaming fixes the first and does nothing for the second.
Cutting real latency
OpenAI's latency-optimization guide organizes the levers well, and they map directly onto agent loops:
- Fewer serial model calls. Every hop in a chain is a full round-trip. Collapse plan-then-act pairs where the plan is trivial, and never use a model call for what a regex or a join could do — the guide's "use fewer tokens / make fewer requests" advice is doubly true when each request feeds the next.
- Parallelize the independent. Retrievals, tool calls, and subagent fan-outs that don't depend on each other should run concurrently; an agent that awaits three searches serially triples its own latency.
- Cache the stable prefix. System prompt + tool definitions re-sent every turn are exactly what prompt caching exists for — cache reads cut both cost and time-to-first-token on every subsequent call.
- Prefetch the predictable. If step 3 always needs the user's calendar, start fetching it during step 2's model call, not after.
- Right-size the model. Routing classification and extraction to a small fast model, reserving the frontier model for the hard reasoning steps, is often the single biggest total-time win.
Streaming as UX, honestly
Stream tokens for prose and structured progress events for tool phases ("searching…", "3 files read") — Anthropic's streaming docs cover the event model, including streaming during tool use. Expose status, not private chain-of-thought. And keep three engineering invariants that streaming tempts you to drop:
- Cancellation: a user who sees the wrong direction at token 50 must be able to stop the run — and cancel any in-flight tool side effects safely (idempotency again).
- Backpressure: a consumer slower than the stream must not silently drop frames or balloon memory.
- A final authoritative result, distinct from the partial stream. Partial output is a preview; downstream systems consume only the committed final — otherwise a mid-stream disconnect becomes a half-acted-on answer.
The agent-specific trap
Long-running agents also need liveness signaling at the task level: an immediate acknowledgment ("on it — this will take a few minutes"), progress at real milestones, and a completion notification. That's cheap to build and worth more perceived speed than any decoding optimization.
Sources: OpenAI — Latency optimization · Anthropic — Streaming Messages.
Related: prompt caching, token budgets, multi-agent orchestration, tool retries & idempotency.