The most common cost mistake in agent systems is optimizing the wrong unit. Teams compare models by price per million tokens, pick the cheapest, and then watch the bill grow anyway — because the cheap model retries more, wanders through longer trajectories, and escalates to humans more often. The unit that matters is cost per completed task, and a more capable model that finishes in 6 tool calls routinely beats a cheaper one that takes 15. Anthropic's guide to building effective agents makes the underlying point: added cost and latency must buy measurable outcome improvement, and simple single-call designs should win whenever they suffice.
Where the tokens actually go
1. Repeated input dwarfs output. An agent loop re-sends its system prompt, tool schemas, and accumulated history on every turn — a 20-turn trajectory can re-read the same context 20 times. Prompt caching attacks exactly this: cache reads are billed at roughly a tenth of fresh input, so structuring prompts as a stable prefix (system + tools first, volatile content last) turns the dominant cost bucket into the cheapest one. In practice this is the single highest-leverage change most teams can make in an afternoon.
2. Context that never gets pruned. Retrieval that stuffs top-20 chunks "to be safe," tool outputs appended verbatim forever, history that grows without compaction — all of it is paid on every subsequent turn. Retrieve less and rerank harder, summarize closed episodes, and cap tool output size at the boundary.
3. Unbounded loops. A stuck agent that retries the same failing tool call is a token furnace. Cap tool-call and reasoning iterations, detect repeated near-identical calls, and treat a budget-exceeded run as an explicit outcome — surfaced as partial success or escalation — rather than letting it silently degrade quality or burn to the cap.
Routing: escalate on evidence, not vibes
Routing routine classification and extraction to small models while reserving frontier models for hard cases is the other structural lever. FrugalGPT (arXiv:2305.05176) demonstrated the cascade pattern — try cheap, escalate on low confidence — matching top-model accuracy at a fraction of the cost. The operational key is the escalation trigger: validation failure, low self-reported confidence calibrated against evals, or schema violations. Route on measured difficulty signals, and make the escalation rate a tracked metric — a rising rate means your cheap tier's job drifted.
Budgets as a product feature
Set per-run and per-tenant budgets the way you set rate limits: enforced at the platform layer, visible in observability, with exhaustion producing a defined partial outcome. A budget that only exists in a dashboard is a report, not a control.
Sources: Anthropic — Building effective agents · Anthropic — Prompt caching · FrugalGPT (arXiv:2305.05176).
Related: token budgets, prompt caching, model selection & adaptation, agent evaluation pitfalls.