Rate limiting and backpressure
Added 2026-07-14 · Last verified 2026-07-14
Model providers can limit requests, input tokens, output tokens, or concurrent work, so admission control must estimate token load rather than count requests alone. Use per-tenant quotas, bounded queues, concurrency semaphores, exponential backoff with jitter, and provider reset headers; shed or defer low-priority work before queues consume all memory or exceed user deadlines. Propagate cancellation through agent and tool calls, and avoid synchronized retry storms or unbounded worker fan-out after a limit clears.
Useful?
Related notes
Source: The Agent Loop — agent-loop.xyz. Curated in the open knowledge base (CC BY 4.0).