Retries look like a reliability feature until an agent turns one uncertain tool call into three charges, three emails, or a synchronized flood against a struggling service. The safe design starts by separating three questions: Did the request time out? Is the failure transient? Is repeating the operation safe? A retry is justified only when the answers line up.
Timeouts are layered budgets
A networked tool needs both a short connection timeout and an overall deadline for the logical operation. The connection timeout stops a dead endpoint from consuming the whole run; the deadline prevents DNS, connection, response, and backoff delays from quietly exceeding the agent's task budget. Each retry spends from the same deadline rather than resetting the clock.
Amazon's Builders' Library recommends choosing timeouts from the downstream service's latency distribution, then accounting for network overhead and deployment effects such as cold connections. For agents, expose the exhausted deadline as a structured outcome—transient_failure, attempts used, and the last provider request ID—so the model can switch strategy or escalate instead of starting another hidden retry loop.
Retry a class of failure, not every error
Connection resets, socket timeouts, throttling, and selected 5xx responses are usually transient. Invalid arguments, failed authorization, missing resources, and business-rule rejection require a changed request or human action; repeating them only adds load. Google Cloud's retry guidance makes the second gate explicit: even a transient response should be retried automatically only when the operation is idempotent or protected by a precondition.
Use exponential backoff with random jitter and a hard attempt/deadline cap. Backoff gives the dependency room to recover; jitter prevents thousands of workers from retrying on the same schedule and causing another outage. Honor Retry-After or provider-specific hints when present. Put the policy in the executor, not in the model prompt, so a hallucinated “try again” cannot bypass it.
The ambiguous-success trap
The most dangerous failure is a timeout after the server committed the write but before the client received the response. From the agent's perspective the outcome is unknown. Retrying with a fresh request may duplicate the effect; refusing to retry may leave the task falsely reported as failed.
An idempotency key resolves that ambiguity. Generate one stable key for the logical action, persist it with the task or checkpoint, and reuse it on every replay. The service stores the first outcome and returns it for the same key. Stripe's idempotent-request contract demonstrates the pattern: repeated create/update requests with the same key return the recorded result rather than performing the action again, while changed parameters are rejected.
If the provider has no native key, build deduplication at the tool boundary: store (operation, target, idempotency_key) → outcome before returning success, and query that record before executing a replay. Conditional writes and resource-version preconditions can make updates safely retryable too.
What to record
For every attempt, capture the logical idempotency key, provider request ID, attempt number, error class, delay, elapsed deadline, and final disposition. That turns “the agent called the tool twice” from a mystery into an auditable decision—and lets evals inject timeout-after-commit failures to prove the duplicate-effect path is actually closed.
Sources: AWS Builders' Library — Timeouts, retries, and backoff with jitter · AWS Builders' Library — Making retries safe with idempotent APIs · Google Cloud — Retry strategy · Stripe — Idempotent requests.
Related: tool use, tool schema design, durable agent execution, rate limiting & backpressure.