Timeouts, retries and circuit breakers

The premise

Every network call fails eventually. Not "might" — will. The difference between a system that degrades and one that collapses is entirely in how it handles that.

Timeouts

Every outbound call needs an explicit timeout. Most client libraries default to none or to something absurd like two minutes.

Without a timeout, a hung downstream service holds your worker, which holds a connection, which holds a slot in the pool. Requests queue. The queue grows. Your service is now down because someone else is slow. This is the single most common way one team's incident becomes four teams' incident.

Set timeouts from measured latency, not vibes: roughly p99 plus headroom. And budget them across the chain — if your API has a 3-second budget and calls three services, they cannot each have a 3-second timeout.

Two kinds worth distinguishing: connection timeout (short, ~1 s — either it connects or it does not) and read timeout (longer — the work takes time).

Retries

Retries turn transient failures into successes. They also turn a struggling service into a dead one.

Only retry what is safe to retry. A timeout is ambiguous: the request may well have succeeded. Retrying a POST /payments after a timeout is how customers get charged twice — unless you have an idempotency key, which is exactly why the two topics belong together.

Retry these: connection failures, 429, 502, 503, 504, timeouts on idempotent operations. Never retry these: 400, 401, 403, 404, 422. They will fail identically forever.

Exponential backoff with jitter. Fixed-interval retries synchronise: 10,000 clients fail at the same moment and all retry one second later, producing a perfectly timed thundering herd against a service that was just recovering.

delay = random(0, min(cap, base * 2 ** attempt))

The randomness is not a refinement — it is the part that actually works. Full jitter spreads the load; without it, backoff just moves the spike.

Forty clients retrying. Without jitter they synchronise; with it, the load spreads.

Cap total attempts (3–5) and cap total elapsed time. A retry that arrives after the user has given up is pure waste.

Beware retry amplification. Three layers each retrying three times is 27 requests for one logical call. Retry at one layer, ideally the outermost that can make the safety judgement.

Circuit breakers

When a dependency is genuinely down, retrying is worse than useless — you are adding load to something already failing and burning your own capacity waiting for timeouts.

A circuit breaker tracks the failure rate and has three states:

• Closed — normal, requests flow.
• Open — failure threshold exceeded, so requests fail immediately without a network call. Fast failure is the feature: you stop wasting your worker on a timeout.
• Half-open — after a cooldown, let one request through. Success closes the circuit; failure re-opens it.

The value is protecting yourself, and giving the failing service room to recover.

The three states, and what moves the breaker between them.

Degrade, do not collapse

When something fails, the question is what the user should see. Ranked:

1. Serve stale. A cached value from ten minutes ago beats an error page.
2. Serve partial. The recommendations widget fails; render the page without it. Never let an optional dependency take down a required page.
3. Queue it. Accept the request, return 202, process when the dependency recovers.
4. Fail clearly. If you must fail, say what happened and whether retrying will help.

Error responses clients can act on

Include a stable machine-readable code, a human message, and whether a retry is worth it:

{
  "error": {
    "code": "insufficient_funds",
    "message": "Card was declined.",
    "retryable": false,
    "request_id": "req_01HX3..."
  }
}

request_id is the one that saves you hours — the user pastes it into a support ticket and you find the exact trace.

Prove you know it

Pick one service you call. Answer: what is the timeout, what happens on a timeout, how many retries with what backoff, and what does the user see if it is down for ten minutes? Then actually test it — block the dependency with a firewall rule and watch. The gap between the answer and the behaviour is always instructive.