Retry Storm Simulator
Explore how client retries can prolong a dependency outage. Compare backoff, retry budgets, and deadline-aware load shedding in an interactive model, then export Envoy configuration fragments.
A dependency serves 1,000 requests per second until something knocks it down at 10s. Pick an incident, then change how clients retry and see whether the system recovers on its own once capacity comes back.
Scenarios
The primary fails over and serves nothing for 15 s at 60% load. Clients already use jittered exponential backoff with 3 attempts.
Not recovered by 120s: the trigger ended at 25s
Observed: the backlog was still growing when the run ended, and 94.1% of the dependency's work went to requests whose caller had already given up. Each of those timeouts produced another retry, the self-sustaining loop called a metastable failure. Model bound: once every response arrives late, retries can hold demand at 3× normal (180% of capacity), so this model has no way to drain on its own.
Load at the dependency
Stacked: what clients send. Lines: what the dependency can serve, and how much of it reaches a caller who is still waiting.
- First attempts
- Retries
- Served after caller gave up
- Successful responses
- Dependency capacity
- Trigger window
- Peak load
- 3×normal demand, at the dependency
- Failed requests
- 89.1%64,170 of 72,000 in 120s
- Wasted work
- 94.1%served after the caller gave up
- Tipping point
- 1.5sshortest trigger whose backlog is still growing at 120s, in this model
The incident
Once the queue holds more than one timeout of work, every response arrives too late.
Retry policy
- User
- Caller ×3
- Dependency
Up to 3 attempts per user request.
Model bound: when everything fails, demand can reach 3× normal, so the backlog is guaranteed to drain only below 33.3% load. You are at 60%, so in this model a long enough trigger leaves it stuck. Real systems differ in the details.
Nominal per-layer ratio, compounded into one dependency-wide rolling 10s budget. An aggregate approximation, not separate budgets at each layer.
Drop queued requests whose caller has already timed out (deadline propagation).
Same incident, one defense at a time
Each row keeps the changes above it. Load a row to see its chart.
No retries
One attempt; failures go straight to the user.
Recovered +21.05s29.2% failed · 1× peak
Immediate retries
3 attempts at each of 1 layer, no delay.
Not recovered by 120s89.2% failed · 3× peak
+ Backoff with jitter
Exponential from 100ms, capped at 2s.
CurrentNot recovered by 120s89.1% failed · 3× peak
+ 10% retry budget
Retries capped at 10% of first attempts.
Recovered +28.65s35.5% failed · 2× peak
+ Shed abandoned work
Server drops queued requests whose caller already timed out.
Recovered +0.1s11.7% failed · 2× peak
What to change
Cap retries as a share of traffic
With a single retrying layer, a per-client budget (the SRE book uses 10% of requests) keeps sustained amplification near 1.1× however many attempts each request is allowed. Requests over the budget fail fast instead of joining the queue.
Stop serving callers who already left
Propagate deadlines and drop queued work once its caller has timed out. Otherwise recovery capacity goes to responses nobody will read.
Backoff alone did not save it
New requests keep arriving whatever the clients do, so backoff only delays retries. They still land on a saturated queue. Backoff helps with short blips and synchronized clients, but it does not cap load.
Or buy headroom
In this model, the backlog always drains below 33.3% load with this policy. Above that it is fine until a trigger of about 1.5s or longer leaves it unrecovered at 120s. Treat both numbers as a direction for real capacity planning, not as thresholds.