The Retry Storm That Took Down Your AI Feature

How a ten-second blip becomes a ten-minute outage

Picture a normal Tuesday. Your AI feature does 2,000 requests per second to a model provider. The provider has a brief degradation, a deploy, a noisy neighbor, a regional blip, and for ten seconds latency spikes and a chunk of requests start timing out.

Here's what your "resilient" stack does next. Every timed-out request retries. Many retry immediately. Each retry is a fresh request against a provider that is already slow, so it's also likely to time out, and trigger another retry. Meanwhile the original 2,000 rps of new traffic keeps arriving on top. Within seconds you're sending 3x, 4x, 5x your normal load at a service that was struggling at 1x. The provider, or your connection pool, or your gateway buckles. Now requests that would have succeeded fail too. The storm sustains itself: it keeps the system saturated long after the original ten-second fault cleared.

This is retry amplification, one of the best-documented failure modes in distributed systems. AWS's Marc Brooker laid out the mechanics and the cure in the Builders' Library piece Timeouts, retries, and backoff with jitter: retries are load multipliers, and the worst time to multiply load is during a partial failure.

The three mistakes that build a storm

1. Fixed-interval or immediate retries

Retrying immediately, or on a fixed timer, is the fastest path to a storm. As Google's SRE Book chapter on handling overload notes, uncontrolled retries are a primary driver of cascading failure. Every client that failed at time T retries at roughly the same moment, producing a synchronized wave. The recovering service gets hit with a spike, fails again, and the whole population retries together on the next tick. You've built a metronome that beats the service to death, the classic thundering herd.

2. Exponential backoff without jitter

Exponential backoff, 1s, 2s, 4s, 8s, is necessary but not sufficient. Without randomness, every client that failed at the same time backs off by the same amount and retries at the same time. The waves get further apart but stay just as tall. AWS's Exponential Backoff And Jitter showed this with simulation: adding jitter, randomizing each client's backoff, flattens the synchronized spikes into a smooth, absorbable trickle. Jitter is not a refinement; it's the part that actually stops the storm.

3. No retry budget or ceiling

Even with perfect backoff and jitter, if every layer of your stack independently retries, the multipliers compound. The browser retries 3x, the SDK retries 3x, the gateway retries 3x, that's 27x amplification for a single user action against a downed dependency. Without a global cap on what fraction of traffic can be retries, a bad enough fault still produces a storm.

The compounding cost: you pay for the storm

Retry storms are uniquely expensive for AI features because every retried request that does reach the model is billed. A user clicks once; your stack sends the prompt five times; you pay input tokens five times. During a multi-minute storm against a degraded-but-not-down provider you can rack up several times your normal token spend on requests that mostly fail or get discarded, a cost spike that lands in the same week as the reliability incident. The outage and the bill arrive together.

# What naive retries do to load during a 30s fault
def amplification(base_rps, layers, retries_per_layer):
    # Each layer independently retries on failure
    mult = retries_per_layer ** layers
    return base_rps, base_rps * mult

base, peak = amplification(base_rps=2000, layers=3, retries_per_layer=3)
print(f"normal: {base:, } rps  ->  worst-case retry peak: {peak:, } rps")
# normal: 2,000 rps  ->  worst-case retry peak: 54,000 rps
# 27x the load, aimed at a service that's already failing.

Building a storm-proof retry layer

The recipe is settled. The discipline is applying all of it, not just the easy half.

Capped exponential backoff with full jitter

import random, time

def retry_delay(attempt, base=0.25, cap=20.0):
    # Full jitter: sleep a random amount in [0, exp_backoff]
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)

def call_with_backoff(fn, max_attempts=4, retry_budget=None):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RetryableError as e:
            # Honor server guidance first, it's the single best signal
            wait = e.retry_after if e.retry_after else retry_delay(attempt)
            if attempt == max_attempts - 1:
                raise
            if retry_budget and not retry_budget.allow():
                raise BudgetExceeded("retry budget exhausted; shedding")
            time.sleep(wait)

A retry budget (the part everyone skips)

A retry budget caps retries as a fraction of total traffic, say, retries may never exceed 10-20% of successful requests over a rolling window. AWS's numbers make the case: a 20% budget keeps total traffic near 1.2x normal during a fault instead of letting it balloon to 3.5x. When a real outage hits and retries would spike, the budget runs out and excess retries are dropped instead of amplifying the fault. This is the single most effective storm circuit-breaker, and the one most stacks don't have.

import collections, time

class RetryBudget:
    """Token-bucket: retries can't exceed `ratio` of recent successes."""
    def __init__(self, ratio=0.2, window_s=10):
        self.ratio, self.window = ratio, window_s
        self.success = collections.deque()
        self.retries = collections.deque()
    def _trim(self, dq, now):
        while dq and dq[0] < now - self.window: dq.popleft()
    def record_success(self): self.success.append(time.time())
    def allow(self):
        now = time.time()
        self._trim(self.success, now); self._trim(self.retries, now)
        budget = max(1, len(self.success)) * self.ratio
        if len(self.retries) < budget:
            self.retries.append(now); return True
        return False   # over budget, shed the retry, don't amplify

Don't retry the non-retryable

A 400 for a malformed prompt or an oversized context will fail identically on every attempt, retrying it is pure waste and pure amplification. Retry only transient classes: timeouts, 429s, 502/503/504. Everything else fails fast. The AWS SDK retry behavior docs codify this: standard mode applies exponential backoff with jitter and distinguishes transient errors from throttling.

Idempotency: the prerequisite nobody mentions

There's a quieter danger in retries that has nothing to do with load and everything to do with correctness. If a request is not idempotent, if running it twice produces two effects, then every retry risks a double-execution. For a read-only completion this is harmless. But the moment your AI feature has side effects, an agent that places an order, sends an email, writes to a database, charges a card, or kicks off a downstream job, a retry of a request that actually succeeded but whose response was lost will do the thing twice.

This is exactly the scenario retries create. A request reaches the provider, the provider does the work, and the response is lost on the way back, a dropped connection, a timeout on your side after the server already committed. Your client sees a failure, retries, and now the action has happened twice. During a retry storm, when timeouts are firing en masse, this double-execution risk is multiplied across thousands of in-flight requests. The storm doesn't just amplify load; it amplifies duplicate side effects.

The fix is an idempotency key: a unique token the client generates per logical operation and attaches to every attempt of that operation. The server deduplicates on the key, so a retry of an already-completed request returns the original result instead of re-executing. Most mature APIs support this; the discipline is using it for anything with a side effect, and load-testing that the dedup actually holds under concurrent retries.

import uuid

def perform_action_with_retries(client, payload, max_attempts=4):
    # One key per logical operation, reused across ALL retries of it
    idem_key = str(uuid.uuid4())
    for attempt in range(max_attempts):
        try:
            return client.post(payload, headers={"Idempotency-Key": idem_key})
        except RetryableError:
            if attempt == max_attempts - 1:
                raise
            time.sleep(retry_delay(attempt))
    # If the first attempt secretly succeeded, the retry returns the
    # SAME result via the key, no double charge, no duplicate order.

Load-test the storm before it tests you

Here's the gap: teams write careful retry code and never verify it under a sustained fault. Unit tests check that a single retry happens. They do not check what 5,000 concurrent clients do when the dependency is degraded for 60 seconds. That's the only test that matters, and it requires injecting a fault during a concurrent load run and watching aggregate request rate to the dependency.

# Fault-injection load test: prove retries don't amplify
async def storm_test(concurrency=5000, fault_window=(20,80)):
    sent = collections.Counter()   # requests/sec actually hitting the dep
    async def client():
        while test_running():
            t = int(time.time())
            sent[t] += 1
            degraded = fault_window[0] <= elapsed() <= fault_window[1]
            await do_request_with_backoff(force_fail=degraded)
    await asyncio.gather(*[client() for _ in range(concurrency)])

    baseline = median(rps for sec, rps in sent.items()
                      if not in_fault(sec))
    peak = max(rps for sec, rps in sent.items() if in_fault(sec))
    amp = peak / baseline
    print(f"baseline {baseline:.0f} rps, fault peak {peak:.0f} rps "
          f"-> {amp:.1f}x amplification")
    # A storm-proof stack should stay near 1x, backoff+jitter+budget
    # should flatten the fault, not amplify it.
    assert amp < 1.5, "retry storm: load amplified during fault"

The bottom line

Retries feel like resilience and behave like a load multiplier. During a fault, exactly when amplification hurts most, naive retries turn a ten-second blip into a self-sustaining outage and triple your token bill on the way down. The cure is old and proven: capped exponential backoff with full jitter, a global retry budget that sheds excess retries (20% keeps you near 1.2x, not 3.5x), honoring Retry-After, and never retrying non-transient errors. The thing teams skip is proving it works, inject a sustained fault under concurrent load and measure amplification. Aim for ~1x. Anything that spikes is a storm waiting for its trigger. And don't forget the correctness half: if your feature has side effects, an idempotency key is what keeps the storm from charging a card twice while it tries to take you down.