429 Cascade: How Provider Rate Limits Become Your Outage

The 429 you didn't plan for

Rate limits are not an edge case; they're a designed-in property of every commercial LLM API. OpenAI's rate limits are enforced across four independent dimensions, requests per minute (RPM), tokens per minute (TPM), requests per day (RPD), and tokens per day (TPD), and exceeding any one returns a 429. Limits scale with usage tiers from Free through Tier 5, where your tier is a function of cumulative spend: roughly $5+ unlocks Tier 2, $50+ Tier 3, and Tier 5 reaches around 2M TPM on flagship models. Anthropic's rate limits follow the same model: per-model RPM, input TPM, and output TPM tied to spend tiers, Tier 1 starts as low as 40K TPM on Opus, with throttled requests returning HTTP 429 and a retry-after header.

The detail that catches teams off guard: TPM is usually the binding constraint, not RPM. A single request with a 100K-token context and a long output can consume a large share of a minute's token budget on its own. You can be nowhere near your requests-per-minute ceiling and still get throttled because a handful of long-context calls drained your tokens-per-minute. The limit you monitor is rarely the limit you hit.

How one 429 cascades into an outage

A 429 is a small, recoverable signal. The cascade is what your own infrastructure does with it. Here is the chain that turns a few percent of throttled calls into a P1.

1. Throttle. The provider returns 429 on, say, 5% of requests during a peak-minute burst. By itself, harmless, those requests could back off and succeed a second later.
2. Retry without backoff. Your client immediately retries the 429s. The retries land in the same already-saturated minute, get throttled again, and add load. (This is the rate-limit flavor of a retry storm.)
3. Queue. Retries and new requests pile up in your worker pool or connection pool waiting for a slot. The pool was sized for normal latency, not for requests sitting in backoff loops.
4. Pool exhaustion. Every worker is now occupied by a request that's waiting, retrying, or backing off. Healthy requests, ones the provider would serve, can't get a worker. The 5% throttle has now blocked 100% of new traffic.
5. Timeout and P1. Upstream callers (the browser, the gateway, the load balancer) hit their own timeouts waiting for the saturated pool, return 5xx to users, and your "5% rate-limited" event is a total feature outage.

The throttle was recoverable. The cascade was not, and it was entirely your stack's doing. Provider rate limits become your outage because the failure propagates through resources the provider doesn't control: your worker pool, your connection pool, your timeout budget.

Shared limits, noisy neighbors, and the org-level trap

There's a structural wrinkle that turns rate limits from a per-feature concern into an organizational one: limits are usually scoped to the account or API key, not to the individual feature. Every team, every cron job, every batch process, and every experiment sharing that key draws from the same RPM and TPM pool. Your carefully capacity-planned chat feature can get throttled because an unrelated team kicked off a large backfill at 2pm, or because a misconfigured retry loop somewhere else in the company is quietly eating the token budget.

This is the noisy-neighbor problem, internalized. It means your feature's effective rate limit isn't the number on the provider's tier table, it's that number minus whatever everyone else on the key is consuming, which is a value that changes minute to minute and that you don't control. A load test that assumes you have the full TPM ceiling to yourself will be optimistic by exactly the amount of background traffic on the key in production.

The mitigations are organizational as much as technical. Use separate API keys or projects per feature so limits are isolated and one team's spike can't starve another's. Where the provider supports it, set per-key or per-project budgets so a runaway job self-limits instead of consuming the shared pool. And when you load-test, test against the budget your feature will actually have in production, the residual after known background consumers, not the theoretical ceiling. A feature that passes at 100% of the tier limit and ships into an account already running at 60% utilization will start cascading the first time the two peaks coincide.

Why staging never sees it

Staging runs at 1x with a clean token budget and never approaches the TPM ceiling, so it never returns a 429, so the cascade path is never exercised. The first time your 429-handling code runs at scale is in production at peak, the worst possible place to discover that it queues instead of sheds. You have to manufacture the 429 deliberately to test it.

# Model TPM exhaustion: it's tokens, not requests, that throttle you
def minutes_to_throttle(rps, avg_input, avg_output, tpm_limit):
    tokens_per_min = rps * 60 * (avg_input + avg_output)
    return tpm_limit / tokens_per_min   # <1.0 means you exceed TPM

# 50 rps of long-context RAG calls vs a 2M TPM ceiling
ratio = 50 * 60 * (12_000 + 800) / 2_000_000
print(f"TPM utilization: {ratio*100:.0f}%")
# TPM utilization: 192%  -> you're throttled at ~half this traffic,
# while RPM (e.g. 3,000/min) may be nowhere near its limit.

Handling 429 correctly: shed, don't queue

The core principle: when the provider says 429, the right response is almost never "queue it and try harder." It's "back off respectfully, shed what you can't serve, and protect the requests that can succeed."

Honor retry-after, it's the provider telling you exactly when to come back

import time, random

def call_respecting_limits(fn, max_attempts=3):
    for attempt in range(max_attempts):
        resp = fn()
        if resp.status != 429:
            return resp
        # The provider tells you when capacity returns, obey it.
        wait = float(resp.headers.get("retry-after", 0)) or \
               min(20,0.5 * 2 ** attempt) + random.uniform(0,0.5)
        if attempt == max_attempts - 1:
            raise RateLimited("shedding after retry-after window")
        time.sleep(wait)

Cap concurrency to your token budget, not your worker count

The cleanest way to never trigger a 429 cascade is to not exceed your TPM in the first place. A client-side token-aware limiter throttles you before the provider does, smoothing your bursts into the rate the provider will actually serve, and keeping your worker pool free.

import asyncio, time

class TokenBudgetLimiter:
    """Client-side TPM governor: never offer the provider more than it allows."""
    def __init__(self, tpm_limit):
        self.tpm = tpm_limit
        self.window_start = time.time()
        self.spent = 0
        self.lock = asyncio.Lock()
    async def acquire(self, est_tokens):
        async with self.lock:
            now = time.time()
            if now - self.window_start >= 60:
                self.window_start, self.spent = now, 0
            if self.spent + est_tokens > self.tpm:
                # We'd exceed TPM, wait for the window instead of 429-ing
                await asyncio.sleep(60 - (now - self.window_start))
                self.window_start, self.spent = time.time(), 0
            self.spent += est_tokens

Isolate with a bulkhead so a 429 storm can't drain the whole pool

Cap the number of workers that can be tied up in provider calls. If the model API is throttling, the bulkhead fills and new requests fail fast with a clear degraded response, instead of waiting on an exhausted pool and taking down unrelated functionality with them. For non-urgent work, both providers offer Batch APIs that are exempt from standard rate limits and discounted ~50%, which is the right escape valve for anything that doesn't need a synchronous answer. OpenAI's rate-limit cookbook walks through exactly these tactics.

The headers the provider already gives you

Most teams discover their rate limits the hard way, by getting 429s, when the provider has been telling them their remaining budget all along. Both OpenAI and Anthropic return rate-limit headers on every response, not just on the throttled ones: remaining requests, remaining tokens, and the reset time for each. These are a real-time fuel gauge for your quota, and almost nobody reads them proactively.

Reading them changes rate-limit handling from reactive to predictive. Instead of slamming into the ceiling and reacting to a 429, you watch remaining-tokens trend toward zero and slow yourself before you cross it. You can surface the headroom on a dashboard, alert when remaining budget drops below a threshold, and shed or queue non-urgent work proactively rather than scrambling after the throttle hits. The 429 should be your last line of defense, not your first signal.

def adjust_pace_from_headers(resp, limiter):
    # Providers expose live quota on every response, use it.
    remaining_tokens = int(resp.headers.get("x-ratelimit-remaining-tokens", 0))
    reset_seconds    = float(resp.headers.get("x-ratelimit-reset-tokens", 0))
    limit_tokens     = int(resp.headers.get("x-ratelimit-limit-tokens", 1))

    headroom = remaining_tokens / max(limit_tokens, 1)
    if headroom < 0.15:
        # Within 15% of the ceiling, proactively slow down, don't wait for 429
        limiter.throttle(until=time.time() + reset_seconds)
        log.warning(f"approaching TPM ceiling: {remaining_tokens:, } tokens left")
    return headroom

Load-test the throttle deliberately

You can't wait for a real 429 to learn how your stack behaves. Drive load past your known TPM/RPM ceiling on purpose and assert that the system sheds gracefully rather than collapsing.

# Push past the rate limit on purpose; assert graceful shedding
async def rate_limit_cascade_test(target_tpm, ceiling_tpm):
    # Ramp offered load from 0.5x to 2x the provider's TPM ceiling
    results = await drive_load(
        offered_tpm_schedule=[0.5,0.8,1.0,1.3,1.6,2.0],
        ceiling=ceiling_tpm, per_step_seconds=60)

    for step in results:
        served = step["success_rate"]
        pool = step["pool_utilization"]
        p99 = step["p99_latency_ms"]
        print(f"offered {step['offered']:.1f}x ceiling | "
              f"served {served*100:4.1f}% | pool {pool*100:3.0f}% | p99 {p99}ms")
        if step["offered"] > 1.0:
            # Past the ceiling, throughput should plateau, NOT collapse,
            # and the pool must not saturate (cascade signature).
            assert served > 0.55, "throughput collapsed past rate limit"
            assert pool < 0.95, "pool saturating, cascade path is live"

The signature of a healthy system: as offered load climbs past the ceiling, served throughput flattens near the provider's actual capacity and excess requests are shed quickly with a clear error. The signature of a cascade: served throughput climbs, peaks, then collapses while pool utilization pins at 100%, the queue-and-die pattern.

The bottom line

Provider rate limits are not a rare failure; they're a designed constraint you will hit, and TPM usually before RPM. A 429 is recoverable. The cascade, immediate retries, queueing, pool exhaustion, upstream timeouts, is what converts a 5% throttle into a 100% outage, and it lives entirely in your stack. Govern your concurrency to your token budget, honor retry-after, bulkhead the provider calls, push non-urgent work to the Batch API, shed instead of queue, and prove all of it by driving load past the ceiling on purpose. Read the provider's rate-limit headers so you slow down before the ceiling rather than after it. The test is cheap. The 3am cascade is not.