Your LLM Timeouts Are Wrong (Both Directions)

One number can't time out an LLM

Most stacks set a single timeout, timeout=30, and move on. That works fine for a REST API where latency is a tight, predictable distribution. It fails for LLMs because an LLM request has multiple distinct phases with wildly different latency characteristics, and a single timeout can only protect one of them well while leaving the others exposed.

A streamed completion has at least three phases worth timing separately, and as OpenAI's own streaming guidance makes clear, the first byte and the full generation are entirely different latency events:

• Connect, establishing the TCP/TLS connection and getting the request accepted. Should be fast and consistent; a slow connect signals network or provider-frontend trouble, and you want to fail it quickly.
• Time to first token (TTFT), the model accepting the request, queueing, and producing the first token. This is where queueing delay and prompt-processing time live, and it's the phase most sensitive to provider load. A laggy interaction is almost always a high p95/p99 TTFT problem.
• Total / inter-token, the full generation. A long, valid answer legitimately takes a long time; the right guard here is usually an inter-token timeout (the stream went silent) rather than a total-duration cap that punishes long outputs.

Collapse all three into one number and you're forced into a bad compromise: set it low enough to catch a hung connection and you kill long valid generations; set it high enough to allow long generations and a hung request occupies a worker for a minute.

Too short: the false-negative tax

A timeout set below your real latency tail kills requests that would have succeeded. Suppose your p99 total latency is 18 seconds for long answers, and someone set timeout=10 because it "felt reasonable." Now the slowest 1-3% of requests, disproportionately your most substantive, highest-value answers, get killed mid-generation. The user sees an error for a response that was actively working. Worse, if that killed request then retries, you've doubled the load and the cost to produce an answer you already had in flight. Short timeouts manufacture failures and then amplify them with retries.

Too long: the pool-exhaustion trap

The opposite error is subtler and more dangerous. A 60-second timeout looks harmless at 1x load, almost nothing ever reaches it. But when the provider slows down, requests that used to finish in 5 seconds now take 40, and every one holds a worker for that whole time. Your worker pool, sized for 5-second requests, can't keep up. The pool fills with slow-but-not-yet-timed-out requests, new requests queue, and the queue depth explodes. The 60-second timeout that never fired at 1x becomes the mechanism by which a provider slowdown turns into a total outage at peak. A timeout that's too long doesn't fail, it lets the system fail around it.

# How a too-long timeout exhausts a pool during a slowdown
def pool_headroom(pool_size, normal_latency_s, slow_latency_s, rps):
    # Little's Law: concurrent requests = arrival rate x time in system
    normal_inflight = rps * normal_latency_s
    slow_inflight   = rps * slow_latency_s
    print(f"normal: {normal_inflight:.0f} in-flight / {pool_size} workers")
    print(f"slow:   {slow_inflight:.0f} in-flight / {pool_size} workers")
    if slow_inflight > pool_size:
        print("=> pool exhausted during slowdown; queue grows unbounded")

pool_headroom(pool_size=200, normal_latency_s=5, slow_latency_s=40, rps=20)
# normal: 100 in-flight / 200 workers
# slow:   800 in-flight / 200 workers
# => pool exhausted during slowdown; queue grows unbounded

Derive timeouts from the distribution, not a guess

The correct timeout comes from your measured latency distribution, phase by phase. The standard practice, set timeouts off the high percentile of observed latency, not the average, applies directly: the average tells you nothing about the tail, and the tail is what times out. The same percentile-driven discipline that AWS recommends for timeouts and retries applies here, just split across phases. Measure each phase under realistic load, take the p99 (or p99.9 for the total), and add modest headroom.

# Derive per-phase timeouts from measured latency, with headroom
import numpy as np

def derive_timeouts(samples):
    connect = np.array([s["connect_ms"]    for s in samples])
    ttft    = np.array([s["ttft_ms"]       for s in samples])
    gap     = np.array([s["max_token_gap_ms"] for s in samples])  # inter-token

    return {
        # connect should be tight: p99 + small buffer
        "connect_ms": int(np.percentile(connect, 99) * 1.3),
        # TTFT carries queueing variance: p99.9 to avoid false kills
        "ttft_ms":    int(np.percentile(ttft, 99.9) * 1.5),
        # inter-token gap: if the stream goes silent past this, it's dead
        "inter_token_ms": int(np.percentile(gap, 99.9) * 2.0),
    }

# Example output for a chat feature:
# {'connect_ms': 900, 'ttft_ms': 6200, 'inter_token_ms': 4800}
# Note: NO single 'total' timeout, long valid answers aren't punished.

Notice what's missing: a single total-duration cap. By guarding connect, TTFT, and inter-token gap separately, you catch every real failure mode, dead connection, stuck queue, silent stream, without ever killing a long answer that's still actively streaming tokens. That's the whole trick.

One more nuance worth wiring in: timeouts should be coordinated down the call stack, not set independently at each layer. If your browser waits 30 seconds, your gateway waits 30 seconds, and your backend waits 30 seconds, then on a slow request all three are blocked for the full duration and a retry can't fire until the outermost layer gives up. The standard fix is a shrinking budget, the outer layer allows the most time, and each inner layer gets a slightly smaller slice, so the inner call fails and surfaces a clean error before the outer one times out. Pair that with a deadline propagated through the request (a header carrying "you have N milliseconds left") and every layer can make a local decision to fail fast instead of doing work whose result will already have been abandoned upstream. Uncoordinated timeouts are how a single slow request ties up capacity at three layers at once.

Wire the phase timeouts into the client

async def call_with_phase_timeouts(client, body, t):
    # 1. connect timeout
    resp = await asyncio.wait_for(client.open(body), timeout=t["connect_ms"]/1000)
    # 2. first-token timeout
    first = await asyncio.wait_for(resp.next_token(), timeout=t["ttft_ms"]/1000)
    yield first
    # 3. inter-token timeout: reset the clock on every token
    while True:
        try:
            tok = await asyncio.wait_for(resp.next_token(),
                                         timeout=t["inter_token_ms"]/1000)
        except asyncio.TimeoutError:
            raise StreamStalled("no token within inter-token budget")
        if tok is None: break    # clean end of stream
        yield tok                # long answers stream freely, never capped

Timeouts, retries, and circuit breakers are one system

A timeout in isolation is half a decision. The other half is what happens when it fires, and that's where timeouts, retries, and circuit breakers stop being three separate features and become one coupled system that you have to tune together. A timeout that fires hands control to your retry logic; a retry that keeps failing should feed your circuit breaker; a breaker that opens should change whether a timeout even gets a chance to fire. Tune any one of them without the others and you get pathological interactions.

The classic failure: an aggressive timeout paired with eager retries. Each request times out at 8 seconds, immediately retries, times out again, retries again. You've turned one slow request into three timed-out requests, tripling the load on an already-slow provider, a retry storm born from a timeout that was set too tight. The opposite pairing is just as bad: a generous timeout with no circuit breaker means doomed requests dwell for the full duration with nothing to cut them off, and your pool exhausts before anything trips.

The coherent design treats them as a budget. Your total time budget for a user-facing request is fixed, say, 25 seconds before the user gives up. Within that budget you allocate: a per-attempt timeout derived from your p99.9, room for one or two retries with jittered backoff, and a circuit breaker that short-circuits the whole thing if the dependency is already known-bad. The per-attempt timeout times the max attempts plus the backoff must fit inside the total budget, otherwise your last retry fires after the user has already left. Compute it backward from the deadline, don't stack three independent numbers and hope they add up.

# Timeouts, retries, and the deadline are ONE budget, not three knobs
def attempt_plan(total_budget_ms, p999_ttft_ms, max_retries=2):
    # Reserve backoff time; the rest is split across attempts
    backoff_ms = sum(min(4000,250 * 2 ** i) for i in range(max_retries))
    per_attempt = (total_budget_ms - backoff_ms) / (max_retries + 1)
    # A per-attempt timeout below p99.9 will false-kill good requests
    if per_attempt < p999_ttft_ms:
        raise ValueError("budget too tight: retries won't fit before deadline")
    return {"per_attempt_ms": int(per_attempt),
            "max_retries": max_retries, "backoff_ms": backoff_ms}

# 25s budget, 6.2s p99.9 TTFT -> ~7s per attempt, 2 retries fit cleanly
print(attempt_plan(25_000,6_200))

Load-test the timeouts, because the right number changes under load

The catch that makes this a testing problem and not just a config problem: your latency distribution is not stationary. TTFT and inter-token gaps stretch under load as provider queueing kicks in. A timeout derived from 1x-load samples will start false-killing requests at 5x load, when the tail fattens. You have to measure the distribution at the load you'll actually run, and re-verify your timeouts hold there.

# Validate timeouts across the load range, the tail moves with load
async def timeout_validation(timeouts):
    for mult in (1,3,5):
        dist = await drive_load(rps_multiplier=mult, duration=120,
                                phase_timeouts=timeouts)
        # False kills: requests timed out that were still streaming
        assert dist.false_timeout_rate < 0.005, \
            f"timeouts too tight at {mult}x, killing live streams"
        # Doomed-request dwell: timed-out requests must exit fast
        assert dist.timeout_p99_ms < timeouts["ttft_ms"] * 1.2, \
            f"doomed requests dwelling too long at {mult}x, pool risk"
        print(f"{mult}x load: false-kill {dist.false_timeout_rate*100:.2f}%  "
              f"pool peak {dist.pool_peak*100:.0f}%")

The bottom line

A single LLM timeout is wrong in both directions: short enough to kill long valid answers, long enough to let a slowdown exhaust your pool. Replace it with three phase timeouts, connect, time-to-first-token, and inter-token gap, each derived from the p99 of your measured distribution under realistic load, with no total-duration cap to punish long outputs. Then validate them across your load range, because the latency tail fattens as you scale and a timeout tuned at 1x betrays you at 5x. Timeouts aren't a config you set once; they're a property of your latency distribution, and that distribution moves.