Context Window Exhaustion: The Failure That Only Shows Up Under Load

The window you bought is not the window you have

Every provider sells context window the way a hard-drive maker sells capacity: one big number on the box. Claude ships a 200K-token window with a 1M tier, Google's Gemini line reaches 1M-2M tokens, and OpenAI's GPT-4.1 family advertises 1M. The implicit promise is that you can fill that window and the model will use all of it equally well.

It won't. In July 2025 Chroma published Context Rot: How Increasing Input Tokens Impacts LLM Performance (Hong, Troynikov, and Huber), a controlled study of 18 models. The headline finding: performance is not uniform across the context window. Even on trivial tasks, copying text, retrieving a single planted fact, reliability declines as input grows, and it declines long before the advertised maximum. Crucially, context rot is not the same as context overflow: overflow is hitting the hard token limit, while rot happens well before it. A model that handles a task perfectly at 100 tokens can fail the identical task at 1,000, despite supporting a million-token window.

This builds on the still-load-bearing Lost in the Middle result from Liu et al.: models attend best to the beginning and end of a long context and worst to the middle, producing a U-shaped accuracy curve. Architecturally it's reinforced by position-embedding decay (RoPE), which biases attention toward the start and end of the sequence. Anthropic's own context-window docs describe the 200K standard window and 1M tier, but capacity and reliable recall are not the same number. Stack a long system prompt, a growing conversation, and a pile of retrieved documents, and the fact your answer depends on lands squarely in the low point of that curve.

Two failure modes, one root cause

Context exhaustion shows up as two distinct symptoms that share one cause, too many tokens in the request.

1. Silent quality decay

This is the dangerous one because nothing breaks. The request returns a 200, the answer is fluent and plausible, and it is subtly wrong: it ignores an instruction buried in the system prompt, contradicts something said earlier in the conversation, or hallucinates a detail that was actually present in a retrieved document the model under-attended to. Your error rate is zero. Your quality is quietly collapsing. No status code, no log line, no alert.

2. Hard context-length errors

This is the one that pages you. When the assembled prompt, system prompt + full conversation history + retrieved context + the user's new turn, exceeds the model's hard limit, the API rejects it outright. OpenAI returns a 400 with context_length_exceeded; other providers return analogous errors. The request never reaches the model. The feature returns an error to the user.

The cruel part is the ordering: quality decay arrives first and silently, then hard failures arrive at peak. A long-running session that has been gradually getting worse will, eventually, tip over the hard limit mid-conversation, exactly when the user is most invested.

Why load is the trigger

Token count per request is not constant. It grows along two axes, and load pushes both.

Conversation length grows within a session. Most chat implementations send the entire history on every turn. A 20-turn support conversation doesn't send the latest message, it sends all 20 turns plus the system prompt plus any tool outputs. Token count grows roughly linearly (often super-linearly once tool calls and retrieved documents accumulate) with conversation depth. The deepest, highest-value conversations sit closest to the ceiling.

Concurrency grows the population of long sessions. At launch or peak you don't just have more requests, you have more simultaneous long-lived sessions. If 2% of sessions reach 50 turns and you go from 100 to 10,000 concurrent sessions, your population of context-stressed requests grows 100x. The tail that was invisible in staging becomes your top incident driver in production.

# Model the token growth you'll actually see in production
def session_tokens(turns, sys_prompt=1200, avg_user=180,
                   avg_assistant=420, retrieved_per_turn=2500):
    # Most chat apps resend the full history every turn
    history = 0
    for _ in range(turns):
        history += avg_user + avg_assistant + retrieved_per_turn
    return sys_prompt + history

for t in (5,10,20,40):
    tk = session_tokens(t)
    print(f"{t:>3} turns -> {tk:>7, } tokens "
          f"({tk/200_000*100:5.1f}% of a 200K window)")
# 40 turns with RAG blows past 120K tokens, two-thirds of the window, 
# and Chroma's data says quality was already rotting at a fraction of that.

A 40-turn RAG conversation is already at two-thirds of a 200K window, and the Context Rot data says accuracy was degrading from a small fraction of that. You're paying for the full window and getting reliable behavior from a slice of it.

Load-test the context, not just the endpoint

Conventional load tests fire short, fixed prompts thousands of times. They measure throughput and latency beautifully and tell you nothing about context exhaustion, because every request is tiny. To find this failure you have to drive the test with contexts that grow the way real sessions do, and assert on quality, not just status.

import asyncio, random

async def simulate_session(client, model, max_turns):
    """Replay a growing conversation, tracking tokens + a quality probe."""
    history, results = [], []
    # Plant a fact early so we can test 'lost in the middle' under growth
    canary = f"The deployment code is ZEPHYR-{random.randint(1000,9999)}."
    history.append({"role": "system",
                    "content": f"Remember this exactly: {canary}"})

    for turn in range(max_turns):
        history.append({"role": "user",
                        "content": "Continue the analysis in detail." if turn else
                                   "Begin a long technical analysis."})
        try:
            resp = await client.chat(model=model, messages=history)
        except ContextLengthError:
            results.append({"turn": turn, "hard_fail": True}); break

        history.append({"role": "assistant", "content": resp.text})

        # Every few turns, probe whether the early canary survived
        if turn % 5 == 4:
            probe = history + [{"role": "user",
                                "content": "What is the deployment code?"}]
            ans = await client.chat(model=model, messages=probe)
            results.append({"turn": turn,
                            "tokens": resp.usage.prompt_tokens,
                            "canary_ok": canary.split()[-1] in ans.text})
    return results

# Fan out hundreds of growing sessions concurrently, the population
# of long sessions is what production actually looks like at peak.
async def run(concurrency=300):
    out = await asyncio.gather(*[
        simulate_session(client, "your-model", max_turns=random.randint(8,45))
        for _ in range(concurrency)])
    flat = [r for s in out for r in s]
    hard = sum(r.get("hard_fail") for r in flat)
    probes = [r for r in flat if "canary_ok" in r]
    recall = sum(r["canary_ok"] for r in probes) / max(len(probes), 1)
    print(f"hard context failures: {hard}")
    print(f"early-fact recall under growth: {recall*100:.1f}%")
    assert hard == 0, "context-length errors under concurrent long sessions"
    assert recall > 0.95, "fact recall rotted below 95% as context grew"

The two assertions catch the two failure modes. The hard_fail counter catches the 400-class exhaustion errors that page you. The canary recall catches the silent decay, if a fact planted at turn zero stops surviving by turn 30, you've quantified context rot for your prompt shape, not a benchmark's.

The cost dimension nobody connects to context

There's a second tax on growing context that has nothing to do with quality or hard failures: money. Because most chat implementations resend the full history on every turn, the input-token count of a conversation grows with its depth, and you pay for input tokens on every single turn. A 40-turn conversation doesn't cost 40 small requests; it costs the sum of 40 requests whose prompts get progressively larger, so the total token spend grows roughly quadratically with conversation length. The deepest conversations are both the most likely to hit the context ceiling and the most expensive per session, and the two curves climb together.

This couples context exhaustion directly to your bill. The same compaction that keeps you inside the usable window, summarizing old turns, trimming retrieved chunks, also caps the token growth that drives cost. Prompt caching, where the provider charges a reduced rate for repeated prefix tokens, helps for the stable system-prompt prefix but does nothing for the growing conversational tail. So the budgeting discipline below isn't only a reliability measure; it's a cost-control measure. A context-exhaustion load test that also records prompt-token count per turn gives you the cost curve for free, and that curve is usually steeper than anyone expected.

Engineering around the ceiling

Once you can see the cliff, you can build guardrails before it. None of these are exotic; what's missing in most stacks is the budget discipline to apply them before the request, not after it 400s.

Budget the window, don't fill it

Set an internal soft limit well below the hard limit, Chroma's data argues for treating maybe 40-50% of the advertised window as your reliable zone for accuracy-sensitive tasks. Track assembled prompt tokens before every call and act when you cross the soft limit instead of waiting for the provider to reject you.

SOFT_LIMIT = 90_000   # well under a 200K hard limit
HARD_LIMIT = 200_000

def assemble(system, history, retrieved, user_turn, count_tokens):
    prompt = [system, *history, *retrieved, user_turn]
    total = count_tokens(prompt)
    if total > SOFT_LIMIT:
        # Compress before the model ever sees an oversized prompt:
        history = summarize_old_turns(history)        # collapse stale turns
        retrieved = rerank_and_trim(retrieved, k=4)   # fewer, better chunks
        prompt = [system, *history, *retrieved, user_turn]
        total = count_tokens(prompt)
    assert total < HARD_LIMIT, "still oversized after compaction"
    return prompt, total

Keep critical instructions out of the middle

Given the Lost in the Middle curve, put non-negotiable instructions and the most relevant retrieved chunk near the end of the prompt, just before the user's turn, where recency attention is strongest. Re-stating a critical constraint right before generation is cheap insurance against it being lost in a long middle.

Summarize and roll the conversation

Don't resend 40 raw turns. Periodically compact older history into a running summary, keeping the last few turns verbatim. This caps token growth at a plateau instead of letting it climb linearly into the failure zone, and it directly slows the rot the canary test measures.

The bottom line

Context window is sold as one number and behaves like two: a hard limit that 400s your request at peak, and a much lower soft limit where quality silently rots. Both are reached faster under load, because conversations grow within sessions and long sessions multiply with concurrency. Test it the way it fails, drive growing, realistic contexts at production concurrency, assert on both hard failures and fact recall, and budget your prompt to stay inside the window you can actually trust, not the one printed on the box.