GPU Saturation: The Cliff Your Self-Hosted LLM Falls Off

The graph that goes vertical

If you plot latency against concurrency for a hosted API, you get a gentle upward curve. Plot the same thing for your own vLLM or TGI deployment and you get something different and alarming: a nearly flat line that, at some concurrency, turns almost vertical. One more concurrent request and p99 jumps from 300ms to 6 seconds. This isn’t gradual saturation, it’s a phase change. And the reason it’s so abrupt is specific to how LLM inference uses GPU memory.

The thing that runs out isn’t compute. It’s the KV cache: the per-request memory that stores the attention keys and values for every token in the context, growing as the request generates more tokens. The vLLM PagedAttention paper notes a 13B model can consume ~1.7GB of KV cache per request. GPU memory is fixed, so the number of requests you can hold in flight is fixed, and once you try to hold more, something has to give. What gives is everyone’s latency, all at once.

Continuous batching: the thing that makes it fast and the thing that hides the cliff

Continuous batching is why self-hosting can be cheap. Instead of static batches where every request waits for the slowest one to finish, the scheduler adds and evicts requests token-by-token, keeping the GPU saturated. Anyscale’s benchmark reported up to 23x throughput and lower p50 latency from this alone. PagedAttention makes it even better by eliminating KV-cache fragmentation, the vLLM blog reports traditional serving wastes 60-80% of memory to fragmentation and over-reservation, which PagedAttention cuts to under 4%, letting you batch far more requests into the same card.

But these wins all push you closer to the edge while making the approach feel safe. The more efficiently you pack the KV cache, the more headroom you appear to have, until you don’t. And because throughput keeps rising right up to the cliff, a load test that watches only throughput sees nothing wrong until the moment everything is wrong.

What actually happens at the cliff: preemption

When the KV cache fills and a new request needs space, the scheduler can’t just queue it cleanly, running requests are already consuming all the memory. So it preempts. vLLM handles this two ways, both costly: recomputation (throw away a request’s KV cache and recompute it later from scratch) or swapping (copy the KV cache out to CPU memory over the PCIe bus and back). Either way, the GPU now spends cycles shuffling state instead of generating tokens. Under sustained overload this becomes a storm: requests get preempted, recomputed, preempted again. Throughput doesn’t just plateau, it inverts, because the system is doing negative-value work. That inversion is the cliff.

Why your dashboards won’t warn you

GPU utilization, the metric most teams watch, is actively misleading here. A preemption storm keeps the GPU at 100% utilization; it’s busy, just busy doing recomputation instead of useful generation. So “GPU at 100%” looks identical whether you’re at peak goodput or in a death spiral. The signals that actually predict the cliff are different and rarely graphed:

• KV cache utilization (the fraction of cache blocks in use). This is your real fuel gauge. vLLM exposes it; most teams never look. When it approaches 100%, you are one request from preemption.
• Preemption count / recompute rate. The instant this leaves zero, you are over the cliff. It should be a paged alert, not a buried metric.
• Queue waiting time. Requests that can’t be admitted pile up; their wait time is pure TTFT inflation invisible to a model that only times generation.

Load-testing to the cliff, on purpose

The goal of the load test is not to confirm the system works at expected load, it’s to locate the cliff and measure how much headroom sits between it and your planned operating point. Ramp concurrency and watch for the inflection where latency goes vertical and preemption leaves zero:

async def find_the_cliff(base_url, concurrencies, prompt, max_tokens=256):
    for c in concurrencies:
        lat, ttft, ok = [], [], 0
        async def one():
            nonlocal ok
            t0 = time.perf_counter(); first = None
            async for chunk in stream(base_url, prompt, max_tokens):
                if first is None: first = time.perf_counter()
            lat.append((time.perf_counter() - t0) * 1000)
            ttft.append((first - t0) * 1000); ok += 1
        await gather_with_concurrency(c, [one() for _ in range(c * 4)])
        m = await scrape_metrics(base_url)          # vLLM /metrics
        cliff = " <== CLIFF" if m["num_preemptions"] > 0 else ""
        print(f"c={c:4d}  ttft_p99={percentile(ttft, 99):7.0f}ms  "
              f"e2e_p99={percentile(lat, 99):7.0f}ms  "
              f"kv_used={m['kv_cache_usage']:.0%}  preempt={m['num_preemptions']}{cliff}")

The scrape against vLLM’s Prometheus endpoint is what makes this honest, you’re not inferring the cliff from latency alone, you’re reading the cause directly:

async def scrape_metrics(base_url):
    text = await http_get(f"{base_url}/metrics")
    def g(name):
        for line in text.splitlines():
            if line.startswith(name): return float(line.split()[-1])
        return 0.0
    return {
        "kv_cache_usage":  g("vllm:gpu_cache_usage_perc"),     # the fuel gauge
        "num_preemptions": g("vllm:num_preemptions_total"),    # 0 = healthy
        "num_waiting":     g("vllm:num_requests_waiting"),     # queue depth
    }

Buying back headroom

Once the test shows where the cliff is, you have real levers to move it, and each is testable by re-running the ramp:

• Cap max_model_len. The KV cache is sized for the worst-case context. If you allow 128K context but real requests use 8K, you’re reserving cache for tokens that never arrive. Right-sizing the max context frees cache for more concurrency.
• Quantize the KV cache. FP8 KV cache roughly halves per-request memory, directly buying more concurrent slots, at a small, measurable quality cost you should evaluate, not assume.
• Set admission control. Better to reject (or queue with a fast 503) the request that would trigger a preemption storm than to let it degrade everyone. A clean rejection is recoverable; a storm is an outage.
• Scale horizontally before the cliff, not after. Autoscaling on GPU utilization is useless here (it’s pinned at 100% on both sides of the cliff). Scale on KV cache utilization and queue depth instead, the signals covered in cold-start latency in autoscaling LLMs.

Gate the cliff distance in CI

Self-hosted serving config drifts: a model swap changes per-request cache size, a context-length bump shrinks your slot count, a quantization revert doubles memory. Any of these can move the cliff under your planned load without anyone noticing. Gate on it:

Why the cliff moves under you

The cruelest part is that the cliff’s location is not fixed, it drifts with the shape of your traffic, which means a number you validated last month can be wrong today. KV cache consumption scales with context length, so a shift toward longer prompts (more RAG chunks, longer conversation history) shrinks the number of concurrent slots without any code change. A 70B model with 8K context needs roughly 20GB of cache per request; push that context to 32K and per-request cache can balloon toward 128GB at batch, per the memory math the production KV-cache analyses lay out. The same fleet that comfortably held 96 concurrent users on short prompts can fall off its cliff at 40 once your average context doubles. If your load test used short fixtures, it certified a cliff your real traffic will never see.

This also reframes how you read a preempted request’s cost. When vLLM swaps a long-context sequence out to CPU and back over a PCIe 4.0 link (~32 GB/s), restoring a single long-context 70B sequence can take hundreds of milliseconds, pure latency added to a request that was already in flight. Recomputation isn’t free either; it re-runs prefill from scratch. Both are why the cliff is a cliff and not a slope: past saturation the system isn’t just slower, it’s spending real wall-clock time doing work it already did. Note too that TGI entered maintenance mode in late 2025, with Hugging Face steering new deployments toward vLLM or SGLang, so the metrics surface you instrument against is increasingly vLLM’s, which makes its gpu_cache_usage_perc and num_preemptions_total the right things to gate on.

# CI: prove the cliff sits safely above planned peak concurrency
- name: GPU saturation gate
  run: |
    python find_cliff.py --url $STAGING_VLLM \
      --planned-peak 96 --ramp "32,64,96,128,160" \
      --require-zero-preemption-through 128 \
      --max-kv-usage-at-peak 0.85
    #   fails if any preemption occurs at or below 128, or KV usage > 85% at peak

Running this against your real deployment, with the real model, real context lengths, and real streaming, is the only way to get a trustworthy cliff location. Driving that concurrency from the browser against your authenticated endpoint, and reading both client-side latency and server-side KV/preemption metrics together, is the workflow Hit is built around: find the cliff before a traffic spike does.

The bottom line

Self-hosted LLMs fail like a phase change, not a slow fade. The cause is KV cache memory exhaustion; the symptom is a preemption storm where the GPU works at 100% while doing negative-value work; and the trap is that throughput and GPU utilization both look healthy right up to the edge. Watch KV cache utilization and preemption count, load-test to locate the cliff, operate well below it, and gate the cliff distance in CI. Do that and a traffic spike becomes a scaling event instead of an outage.