Inter-Token Latency: The Stutter Your Users Feel But You Never Measure

The complaint that has no row in your logs

You have seen the ticket. "The assistant feels slow / laggy / janky." You check the dashboard: TTFT is healthy, error rate is zero, p99 request duration is within budget. You cannot reproduce it. You close it as "works on my machine" and it comes back next week from a different user.

What that user felt has a name and a number, and your tooling never recorded it. After the first token appears, the response streams token by token. If those tokens arrive at an even 40ms apart, the text flows like someone typing fast and confident. If they arrive at an uneven 40ms, then 90ms, then a 300ms hitch, then 60ms, the eye catches every stall. That irregular cadence, inter-token latency, is the stutter. It is perfectly compatible with a great TTFT and a fine total duration, which is exactly why it survives every load test built for stateless REST.

What inter-token latency actually is

Inter-token latency is the wall-clock gap between the model emitting one token and emitting the next. Its inverse, aggregated, is your throughput in tokens per second. The closely related average is Time Per Output Token (TPOT), and NVIDIA's inference team treats TTFT and TPOT as the two latencies that together define a serving SLA, TTFT for the prefill phase, TPOT for the decode phase, as laid out in their LLM inference benchmarking fundamentals. As the LLM Inference Handbook notes, there is a subtle distinction worth keeping straight: ITL strictly measures the time between two consecutive tokens, while TPOT is usually computed as an end-to-end average; some benchmark tools use them interchangeably. The useful identity is simple:

end_to_end_latency  =  TTFT  +  (TPOT × output_tokens)
tokens_per_second   =  1000 / TPOT_ms        # steady-state decode rate

That second line is the one that matters for UX. If your average TPOT is 40ms, you stream at 25 tok/s. If a noisy neighbor on the same GPU pushes your TPOT to 150ms under load, you have collapsed to under 7 tok/s, and the user watches the words crawl out. The average request duration may barely move, because the model still finishes; it just finished while the human was staring at a stalling cursor. A TPOT of 100ms, as the same handbook spells out, means roughly 10 tokens/sec, or about 450 words per minute of output.

The number that anchors everything: human reading speed

To know whether a token rate is "fast enough, " you need the rate humans actually consume text. Decades of reading research, summarized in Marc Brysbaert's widely cited meta-analysis of silent reading rates, put comfortable adult reading of non-fiction at roughly 238 words per minute, call it about 4 words per second. Because a token averages around 0.75 words for English, that maps to a comfortable consumption rate near 5-6 tokens per second, with fast readers pushing past 8.

Practitioner benchmarks land in exactly the same place. A 2026 tokens-per-second guide observes that you only need around 10 tok/s to keep up with reading, that above roughly 50 tok/s a model is indistinguishable from any faster one for a reading task, and that the pain starts below 8 tok/s, where responses feel like they are being typed out slowly, with individual words appearing with noticeable gaps below 5 tok/s. That gives you a hard floor and a target band:

• Below ~5 tok/s, the user reads faster than you generate. They wait between words. This is where "laggy" lives.
• ~7-15 tok/s, comfortable. The text stays slightly ahead of the eye; the stream feels alive.
• Above ~20-50 tok/s, the text outruns reading. Extra speed is invisible to a human reader (though it still helps tools, agents, and downstream parsing).

The practical conclusion is uncomfortable for anyone optimizing only raw throughput: past about 50 tok/s, a human reading the answer cannot tell the difference between fast and faster. But they absolutely notice a hitch, a single 400ms gap in an otherwise smooth stream reads as a glitch, the same way a dropped frame in 60fps video is more jarring than a steady 30fps. Smoothness beats peak speed.

Where the stutter comes from under load

On a single warm request, decode is metronome-steady. The stutter is born under concurrency, and the causes are mechanical:

Continuous batching contention

Modern servers like vLLM and TGI use continuous (in-flight) batching: many users' decode steps share each forward pass on the GPU. When a burst of new requests joins the batch, the per-token compute for everyone already in the batch goes up, so everyone's TPOT rises together. Your stream stutters because someone else hit submit. This is the central trade-off documented across vLLM's serving guidance: higher batch size lifts aggregate throughput but widens per-user inter-token latency. As the inference metrics handbook puts it, even when overall tokens/sec is high, occasional spikes in TPOT cause choppy output.

Network and proxy buffering

Even if the GPU emits tokens evenly, a proxy, load balancer, or CDN that buffers Server-Sent Events will deliver them in clumps. The model streamed smoothly; your edge re-chunked it into bursts. The user sees four words appear at once, then a pause, then four more, pure ITL artifact introduced after generation.

Token-by-token vs. chunked delivery

Some gateways flush on a timer or a byte threshold rather than per token, trading a smooth visual stream for fewer writes. Cheaper for you, choppier for them.

Speculative decoding and its variance tax

One subtlety worth knowing: speculative decoding, where a small draft model proposes several tokens that the big model verifies in one pass, raises average throughput but can increase variance. When the draft guesses well, several tokens land at once; when it guesses badly, you fall back to slow single-token decode. The result is a stream that is faster on average and bumpier in cadence, which is precisely the trade-off that looks great on a tokens/sec dashboard and worse to a human eye. This is the clearest illustration of why a single average throughput number is not enough: two systems with identical mean tok/s can feel completely different depending on the shape of their gap distribution. You have to measure the distribution, not the headline rate.

Measuring inter-token latency correctly

You cannot get ITL by awaiting the response body, that throws away every intermediate timestamp. You have to read the stream and record the arrival time of each chunk. In the browser, fetch() exposes the body as a ReadableStream, so you can capture the full gap distribution with no backend:

// Capture the inter-token gap distribution from a streaming endpoint
async function measureITL(url, body, headers) {
  const t0 = performance.now();
  const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body) });
  const reader = res.body.getReader();
  const dec = new TextDecoder();

  let ttft = null, last = t0, gaps = [], tokens = 0;
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    const now = performance.now();
    // count SSE token deltas in this chunk
    const deltas = dec.decode(value, { stream: true })
      .split('\n')
      .filter(l => l.startsWith('data:') && !l.includes('[DONE]'));
    for (const _ of deltas) {
      if (ttft === null) ttft = now - t0;     // first token
      else gaps.push(now - last);             // one inter-token gap
      last = now;
      tokens++;
    }
  }
  return { ttft_ms: ttft, gaps_ms: gaps, tokens };
}

The gaps array is the whole story. Summarize it as a distribution, not an average, and compare it against the human reading floor:

import numpy as np

def itl_report(gaps_ms, tokens, total_s):
    p50, p95, p99 = np.percentile(gaps_ms, [50,95,99])
    tps = tokens / total_s
    stalls = np.mean(np.array(gaps_ms) > 200) * 100   # human-visible hitches
    print(f"ITL  p50 {p50:.0f}ms - p95 {p95:.0f}ms - p99 {p99:.0f}ms")
    print(f"Throughput {tps:.1f} tok/s   (reading floor ~5-6 tok/s)")
    print(f"{stalls:.1f}% of gaps exceed 200ms (visible stutter)")
    # Smoothness: a stream is 'janky' if p99 gap is many times the median
    jank = p99 / max(p50,1)
    print("SMOOTH" if jank < 4 and tps > 7 else "JANKY")

The jank ratio is the trick most teams miss. A stream where p99 is 3x the median feels even; a stream where p99 is 12x the median feels broken at exactly the same average throughput. Report the ratio, not just the mean.

Turn smoothness into a CI gate

"It feels laggy" is unactionable; a failing assertion is not. Once you can measure the gap distribution under realistic concurrency, you can gate on it the way you gate a unit test:

assert tps_p50      > 10,  f"Decode throughput below comfort: {tps_p50} tok/s"
assert itl_p99_ms   < 250, f"Inter-token p99 stutter: {itl_p99_ms}ms"
assert jank_ratio   < 5,   f"Stream is uneven: p99/p50 = {jank_ratio}x"

Now a model swap that quietly halves decode speed, a batching config that boosts throughput at the cost of per-user smoothness, or a new edge proxy that re-buffers SSE all turn into a red build, before a single user files an unreproducible "feels janky" ticket. That is the entire point: move the discovery from the support queue to the pull request.

The bottom line

TTFT decides whether the user waits. Inter-token latency decides whether they enjoy what comes next. It is bounded below by how fast humans read, roughly 5-6 tok/s, bounded above by the point where extra speed is invisible, and ruined by uneven gaps that no average will ever show you. Measure the gap distribution under load, watch the p99 and the jank ratio rather than the mean, and gate on smoothness. Do that and the "it feels laggy" tickets stop arriving mysteriously, because you caught the stutter before it shipped.