When the Stream Drops at Token 500: Streaming Failures Under Load

The 200 that wasn't

Streaming responses don't fail the way REST responses do. A REST call either returns a complete body or throws. A streamed completion opens a connection, returns a 200 almost immediately, and then delivers tokens over several seconds. If the connection dies after the headers and the first few chunks, from the server's point of view the request succeeded. From the user's point of view, the assistant stopped talking mid-sentence.

This is the most under-tested failure mode in AI applications, and it's structural: with Server-Sent Events and chunked transfer the status code is set before the payload finishes, so any tool that keys on status codes, including most load testers and most uptime monitors, reports green while users get truncated, broken answers. The streamed body is delivered as a ReadableStream, which is exactly what lets you detect the failure if you read it to completion instead of awaiting the whole response.

The failure modes hiding inside a stream

• Mid-stream connection drop. The classic: partial response, 200 status, silent truncation.
• Idle-timeout severance. A proxy or load balancer with a 30s idle timeout kills a stream that pauses between tokens, common when the model "thinks" mid-generation.
• Malformed SSE. Under load, partial or interleaved data: frames break naïve client parsers, which silently drop tokens or crash the render.
• Backpressure stalls. A slow client that can't drain the stream causes server-side buffering, which under concurrency starves other connections.

Measure stream-completion rate, not just status

The metric conventional tools don't give you is the one that matters: of the streams that started, what fraction finished? You measure it by reading to the end and checking for the terminal event:

async function runStream(url, body, headers) {
  const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body) });
  if (!res.ok) return { ok: false, reason: 'http_' + res.status };

  const reader = res.body.getReader();
  let chunks = 0, sawDone = false;
  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      chunks++;
      if (isTerminalSSE(value)) sawDone = true;   // e.g. `data: [DONE]`
    }
  } catch (e) {
    return { ok: false, reason: 'mid_stream_drop', chunks };  // ← the silent one
  }
  // A "successful" HTTP request with no terminal event = truncated answer
  return { ok: sawDone, reason: sawDone ? 'complete' : 'truncated_no_done', chunks };
}

# Aggregate across a concurrent burst
def stream_health(results):
    n = len(results)
    complete = sum(r['ok'] for r in results)
    drops    = sum(r['reason'] == 'mid_stream_drop' for r in results)
    trunc    = sum(r['reason'] == 'truncated_no_done' for r in results)
    print(f"stream-completion rate: {complete/n*100:.1f}%")
    print(f"mid-stream drops: {drops}  ·  silent truncations: {trunc}")
    assert complete/n > 0.995, "stream completion below 99.5% under load"

Run that loop at 1x, 3x, and 5x expected concurrency and you'll see the completion rate degrade long before status codes do. That degradation curve is the early warning conventional dashboards can't show you.

Design the client to survive the drop

Detection is half the job; graceful recovery is the other half. A resilient streaming client retries with backoff, preserves what it already received, and never renders a truncated answer as if it were final:

async def stream_with_resilience(endpoint, prompt, max_retries=3):
    received = []
    for attempt in range(max_retries):
        try:
            async for tok in endpoint.stream(prompt):
                received.append(tok)
            return "".join(received), "complete"
        except ConnectionError:
            if attempt < max_retries - 1:
                await asyncio.sleep(2 ** attempt)   # exponential backoff
                continue
            return "".join(received), "best_effort"   # mark it, don't fake it

The key discipline: a best-effort partial response must be labeled as partial, so the UI can offer a retry instead of presenting half a sentence as the authoritative answer.

The bottom line

If your AI feature streams, "the request succeeded" is not the same as "the user got a complete answer." Load-test the stream, not the status code. Track stream-completion rate as a first-class SLO, exercise it at 3-5x concurrency, and build clients that detect drops, recover with backoff, and never pass off a truncated response as the real thing.