Load Testing the Realtime API: WebSockets Break Differently

A different shape of failure

OpenAI's Realtime API, generally available in 2025, enables low-latency speech-to-speech and multimodal interaction over a persistent connection, WebSocket for server-to-server, WebRTC for browser clients. Unlike a chat completion, which opens a connection, does its work, and closes, a realtime session stays open for the entire conversation: audio streams in, the model streams audio back, and the socket lives for the whole call, frequently many minutes.

That single architectural difference changes everything about how the feature scales and fails. A REST endpoint's capacity is measured in requests per second, and connections are ephemeral, a server handling 1,000 rps might have only a few dozen connections open at any instant. A realtime endpoint's capacity is measured in simultaneous open connections, and each one is sticky. Ten thousand concurrent voice users means ten thousand live sockets, ten thousand session buffers, and ten thousand streams competing for real-time GPU inference, all at once, all held for minutes.

Where the connection ceiling actually lives

The limit isn't a single number; it's the lowest of several ceilings stacked across your path, and you hit whichever is smallest first.

• Provider session limits. As of February 2025 OpenAI no longer caps simultaneous sessions (it previously limited Tier 5 to about 100), but per-minute token caps and media-bitrate norms still apply, and sessions have historically had maximum durations. A long call can hit a duration cap and drop mid-conversation.
• Your gateway / proxy. Load balancers, reverse proxies, and API gateways all have max-connection and file-descriptor limits, plus idle-timeout settings that can sever a socket during a natural pause. Defaults are frequently far below what a realtime feature needs.
• Your application server. Each open WebSocket consumes a file descriptor, an event-loop slot, and memory for session state and audio buffers. Servers tuned for short HTTP requests run out of file descriptors or memory long before CPU.
• GPU streaming capacity. Real-time audio generation needs sustained, low-jitter inference for every active session at once. Past a certain concurrency the model can't keep every stream real-time, and audio starts to stutter or lag, a quality failure, not an error, so monitoring stays green while users hear garbage. OpenAI's writeup on delivering low-latency voice AI at scale underscores how hard sustained real-time streaming is at high concurrency.

Why your existing load tests can't see it

Here's the practical trap. The overwhelming majority of load-testing tooling is built on the request/response model: fire a request, wait for a response, measure it, repeat. That model literally cannot express the realtime workload, which is "open a bidirectional stream, send audio frames continuously, receive audio frames continuously, hold for minutes, and measure jitter the whole time." A tool that can't hold thousands of stateful WebSocket sessions open while streaming both directions can't fire this test at all, so teams skip it, ship the feature, and learn their concurrent-connection ceiling from production. (Note too that for browser media OpenAI recommends WebRTC over raw WebSocket, your test client should match the transport real users use.)

# Why connection count, not request rate, is the binding constraint
def concurrent_connections(active_users):
    # Every active voice user holds ONE socket for the whole call
    return active_users   # not divided by anything, all open at once

def fd_headroom(open_conns, fds_per_conn, server_fd_limit):
    needed = open_conns * fds_per_conn
    print(f"{open_conns:, } sessions need ~{needed:, } FDs "
          f"vs ulimit {server_fd_limit:, }")
    if needed > server_fd_limit:
        print("=> file-descriptor exhaustion; new connections refused")

fd_headroom(open_conns=8000, fds_per_conn=2, server_fd_limit=1024)
# 8,000 sessions need ~16,000 FDs vs ulimit 1,024
# => file-descriptor exhaustion; new connections refused
# (default ulimit kills you long before CPU does.)

Load-test the connections, not the requests

The test you need ramps the number of concurrent, persistent, full-duplex sessions and watches for the connection ceiling and the quality cliff, not throughput. Hold each session open for a realistic duration, stream audio frames continuously, and measure both whether new connections are accepted and whether existing ones stay real-time.

# Ramp concurrent realtime sessions; find the ceiling and the quality cliff
import asyncio, time

async def realtime_session(url, duration_s, jitter_log):
    ws = await connect_ws(url)              # open + hold a full-duplex socket
    if ws is None:
        return {"connected": False}         # refused = we hit a ceiling
    last = time.time()
    try:
        async for frame in stream_audio(ws, duration_s):  # send + receive
            now = time.time()
            jitter_log.append((now - last) * 1000)        # inter-frame gap ms
            last = now
        return {"connected": True, "completed": True}
    except (ConnectionClosed, asyncio.TimeoutError):
        return {"connected": True, "completed": False}     # dropped mid-call

async def ramp(url, steps=(500,2000,5000,8000,12000), hold_s=180):
    for n in steps:
        jitter = []
        sessions = [realtime_session(url, hold_s, jitter) for _ in range(n)]
        res = await asyncio.gather(*sessions)
        accepted = sum(r["connected"] for r in res) / n
        completed = sum(r.get("completed") for r in res) / n
        p99_jitter = sorted(jitter)[int(len(jitter)*0.99)] if jitter else 0
        print(f"{n:>6} concurrent | accepted {accepted*100:5.1f}% | "
              f"completed {completed*100:5.1f}% | p99 jitter {p99_jitter:.0f}ms")
        # Two distinct failures to catch:
        assert accepted > 0.99, f"connection ceiling hit at {n} sessions"
        assert p99_jitter < 200, f"audio quality cliff at {n} sessions"

The two assertions catch the two distinct realtime failures. The accepted rate catches the hard ceiling, the point where new sockets are refused because something ran out of connections, file descriptors, or session slots. The jitter assertion catches the soft cliff, the point where the GPU can't keep every stream real-time and audio starts lagging, which never shows up as an error code but makes the feature unusable.

There's a third failure the ramp surfaces that a steady-state test never will: behavior at the connection-establishment rate, not just the steady connection count. A realtime feature that comfortably holds 8,000 open sockets can still fall over if 8,000 users try to connect in the same ten-second window, a launch, a push notification, the top of the hour for a scheduled event. WebSocket and WebRTC handshakes are expensive (TLS negotiation, session setup, the model allocating buffers), so the connect rate has its own ceiling distinct from the connection count. Ramp the arrival rate as a separate dimension, hold the count steady while spiking new connections per second, and you'll find whether your stack can absorb a synchronized arrival surge or whether the handshake path becomes the bottleneck. The two ceilings fail differently and need separate headroom.

The metrics a request dashboard can't show you

Even teams that do load-test realtime features often measure the wrong things, because their observability was built for request/response traffic. A standard dashboard shows requests per second, error rate, and request-duration percentiles, all of which are nearly meaningless for a long-lived bidirectional stream. You need a different metric vocabulary, and most of it has to be instrumented deliberately because nothing reports it by default.

Concurrent open connections is the headline number, the actual count of live sockets right now, sampled continuously, because this is the resource that hits a ceiling. Connection duration distribution tells you how long sessions last, which drives your capacity math: a feature with 2-minute calls and one with 20-minute calls have wildly different concurrent-connection footprints at the same signup rate. Inter-frame jitter (the variance in audio-frame arrival timing) is your real-time quality SLO, it's the thing that degrades silently before connections start getting refused. New-connection rate and connection-failure rate separate the establishment ceiling from the steady-state ceiling.

And one realtime-specific failure deserves its own alert: the half-open or zombie connection. A socket whose TCP state says "open" but across which no frames are flowing is worse than a closed one, it's holding a slot, a buffer, and a session against your ceiling while doing nothing. Under load these accumulate, eating your concurrent-connection budget invisibly. Track the gap between "sockets open" and "sockets actively streaming"; a widening gap is zombie buildup, and it's a leak that an HTTP-shaped dashboard will never surface.

Engineering for the connection ceiling

Once you know your real ceilings, the fixes are concrete. Raise file-descriptor limits and tune your event-loop server for many idle-but-open connections rather than many fast requests. Set WebSocket idle timeouts long enough to survive natural conversational pauses, with application-level keepalive pings to distinguish a thinking user from a dead socket. Pre-provision GPU streaming capacity against your concurrent-session projection, not your request rate. Add admission control so that when you approach the connection ceiling you reject new sessions cleanly, with a "lines are busy, try again" message, instead of accepting them and degrading every existing call's audio. And monitor concurrent open connections and audio jitter as first-class metrics, because neither shows up in a request-rate dashboard.

The bottom line

Realtime and voice features fail on a different axis than everything else you've load-tested. The binding constraint is concurrent open connections, sticky, minutes-long, full-duplex, not request throughput, and the ceiling is the lowest of your provider's per-minute caps, your gateway, your file descriptors, and your GPU's real-time streaming capacity. Worse, the quality failure (audio jitter) arrives before the hard failure (refused connections) and never trips an error code. Standard request/response load tools can't even fire this test. Ramp real concurrent sessions, hold them, stream both directions, and assert on both connection acceptance and jitter, that's the only way to know how many users your realtime feature can actually hold at once.