Voice AI Has a 300ms Budget. Here's How to Spend It Under Load

Silence is the failure mode

Text AI can hide latency behind a typing indicator and a streaming cursor. Voice has nowhere to hide. When a person stops speaking and the assistant does not respond within a few hundred milliseconds, the human brain, wired by a lifetime of conversation, registers the silence as something is wrong. The user starts to repeat themselves, talks over the response, or simply concludes the thing is dumb. There is no spinner in a phone call. The latency is the product.

The benchmark is not arbitrary. Linguists studying turn-taking across ten languages found that the median gap between conversational turns is around 200ms and the cross-linguistic pattern is strikingly stable, as documented in Stivers et al.'s widely cited PNAS study on universals and cultural variation in turn-taking. People plan their reply while the other person is still talking and launch it almost immediately. Your voice agent cannot plan ahead, and it has to do speech recognition, reasoning, and speech synthesis in the gap a human fills with nothing. That is the difficulty in one sentence.

The four-stage budget

A conversational voice pipeline is serial by nature. The realtime voice literature converges on a familiar breakdown of the "user stops talking to user hears reply" budget; one representative allocation from a voice AI pipeline latency analysis puts network at ~50ms, VAD/turn-taking around 200ms, LLM TTFT ~250ms, and TTS first audio ~100ms, totaling roughly 600ms for a human-feeling agent. Here is a realistic allocation for a sub-800ms target:

Stage                         Target     Notes
-----------------------------------------------------------------
End-of-speech detection (VAD)  ~50ms     deciding the user actually stopped
Speech-to-text (final)         ~150ms    streaming STT, partials already in
Network + orchestration        ~50ms     hops between services
LLM time-to-first-token       ~250ms    the big one; prefill-bound
Text-to-speech (first audio)  ~150ms    time to first audio chunk, not full
Playback buffer / jitter       ~50ms     client-side
-----------------------------------------------------------------
TOTAL to first audio out      ~700ms    inside the 'feels natural' band

Notice what dominates: LLM time-to-first-token and TTS time-to-first-audio. Those are the two levers worth the most. Notice also what is missing from the naive mental model, VAD endpointing and the playback buffer are real, non-negotiable costs that teams routinely forget to budget, and they alone can eat 100ms before any AI runs.

How to spend the budget

Stream everything; await nothing

The single biggest win is refusing to wait for any stage to finish. As a 2025 breakdown of voice AI latency quantifies, streaming STT can begin before the user finishes speaking (saving 100-200ms), streaming TTS can start playing before full synthesis (saving 200-400ms), and feeding the LLM's first sentence into TTS while generation continues yields combined savings of 300-600ms. Stream partial transcripts into the LLM as the user speaks. Stream the LLM's first sentence into TTS before the full answer is generated. Stream the first audio chunk to the user before the full sentence is synthesized. The pipeline should be a set of overlapping streams, not a relay race of complete handoffs.

Endpoint aggressively but smartly

Voice activity detection decides when the user has stopped. Too eager and you cut them off mid-thought; too patient and you add dead air to every turn. Semantic / smart endpointing, using a small model to predict whether an utterance is actually complete, lets you respond fast on clear sentence endings without clipping someone who paused to think.

Handle barge-in as a latency feature

Real conversations are full of interruptions, and a voice agent that cannot be cut off feels robotic no matter how fast it starts. Barge-in handling is itself a latency budget: when the user starts talking over the agent, you need to detect speech, stop TTS within a single audio chunk, discard whatever the LLM was about to say, and start a fresh STT stream, ideally in under 100ms. An agent that keeps talking for half a second after the user interrupts feels deaf, and that perceived rudeness is a latency failure as surely as dead air is. Budget for the interruption path, not just the happy-path turn.

Pick the right models, not the biggest

A 250ms LLM TTFT budget rules out the slowest frontier models for the conversational path. Many production voice stacks route to fast, smaller models for the turn-by-turn dialogue and reserve heavyweight reasoning for explicit, asynchronous "let me look that up" moments, with a filler phrase ("one sec, checking that") covering the longer latency, exactly as a human would.

The part everyone skips: this all falls apart under load

Every budget above is a single-call, warm-system budget. The failure that ships is testing one happy-path call, hearing it feel snappy, and shipping. Then fifty concurrent calls arrive and every shared stage degrades at once:

• STT queues; final transcripts arrive later.
• The LLM batches more requests, so TTFT and per-token time both rise, the prefill/decode trade-off NVIDIA documents for inference serving applies directly here.
• TTS contends for its own GPUs and slows its time-to-first-audio.
• Tails compound. Because the stages are serial, the worst case is the sum of each stage's tail, not the max. A p99 STT plus a p99 LLM plus a p99 TTS can blow a budget that every individual p50 met comfortably.

This is why a voice product can demo flawlessly and then feel broken in production: the demo was n=1, and the budget is a tail property under concurrency. The only honest test is a concurrent one that measures end-to-end first-audio latency as a distribution.

Measuring the real budget under load

You measure what the user hears: time from end-of-speech to first audio out, end to end, under concurrency. Instrument each stage so you know which one ate the budget when a turn goes long:

# Per-turn voice latency budget tracker (server-side spans)
import time

class TurnBudget:
    def __init__(self):  self.t = {}
    def mark(self, label): self.t[label] = time.perf_counter()
    def report(self, budget_ms=800):
        eos   = self.t['eos']                       # user stopped speaking
        spans = {
          'stt':   self.t['stt_final'] - eos,
          'llm':   self.t['llm_first_token'] - self.t['stt_final'],
          'tts':   self.t['tts_first_audio'] - self.t['llm_first_token'],
        }
        total = (self.t['tts_first_audio'] - eos) * 1000
        for k, v in spans.items():
            print(f"  {k:4} {v*1000:6.0f} ms")
        print(f"  TOTAL {total:6.0f} ms  budget {budget_ms} ms  "
              + ("OK" if total <= budget_ms else "OVER"))
        return total

Then aggregate first-audio latency across a concurrent run into percentiles and check the tail against the budget, because the median feeling natural while the p95 feels broken is the most common, most expensive voice failure:

import numpy as np

def voice_load_report(first_audio_ms, budget_ms=800):
    p50, p95, p99 = np.percentile(first_audio_ms, [50,95,99])
    print(f"first-audio  p50 {p50:.0f}  p95 {p95:.0f}  p99 {p99:.0f} ms")
    over = np.mean(np.array(first_audio_ms) > budget_ms) * 100
    print(f"{over:.1f}% of turns exceed the {budget_ms}ms budget")
    # Natural conversation needs the BULK of turns well under budget
    print("PASS" if p95 <= budget_ms else "FAIL, tail breaks the conversation")

Gate on the turn budget

Make the voice budget a release gate, evaluated under concurrency rather than on a single warm call:

assert p95_first_audio_ms < 800,  f"voice p95 over budget: {p95_first_audio_ms}ms"
assert p99_first_audio_ms < 1200, f"voice p99 tail breaks turns: {p99_first_audio_ms}ms"
assert llm_ttft_p95_ms    < 350,  "LLM stage eating the conversational budget"

Now a model swap with a slower cold path, a TTS provider change, an endpointing tweak that adds dead air, or a concurrency level your pipeline cannot sustain all surface as a red build, before a customer sits through three seconds of dead air and hangs up.

The bottom line

Human conversation runs on a ~200ms turn-taking gap, and a voice AI has to fake that with a serial STT-to-LLM-to-TTS pipeline inside a roughly 500-800ms budget to first audio. The way to spend it is to stream every stage instead of awaiting it, endpoint smartly, route to fast models for the dialogue path, and use natural filler to cover the rest. But the budget is a tail property under load, the stages are serial, so their tails sum, which means the only test that counts is a concurrent one measuring end-to-end first-audio latency as a distribution. Gate on that, and your voice product stops feeling broken the moment it gets popular.