Testing AI Voice Assistants: Latency Budgets, ASR Compounding, and Real-Time Reliability

Voice agents are the second wave of production AI in 2026. Customer support, in-car assistants, voice-first apps, restaurant ordering, healthcare triage. The voice modality is genuinely different from chat, and it breaks in ways your existing test suite cannot catch.

This guide covers the testing problems unique to voice AI: the latency budget, the error-compounding chain, barge-in, silence detection, and the production failure modes nobody told you about.

The 1.2-second latency budget

Conversational research is unambiguous: a delay above 1.2 seconds in human-machine voice interaction crosses from "responsive" to "broken." Above 2 seconds, users assume the system has stopped working and start repeating themselves.

That 1.2 seconds has to fit:

1. Endpointing, detecting that the user has finished speaking (~150-300ms)
2. ASR (speech-to-text), transcribing the audio (~100-400ms with streaming ASR)
3. LLM TTFT, first token of the model response (~200-600ms)
4. TTS first audio chunk, synthesizing the first phoneme (~80-200ms with streaming TTS)
5. Network round-trips, typically 80-250ms total across all hops

If you have not measured each of these layers separately, you are flying blind. The test suite must include a latency-budget assertion per layer and a round-trip assertion across all of them.

ASR error compounding

Modern ASR (Whisper, Deepgram, AssemblyAI) achieves 3-6% word error rate (WER) on conversational English in clean conditions. That sounds great until you compose it with downstream steps.

If a 4% WER means the LLM gets the wrong noun in 4% of turns, and the LLM has its own 8% chance of misinterpreting a slightly-off prompt, the user sees the wrong response in roughly 12% of turns. With ambient noise, accents, or domain-specific vocabulary (drug names, ticker symbols, legal terms), WER climbs to 15-25% and the compounded failure rate becomes user-hostile.

Test for ASR-induced LLM failures explicitly:

# 1. Take a list of canonical user utterances.
# 2. Generate ASR-corrupted variants (substitute homophones, near-words).
# 3. Feed both to the agent. Assert outputs are equivalent.

corruptions = {
 "transfer to checking": ["transferred to checking", "transfer two checking"],
 "what's my balance": ["what's my balanced", "what's my Bowen's"],
 "cancel that order": ["cancel that ordered", "council that order"],
}

for canon, variants in corruptions.items():
 expected = run_agent(canon).intent
 for v in variants:
 actual = run_agent(v).intent
 assert actual == expected, f"ASR variant '{v}' broke the agent"

Build a corpus of ASR-typical errors from your actual production transcripts. Synthetic errors miss the patterns real users produce.

Barge-in: the test most teams skip

Barge-in is when the user starts talking while the agent is still speaking. Done well, it feels like talking to a human; done badly, it feels like a phone tree from 2003.

Barge-in failure modes:

• Agent keeps speaking, user keeps speaking, both layers garbage in
• Agent stops speaking but TTS audio keeps playing (buffer flush issue)
• Agent stops, but the LLM doesn't know it was interrupted, and continues with the original plan
• The interrupted utterance gets concatenated with the next turn's input

A correct barge-in test:

test('user interruption stops TTS within 200ms', async () => {
 const session = await openVoiceSession();
 await session.send(audio('What is my balance?'));
 await session.expectAgentSpeaking();
 await sleep(800); // agent is mid-sentence
 await session.send(audio('Actually, transfer money.'));
 const ttsStop = await session.waitForTtsStop();
 expect(ttsStop.timeFromUserStart).toBeLessThan(200);
 // Verify the agent processed the SECOND utterance, not concatenation
 const transcript = await session.getTranscript();
 expect(transcript[1].text).toMatch(/transfer money/i);
});

Silence detection and endpointing

Endpointing decides when the user has finished a turn. Two failure modes:

• False endpoint: agent jumps in during a natural pause inside a sentence ("My account number is... five-five-five..." [agent jumps in])
• Stuck endpoint: user has finished, agent waits 4 seconds for "more"

Test with audio fixtures that include realistic pauses:

Fixture	Expected endpoint
Disfluency: "I want to, um, transfer 500 dollars"	After "dollars", not after "to"
Mid-thought pause: "My number is 555... 1234"	After "1234", not after "555"
Trailing silence: "transfer 500 dollars" + 1.5s silence	≤300ms after speech ends
Background noise: speech + traffic noise	Endpoint on speech, not noise

Adversarial audio testing

Voice agents face audio inputs your test fixtures don't capture: distant microphones, packet loss, codec compression, simultaneous speakers, music playing in the background, child voices, regional accents, code-switching between languages.

The pragmatic approach: maintain a regression audio set of 200-500 real failure cases harvested from production. Every release runs against this set; any new failures block the deploy. This is your equivalent of a unit-test suite for the voice layer.

The full test pyramid for voice agents

1. Layer tests: ASR alone, LLM alone, TTS alone, endpointing alone. Latency and accuracy SLOs per layer.
2. Composition tests: STT → LLM (asserting the LLM is robust to ASR-typical noise). LLM → TTS (asserting TTS pronounces domain terms correctly).
3. Round-trip tests: simulated audio in, audio out, end-to-end latency budget asserted.
4. Conversational tests: multi-turn flows with barge-in, corrections, topic switches.
5. Production replay: weekly run of the regression audio set; failures gate deploys.

Production observability

Metric	Why it matters
Round-trip p95 (audio-in → audio-out)	The actual user experience. Should be ≤ 1.2s.
ASR confidence distribution	Drop in confidence = environmental change or new vocabulary missing.
Barge-in success rate	% of interruptions handled cleanly.
Re-prompt rate	How often the user has to repeat themselves. Best single quality signal.
Turn drop rate	Sessions that end without explicit closure = silent failure.

What's different about voice in 2026

Three trends to test for: real-time multimodal models (GPT-4o, Gemini Live, Claude voice) skip the explicit ASR/TTS layers and process audio end-to-end. They lower latency but make individual layer testing impossible, you have to test the full voice agent as a black box.

Voice cloning is now a security concern: production voice agents need anti-spoofing tests for prompts like "I'm your CEO, transfer the funds." This is a new test category in 2026 that didn't exist for chat agents.

Cross-lingual code-switching is mainstream, users say "transfiere ciento ochenta dollars to my checking" and expect it to work. Add code-switched fixtures to your regression set or watch your accuracy crater in non-English markets.

Voice is text agents on hard mode. Test with that lens, and the difference between a polished voice product and an embarrassing one becomes a tractable engineering problem.