Latency Testing for AI: How Slow Is Too Slow?

Here's a question that trips up most teams building AI features: how slow is too slow?

Perceived vs. Actual Latency: Why Streaming Changes Everything

The most important distinction in AI latency testing: perceived latency and actual latency are completely different metrics, and only perceived latency affects user experience.

Actual latency is what you measure with a timer. Perceived latency is what your users feel. And streaming is the tool that decouples them.

Consider this scenario: an AI chatbot response takes 3 seconds to generate. Without streaming, users wait 3 seconds and see the entire response appear instantly. Feels slow. With streaming, users see text appearing immediately (first token in 200ms) and the remaining text trickles in. Feels instant, even though total time is identical.

async function testPerceivedLatency() {
 // Test 1: Without streaming (actual latency = perceived latency)
 async function testBlockingResponse() {
 const startTime = Date.now();

 const response = await client.messages.create({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: 1000,
 messages: [
 {
 role: "user",
 content: "Explain quantum entanglement in 500 words"
 }
 ]
 });

 const totalLatency = Date.now() - startTime;
 console.log(`Total latency (blocking): ${totalLatency}ms`);

 // Users perceive this as: [wait 3000ms] [see entire response]
 // This feels slow
 expect(totalLatency).toBeLessThan(4000);
 }

 // Test 2: With streaming (TTT is what matters)
 async function testStreamingResponse() {
 let firstTokenTime = null;
 let allTokensTime = null;

 const startTime = Date.now();

 const stream = await client.messages.stream({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: 1000,
 messages: [
 {
 role: "user",
 content: "Explain quantum entanglement in 500 words"
 }
 ]
 });

 // Track metrics
 let tokenCount = 0;
 for await (const event of stream) {
 if (event.type === "content_block_delta" && !firstTokenTime) {
 firstTokenTime = Date.now() - startTime;
 console.log(`Time to first token: ${firstTokenTime}ms`);
 }

 if (event.type === "message_stop") {
 allTokensTime = Date.now() - startTime;
 console.log(`Total streaming latency: ${allTokensTime}ms`);
 }

 if (event.type === "content_block_delta") {
 tokenCount++;
 }
 }

 // Users perceive this as: [wait 200ms] [see text appearing]
 // This feels instant even though total time is similar
 expect(firstTokenTime).toBeLessThan(500);
 }

 await testBlockingResponse();
 await testStreamingResponse();
}

"Streaming isn't just a feature, it's a latency testing strategy. The same API response that feels unacceptably slow without streaming feels instant with it."

This is why every AI feature should support streaming. It's the only way to make AI latency acceptable in user-facing applications, not just better UX.

Use-Case-Specific Latency Thresholds

Different AI features have wildly different acceptable latency profiles. There's no universal "AI should respond in X milliseconds" rule. Context matters enormously.

Use Case	Acceptable TTT	Acceptable Total	User Tolerance
Search results	<100ms	<500ms	Very low, users expect instant
Chat/conversational	200-500ms	3-5s	Moderate, streaming hides latency
Recommendations	<100ms	<1000ms	Low, feels like page slowdown
Code generation	500ms-1s	5-10s	High, users expect computation
Long-form analysis	1-2s	15-30s	Very high, explicit wait expected

Why these differences? It comes down to user mental models and context. Users expect search to be instant (sub-500ms total, including network). Chat can be slower because users expect thinking. Recommendations feel worse at 2 seconds than chat feels at 3 seconds because recommendations should be "free" computationally, the user wasn't asking for heavy lifting.

to test these thresholds:

async function testLatencyByUseCase() {
 // Test: Search results must be fast
 async function testSearchLatency() {
 const searchQuery = "best JavaScript frameworks 2026";

 const startTime = Date.now();
 const results = await aiSearch(searchQuery);
 const latency = Date.now() - startTime;

 // For search, p95 should be under 500ms
 expect(latency).toBeLessThan(500);

 // Don't rely on average, test percentiles
 // This requires running multiple queries
 const latencies = [];
 for (let i = 0; i < 20; i++) {
 const start = Date.now();
 await aiSearch(searchQuery);
 latencies.push(Date.now() - start);
 }

 latencies.sort((a, b) => a - b);
 const p95 = latencies[Math.floor(latencies.length * 0.95)];
 const p99 = latencies[Math.floor(latencies.length * 0.99)];

 expect(p95).toBeLessThan(600); // Allow some variance
 expect(p99).toBeLessThan(1000); // Extreme outliers
 }

 // Test: Chat can be slower if streaming is enabled
 async function testChatLatency() {
 let ttfTokenTime = null;
 let totalTime = null;

 const startTime = Date.now();

 const stream = await client.messages.stream({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: 1000,
 messages: [{ role: "user", content: "What is machine learning?" }]
 });

 for await (const event of stream) {
 if (event.type === "content_block_delta" && !ttfTokenTime) {
 ttfTokenTime = Date.now() - startTime;
 }
 if (event.type === "message_stop") {
 totalTime = Date.now() - startTime;
 }
 }

 // Chat can tolerate slower TTT if content is streaming
 expect(ttfTokenTime).toBeLessThan(1000);
 // Total time can be longer
 expect(totalTime).toBeLessThan(5000);
 }

 // Test: Recommendations must be fast
 async function testRecommendationLatency() {
 const recommendations = [];
 const latencies = [];

 for (let i = 0; i < 10; i++) {
 const start = Date.now();
 const recs = await getAIRecommendations(userId);
 latencies.push(Date.now() - start);
 recommendations.push(recs);
 }

 const avgLatency = latencies.reduce((a, b) => a + b) / latencies.length;
 const maxLatency = Math.max(...latencies);

 // Recommendations should be consistently fast
 expect(avgLatency).toBeLessThan(800);
 expect(maxLatency).toBeLessThan(1500);
 }

 await testSearchLatency();
 await testChatLatency();
 await testRecommendationLatency();
}

Performance Budgets for AI Features

The key insight: your application's latency budget should be split separately for your code and the AI model. They have different optimization strategies and different bottlenecks.

For a search feature that needs to respond in 500ms total:

• Your application: 200ms (query parsing, database lookup, response formatting)
• AI model: 300ms (LLM inference)

If the model is slow, you can optimize it (use a faster model, caching, etc.). If your application is slow, you can optimize your code. But you need to know which is the bottleneck.

async function testLatencyBreakdown() {
 async function measureComponentLatency() {
 const measurements = {
 applicationSetup: 0,
 modelInference: 0,
 applicationFormatting: 0
 };

 // Measure application setup
 let start = Date.now();
 const query = parseUserQuery(userInput);
 const context = await fetchContext(query);
 measurements.applicationSetup = Date.now() - start;

 // Measure model inference
 start = Date.now();
 const response = await client.messages.create({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: 500,
 system: `Context: ${JSON.stringify(context)}`,
 messages: [{ role: "user", content: query }]
 });
 measurements.modelInference = Date.now() - start;

 // Measure application formatting
 start = Date.now();
 const formatted = formatResponse(response);
 measurements.applicationFormatting = Date.now() - start;

 console.log("Latency breakdown:", measurements);
 console.log(
 "Total:",
 measurements.applicationSetup +
 measurements.modelInference +
 measurements.applicationFormatting
 );

 return measurements;
 }

 const breakdown = await measureComponentLatency();

 // Example assertions
 expect(breakdown.applicationSetup).toBeLessThan(100); // Your code should be fast
 expect(breakdown.modelInference).toBeLessThan(300); // Model inference
 expect(breakdown.applicationFormatting).toBeLessThan(50); // Final formatting

 const total =
 breakdown.applicationSetup +
 breakdown.modelInference +
 breakdown.applicationFormatting;
 expect(total).toBeLessThan(500);
}

Percentiles, Not Averages

Here's a latency testing principle that applies to all software but is especially critical with AI: percentiles matter far more than averages.

If your average latency is 500ms but your p99 is 10 seconds, users are experiencing terrible performance 1% of the time. That 1% might be your peak traffic or your most valuable users. One slow request ruins the session.

async function testLatencyPercentiles() {
 async function testPecentileDistribution() {
 const latencies = [];

 // Simulate 100 requests to collect distribution
 for (let i = 0; i < 100; i++) {
 const start = Date.now();
 await client.messages.create({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: 100,
 messages: [
 {
 role: "user",
 content: "Respond with a single sentence"
 }
 ]
 });
 latencies.push(Date.now() - start);
 }

 // Sort for percentile calculation
 latencies.sort((a, b) => a - b);

 const p50 = latencies[Math.floor(latencies.length * 0.5)];
 const p95 = latencies[Math.floor(latencies.length * 0.95)];
 const p99 = latencies[Math.floor(latencies.length * 0.99)];
 const avg = latencies.reduce((a, b) => a + b) / latencies.length;

 console.log(`Average: ${avg}ms`);
 console.log(`P50 (median): ${p50}ms`);
 console.log(`P95: ${p95}ms`);
 console.log(`P99: ${p99}ms`);

 // Set assertions based on percentiles, not average
 expect(avg).toBeLessThan(1000);
 expect(p50).toBeLessThan(800);
 expect(p95).toBeLessThan(2000);
 expect(p99).toBeLessThan(3000);
 }

 await testPecentileDistribution();
}

The rule: your p95 latency is what typical users experience when things aren't perfect. Your p99 is your worst-case customer's experience. Never ignore them.

Testing Under Load and Network Conditions

Latency tests should simulate real conditions, not just measure in a vacuum. Model latency increases under load, and network conditions affect perception.

• Test with realistic concurrency: If your system handles 100 concurrent users, test latency with that load
• Test with network latency: Add artificial network delays to simulate regional users
• Test with variable model performance: Model latency varies based on token count, prompt complexity, and server load

async function testLatencyUnderRealisticConditions() {
 // Simulate concurrent requests
 async function testConcurrentLatency() {
 const concurrentRequests = 20;
 const results = [];

 const promises = Array(concurrentRequests)
 .fill(null)
 .map(async () => {
 const start = Date.now();
 await client.messages.create({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: 200,
 messages: [
 { role: "user", content: "What is cloud computing?" }
 ]
 });
 return Date.now() - start;
 });

 const latencies = await Promise.all(promises);
 const avgLatency = latencies.reduce((a, b) => a + b) / latencies.length;

 console.log(`Average latency under load: ${avgLatency}ms`);
 expect(avgLatency).toBeLessThan(2000); // Slower than single request
 }

 // Test with different token counts (affects model latency)
 async function testLatencyByOutputLength() {
 const testCases = [
 { maxTokens: 50, expectedMaxLatency: 1000 },
 { maxTokens: 500, expectedMaxLatency: 2000 },
 { maxTokens: 2000, expectedMaxLatency: 5000 }
 ];

 for (const testCase of testCases) {
 const start = Date.now();
 await client.messages.create({
 model: "claude-3-5-sonnet-20241022",
 max_tokens: testCase.maxTokens,
 messages: [
 {
 role: "user",
 content: `Generate ${testCase.maxTokens} words about AI`
 }
 ]
 });
 const latency = Date.now() - start;
 expect(latency).toBeLessThan(testCase.expectedMaxLatency);
 }
 }

 await testConcurrentLatency();
 await testLatencyByOutputLength();
}

Monitoring Latency in Production

Testing latency locally is important, but production latency tells the real story. You need continuous monitoring of percentiles, not just averages.

// Production monitoring setup
const latencyMetrics = {
 recordLatency(feature, latency) {
 // Record all latencies, not just averages
 metrics.histogram(`ai.${feature}.latency_ms`, latency);

 // Tag by percentile bucket for analysis
 if (latency < 500) metrics.increment(`ai.${feature}.fast`);
 else if (latency < 1000)
 metrics.increment(`ai.${feature}.normal`);
 else if (latency < 3000)
 metrics.increment(`ai.${feature}.slow`);
 else metrics.increment(`ai.${feature}.very_slow`);
 }
};

// Monitor TTT separately for streaming
const streamingMetrics = {
 recordTimeToFirstToken(feature, ttf) {
 metrics.histogram(`ai.${feature}.ttf_ms`, ttf);

 // Alert if TTF exceeds threshold
 if (ttf > 1000) {
 metrics.increment(`ai.${feature}.ttf_alert`);
 }
 }
};