How to A/B Test AI Models in Production Without Losing Your Mind

You have two recommendation models. Your PM says they "feel better." That's not data. You need an A/B test. But unlike traditional A/B tests, your models will never give you the exact same prediction twice, and the business impact takes days to materialize.

This is where most teams break their A/B testing framework and resort to "we're 70% confident this is better, let's ship it."

The Fundamental Problem: Non-Determinism

Your classic A/B test for a button color runs like this: 50% of users see blue, 50% see red. Everyone's deterministic. You run the test for a week and calculate statistical significance. Done.

Your AI A/B test runs like this: user logs in, model A generates a prediction, model B generates a prediction. They're different even for the same input. Now your test is measuring both "are the models different" (yes, obviously) and "which difference matters for business."

Worse, the business impact is delayed. A user sees a recommendation now. They click on it in 3 hours. They make a purchase tomorrow. That signal arrives too late for a rapid iterate cycle.

Most A/B test frameworks were built for deterministic systems and lagged metrics. AI systems violate both assumptions.

Sample Size: What You Actually Need

You want 95% confidence your new model is better. Sounds statistical and rigorous. Then you calculate sample size and find you need 2 million requests to detect a 1% improvement. For a system with 10 million daily requests, that's a week of test. For a system with 100k daily requests, that's 20 weeks.

Here's the insight that changes everything: you don't need to detect tiny improvements with high confidence. You need to detect real business impact fast.

Sample size formula for comparing proportions (simplified):

n = (z_alpha + z_beta)^2 * (p1*(1-p1) + p2*(1-p2)) / (p1 - p2)^2

Where:
 n = sample size per group
 z_alpha = z-score for significance level (1.96 for 95% confidence)
 z_beta = z-score for power (0.84 for 80% power)
 p1, p2 = success rates for models A and B
 p1 - p2 = minimum detectable effect

Example: You want to detect a 2% improvement in click-through rate (from 10% to 10.2%) with 95% confidence and 80% power.

n = (1.96 + 0.84)^2 * (0.1*0.9 + 0.102*0.898) / (0.002)^2 = approximately 78,000 per group

At 100k requests per day, you need 1.5 days of traffic. Reasonable.

But here's what many teams miss: the metric you're measuring in production (CTR) might be different from the metric you care about (long-term user satisfaction). You might get statistical significance on short-term clicks but still hurt long-term engagement.

This is why you run multiple tiers of tests:

Tier 1: Fast metrics (model agreement, latency, server errors). These tell you if the model works. Run these in shadow mode (model B runs alongside A but doesn't affect users) to get fast signal. Target: 100-500 samples per variant.

Tier 2: User-observable metrics (clicks, engagement, conversion). These tell you if users prefer it. Run for the calculated sample size. Target: whatever you calculated above.

Tier 3: Long-term business metrics (retention, lifetime value, churn). These tell you if it actually mattered. Run for weeks or months after launch. This is offline validation, not live testing.

Sequential Analysis: Stop When You Know

The classic A/B test runs for a fixed duration. You wait the full period, then calculate significance. But with sequential analysis, you check results continuously and stop early when a winner is clear.

This is mathematically valid if you correct for peeking (don't just check p-values naively or you'll get false positives). Use Pocock's method or O'Brien-Fleming's method for predetermined checkpoints.

Here's the practical implementation in TypeScript:

interface ABTestResult {
 variantA_successes: number;
 variantA_trials: number;
 variantB_successes: number;
 variantB_trials: number;
 confidence: number;
 shouldStop: boolean;
 winner: 'A' | 'B' | null;
}

class SequentialABTest {
 private alphaLevel = 0.05;
 private powerLevel = 0.2;
 private maxSampleSize: number;

 constructor(expectedBaseline: number, minimumDetectableEffect: number) {
 // Calculate required sample size
 const z_alpha = 1.96; // 95% CI
 const z_beta = 0.84; // 80% power
 const p1 = expectedBaseline;
 const p2 = expectedBaseline + minimumDetectableEffect;

 this.maxSampleSize = Math.ceil(
 Math.pow(z_alpha + z_beta, 2) *
 (p1 * (1 - p1) + p2 * (1 - p2)) /
 Math.pow(p1 - p2,2)
 );
 }

 evaluate(result: ABTestResult): {
 status: 'continue' | 'stop_winner_a' | 'stop_winner_b' | 'stop_inconclusive';
 confidence: number;
 } {
 const totalA = result.variantA_trials;
 const totalB = result.variantB_trials;

 // Stop if we've hit max sample size
 if (totalA >= this.maxSampleSize || totalB >= this.maxSampleSize) {
 return this.concludeTest(result);
 }

 // Bayesian approach: calculate probability that B is better than A
 const probBBetterThanA = this.bayesianComparison(
 result.variantA_successes,
 result.variantA_trials,
 result.variantB_successes,
 result.variantB_trials
 );

 // If we're >95% confident in a winner, stop
 if (probBBetterThanA > 0.95) {
 return {
 status: 'stop_winner_b',
 confidence: probBBetterThanA
 };
 }

 if (probBBetterThanA < 0.05) {
 return {
 status: 'stop_winner_a',
 confidence: 1 - probBBetterThanA
 };
 }

 return {
 status: 'continue',
 confidence: Math.max(probBBetterThanA, 1 - probBBetterThanA)
 };
 }

 private bayesianComparison(
 successes_a: number,
 trials_a: number,
 successes_b: number,
 trials_b: number
 ): number {
 // Simplified Beta-Binomial conjugate prior
 // Prior: Beta(1,1) (uniform)
 // Posterior: Beta(successes + 1, failures + 1)

 const alpha_a = successes_a + 1;
 const beta_a = (trials_a - successes_a) + 1;

 const alpha_b = successes_b + 1;
 const beta_b = (trials_b - successes_b) + 1;

 // Estimate: P(B > A) via sampling
 // In production, use a more sophisticated method like MCMC
 const rate_a = alpha_a / (alpha_a + beta_a);
 const rate_b = alpha_b / (alpha_b + beta_b);

 // Rough approximation for illustration
 // Probability that B's true success rate exceeds A's
 const diff = rate_b - rate_a;
 const se = Math.sqrt(
 (rate_a * (1 - rate_a) / trials_a) +
 (rate_b * (1 - rate_b) / trials_b)
 );

 // Normal approximation to compute P(B > A)
 const z = diff / se;
 return this.normalCDF(z);
 }

 private normalCDF(z: number): number {
 // Approximation of standard normal CDF
 const t = 1 / (1 + 0.2316419 * Math.abs(z));
 const d = 0.3989423 * Math.exp(-z * z / 2);
 const prob = d * t * (0.319381530 + t * (-0.356563782 + t * (1.781477937 + t * (-1.821255978 + t * 1.330274429))));

 return z >= 0 ? 1 - prob : prob;
 }

 private concludeTest(result: ABTestResult): {
 status: 'stop_winner_a' | 'stop_winner_b' | 'stop_inconclusive';
 confidence: number;
 } {
 const rateA = result.variantA_successes / result.variantA_trials;
 const rateB = result.variantB_successes / result.variantB_trials;

 if (rateB > rateA) {
 return {
 status: 'stop_winner_b',
 confidence: (rateB - rateA) / rateB
 };
 } else if (rateA > rateB) {
 return {
 status: 'stop_winner_a',
 confidence: (rateA - rateB) / rateA
 };
 }

 return {
 status: 'stop_inconclusive',
 confidence: 0.5
 };
 }
}

// Usage
const test = new SequentialABTest(0.10,0.02); // 10% baseline, 2% MDE

const result: ABTestResult = {
 variantA_successes: 1050,
 variantA_trials: 10500,
 variantB_successes: 1120,
 variantB_trials: 10500,
 confidence: 0,
 shouldStop: false
};

const decision = test.evaluate(result);
console.log(decision);
// { status: 'stop_winner_b', confidence: 0.87 }

This approach stops your test when you have clear winners. If A is 3% better after 1000 samples and there's low variance, you stop early. If results are noisy, you run longer.

Deployment Patterns: Shadow, Canary, Then Full

Shadow mode: Model B runs alongside Model A but never affects users. You log both outputs and see how they differ. This catches obvious bugs and model degradation without risk. Run for 2-3 days minimum. If Model B makes clearly different predictions that seem worse, kill it now.

Canary deployment: Send 1% of traffic to Model B. Monitor key metrics. If error rates spike, latency degrads, or prediction quality drops, abort immediately. If everything looks healthy after a few hours, move to 5%, then 10%, then 50%, then 100%.

Blue-green deployment: Have two complete infrastructure setups (blue and green). Run Model A on blue, Model B on green. Full traffic to one at a time. If Model B fails, switch back to blue instantly. More infrastructure, but safer for critical systems.

Choose based on risk: shadow mode for experimenting, canary for lower-risk changes, blue-green for critical systems where you can't afford downtime.

Stratified Analysis: Look Deeper

You have statistical significance. But Model B might be better for segment X and worse for segment Y, and the winner on the aggregate metric is hiding a lose-lose situation.

Stratify your results by:

• User segment: New vs. returning, by geographic region, by user cohort (age, interests, etc.)
• Traffic type: Organic search vs. paid ads, desktop vs. mobile, app vs. web
• Time of day / day of week: Morning vs. night behavior can differ dramatically
• Model confidence: Predictions where the model was very confident vs. uncertain

For each stratum, ask: does Model B win or lose? If it's mixed, you have a decision to make: ship with segment-specific routing (Model A for segment X, Model B for segment Y), or choose the option that wins overall.

The Gotchas That Will Bite You

Selection bias in labeling: You measure clicks as a proxy for "good recommendation, " but users are more likely to click on obvious recommendations. Your test might prefer a model that gives safe, obvious answers over one that takes interesting risks.

Time-lag between prediction and outcome: A recommendation shown today affects purchase behavior tomorrow or next week. If you check results after 24 hours, you're measuring early engagement, not long-term value. Run Tier 3 tests offline or in the background.

Multiple comparisons: If you test 10 metrics and get one at p = 0.05, you're probably looking at a false positive. Adjust your significance threshold for the number of metrics you're checking (Bonferroni correction) or pre-commit to a primary metric.

Segment effects hiding aggregate results: A model can be 2% better overall but 50% worse for your highest-value user segment. Always stratify.

The Practical Path

Start with Tier 1 tests (fast metrics in shadow mode). They're cheap and fast and catch obvious problems. Once you have confidence the model works, move to Tier 2 (live testing with calculated sample sizes and sequential analysis).

Only run Tier 3 (long-term business metrics) on changes you've already validated in Tiers 1 and 2. And treat Tier 3 results as offline validation: measure them passively, don't rig your test on them.

This layered approach lets you move fast without losing rigor. Most teams either move slow because they over-test, or move fast and break everything. This gives you both.