Probabilistic Testing: Because Your AI Never Gives the Same Answer Twice

Your AI test suite has a problem that's been hiding in plain sight. You're testing outputs as if they're deterministic, but they're not. You run the same prompt through the same model three times and get three different answers. So you write your assertion to pass if it gets any of them. This is fragile, doesn't tell you what changed, and completely misses the real issue: you don't understand your model's output distribution.

Probabilistic testing flips this. Instead of asking "does it output X?", you ask "what's the expected distribution of outputs, and is this run within reasonable confidence bounds?" It's the difference between binary pass/fail and actually understanding your system.

The Problem With Deterministic Testing of Probabilistic Systems

Deterministic tests assume reproducibility. Feed in input A, get output B, every time. This works for sorting algorithms and database queries. It doesn't work for language models, classifiers with stochastic components, or any system with inherent randomness.

When you test an LLM, you have three choices: set temperature to 0 (removes the variation you care about testing), accept any output (no real assertion), or write brittle regex matchers that break on punctuation changes. None of these options actually validate model quality. You're just checking that the model runs without crashing.

Real testing for AI means measuring confidence intervals, not checking for exact outputs.

The math is simpler than you'd think, and you don't need a statistics degree. You need to understand two concepts: running the same prompt multiple times to measure variance, and comparing results statistically to detect real changes.

Measuring Output Distributions with Bootstrap Sampling

Bootstrap sampling is the practical foundation of probabilistic testing. Run your test multiple times, collect the results, and measure the distribution. This tells you what's normal and what's anomalous.

interface ScoredOutput {
 output: string;
 score: number; // 0-1 rating from evaluator
 tokens: number;
}

interface DistributionStats {
 mean: number;
 stdDev: number;
 p5: number; // 5th percentile
 p95: number; // 95th percentile
 median: number;
 sampleSize: number;
}

const bootstrapDistribution = (samples: number[]): DistributionStats => {
 const sorted = samples.sort((a, b) => a - b);
 const n = sorted.length;

 const mean = samples.reduce((a, b) => a + b) / n;
 const variance = samples.reduce((sq, x) => sq + Math.pow(x - mean, 2)) / n;
 const stdDev = Math.sqrt(variance);

 return {
 mean,
 stdDev,
 p5: sorted[Math.floor(n * 0.05)],
 p95: sorted[Math.floor(n * 0.95)],
 median: sorted[Math.floor(n * 0.5)],
 sampleSize: n
 };
};

// Test quality consistency across multiple runs
const testQualityDistribution = async (
 prompt: string,
 runs = 30,
 expectedMeanScore = 0.85
) => {
 const scores: number[] = [];

 for (let i = 0; i < runs; i++) {
 const response = await model.complete(prompt, { temperature: 0.7 });
 const score = await evaluator.score(response);
 scores.push(score);
 }

 const stats = bootstrapDistribution(scores);

 // Check that mean is within 5% of expected
 expect(stats.mean).toBeGreaterThan(expectedMeanScore - 0.05);
 expect(stats.mean).toBeLessThan(expectedMeanScore + 0.05);

 // Check that variation is reasonable
 expect(stats.stdDev).toBeLessThan(0.15); // allow 15% variance

 console.log('Distribution:', {
 mean: stats.mean.toFixed(3),
 stdDev: stats.stdDev.toFixed(3),
 range: `${stats.p5.toFixed(2)} - ${stats.p95.toFixed(2)}`
 });
};

await testQualityDistribution(
 "Explain quantum computing in one sentence",
 50
);

This test runs your prompt 50 times and validates that the quality scores follow an expected distribution. You're measuring consistency and detecting when something changed significantly. The 5th to 95th percentile range tells you the normal bounds for that prompt.

Hypothesis Testing for Model Changes

When you update your model or change your prompt, how do you know if it actually improved? Run the same test suite against both versions and compare distributions. This is hypothesis testing.

interface ComparisonResult {
 pValue: number; // probability that difference is random
 significantlyDifferent: boolean;
 baselineMean: number;
 currentMean: number;
 improvement: string;
}

// Two-sample t-test for comparing distributions
const twoSampleTTest = (
 baselineScores: number[],
 currentScores: number[]
): ComparisonResult => {
 const n1 = baselineScores.length;
 const n2 = currentScores.length;

 const mean1 = baselineScores.reduce((a, b) => a + b) / n1;
 const mean2 = currentScores.reduce((a, b) => a + b) / n2;

 const var1 = baselineScores.reduce((sq, x) => sq + Math.pow(x - mean1,2)) / n1;
 const var2 = currentScores.reduce((sq, x) => sq + Math.pow(x - mean2,2)) / n2;

 // Welch's t-statistic (doesn't assume equal variances)
 const tStat = (mean2 - mean1) / Math.sqrt(var1/n1 + var2/n2);

 // Approximate p-value (simplified - in production use jStat or similar)
 const pValue = 2 * (1 - normalCDF(Math.abs(tStat)));

 return {
 pValue,
 significantlyDifferent: pValue < 0.05, // 95% confidence
 baselineMean: mean1,
 currentMean: mean2,
 improvement: mean2 > mean1
 ? `+${((mean2 - mean1) / mean1 * 100).toFixed(1)}%`
 : `${((mean2 - mean1) / mean1 * 100).toFixed(1)}%`
 };
};

// Before/after testing when deploying new model
const validateModelUpdate = async (newModel, oldModel) => {
 const testPrompts = [
 "Summarize the main points",
 "Translate to Spanish",
 "Generate a creative title"
 ];

 const results = [];

 for (const prompt of testPrompts) {
 const baselineScores: number[] = [];
 const currentScores: number[] = [];

 // Test old model 20 times
 for (let i = 0; i < 20; i++) {
 const response = await oldModel.complete(prompt);
 baselineScores.push(await evaluator.score(response));
 }

 // Test new model 20 times
 for (let i = 0; i < 20; i++) {
 const response = await newModel.complete(prompt);
 currentScores.push(await evaluator.score(response));
 }

 const comparison = twoSampleTTest(baselineScores, currentScores);
 results.push({
 prompt: prompt.substring(0,30),
 ...comparison
 });
 }

 // Only deploy if all tests show improvement or no significant change
 const allPositive = results.every(r => !r.significantlyDifferent || r.currentMean > r.baselineMean);

 console.table(results);
 expect(allPositive).toBe(true);
};

await validateModelUpdate(newModel, oldModel);

This test compares two models on the same prompts and tells you whether differences are statistically significant. A p-value below 0.05 means you have 95% confidence the difference isn't random. If your new model's p-value is 0.8, you can't claim improvement even if the raw scores are slightly higher.

Measuring Consistency as a Feature

Sometimes low variance is bad (you want exploration) and sometimes it's good (you want reliable summaries). Probabilistic testing lets you measure variance as a first-class metric, not a bug.

// Define consistency requirements per task
interface ConsistencyTarget {
 task: string;
 maxStdDev: number; // higher = more variation tolerated
 minAgreement: number; // percentage of outputs that should be similar
}

const validateConsistency = async (
 prompt: string,
 consistencyTarget: ConsistencyTarget,
 runs = 30
) => {
 const outputs: string[] = [];

 for (let i = 0; i < runs; i++) {
 const response = await model.complete(prompt);
 outputs.push(response);
 }

 // Semantic similarity using embeddings
 const embeddings = await Promise.all(
 outputs.map(o => embed(o))
 );

 // Measure pairwise similarity
 const similarities: number[] = [];
 for (let i = 0; i < embeddings.length; i++) {
 for (let j = i + 1; j < embeddings.length; j++) {
 const sim = cosineSimilarity(embeddings[i], embeddings[j]);
 similarities.push(sim);
 }
 }

 const avgSimilarity = similarities.reduce((a, b) => a + b) / similarities.length;
 const percentSimilar = similarities.filter(s => s > 0.8).length / similarities.length;

 console.log(`Task: ${consistencyTarget.task}`);
 console.log(`Average similarity: ${avgSimilarity.toFixed(2)}`);
 console.log(`High similarity (>0.8): ${(percentSimilar * 100).toFixed(0)}%`);

 expect(percentSimilar).toBeGreaterThan(consistencyTarget.minAgreement);
};

// Run for a summarization task - expect high consistency
await validateConsistency(
 "Summarize the key findings from this research paper",
 {
 task: "summarization",
 maxStdDev: 0.1,
 minAgreement: 0.7 // 70% of summaries should be similar
 }
);

This approach measures whether your model produces consistent outputs. For summarization, you want high consistency. For creative writing, you might expect lower consistency. Either way, you can define and validate it.

// Store baseline distributions in git
const BASELINE_DISTRIBUTIONS = {
 summarization: {
 mean: 0.87,
 stdDev: 0.09,
 p5: 0.71,
 p95: 0.95,
 sampleSize: 100
 },
 classification: {
 mean: 0.92,
 stdDev: 0.06,
 p5: 0.82,
 p95: 0.98,
 sampleSize: 100
 }
};

// Test against established baseline
const testAgainstBaseline = async (task: string, runs = 20) => {
 const baseline = BASELINE_DISTRIBUTIONS[task];
 const scores: number[] = [];

 for (let i = 0; i < runs; i++) {
 const result = await evaluator.evaluate(task);
 scores.push(result.score);
 }

 const current = bootstrapDistribution(scores);

 // Warn if we drift outside expected range
 const driftPct = Math.abs(current.mean - baseline.mean) / baseline.mean * 100;
 if (driftPct > 10) {
 console.warn(`⚠️ ${task} drifted ${driftPct.toFixed(1)}%`);
 }

 // Fail if we're completely outside baseline
 expect(current.p5).toBeGreaterThan(baseline.p5 - 0.05);
 expect(current.p95).toBeLessThan(baseline.p95 + 0.05);
};

await testAgainstBaseline('summarization', 30);