Continuous AI Evaluation: Testing Doesn't Stop at Deploy

Your eval team ran 10,000 test cases. All green. You deployed on Tuesday.

By Friday, engagement tanked 8%.

The evaluation was thorough. But it was static, a snapshot of one moment. Real users hit edge cases you didn't think to test. Data distribution shifted slightly. The model behaves differently when talking to thousands of users than to your test harness.

Pre-deploy testing fails not because it's wrong, but because it's incomplete. You need evaluation that continues after deploy.

The Limitations of Pre-Deploy Testing

Pre-deploy evaluation assumes the world is stable. It's not.

Test data is not prod data. Your 10,000 test cases represent maybe 0.001% of possible user inputs.

Real issues emerge in production:

• Edge cases you didn't think of: Actual users are more creative at finding weird inputs than your test harness
• Data distribution shift: User queries change seasonally, by region, by trend. Your model might be trained on 2023 data but it's now 2026
• Cascading failures: Individual components test fine in isolation. They interact badly in production
• Slow degradation: Quality doesn't usually crash overnight. It drifts gradually, and you miss it without continuous monitoring
• Emergent behavior: LLMs sometimes exhibit surprising behavior in specific user contexts that never appeared in evals

The solution: build evaluation infrastructure that's always on, always learning.

Shadow Testing: Safe Validation Before Go-Live

Shadow mode means running a new algorithm or model without showing results to users. You collect metrics silently, compare to production, validate before real traffic.

How to Implement Shadow Testing

Before deploying a new recommendation algorithm, model version, or prompt engineering approach, run it in parallel with production:

async function shadowTestNewAlgorithm(
 userId: string,
 request: Request
): Promise<{
 productionResponse: Response;
 shadowResponse: Response;
 metrics: ShadowMetrics;
}> {
 // Run both algorithms
 const productionTask = runProductionAlgorithm(userId, request);
 const shadowTask = runShadowAlgorithm(userId, request);

 const [productionResponse, shadowResponse] = await Promise.all([
 productionTask,
 shadowTask,
 ]);

 // Serve production to user
 await sendToUser(productionResponse);

 // Collect metrics on shadow in background
 const metrics = await evaluateShadowResponse(
 shadowResponse,
 productionResponse,
 userId
 );

 // Log for later analysis
 await logShadowMetrics({
 userId,
 timestamp: Date.now(),
 productionQuality: metrics.productionScore,
 shadowQuality: metrics.shadowScore,
 deltaQuality: metrics.shadowScore - metrics.productionScore,
 });

 return {
 productionResponse,
 shadowResponse,
 metrics,
 };
}

async function analyzeShadowMetrics(
 window: { startDate: Date; endDate: Date }
): Promise<{
 shadowWins: number;
 shadowLoses: number;
 avgDelta: number;
 statisticallySignificant: boolean;
}> {
 const metricsInWindow = await fetchMetrics(window);

 const wins = metricsInWindow.filter(m => m.deltaQuality > 0).length;
 const loses = metricsInWindow.filter(m => m.deltaQuality < 0).length;
 const avgDelta =
 metricsInWindow.reduce((sum, m) => sum + m.deltaQuality, 0) /
 metricsInWindow.length;

 // Use t-test to check if difference is significant
 const significant = await runTTest(metricsInWindow, 0.05);

 return {
 shadowWins: wins,
 shadowLoses: loses,
 avgDelta,
 statisticallySignificant: significant,
 };
}

Run shadow tests for 1-2 weeks before making a call. The key metric: does the shadow version outperform production? By how much? Is the difference statistically significant or just noise?

When to Graduate From Shadow to Production

Set clear promotion criteria before you start:

• Shadow must outperform production by at least 3% on your primary metric
• No regressions on any secondary metric (e.g., no increase in hallucinations)
• Improvement is statistically significant (p < 0.05)
• Passes safety checks (no toxic outputs, hallucinations within tolerance)

If shadow doesn't meet criteria, iterate. Don't force it out to save face.

Canary Evaluations: Gradual Rollout with Validation

Once shadow validation passes, don't flip a switch. Roll out gradually to subsets of users, validating at each step.

enum CanaryStage {
 SHADOW = "shadow",
 CANARY_1PCT = "canary_1pct",
 CANARY_10PCT = "canary_10pct",
 CANARY_50PCT = "canary_50pct",
 FULL = "full",
}

interface CanaryConfig {
 stage: CanaryStage;
 trafficPercentage: number;
 holdDurationHours: number;
 rollbackThreshold: number; // e.g., 0.05 (5% quality drop)
 metricsToMonitor: string[];
}

const CANARY_PROGRESSION: Record = {
 [CanaryStage.SHADOW]: {
 stage: CanaryStage.SHADOW,
 trafficPercentage: 0,
 holdDurationHours: 336, // 2 weeks
 rollbackThreshold: 0.1,
 metricsToMonitor: ["quality", "latency", "errors"],
 },
 [CanaryStage.CANARY_1PCT]: {
 stage: CanaryStage.CANARY_1PCT,
 trafficPercentage: 1,
 holdDurationHours: 24,
 rollbackThreshold: 0.05,
 metricsToMonitor: ["quality", "latency", "errors", "user_feedback"],
 },
 [CanaryStage.CANARY_10PCT]: {
 stage: CanaryStage.CANARY_10PCT,
 trafficPercentage: 10,
 holdDurationHours: 48,
 rollbackThreshold: 0.03,
 metricsToMonitor: ["quality", "latency", "errors", "user_feedback"],
 },
 [CanaryStage.CANARY_50PCT]: {
 stage: CanaryStage.CANARY_50PCT,
 trafficPercentage: 50,
 holdDurationHours: 72,
 rollbackThreshold: 0.02,
 metricsToMonitor: ["quality", "latency", "errors", "user_feedback", "churn"],
 },
 [CanaryStage.FULL]: {
 stage: CanaryStage.FULL,
 trafficPercentage: 100,
 holdDurationHours: Infinity,
 rollbackThreshold: 0.015,
 metricsToMonitor: ["quality", "latency", "errors", "user_feedback", "churn"],
 },
};

async function manageCanaryRollout(
 changeId: string,
 currentStage: CanaryStage
): Promise<"promote" | "hold" | "rollback"> {
 const config = CANARY_PROGRESSION[currentStage];
 const metrics = await getMetricsSince(
 changeId,
 Date.now() - config.holdDurationHours * 3600 * 1000
 );

 const baseline = await getBaselineMetrics(changeId);

 // Check if metrics have degraded beyond threshold
 const qualityRegression =
 (baseline.quality - metrics.quality) / baseline.quality;

 if (Math.abs(qualityRegression) > config.rollbackThreshold) {
 console.error(
 `Quality dropped ${(qualityRegression * 100).toFixed(1)}%. Rolling back.`
 );
 return "rollback";
 }

 // Check if we should hold
 if (metrics.hasHighErrorRate) {
 console.warn("Error rate elevated. Holding before next stage.");
 return "hold";
 }

 // Good to promote
 console.log(`${currentStage} passed validation. Promoting.`);
 return "promote";
}

async function rolloutCanary(
 changeId: string,
 stages: CanaryStage[] = [
 CanaryStage.CANARY_1PCT,
 CanaryStage.CANARY_10PCT,
 CanaryStage.CANARY_50PCT,
 CanaryStage.FULL,
 ]
): Promise {
 let currentStage = CanaryStage.SHADOW;

 for (const nextStage of stages) {
 console.log(
 `Promoting ${changeId} from ${currentStage} to ${nextStage}`
 );
 await setCanaryTraffic(changeId, nextStage);

 const decision = await manageCanaryRollout(changeId, nextStage);

 if (decision === "rollback") {
 console.error(`Rolling back ${changeId}`);
 await revertCanary(changeId);
 return;
 }

 if (decision === "hold") {
 console.warn(`Holding ${changeId} at ${nextStage}`);
 // Don't promote yet; check again later
 return;
 }

 currentStage = nextStage;
 console.log(`${changeId} now at ${(CANARY_PROGRESSION[nextStage].trafficPercentage)}% traffic`);
 }

 console.log(`${changeId} is now fully deployed`);
}

Real-Time Quality Monitoring

With production traffic, you need immediate visibility into quality degradation. Set up continuous monitoring that samples and evaluates outputs in real-time:

interface QualityAlert {
 severity: "warning" | "critical";
 metric: string;
 currentValue: number;
 baseline: number;
 percentChange: number;
 timestamp: Date;
}

class ProductionQualityMonitor {
 private sampleRate = 0.01; // Evaluate 1% of requests
 private alertThreshold = 0.05; // 5% drop triggers alert
 private baselineWindow = 24 * 3600 * 1000; // 24 hours

 async evaluateSample(
 request: Request,
 response: Response
 ): Promise {
 // Randomly sample
 if (Math.random() > this.sampleRate) {
 return null;
 }

 // Evaluate this response
 const score = await evaluateQuality(response);

 // Get baseline
 const baseline = await this.getBaseline();

 // Check for degradation
 const percentChange = (baseline - score) / baseline;

 if (percentChange > this.alertThreshold) {
 const alert: QualityAlert = {
 severity: percentChange > 0.1 ? "critical" : "warning",
 metric: "quality_score",
 currentValue: score,
 baseline: baseline,
 percentChange,
 timestamp: new Date(),
 };

 // Log immediately
 await logAlert(alert);

 return alert;
 }

 return null;
 }

 private async getBaseline(): Promise {
 const startTime = Date.now() - this.baselineWindow;
 const historicalScores = await fetchQualityScores(startTime, Date.now());
 return (
 historicalScores.reduce((a, b) => a + b, 0) / historicalScores.length
 );
 }
}

Production Regression Detection

Beyond point-in-time monitoring, detect slow degradation trends. Quality might drift 1% per week without ever hitting an alert threshold:

async function detectRegressionTrend(
 metricKey: string,
 window: number = 7 * 24 * 3600 * 1000 // 7 days
): Promise<{
 trend: "stable" | "improving" | "degrading";
 regressionProbability: number;
}> {
 const dataPoints = await fetchMetricTimeseries(metricKey, window);

 // Simple linear regression
 let sumX = 0,
 sumY = 0,
 sumXY = 0,
 sumX2 = 0;
 const n = dataPoints.length;

 for (let i = 0; i < n; i++) {
 sumX += i;
 sumY += dataPoints[i].value;
 sumXY += i * dataPoints[i].value;
 sumX2 += i * i;
 }

 const slope = (n * sumXY - sumX * sumY) / (n * sumX2 - sumX * sumX);
 const avgValue = sumY / n;
 const percentChangePerDay = (slope / avgValue) * 100;

 let trend: "stable" | "improving" | "degrading" = "stable";
 if (percentChangePerDay < -0.5) trend = "degrading";
 if (percentChangePerDay > 0.5) trend = "improving";

 // Calculate probability of continued regression
 const regressionProbability = trend === "degrading" ? 0.7 : 0.1;

 return {
 trend,
 regressionProbability,
 };
}

Feedback Loops: Let Production Data Improve Evaluation

Don't just monitor in one direction. Use production feedback to retrain your evaluation models and catch real issues faster:

async function collectProductionFeedback(): Promise {
 // Collect user signals: thumbs up/down, explicit ratings, session metrics
 const recentFeedback = await fetchUserFeedback({
 since: Date.now() - 24 * 3600 * 1000,
 });

 // For responses with thumbs down, log for analysis
 const negativeExamples = recentFeedback.filter(f => !f.helpful);

 // Periodically retrain evaluation models using real feedback
 if (negativeExamples.length > 100) {
 console.log("Retraining evaluation model with recent negative examples...");

 const trainingData = negativeExamples.map(ex => ({
 input: ex.prompt,
 output: ex.response,
 label: 0, // Bad
 }));

 // Also sample positive examples
 const positiveExamples = recentFeedback.filter(f => f.helpful);
 const positiveData = positiveExamples.map(ex => ({
 input: ex.prompt,
 output: ex.response,
 label: 1, // Good
 }));

 await retrainEvaluationModel([...trainingData, ...positiveData]);
 }
}

async function retrainEvaluationModel(
 trainingData: Array<{ input: string; output: string; label: number }>
): Promise {
 const modelPath = "/models/production_evaluator_v2";

 // Fine-tune a model on production data
 const model = await loadBaseModel();
 const finetuned = await model.finetune(trainingData);

 // Validate new model against held-out test set
 const testAccuracy = await validateModel(finetuned);

 if (testAccuracy > 0.92) {
 // Good enough, promote
 await promoteEvaluationModel(modelPath, finetuned);
 console.log(`New evaluator promoted. Accuracy: ${(testAccuracy * 100).toFixed(1)}%`);
 } else {
 console.warn(`New evaluator accuracy too low: ${testAccuracy}`);
 }
}

Building the Continuous Evaluation Dashboard

All of this data is worthless if nobody sees it. Build a real-time dashboard that teams check daily:

Key metrics to surface:

• Quality scores over time (last 24h, 7d, 30d)
• Error rates and latency trends
• Regression detection status
• Canary deployment progress
• User feedback aggregates
• Drift alerts and flagged outputs
• A/B test results (ongoing and recently completed)

Make it obvious when something needs attention. Use red/yellow/green status indicators. Have a one-click rollback button for critical issues.

The Shift From Gates to Guardrails

Traditional testing is a gate: you test, you either deploy or don't.

Continuous evaluation is a guardrail: you deploy with monitoring, and the system can automatically rollback if things break.

Pre-deploy testing asks "Is this safe to ship?" Post-deploy monitoring asks "Is this still working?" Both matter.

This shift requires cultural change. Engineers need to be comfortable deploying with monitoring. You need automated rollback systems. You need on-call rotations that respond to eval alerts fast.

But the payoff is huge: you can ship faster, catch real issues faster, and iterate based on actual user behavior instead of guesses.