Testing AI Recommendations: Why "Relevant" Isn't Good Enough

Your recommendation engine nailed the accuracy test. Users are clicking the suggestions. Everything looks perfect on paper.

Then engagement drops 12% in production.

This happens constantly in AI systems, and the culprit isn't usually the algorithm, it's the testing framework. You measured relevance when you should have measured delight.

Why Accuracy Metrics Are Dangerously Incomplete

Accuracy-based testing, precision@k, recall@k, mean average precision, captures one dimension of recommendation quality. They tell you whether your model can match users to items.

They don't tell you if users will stay.

Consider a music streaming service. Your system recommends Drake because the user listened to Drake yesterday. Technically relevant. High precision. But it's boring, users already know Drake exists.

Or a movie platform that recommends superhero films to superhero fans. Accurate? Absolutely. But it creates filter bubbles where users never discover outside their comfort zone, eventually leading to churn.

The gap between "relevant" and "delightful" is where retention lives.

Building a Real Evaluation Framework

Practical recommendation testing requires five dimensions working together:

1. Diversity Metrics

Measure whether recommendations cover different categories, genres, or item types. A system recommending five nearly-identical movies isn't helping users explore.

Use Jaccard similarity to measure item overlap across recommendations, or track intra-list diversity (how different items are from each other):

function calculateIntraListDiversity(items: Item[]): number {
 let totalDissimilarity = 0;
 for (let i = 0; i < items.length; i++) {
 for (let j = i + 1; j < items.length; j++) {
 totalDissimilarity += 1 - calculateSimilarity(items[i], items[j]);
 }
 }
 const pairCount = (items.length * (items.length - 1)) / 2;
 return totalDissimilarity / pairCount;
}

function calculateSimilarity(a: Item, b: Item): number {
 // Use embedding distance, category overlap, or hybrid approach
 return 1 - euclideanDistance(a.embedding, b.embedding);
}

2. Novelty Scores

Track how often you recommend items users haven't encountered before. Novelty prevents the "I already know this" problem.

Calculate the percentage of fresh recommendations (items not in user's interaction history) for each user, then average across cohorts:

function calculateNovelty(
 recommendations: Item[],
 userHistory: Set
): number {
 const freshItems = recommendations.filter(
 item => !userHistory.has(item.id)
 );
 return freshItems.length / recommendations.length;
}

Set targets per user segment. New users need higher novelty. Long-time users might tolerate some familiarity if it's still valuable.

3. Serendipity Detection

This is the magic metric, recommendations that surprise users positively. Items that are unexpected but resonate strongly.

Serendipity is hard to measure directly (it requires user intent analysis), but you can approximate it by finding recommendations that:

• Were not predicted by popularity bias
• Fall outside user's typical interaction pattern
• Have high engagement despite low prior exposure

function estimateSerendipity(
 recommendation: Item,
 userProfile: UserProfile,
 engagementScore: number
): number {
 const popularityBias = recommendation.globalPopularity;
 const profileAlignment = calculateProfileAlignment(
 recommendation,
 userProfile
 );
 const unexpectednessScore = 1 - profileAlignment;

 return (engagementScore * unexpectednessScore) / (1 + popularityBias);
}

4. Coverage & Long-Tail Analysis

Does your system recommend 80% of your catalog, or just the same 200 items for everyone? Coverage measures inventory utilization.

Track catalog coverage (% of items recommended at least once) and user coverage (% of users who received non-default recommendations):

function calculateCatalogCoverage(
 allRecommendations: Recommendation[],
 catalogSize: number
): number {
 const recommendedItems = new Set(
 allRecommendations.map(r => r.itemId)
 );
 return recommendedItems.size / catalogSize;
}

function calculateLongTailRatio(
 recommendations: Recommendation[]
): number {
 const itemFrequency = new Map();

 recommendations.forEach(rec => {
 itemFrequency.set(
 rec.itemId,
 (itemFrequency.get(rec.itemId) || 0) + 1
 );
 });

 const sorted = Array.from(itemFrequency.values()).sort((a, b) => b - a);
 const top20Percent = Math.ceil(sorted.length * 0.2);
 const top20Count = sorted.slice(0, top20Percent).reduce((a, b) => a + b, 0);

 return 1 - (top20Count / recommendations.length);
}

A/B Testing Recommendations Beyond Click-Through

A/B testing recommendations requires rethinking your success metrics. Click-through rate alone is a trap.

Set up holistic evaluation in your test groups:

interface RecommendationTestMetrics {
 clickThroughRate: number;
 conversionRate: number;
 averageSessionDuration: number;
 diversityScore: number;
 noveltyScore: number;
 userRetention7d: number;
 userRetention30d: number;
 cartValue?: number;
 timeToChurn?: number;
}

function evaluateRecommendationVariant(
 variant: 'control' | 'treatment',
 userSample: User[]
): RecommendationTestMetrics {
 return {
 clickThroughRate: calculateCTR(variant, userSample),
 conversionRate: calculateConversion(variant, userSample),
 averageSessionDuration: calculateAvgSession(variant, userSample),
 diversityScore: calculateDiversity(variant, userSample),
 noveltyScore: calculateNovelty(variant, userSample),
 userRetention7d: calculateRetention(variant, userSample, 7),
 userRetention30d: calculateRetention(variant, userSample, 30),
 cartValue: calculateAvgCartValue(variant, userSample),
 };
}

Run Maturity Curves

Recommendations often have delayed effects. Users need time to warm up to serendipitous suggestions. Run longer test windows, at least 4 weeks, before declaring winners.

Plot engagement curves by test day to spot delayed activation or late-stage issues.

Business Metric Alignment: The Silent Killer

Technical metrics don't matter if they don't drive business outcomes. Align your evaluation framework to what actually generates revenue or retention.

For e-commerce, it's order value and repurchase rate. For streaming, it's churn. For social, it's daily active users and time spent.

If your metrics improve but churn increases, your testing framework is broken.

Build a scorecard that weights technical metrics against business KPIs:

function calculateRecommendationScore(metrics: RecommendationTestMetrics): {
 const weights = {
 retention30d: 0.35, // Business critical
 conversionRate: 0.25, // Direct revenue
 cartValue: 0.15, // AOV impact
 diversityScore: 0.15, // Long-term health
 noveltyScore: 0.10, // Engagement health
 };

 let score = 0;
 score += metrics.userRetention30d * weights.retention30d;
 score += metrics.conversionRate * weights.conversionRate;
 score += (metrics.cartValue || 0) * weights.cartValue;
 score += metrics.diversityScore * weights.diversityScore;
 score += metrics.noveltyScore * weights.noveltyScore;

 return score;
}

Detecting and Preventing Filter Bubbles

Filter bubbles form slowly. Users get comfortable, engagement looks flat, then suddenly they're inactive.

Monitor these signals:

• Category entropy: How diverse are user browsing patterns over time? Declining entropy = narrowing bubble
• Cold-start fatigue: Are new users seeing diverse recommendations, or are they immediately pigeon-holed?
• Cross-category discovery: What % of users interact outside their primary category each month? Trending down = bubble formation
• Recommendation-to-organic ratio: Are users finding items through recs or organically? Heavy rec-dependence signals potential bubbles

Set up automated alerts when entropy drops below thresholds per user cohort. Implement mandatory diversity floors, always include 20-30% recommendations outside the user's primary cluster.

Running Production Evaluations

Shadow mode is your friend. Deploy new recommendation logic without showing it to users, collect metrics silently, validate before going live:

async function evaluateRecommendationVariant(
 userId: string,
 productionAlgo: RecommendationEngine,
 candidateAlgo: RecommendationEngine
): Promise {
 const productionRecs = await productionAlgo.recommend(userId, 10);
 const candidateRecs = await candidateAlgo.recommend(userId, 10);

 // Log both, serve only production
 await logMetrics({
 userId,
 variant: 'production',
 recommendations: productionRecs,
 timestamp: Date.now(),
 });

 await logMetrics({
 userId,
 variant: 'candidate',
 recommendations: candidateRecs,
 timestamp: Date.now(),
 });

 // Later: analyze shadow metrics
 // If candidate beats all thresholds, promote to production
}