Building an AI Quality Dashboard: The 12 Metrics That Actually Matter

You just shipped your first AI feature. It works. Users like it. Now what?

The Core Metric Taxonomy

Think of AI quality metrics in tiers. Some are leading indicators (they warn of problems), some are lagging indicators (they measure problems), and some are operational (they keep the lights on).

Metric	Category	What It Measures	Alert Threshold
Accuracy	Lagging (Quality)	Correct outputs vs total outputs	Below baseline -5%
Hallucination Rate	Leading (Quality)	False claims as % of outputs	Above 3% (varies by domain)
Latency (p50/p95/p99)	Operational	Response time at percentiles	p95 > 2x baseline
Cost per Query	Financial	Total cost / total queries	Monthly trend increase
Bias Score	Leading (Quality)	Fairness across segments	Variance > threshold
Safety Score	Lagging (Compliance)	Unsafe outputs / total	Any unsafe output
User Satisfaction	Lagging (UX)	Thumbs up/down ratio	<75% satisfaction
Error Rate	Operational	Failed requests %	>0.5%
Coverage	Leading (Quality)	Queries system can handle	<95% coverage
Data Freshness	Leading (Quality)	Age of training/retrieval data	Data >1 year old
Drift Score	Leading (Quality)	Model behavior change	Significant shift detected
Uptime	Operational	System availability %	<99.5%

Accuracy: The Trap of False Simplicity

Accuracy is the metric everyone wants. It's also the most dangerous metric if you get it wrong.

Here's the trap: if you measure accuracy as "percentage of outputs that are correct, " you'll optimize for the wrong things. A recommendation system that's 95% accurate on popular items might be 40% accurate on niche items. That single 95% number hides everything important.

Slice accuracy by everything that matters:

const accuracyMetrics = {
 // Overall accuracy (should trend upward or stay flat)
 overall: 0.92,

 // Broken down by use case
 byUseCase: {
 simpleFactual: 0.97,
 complexReasoning: 0.78,
 codeGeneration: 0.85
 },

 // Broken down by user segment
 bySegment: {
 powerUsers: 0.95,
 newUsers: 0.88,
 freeUsers: 0.82
 },

 // Broken down by query complexity
 byQueryLength: {
 short: 0.94, // < 100 words
 medium: 0.91, // 100-500 words
 long: 0.87 // > 500 words
 },

 // Critical: accuracy over time (detect drift)
 trend7Day: 0.91,
 trend30Day: 0.90,
 trend90Day: 0.89,

 // By response type (text vs code vs structured)
 byResponseType: {
 freeformText: 0.93,
 structuredData: 0.96,
 code: 0.81,
 markdown: 0.94
 }
};

// Alert if accuracy drops in ANY segment
function checkAccuracyRegression() {
 const previousWeekAccuracy = 0.920;
 const currentWeekAccuracy = 0.912;

 if (currentWeekAccuracy < previousWeekAccuracy * 0.95) {
 alert("Accuracy regression detected");
 }

 // Also check for divergence (one segment drops)
 if (accuracyMetrics.bySegment.freeUsers < 0.80) {
 alert(
 "Free user accuracy below threshold, potential UX impact"
 );
 }
}

"Accuracy without context is theater. What matters is: accurate for whom, on what type of query, and is it getting better or worse?"

Your accuracy dashboard should show trends, not just current numbers. A 5% regression is fine if it's recovering. A 0.5% regression that's accelerating is an emergency.

Hallucination Rate: Your Leading Indicator

Hallucinations (false claims presented as fact) are your most important leading indicator. When hallucination rate increases, accuracy is about to drop. When hallucination rate decreases, quality improvements are coming.

The problem: hallucination detection is expensive. You can't manually verify every output. to do it at scale:

// Automated hallucination detection strategy
async function detectHallucinations(output, sourceContext) {
 // Strategy 1: Contradiction detection
 // If the output contradicts itself, it's probably hallucinating
 const selfContradictions = await analyzeInternalConsistency(output);

 // Strategy 2: Source verification
 // For retrieval-augmented outputs, check if claims are in sources
 const unsubstantiatedClaims = findClaimsNotInSources(
 output,
 sourceContext
 );

 // Strategy 3: Confidence scoring
 // Re-run the same prompt and see if you get consistent answers
 const confidenceScore = await measureOutputConsistency(output);

 // Strategy 4: Fact-checking with a second model
 // Use a smaller, cheaper model as a fact-checker
 const factsToCheck = extractFactualClaims(output);
 const factCheckResults = await runFactChecker(factsToCheck);

 return {
 selfContradictions: selfContradictions.length,
 unsubstantiatedClaims: unsubstantiatedClaims.length,
 confidenceScore: confidenceScore,
 factCheckFailures: factCheckResults.failures
 };
}

// Monitor hallucination rate continuously
function trackHallucinationTrend() {
 const dailyHallucinationRates = [
 { date: "2026-04-01", rate: 0.032 },
 { date: "2026-04-02", rate: 0.035 },
 { date: "2026-04-03", rate: 0.041 },
 { date: "2026-04-04", rate: 0.052 }
 ];

 // Alert if trending upward
 const recentTrend = dailyHallucinationRates.slice(-7);
 const trend = calculateTrend(recentTrend);

 if (trend > 0.002) {
 // Increasing by >0.2% per day
 alert("Hallucination rate trending upward, investigate immediately");
 }

 // Alert if abnormally high
 if (recentTrend[recentTrend.length - 1].rate > 0.05) {
 alert("Hallucination rate critically high");
 }
}

Setting thresholds: this varies massively by domain. A recommendation system might tolerate 5% hallucination rate (false recommendations). A medical AI must be near zero. Define your threshold explicitly.

Latency Percentiles: Know Your Outliers

Monitoring p50 latency is useless. p95 and p99 tell you what real users experience.

// Track latency at percentiles
function trackLatencyPercentiles() {
 const latencies = [
 450,480,520,550,580,610,640,670,700,750,800,850,
 900,950,1000,1100,1200,1500,2000,3000
 ]; // Sorted

 const n = latencies.length;
 const p50 = latencies[Math.floor(n * 0.5)]; // 670ms
 const p95 = latencies[Math.floor(n * 0.95)]; // 2400ms
 const p99 = latencies[Math.floor(n * 0.99)]; // 2980ms

 console.log(`P50: ${p50}ms`);
 console.log(`P95: ${p95}ms`);
 console.log(`P99: ${p99}ms`);

 // Alert thresholds should be based on user experience
 const targetP95 = 2000; // 2 seconds for search
 const targetP99 = 5000; // 5 seconds is acceptable max

 if (p95 > targetP95) {
 alert("P95 latency exceeds target");
 }

 if (p99 > targetP99) {
 alert("P99 latency critically high, worst 1% of users suffering");
 }
}

Cost Per Query: Your Financial Health

Track this obsessively. Cost per query is the leading indicator of whether your AI feature is sustainable.

// Cost tracking per model, per feature
function trackCostMetrics() {
 const costMetrics = {
 totalMonthlySpend: 45000,
 totalQueriesThisMonth: 2100000,
 costPerQuery: 0.0214, // $0.0214 per query

 byModel: {
 "claude-3-5-sonnet": {
 queries: 1200000,
 costPerQuery: 0.003,
 totalCost: 3600
 },
 "claude-3-opus": {
 queries: 500000,
 costPerQuery: 0.015,
 totalCost: 7500
 },
 "gpt-4": {
 queries: 400000,
 costPerQuery: 0.03,
 totalCost: 12000
 }
 },

 byFeature: {
 search: {
 queries: 1000000,
 costPerQuery: 0.012,
 userValue: "high"
 },
 recommendations: {
 queries: 500000,
 costPerQuery: 0.025,
 userValue: "medium"
 },
 analysis: {
 queries: 600000,
 costPerQuery: 0.045,
 userValue: "very high"
 }
 },

 trend: {
 costPerQuery7DaysAgo: 0.0205,
 costPerQuery30DaysAgo: 0.0189,
 costPerQuery90DaysAgo: 0.0178
 }
 };

 // Alert if cost per query is increasing
 if (
 costMetrics.costPerQuery >
 costMetrics.trend.costPerQuery30DaysAgo * 1.1
 ) {
 alert("Cost per query increased 10%, investigate efficiency");
 }

 // Identify expensive features to optimize
 const expensiveFeatures = Object.entries(
 costMetrics.byFeature
 ).filter(([_, data]) => data.costPerQuery > 0.03);

 if (expensiveFeatures.length > 0) {
 console.log("Consider optimizing:", expensiveFeatures);
 }
}

Drift Score: Detecting Model Behavior Change

Model drift is when your model's behavior changes without you changing anything. Detecting it requires comparing current outputs against historical baselines.

// Detect model drift by comparing output distributions
async function calculateDriftScore() {
 // Compare output distribution: length, token diversity, etc.
 const historicalOutputs = await getHistoricalOutputs();
 const currentOutputs = await getCurrentOutputs();

 const metrics = {
 // Measure 1: Output length distribution
 historicalLengthMean: getAvgLength(historicalOutputs),
 currentLengthMean: getAvgLength(currentOutputs),

 // Measure 2: Vocabulary diversity (unique tokens)
 historicalDiversity: getVocabDiversity(historicalOutputs),
 currentDiversity: getVocabDiversity(currentOutputs),

 // Measure 3: Semantic similarity (are answers covering same topics?)
 historicalSimilarity: getAvgSemanticConsistency(
 historicalOutputs
 ),
 currentSimilarity: getAvgSemanticConsistency(currentOutputs),

 // Measure 4: Topic distribution (what's being discussed?)
 historicalTopics: analyzeTopicDistribution(historicalOutputs),
 currentTopics: analyzeTopicDistribution(currentOutputs)
 };

 // Calculate drift
 const drift = {
 lengthDrift: Math.abs(
 metrics.historicalLengthMean - metrics.currentLengthMean
 ),
 diversityDrift: Math.abs(
 metrics.historicalDiversity - metrics.currentDiversity
 ),
 topicDrift: calculateJSDivergence(
 metrics.historicalTopics,
 metrics.currentTopics
 )
 };

 return drift;
}

// Example alert
function checkForDrift() {
 const drift = calculateDriftScore();

 if (drift.topicDrift > 0.15) {
 alert("Significant topic drift detected, model behavior changed");
 }
}

Building the Dashboard UI

The metrics are only useful if you can see them. A good AI quality dashboard should answer these questions at a glance:

• Is quality improving or degrading?
• Where are the problems (which segment, use case, model)?
• Are we within budget and SLOs?
• What needs immediate attention?

// Dashboard structure
const dashboardLayout = {
 topBar: {
 // Health status
 healthStatus: "HEALTHY", // RED, YELLOW, GREEN
 lastUpdated: "2 minutes ago",
 alertCount: 2
 },

 mainMetrics: {
 // The 4-8 most important metrics in big cards
 accuracy: { value: 0.92, trend: "up 1.2%", status: "green" },
 latencyP95: { value: "1.8s", trend: "down 0.2s", status: "green" },
 costPerQuery: {
 value: "$0.0214",
 trend: "up 2%",
 status: "yellow"
 },
 hallucination: { value: "3.2%", trend: "up 0.4%", status: "yellow" },
 userSatisfaction: {
 value: "82%",
 trend: "stable",
 status: "green"
 },
 uptime: { value: "99.97%", trend: "stable", status: "green" }
 },

 details: {
 // Breakdown by segment/use case
 accuracyBySegment: [
 { segment: "power users", accuracy: 0.96, samples: 50000 },
 { segment: "regular users", accuracy: 0.92, samples: 120000 },
 { segment: "new users", accuracy: 0.85, samples: 30000 }
 ],

 latencyByPercentile: [
 { percentile: "p50", value: "0.65s" },
 { percentile: "p95", value: "1.8s" },
 { percentile: "p99", value: "4.2s" }
 ],

 costByFeature: [
 { feature: "search", costPerQuery: "$0.012", queries: 1000000 },
 {
 feature: "recommendations",
 costPerQuery: "$0.025",
 queries: 500000
 },
 {
 feature: "analysis",
 costPerQuery: "$0.045",
 queries: 600000
 }
 ]
 },

 alerts: [
 { severity: "warning", message: "Cost per query up 2% this week" },
 { severity: "warning", message: "Hallucination rate trending upward" }
 ],

 trends: {
 // 30/90 day trends
 accuracyTrend: [
 { date: "2026-03-05", accuracy: 0.89 },
 // ... more data points
 { date: "2026-04-04", accuracy: 0.92 }
 ]
 }
};

Alerting Strategy

Metrics without alerting are just theater. Your alerting should answer: "What actually requires a human right now?"

• Page-worthy (p0): Safety violations, uptime below 99%, error rate above 2%
• Urgent (p1): Accuracy drop >5%, cost per query up >20%, p99 latency >5x baseline
• Important (p2): Hallucination rate trending up, accuracy segment divergence
• Info (p3): Cost trends, minor latency shifts, user satisfaction changes