Snapshot Testing for AI: Because assertEqual Is Dead

You've seen the test failure:

Expected: "The quick brown fox"
Received: "The swift brown fox"

Locally it passes. In CI it fails. Your LLM generated a synonym instead of exact text. Standard snapshot testing has become a pain-point for AI systems.

The problem: snapshots are written for deterministic outputs. AI isn't deterministic. You need a new approach.

The Case Against Exact-Match Snapshots

Jest snapshots, Percy visual snapshots, and similar tools assume determinism. You run a function, store the output, compare future outputs byte-for-byte.

AI systems violate this assumption fundamentally:

• Temperature sampling means the same prompt produces slightly different outputs
• Semantic equivalence matters more than word-for-word matching
• Context windows shift results subtly between model versions
• Hallucinations are probabilistic, you can't snapshot them away

The result: either you lock tests with low temperature (killing diversity) or you maintain hundreds of false-positive failures that developers ignore.

Developers ignoring test failures is how bugs ship.

Semantic Snapshots: Testing Meaning, Not Text

The core insight: you don't need exact matches. You need semantic similarity.

Convert outputs to embeddings, store the embedding snapshot, and compare new outputs against it using cosine similarity. When similarity dips below a threshold, flag it for review.

How Semantic Snapshots Work

Here's the mental model: instead of storing "The assistant generated this exact sentence", you store the semantic representation of that sentence in vector space. New outputs are compared to that vector.

This lets minor variations (synonyms, rephrasing) pass while catching actual meaning shifts.

import Anthropic from "@anthropic-ai/sdk";

interface SemanticSnapshot {
 id: string;
 embedding: number[];
 text: string;
 model: string;
 timestamp: number;
}

async function generateSemanticSnapshot(
 prompt: string,
 model: string = "claude-3-5-sonnet-20241022"
): Promise {
 const client = new Anthropic();

 // Generate the output
 const message = await client.messages.create({
 model,
 max_tokens: 1024,
 messages: [{ role: "user", content: prompt }],
 });

 const text =
 message.content[0].type === "text" ? message.content[0].text : "";

 // Get embedding via a separate API (e.g., OpenAI embeddings)
 // For this example, simulating with a hash-based approach
 const embedding = await getEmbedding(text);

 return {
 id: generateUUID(),
 embedding,
 text,
 model,
 timestamp: Date.now(),
 };
}

async function compareWithSnapshot(
 newOutput: string,
 snapshot: SemanticSnapshot,
 threshold: number = 0.85
): Promise<{
 passes: boolean;
 similarity: number;
 flagged: boolean;
}> {
 const newEmbedding = await getEmbedding(newOutput);
 const similarity = cosineSimilarity(newEmbedding, snapshot.embedding);

 return {
 passes: similarity >= threshold,
 similarity,
 flagged: similarity < threshold && similarity > 0.7, // Needs review
 };
}

function cosineSimilarity(a: number[], b: number[]): number {
 let dotProduct = 0;
 let normA = 0;
 let normB = 0;

 for (let i = 0; i < a.length; i++) {
 dotProduct += a[i] * b[i];
 normA += a[i] * a[i];
 normB += b[i] * b[i];
 }

 return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

async function getEmbedding(text: string): Promise {
 // In production, use actual embedding service
 // This is a placeholder showing the interface
 const response = await fetch("https://api.openai.com/v1/embeddings", {
 method: "POST",
 headers: {
 Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 input: text,
 model: "text-embedding-3-small",
 }),
 });

 const data = await response.json();
 return data.data[0].embedding;
}

Building a Semantic Snapshot Test Suite

Set up a system that stores snapshots, compares new outputs, and flags drift automatically:

interface SnapshotTestResult {
 testId: string;
 passed: boolean;
 similarity: number;
 action: "pass" | "flag" | "fail";
 requiresReview: boolean;
 diff?: string;
}

class SemanticSnapshotTester {
 private snapshots: Map = new Map();
 private similarityThreshold = 0.85;
 private reviewThreshold = 0.70;

 async testOutput(
 testId: string,
 output: string,
 snapshot?: SemanticSnapshot
 ): Promise {
 if (!snapshot) {
 throw new Error(`No snapshot found for test ${testId}`);
 }

 const { similarity } = await compareWithSnapshot(
 output,
 snapshot,
 this.similarityThreshold
 );

 const passed = similarity >= this.similarityThreshold;
 const requiresReview =
 !passed && similarity >= this.reviewThreshold;

 let action: "pass" | "flag" | "fail" = "pass";
 if (requiresReview) action = "flag";
 if (!passed && similarity < this.reviewThreshold) action = "fail";

 return {
 testId,
 passed,
 similarity,
 action,
 requiresReview,
 diff: generateDiff(snapshot.text, output),
 };
 }

 flagForReview(result: SnapshotTestResult): void {
 if (result.requiresReview) {
 console.log(
 `[REVIEW NEEDED] ${result.testId}: similarity ${result.similarity.toFixed(3)}`
 );
 console.log(result.diff);
 }
 }
}

function generateDiff(original: string, updated: string): string {
 // Simplified diff, in production use a real diff library
 const changes: string[] = [];
 if (original.length !== updated.length) {
 changes.push(
 `Length: ${original.length} → ${updated.length}`
 );
 }
 return changes.join("\n");
}

Drift Detection & Automated Alerts

Track semantic drift over time. When outputs consistently diverge from snapshots, even if individual tests "pass", that's a signal of model regression or environmental change.

interface DriftMetrics {
 testId: string;
 window: { start: Date; end: Date };
 averageSimilarity: number;
 similarityTrend: "stable" | "improving" | "degrading";
 alertTriggered: boolean;
}

function calculateDrift(
 similarityHistory: number[],
 window: number = 30
): DriftMetrics {
 const recentHistory = similarityHistory.slice(-window);
 const averageSimilarity =
 recentHistory.reduce((a, b) => a + b, 0) / recentHistory.length;

 // Calculate trend using simple linear regression
 let sumX = 0,
 sumY = 0,
 sumXY = 0,
 sumX2 = 0;
 for (let i = 0; i < recentHistory.length; i++) {
 sumX += i;
 sumY += recentHistory[i];
 sumXY += i * recentHistory[i];
 sumX2 += i * i;
 }

 const n = recentHistory.length;
 const slope = (n * sumXY - sumX * sumY) / (n * sumX2 - sumX * sumX);
 const trend: "stable" | "improving" | "degrading" =
 Math.abs(slope) < 0.01 ? "stable" : slope > 0 ? "improving" : "degrading";

 const alertTriggered = trend === "degrading" && averageSimilarity < 0.80;

 return {
 testId: "",
 window: { start: new Date(), end: new Date() },
 averageSimilarity,
 similarityTrend: trend,
 alertTriggered,
 };
}

When to Update Snapshots

Updating snapshots is risky. You need clear criteria:

• Code changes: If you changed the prompt, regenerate intentionally
• Model upgrades: If you upgraded the model version, validate new outputs before snapshotting
• Systematic improvements: If most tests show drift in a positive direction (higher quality), review and update cohesively
• Never auto-update: Always manual review before snapshot changes

Create a workflow where snapshot updates require code review and explicit approval:

async function proposeSnapshotUpdate(
 testId: string,
 newOutput: string,
 reason: "model-upgrade" | "prompt-change" | "manual-review"
): Promise<{
 approved: boolean;
 newSnapshot?: SemanticSnapshot;
}> {
 // Don't update automatically, create a GitHub issue or notification
 console.log(`
 📸 Snapshot Update Required: ${testId}
 Reason: ${reason}
 Old snapshot: ${/* old */}
 New output: ${newOutput}

 Please review and approve before updating.
 `);

 // In real system, wait for manual approval
 const approved = await waitForManualApproval(testId);

 if (approved) {
 const newSnapshot = await generateSemanticSnapshot(newOutput);
 return { approved: true, newSnapshot };
 }

 return { approved: false };
}

Fuzzy Matching for Structured Outputs

For structured outputs (JSON, YAML), combine semantic matching with schema validation:

interface StructuredSnapshot {
 schema: object;
 examples: Array<{ output: object; embedding: number[] }>;
 requiredFields: string[];
}

async function validateStructuredOutput(
 output: string,
 snapshot: StructuredSnapshot,
 semanticThreshold: number = 0.80
): Promise<{
 valid: boolean;
 schemaMatch: boolean;
 semanticMatch: boolean;
}> {
 let parsed: object;
 try {
 parsed = JSON.parse(output);
 } catch {
 return {
 valid: false,
 schemaMatch: false,
 semanticMatch: false,
 };
 }

 // Check schema
 const schemaMatch = validateAgainstSchema(
 parsed,
 snapshot.schema,
 snapshot.requiredFields
 );

 // Check semantic similarity to examples
 const embedding = await getEmbedding(JSON.stringify(parsed));
 const semanticScores = snapshot.examples.map((ex) =>
 cosineSimilarity(embedding, ex.embedding)
 );
 const maxSemanticScore = Math.max(...semanticScores);
 const semanticMatch = maxSemanticScore >= semanticThreshold;

 return {
 valid: schemaMatch && semanticMatch,
 schemaMatch,
 semanticMatch,
 };
}

function validateAgainstSchema(
 obj: object,
 schema: object,
 requiredFields: string[]
): boolean {
 for (const field of requiredFields) {
 if (!(field in obj)) return false;
 }
 return true;
}

Integrating Into CI/CD

Set up your test runner to handle semantic snapshots gracefully:

async function runSemanticSnapshotTests(
 testSuite: Array<{ id: string; prompt: string }>
): Promise<{
 passed: number;
 flagged: number;
 failed: number;
}> {
 let passed = 0,
 flagged = 0,
 failed = 0;
 const results: SnapshotTestResult[] = [];

 for (const test of testSuite) {
 const output = await generateAIOutput(test.prompt);
 const snapshot = await loadSnapshot(test.id);
 const result = await tester.testOutput(test.id, output, snapshot);

 results.push(result);

 if (result.action === "pass") passed++;
 else if (result.action === "flag") flagged++;
 else failed++;

 if (result.requiresReview) {
 console.warn(`⚠️ Review needed: ${result.testId}`);
 }
 }

 // Fail CI only on hard failures, flag others for human review
 if (failed > 0) {
 process.exit(1);
 }

 console.log(
 `Results: ${passed} passed, ${flagged} flagged, ${failed} failed`
 );
}

export { runSemanticSnapshotTests, SemanticSnapshotTester };