Embedding Quality Testing: Because Garbage In, Garbage Out Still Applies

You've probably experienced this: a RAG system that worked great last month suddenly returns irrelevant search results. Or your semantic search starts surfacing wrong documents. The model still reports good accuracy metrics. The API endpoint responds fine. But everything is subtly broken.

This is almost always an embedding problem. The embeddings drift. The embedding model changes. The vector database gets corrupted. Something shifts in how your text maps to semantic space, and suddenly "what is AI quality?" doesn't find your blog post about AI QA.

The fix is testing embeddings as a first-class concern, not an afterthought.

Why Embedding Testing Is Different

Traditional ML models map inputs to outputs. Embeddings are weirder: they're just vectors that capture semantic meaning. You can't easily test an embedding in isolation. You have to test it in context: "Does this embedding cluster with other similar embeddings?"

This also means embedding failures are sneaky. Your embedding model could be working perfectly, but:

• The vector database could have stale data
• The embedding model could have been updated and the existing vectors not re-indexed
• Input preprocessing could change (lowercasing, tokenization, language) and break semantic alignment
• The corpus could shift (new documents added, old ones removed)

Embedding problems hide in production because there's no obvious error. The system just returns wrong results confidently.

You need continuous monitoring that verifies: "Are semantically similar items actually close in vector space?"

Test 1: Cosine Similarity Distribution

The foundation of embedding quality is that similar items have high cosine similarity, dissimilar items have low cosine similarity. Test that your similarity scores actually reflect semantic distance:

interface EmbeddingTestPair {
 text1: string;
 text2: string;
 expectedSimilarity: "high" | "medium" | "low";
 similarityThreshold?: { min?: number; max?: number };
}

async function testCosineSimilarity(
 embeddingModel: EmbeddingProvider,
 testPairs: EmbeddingTestPair[]
) {
 const results = [];

 for (const pair of testPairs) {
 const [emb1, emb2] = await Promise.all([
 embeddingModel.embed(pair.text1),
 embeddingModel.embed(pair.text2)
 ]);

 // Cosine similarity between vectors
 const similarity = cosineSimilarity(emb1, emb2);

 // Expected ranges
 const expectedRange =
 pair.similarityThreshold ||
 getExpectedRange(pair.expectedSimilarity);

 const pass =
 similarity >= expectedRange.min &&
 similarity <= expectedRange.max;

 results.push({
 text1: pair.text1,
 text2: pair.text2,
 expected: pair.expectedSimilarity,
 similarity: similarity.toFixed(3),
 pass
 });
 }

 return results;
}

// Expected ranges (adjust for your embedding model)
function getExpectedRange(similarity: string) {
 switch (similarity) {
 case "high":
 return { min: 0.80, max: 1.0 }; // Semantically very similar
 case "medium":
 return { min: 0.50, max: 0.79 }; // Related but not identical
 case "low":
 return { min: 0.0, max: 0.49 }; // Unrelated
 default:
 return { min: 0, max: 1 };
 }
}

// Test cases
const testCases: EmbeddingTestPair[] = [
 {
 text1: "How do I book a flight?",
 text2: "I want to reserve a plane ticket",
 expectedSimilarity: "high"
 },
 {
 text1: "What's the weather today?",
 text2: "How do I book a flight?",
 expectedSimilarity: "low"
 },
 {
 text1: "AI quality engineering",
 text2: "Testing machine learning models",
 expectedSimilarity: "medium"
 }
];

// Run test
const results = await testCosineSimilarity(embeddingProvider, testCases);
const failures = results.filter(r => !r.pass);
if (failures.length > 0) {
 console.error("Embedding similarity test failed", failures);
 process.exit(1);
}

This catches gross failures: if your cosine similarity distribution is inverted (dissimilar items are closer than similar items), you'll catch it immediately.

Test 2: Nearest Neighbor Accuracy (Retrieval Precision)

Here's the real test: given a query, do the nearest vectors in your database actually match semantically?

interface RetrievalTestCase {
 query: string;
 corpus: string[]; // Documents to search over
 expectedMatches: number[]; // Indices of relevant documents
 topK: number; // How many results to return
 minPrecision: number; // E.g., 0.8 = 80% of top-k should be relevant
}

async function testRetrieval(
 embeddingModel: EmbeddingProvider,
 vectorDb: VectorDatabase, // Pinecone, Weaviate, Chroma
 testCases: RetrievalTestCase[]
) {
 const results = [];

 for (const testCase of testCases) {
 // Embed and index corpus
 const corpusEmbeddings = await Promise.all(
 testCase.corpus.map(doc => embeddingModel.embed(doc))
 );

 await vectorDb.upsertBatch(
 testCase.corpus.map((doc, idx) => ({
 id: idx.toString(),
 vector: corpusEmbeddings[idx],
 metadata: { text: doc }
 }))
 );

 // Search for query
 const queryEmbedding = await embeddingModel.embed(testCase.query);
 const searchResults = await vectorDb.search(
 queryEmbedding,
 testCase.topK
 );

 // Check if returned documents are actually relevant
 const retrievedIndices = searchResults.map(r => parseInt(r.id));
 const correctRetrievals = retrievedIndices.filter(idx =>
 testCase.expectedMatches.includes(idx)
 ).length;

 const precision = correctRetrievals / testCase.topK;

 results.push({
 query: testCase.query,
 topK: testCase.topK,
 precisionRequired: testCase.minPrecision,
 precisionAchieved: precision.toFixed(3),
 pass: precision >= testCase.minPrecision,
 retrieved: searchResults.map(r => r.metadata.text)
 });
 }

 return results;
}

// Test cases
const retrievalTests: RetrievalTestCase[] = [
 {
 query: "How do I optimize my database?",
 corpus: [
 "Database performance tuning: indexing strategies",
 "SQL optimization techniques",
 "How to cook chicken",
 "NoSQL databases for scalability",
 "Cloud infrastructure setup"
 ],
 expectedMatches: [0,1,3], // First 3 are relevant
 topK: 5,
 minPrecision: 0.6 // At least 3 out of 5 should be relevant
 },
 {
 query: "What's the weather?",
 corpus: [
 "Weather forecasting models",
 "Meteorology basics",
 "How to build a house",
 "Climate change data",
 "Cooking recipes"
 ],
 expectedMatches: [0,1,3],
 topK: 5,
 minPrecision: 0.6
 }
];

const retrievalResults = await testRetrieval(
 embeddingModel,
 vectorDb,
 retrievalTests
);

This is the most important test. It directly measures what your users care about: "Did the search find the right documents?"

Test 3: Embedding Drift Detection

Over time, embeddings can drift for several reasons:

• The embedding model was updated (new version of text-embedding-3-large)
• Input preprocessing changed
• The vector database was migrated
• Model quantization or distillation introduced subtle changes

Detect drift by comparing embedding distributions:

interface EmbeddingSnapshot {
 timestamp: string;
 embeddings: number[][];
 corpusSize: number;
 metadata: Record;
}

async function detectEmbeddingDrift(
 currentSnapshot: EmbeddingSnapshot,
 previousSnapshot: EmbeddingSnapshot,
 driftThreshold = 0.15
) {
 // Compare embedding distributions
 const currentStats = calculateEmbeddingStats(currentSnapshot.embeddings);
 const previousStats = calculateEmbeddingStats(previousSnapshot.embeddings);

 // Check L2 norm drift (overall magnitude)
 const meanNormCurrent = currentStats.meanNorm;
 const meanNormPrevious = previousStats.meanNorm;
 const normDrift = Math.abs(
 (meanNormCurrent - meanNormPrevious) / meanNormPrevious
 );

 // Check centroid drift (average embedding direction)
 const centroidCurrent = calculateCentroid(currentSnapshot.embeddings);
 const centroidPrevious = calculateCentroid(previousSnapshot.embeddings);
 const centroidDistance = cosineSimilarity(
 centroidCurrent,
 centroidPrevious
 );
 const centroidDrift = 1 - centroidDistance; // Convert to distance

 // Check variance drift
 const varianceCurrent = currentStats.variance;
 const variancePrevious = previousStats.variance;
 const varianceDrift = Math.abs(
 (varianceCurrent - variancePrevious) / variancePrevious
 );

 const driftDetected =
 normDrift > driftThreshold ||
 centroidDrift > driftThreshold ||
 varianceDrift > driftThreshold;

 return {
 normDrift: normDrift.toFixed(4),
 centroidDrift: centroidDrift.toFixed(4),
 varianceDrift: varianceDrift.toFixed(4),
 driftDetected,
 severity: driftDetected ? (
 centroidDrift > 0.3 ? "HIGH" : "MEDIUM"
 ) : "NONE",
 recommendation: driftDetected
 ? "Re-embed corpus with new embedding model"
 : "No action needed"
 };
}

function calculateEmbeddingStats(embeddings: number[][]) {
 const norms = embeddings.map(e => Math.sqrt(
 e.reduce((sum, val) => sum + val * val, 0)
 ));
 const meanNorm = norms.reduce((a, b) => a + b, 0) / norms.length;
 const variance = norms.reduce((sum, norm) =>
 sum + Math.pow(norm - meanNorm, 2), 0
 ) / norms.length;

 return {
 meanNorm,
 variance,
 minNorm: Math.min(...norms),
 maxNorm: Math.max(...norms)
 };
}

Test 4: Embedding Model Update Detection

The sneakiest failure: your embedding model gets updated (silently, by the provider), and suddenly all your embeddings are in a different space:

interface EmbeddingModelVersionTest {
 sampleTexts: string[];
 previousModelVersion: string;
 currentModelVersion: string;
}

async function detectModelVersionChange(
 embeddingModel: EmbeddingProvider,
 previousEmbeddings: Record,
 sampleTexts: string[],
 similarityThreshold = 0.95
) {
 const currentEmbeddings = await Promise.all(
 sampleTexts.map(text => embeddingModel.embed(text))
 );

 const similarities = currentEmbeddings.map((current, idx) => {
 const previous = previousEmbeddings[sampleTexts[idx]];
 return cosineSimilarity(current, previous);
 });

 const avgSimilarity =
 similarities.reduce((a, b) => a + b, 0) / similarities.length;

 return {
 averageSimilarity: avgSimilarity.toFixed(4),
 modelChanged: avgSimilarity < similarityThreshold,
 affectedPercentage: (
 (similarities.filter(s => s < similarityThreshold).length /
 similarities.length) *
 100
 ).toFixed(1),
 recommendation:
 avgSimilarity < similarityThreshold
 ? "Embedding model version changed. Re-embed entire corpus."
 : "No version change detected"
 };
}

// Before deploying an embedding model update:
// 1. Capture baseline embeddings of sample texts
// 2. After update, compare new embeddings to baseline
// 3. If similarity < 0.95, you need to re-index

Building Your Embedding Test Suite

Start with this:

1. 50 similarity test pairs (high/medium/low) covering your domain
2. 10 retrieval test cases testing search over representative corpus
3. Baseline snapshot of current embeddings and their statistics
4. Continuous drift monitoring every 24 hours
5. Model version monitoring whenever the embedding model could change

Integrate into CI/CD:

- name: Test Embedding Quality
 run: |
 npm run test:embeddings \
 --similarity-threshold 0.8 \
 --retrieval-precision 0.75 \
 --drift-threshold 0.15 \
 --fail-on-regression true

If any embedding test fails, block deployment. Stale or degraded embeddings are silent killers.

When Things Break (And They Will)

If embedding tests start failing:

1. Check if the embedding model changed (compare with baseline)
2. Check if corpus changed significantly
3. Check if vector database was migrated or corrupted
4. Check if input preprocessing changed
5. If nothing obvious: re-embed a sample of corpus and compare similarity distributions

The nuclear option: re-embed your entire corpus with the current model. This is usually necessary after a major embedding model update.