Testing AI Search and Retrieval: Your Users Deserve Better Than "Close Enough"

Your RAG system has been live for three months. Users use it. It seems to work. Then you measure it carefully. You pull a random sample of queries and manually check: Is the top result actually what the user was looking for? You find that 40% of the time, the top result is relevant. The second or third result is better. Users aren't scrolling to the second result. They're closing your app and using Google. This is what happens when you don't test search systematically. Search quality is invisible until it's invisibly killing your product. Let's talk about how to measure search and retrieval quality so you catch these problems before your users do.

Why Search Testing Is Hard (And Why You Need to Do It Anyway)

Search quality is not binary. A result is not simply "right" or "wrong." It's relevant, somewhat relevant, tangentially relevant, or irrelevant. This ambiguity makes testing hard. Add to that: different queries have different information needs. A query like "what's the capital of France" has a single right answer. A query like "I'm looking for ways to reduce AWS costs" has many valid answers ranked by usefulness. This is why you can't just count right vs. wrong. You need a framework that captures ranking quality. And then there's the silent killer: irrelevant results rank high, causing users to silently leave.

Search quality is invisible when it's good, and painfully visible when it's bad. You need metrics that catch the in-between cases before users do.

The Metrics That Actually Matter

Precision@K and Recall@K

Start with the basics:

def precision_at_k(retrieved_docs, relevant_docs, k=10):
 """
 Of the top K results, how many are relevant?
 Precision@10 = 8/10 = 0.8 (80% of top 10 are relevant)
 """
 top_k = set(retrieved_docs[:k])
 relevant_set = set(relevant_docs)
 return len(top_k & relevant_set) / k

def recall_at_k(retrieved_docs, relevant_docs, k=10):
 """
 Of all relevant documents, how many are in the top K?
 If there are 100 relevant docs and 8 are in top 10, recall@10 = 8/100 = 0.08
 """
 top_k = set(retrieved_docs[:k])
 relevant_set = set(relevant_docs)
 return len(top_k & relevant_set) / len(relevant_set)

Precision answers: "Are the results I'm showing relevant?" High precision = few false positives. Recall answers: "Did I find all the relevant results?" High recall = few false negatives. Trade-off: You can optimize for either, but not both. High precision means you return only the best results (users see good stuff, but miss some). High recall means you return everything plausibly relevant (users see everything, but wade through junk). Most products optimize for precision first. Users would rather see 5 great results than 50 mediocre ones.

NDCG: Normalized Discounted Cumulative Gain

NDCG captures ranking quality. It rewards putting relevant results at the top and penalizes burying them.

def ndcg(ranked_docs, relevant_docs, k=10, relevance_labels=None):
 """
 NDCG measures ranking quality.
 - Puts high value on relevant results at the top
 - Discounts results that appear lower (ranking matters)
 - Normalized so score is 0-1
 """
 # DCG (Discounted Cumulative Gain)
 dcg = 0
 for i, doc in enumerate(ranked_docs[:k], 1):
 relevance = relevance_labels.get(doc, 0) if relevance_labels else (1 if doc in relevant_docs else 0)
 dcg += relevance / math.log2(i + 1)

 # Ideal DCG (if results were perfectly ranked)
 ideal_docs = sorted(ranked_docs, key=lambda d: relevance_labels.get(d, 0), reverse=True)[:k]
 idcg = 0
 for i, doc in enumerate(ideal_docs, 1):
 relevance = relevance_labels.get(doc, 0) if relevance_labels else 1
 idcg += relevance / math.log2(i + 1)

 return dcg / idcg if idcg > 0 else 0

# Example:
# Query: "python list methods"
# Retrieved: [A (relevant), B (irrelevant), C (relevant), D (irrelevant)]
# Precision@4 = 0.5 (50% are relevant)
# NDCG@4 = 0.63 (penalizes having C in position 3 instead of position 2)

NDCG is the gold standard for search evaluation. If you pick one metric, pick NDCG.

MRR: Mean Reciprocal Rank

For queries where there's one good answer, measure how far down you need to go:

def mrr(ranked_docs, relevant_docs):
 """
 Mean Reciprocal Rank: 1 / position of first relevant result
 MRR = 1.0 if first result is relevant
 MRR = 0.5 if first relevant is at position 2
 MRR = 0.33 if first relevant is at position 3
 """
 for i, doc in enumerate(ranked_docs, 1):
 if doc in relevant_docs:
 return 1.0 / i
 return 0 # No relevant result found

Use MRR when users care about finding the one good answer fast. (Customer support lookups, fact verification.)

MAP: Mean Average Precision

Average precision across all results, averaged across queries:

def map_score(ranked_docs, relevant_docs, k=None):
 """
 Mean Average Precision: average of precision@i for each relevant doc
 Captures: "How good are my rankings overall?"
 """
 if k:
 ranked_docs = ranked_docs[:k]

 ap = 0
 num_relevant = 0
 for i, doc in enumerate(ranked_docs, 1):
 if doc in relevant_docs:
 ap += precision_at_k(ranked_docs, relevant_docs, i)
 num_relevant += 1

 return ap / len(relevant_docs) if relevant_docs else 0

Building Your Evaluation Dataset

You can't measure search quality without ground truth. You need an evaluation dataset: queries paired with relevant documents and relevance judgments. **Step 1: Collect Real Queries** Sample 300-500 actual queries from your logs. These should be representative of your usage:

# Bad: cherry-picked queries that work well
queries = ["Python list", "AWS pricing", "Redis commands"]

# Good: random sample from production
queries = random_sample_from_logs(production_queries, size=300)

# Better: stratified sample (easy, medium, hard)
easy_queries = [q for q in production_queries if len(results) > 100]
hard_queries = [q for q in production_queries if len(results) < 5]
queries = sample(easy_queries, 100) + sample(hard_queries, 100) + sample(production_queries, 100)

**Step 2: Get Relevance Judgments** For each query, have humans rate documents as: - **Relevant (1)**: Answers the query well - **Somewhat relevant (0.5)**: Partially answers the query - **Irrelevant (0)**: Doesn't address the query

# Relevance judgment format
evaluation_dataset = [
 {
 "query": "How do I optimize Python loops?",
 "documents": {
 "doc_42": 1, # Relevant: Python loop optimization guide
 "doc_115": 1, # Relevant: Performance tips article
 "doc_203": 0.5, # Somewhat: General Python performance
 "doc_501": 0, # Irrelevant: Python syntax basics
 }
 },
 # ... 299 more queries
]

Aim for 3-5 raters per query. Compute inter-rater agreement:

from sklearn.metrics import cohen_kappa_score

# Check agreement between pairs of raters
for rater1, rater2 in pairs_of_raters:
 kappa = cohen_kappa_score(rater1.ratings, rater2.ratings)
 if kappa < 0.6:
 # Low agreement. Clarify your rubric.
 pass

This dataset becomes your test set. You'll use it to measure every search improvement.

Testing Different Search Strategies

Semantic Search (Embeddings)

Embed documents and queries into vector space, retrieve by similarity:

def semantic_search(query, documents, embedding_model, top_k=10):
 """
 Semantic search: find documents with similar meaning to the query
 """
 query_embedding = embedding_model.embed(query)

 similarities = []
 for doc in documents:
 doc_embedding = embedding_model.embed(doc)
 similarity = cosine_similarity(query_embedding, doc_embedding)
 similarities.append((doc, similarity))

 ranked = sorted(similarities, key=lambda x: x[1], reverse=True)
 return [doc for doc, _ in ranked[:top_k]]

# Test this
retrieved = semantic_search("Python list optimization", documents, embedding_model)
ndcg = ndcg_score(retrieved, relevant_docs)
print(f"Semantic search NDCG: {ndcg:.3f}")

**Pros**: Catches meaning, handles paraphrasing, great for general questions. **Cons**: Misses exact matches, sensitive to embedding quality, can be slow at scale.

Keyword Search (BM25)

Traditional full-text search using term frequency and inverse document frequency:

from rank_bm25 import BM25Okapi

def keyword_search(query, documents, top_k=10):
 """
 Keyword search: find documents with matching terms
 """
 corpus = [doc.split() for doc in documents]
 bm25 = BM25Okapi(corpus)

 query_tokens = query.split()
 scores = bm25.get_scores(query_tokens)

 ranked_docs = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
 return [doc for doc, _ in ranked_docs[:top_k]]

# Test this
retrieved = keyword_search("Python list optimization", documents)
ndcg = ndcg_score(retrieved, relevant_docs)
print(f"Keyword search NDCG: {ndcg:.3f}")

**Pros**: Exact matches, fast, interpretable, no model needed. **Cons**: Misses paraphrased content, can't handle synonyms well, query-document vocabulary mismatch.

Hybrid Search

Combine semantic and keyword search, rank results by weighted combination:

def hybrid_search(query, documents, embedding_model, semantic_weight=0.6, top_k=10):
 """
 Hybrid: get results from both semantic and keyword search, combine scores
 """
 semantic_results = semantic_search(query, documents, embedding_model, top_k=50)
 keyword_results = keyword_search(query, documents, top_k=50)

 # Combine scores
 scores = {}
 for i, doc in enumerate(semantic_results, 1):
 scores[doc] = semantic_weight / i

 for i, doc in enumerate(keyword_results, 1):
 scores[doc] = scores.get(doc, 0) + (1 - semantic_weight) / i

 ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
 return [doc for doc, _ in ranked[:top_k]]

# Test this
retrieved = hybrid_search(query, documents, embedding_model)
ndcg = ndcg_score(retrieved, relevant_docs)
print(f"Hybrid search NDCG: {ndcg:.3f}")

Hybrid search almost always outperforms either approach alone.

Testing Re-Ranking

Many systems retrieve a large candidate set, then re-rank with a learned model:

def retrieve_and_rerank(query, documents, retriever, reranker, top_k=10):
 """
 Two-stage pipeline: retrieve candidates, then re-rank them
 """
 candidates = retriever.retrieve(query, top_k=100) # Get top 100
 rescored = reranker.score(query, candidates) # Re-rank them

 ranked = sorted(zip(candidates, rescored), key=lambda x: x[1], reverse=True)
 return [doc for doc, _ in ranked[:top_k]]

# Test the re-ranker separately
def test_reranker(reranker, evaluation_dataset):
 """
 Measure: does re-ranking improve ranking quality?
 If not, it's hurting your retrieval.
 """
 original_ndcg = []
 reranked_ndcg = []

 for query, relevant_docs in evaluation_dataset:
 candidates = initial_retrieve(query) # From your retriever

 # NDCG before re-ranking
 original_ndcg.append(ndcg_score(candidates, relevant_docs))

 # NDCG after re-ranking
 rescored = reranker.score(query, candidates)
 reranked = sorted(candidates, key=rescored.__getitem__, reverse=True)
 reranked_ndcg.append(ndcg_score(reranked, relevant_docs))

 improvement = np.mean(reranked_ndcg) - np.mean(original_ndcg)
 print(f"Re-ranker improvement: {improvement:+.3f}")

 return improvement > 0 # Pass if re-ranker helps

**Critical**: A bad re-ranker can destroy good retrieval. Test it.

Connecting to Real-World Impact

Better NDCG should correlate with user behavior:

# Track these metrics in production alongside NDCG
metrics_to_track = {
 "ndcg": search_quality_metric,
 "user_clicks_on_first_result": percentage_who_click_first,
 "scroll_depth": average_results_users_check,
 "time_to_answer": how_long_before_user_gets_answer,
 "task_completion_rate": percentage_who_complete_their_goal,
}

# When NDCG improves, these should improve too
# If NDCG goes up but task completion stays flat, something's wrong with your metric

From Testing to Deployment

Build a CI/CD pipeline that gates deployments on search quality:

def test_search_quality_gate(new_retriever, evaluation_dataset, baseline_ndcg=0.68):
 """
 CI/CD gate: don't deploy if search quality regresses
 """
 ndcg_scores = []
 for query, relevant_docs in evaluation_dataset:
 retrieved = new_retriever.retrieve(query)
 ndcg_scores.append(ndcg_score(retrieved, relevant_docs))

 mean_ndcg = np.mean(ndcg_scores)

 if mean_ndcg < baseline_ndcg * 0.95: # Allow 5% regression
 raise Exception(f"Search quality regressed: {mean_ndcg:.3f} < {baseline_ndcg * 0.95:.3f}")

 print(f"✓ Search quality gate passed: {mean_ndcg:.3f}")
 return True

Start Measuring Today

If you have a search or RAG system and you haven't measured it: 1. Pull 300 random queries from production 2. Hand-label relevance judgments (3-5 hours of work) 3. Measure your current NDCG 4. That number is your baseline 5. Now you can improve it The best time to measure search quality was when you shipped. The second-best time is today.