LLM-as-Judge: When AI Tests AI (And When It Shouldn't)

You've got a new LLM-based customer support bot. It generates responses to support tickets. How do you evaluate it at scale? Hiring 50 human raters is expensive. Running human evaluation for every model candidate is slow. So you train another LLM to judge the first one. Call the first one the "model, " the second one the "judge." This is happening everywhere right now. And it's working. Sometimes. The problem is, it's also failing silently in ways that are hard to detect. An LLM judge might rate a model's outputs consistently, but consistently wrong. Or it might be consistently biased toward certain types of outputs. The real danger is confidence: you run your LLM judge, get a score, and think you're done evaluating. You're not. This post explains what LLM judges are actually good at, where they fail, and how to build a reliable evaluation pipeline around them.

When LLM Judges Shine

LLM judges are surprisingly good at subjective evaluation. If you're measuring something like: - **Writing quality**: Is this response well-written, clear, professional? - **Relevance**: Does this answer address the user's question? - **Helpfulness**: Would a user find this response useful? - **Style adherence**: Does this sound like our brand voice? LLM judges often outperform humans at scale. They're consistent. They don't get tired. They can evaluate thousands of outputs in seconds. Here's a simple LLM judge for customer support quality:

def evaluate_support_response(user_query, model_response, judge_model="gpt-4"):
 """
 LLM-as-judge for subjective support quality evaluation
 Returns: score (1-5), reasoning, confidence
 """
 prompt = f"""You are an expert customer support evaluator.
Rate this response on a scale of 1-5 (1=poor, 5=excellent).

User Query: {user_query}
Response: {model_response}

Criteria:
- Addresses the user's concern directly
- Clear and professional tone
- Actionable guidance
- Appropriate length

Respond with JSON: {{"score": int, "reasoning": str, "confidence": float}}"""

 response = judge_model.create_message(prompt)
 return json.loads(response)

This works. LLMs have learned to recognize quality writing. They understand context. They can balance multiple criteria at once. But there's a ceiling. And it's lower than you think.

The Hidden Biases in LLM Judges

LLM judges are biased. Not in the discriminatory sense necessarily, but in the sense that their evaluations are predictable and systematic.

Position Bias

Show an LLM judge two responses and ask it to compare them. Then show them the same two responses in reverse order. You'll get different answers. LLM judges tend to favor the first item presented, or sometimes the second. This bias is persistent even after you warn judges about it.

# This returns different results depending on order
judge.compare(response_A, response_B) # Maybe: A > B
judge.compare(response_B, response_A) # Maybe: B > A (inconsistent!)

Fix: Always randomize presentation order. Run pairwise comparisons multiple times with different orderings.

Verbosity Bias

LLM judges often prefer longer responses. Even when a short, concise answer is objectively better, a longer response with more "reasoning" will score higher. This is because LLMs themselves tend to associate length with thoughtfulness. If your model learns that it can game the judge by being verbose, it will. Users will hate it.

Self-Agreement Bias

If you're using the same model class to generate responses and to judge them (e.g., both are GPT-4), you'll see inflated agreement. The judge will be more lenient with outputs that "look like" what it would generate. This is insidious because your evaluation scores will look good. But when you deploy to production and real humans use your product, you'll discover problems the judge never caught.

Calibrating Your Judge Against Ground Truth

Before you deploy an LLM judge, you need to validate it against human evaluators.

Step 1: Build a Labeled Dataset

Have 3-5 human raters score 200-500 representative examples. For each example, record: - The input (user query, prompt, etc.) - The model's output - Each human's rating - The majority label (or consensus score) This is your ground truth. It's expensive upfront but it's the foundation for everything that follows.

Step 2: Compute Inter-Rater Reliability

Before comparing human judges to LLM judges, check that humans agree with each other:

from sklearn.metrics import cohen_kappa_score, krippendorff_alpha

# Compute Cohen's kappa between pairs of human raters
kappas = []
for i, j in combinations(human_raters, 2):
 kappa = cohen_kappa_score(i.ratings, j.ratings)
 kappas.append(kappa)

human_agreement = np.mean(kappas)
print(f"Human inter-rater agreement (kappa): {human_agreement:.3f}")

# Kappa < 0.6: Poor agreement. Your task is too ambiguous or poorly defined.
# 0.6-0.75: Moderate. LLM judges can improve with calibration.
# 0.75+: Good. LLM judges can match human performance.

If humans only agree 45% of the time, your task is too subjective to evaluate reliably with any judge. Go back and tighten your evaluation rubric.

Step 3: Evaluate Your Judge

Now run your LLM judge on the same labeled dataset. Compare:

def evaluate_judge_calibration(human_labels, judge_labels):
 """Measure agreement between LLM judge and human consensus"""

 # Exact agreement
 exact_match = np.mean(judge_labels == human_labels)

 # Within-1 accuracy (for 1-5 scale, allowing 1-point error)
 within_one = np.mean(np.abs(judge_labels - human_labels) <= 1)

 # Correlation
 correlation = np.corrcoef(judge_labels, human_labels)[0,1]

 # Cohen's kappa
 kappa = cohen_kappa_score(judge_labels, human_labels)

 return {
 "exact_match": exact_match,
 "within_one": within_one,
 "correlation": correlation,
 "kappa": kappa
 }

If your judge's kappa is <0.5 with humans, it's not ready. Train on more examples or redesign your rubric.

The Factual Accuracy Problem

Here's where LLM judges completely break down: factual evaluation. If your model generates a factual claim, "The capital of France is Paris" or "React version 18 introduces concurrent rendering", you cannot evaluate it with another LLM. LLM judges will confidently evaluate incorrect facts as correct, especially if they're plausibly written. For factual tasks, you have three options: **Option 1: Use a Knowledge Base** Compare the model's claims against a known database of facts. This only works if you have a complete, up-to-date knowledge base (hard for current events, evolving domains).

def evaluate_factual_claim(claim, knowledge_base):
 """Check model claims against ground truth"""
 verified = knowledge_base.search(claim)
 return verified is not None

**Option 2: Task-Specific Evaluation** For code generation, run the code and check if it passes tests. For math, compute the answer and compare. For SQL queries, run them and check results.

def evaluate_code_generation(prompt, generated_code, test_cases):
 """Don't ask an LLM if the code is good. Run it."""
 try:
 exec(generated_code)
 passed = sum(1 for test in test_cases if test(generated_code))
 return passed / len(test_cases)
 except:
 return 0.0

**Option 3: Hybrid Approach** Use an LLM judge for style and clarity, but use external verification for facts. Have the LLM generate explanations and cite sources, then verify the citations.

The golden rule: If there's a ground truth, measure it directly. If there's no ground truth, you're in subjective territory and LLM judges can help. If you can't tell the difference, you're not ready to evaluate at scale.

Building Production Evaluation Pipelines

to structure an LLM judge in production:

Architecture

class LLMJudgePipeline:
 def __init__(self, judge_model, human_agreement_threshold=0.65):
 self.judge = judge_model
 self.threshold = human_agreement_threshold

 def evaluate(self, outputs):
 """Evaluate model outputs with fallback to human review"""

 scores = []
 low_confidence = []

 for output in outputs:
 score = self.judge.score(output)

 # If judge's confidence is low, flag for human review
 if score["confidence"] < 0.7:
 low_confidence.append(output)

 scores.append(score)

 return {
 "scores": scores,
 "needs_human_review": low_confidence,
 "automation_rate": 1.0 - (len(low_confidence) / len(outputs))
 }

Continuous Calibration

Your LLM judge will drift over time. New types of outputs it hasn't seen before. New edge cases. You need to: 1. Continuously sample outputs and have humans rate them 2. Compare human ratings to judge ratings 3. Re-baseline your judge if agreement drops below threshold 4. Alert your team if calibration fails

A Framework for Deciding: Judge or No Judge?

Should you use an LLM judge? - **Highly objective** (right answer exists, can be verified): No judge. Verify directly. - **Moderately subjective** (rubric can be clear, humans agree 70%+): Yes, use judge with human calibration. - **Highly subjective** (no ground truth, humans disagree even with rubric): Maybe. Use multiple judges, ensemble their scores. - **Mixed** (some objective, some subjective aspects): Hybrid. Separate the objective parts, judge the subjective parts. The future of AI evaluation isn't LLM judges replacing human judgment. It's hybrid systems where LLMs do the scalable work and humans provide the calibration signal.

What to Do Monday Morning

If you're already running an LLM judge in production: 1. Immediately measure agreement between your judge and human raters on a random sample 2. If agreement is <60%, flag this as a quality risk 3. Implement a human review queue for low-confidence cases 4. Set up continuous calibration monitoring If you're considering using an LLM judge: 1. Build a 300-example labeled dataset with human raters 2. Check inter-rater reliability 3. Run your judge and compare to human labels 4. Only proceed if kappa > 0.5 LLM judges are powerful. But they're not magic. Treat them like any other test signal, validate them, calibrate them, and keep humans in the loop.