Your AI Summarizer Is Lying to You: How to Catch It

Last month, a major financial services firm deployed an AI summarizer to condense earnings call transcripts. It worked beautifully, executives loved the clean, punchy summaries. Then a lawyer caught it. The model had consistently misattributed quotes, collapsed different scenarios into one, and invented financial metrics that didn't exist in the source material.

The summarizer had a ROUGE-1 score of 0.68. Everyone assumed it was working fine.

This is the lie we tell ourselves about AI summarization: fluency equals accuracy. A model that produces grammatically perfect, well-structured summaries might be silently hallucinating. Traditional metrics like ROUGE and BERTScore were never designed to catch this. They measure surface-level overlap, not whether the summary is true.

Why Your Standard Metrics Are Failing You

Let's start with the elephant in the room: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is everywhere, and it's inadequate.

ROUGE works by comparing your model's summary to reference summaries using n-gram overlap. High overlap = good summary. The problem? A summary can be completely factually wrong while nailing ROUGE scores.

The ROUGE Trap

Consider a source document:

CEO Sarah Thompson announced the acquisition of TechCorp for $150M, expected to close in Q3 2026. The deal includes $50M in earnouts.

Reference summary: "Thompson's TechCorp acquisition of $150M closes Q3 2026 with $50M earnouts."

Model A summary: "Thompson acquired TechCorp for $150M in Q3 2026 with $50M earnouts." ROUGE-1: 0.89

Model B summary: "Thompson bought CompanyX for $200M in Q2 2026 with $75M earnouts." ROUGE-1: 0.71

Model A wins on ROUGE. Model A is also correct. But swap the numbers in Model B, and ROUGE still tanks even though you only hallucinated the amount. Change it to $150M but wrong entity names? ROUGE still thinks you're decent.

The metric is bag-of-words matching. It has no concept of semantic units, entity identity, or numerical accuracy. It's a proxy for a proxy.

BERTScore's Hidden Flaw

BERTScore promised to fix ROUGE by using contextual embeddings. It measures semantic similarity between generated and reference text. Much better, right?

Sort of. BERTScore still compares to references, and if your references aren't Practical, you're measuring something incomplete. More critically, BERTScore doesn't care about information preservation. A summary can capture similar semantic space while dropping critical specifics.

Example: "The company reported strong growth" has high BERTScore similarity to "Revenue increased significantly, " but they're different claims with different evidence requirements.

The Three Tests You Actually Need

1. Factual Consistency Scoring

This measures whether claims in the summary are supported by the source. It's fundamentally different from ROUGE/BERTScore because it asks: "Is this true?"

There are several approaches:

• Entailment-based: Use an NLI (natural language inference) model to check if the source entails each claim in the summary
• Question generation: Generate questions from the summary, then check if the source answers them correctly
• Semantic role labeling: Extract facts from both, compare structured representations

The entailment approach is most practical:

from transformers import pipeline

# Load a cross-encoder for entailment
verifier = pipeline(
 "zero-shot-classification",
 model="cross-encoder/qnli-distilroberta-base"
)

source = "CEO Sarah Thompson announced acquisition of TechCorp for $150M."
summary = "Thompson bought CompanyX for $200M."

score = verifier(source, summary)
# Predicts 'contradiction' with high confidence

# In production:
# - Extract claim-level facts from summary
# - Run NLI on each claim against source
# - Flag low entailment claims as hallucinated

This catches when summarizers invent entities (CompanyX), corrupt numbers ($200M vs $150M), or reverse relationships. It's not perfect, NLI models have their own blind spots, but it's dramatically better than ROUGE.

2. Information Density and Entity Preservation

Summarization is about compression, but bad summarizers compress away meaning. A metric that captures this:

from collections import Counter
import spacy

def entity_preservation_score(source_text, summary_text, nlp):
 """Measure what % of source entities made it to summary"""
 source_doc = nlp(source_text)
 summary_doc = nlp(summary_text)

 source_entities = {
 (ent.text, ent.label_)
 for ent in source_doc.ents
 if ent.label_ in ["PERSON", "ORG", "GPE", "MONEY", "DATE"]
 }

 summary_entities = {
 (ent.text, ent.label_)
 for ent in summary_doc.ents
 if ent.label_ in ["PERSON", "ORG", "GPE", "MONEY", "DATE"]
 }

 if not source_entities:
 return 1.0

 preserved = len(source_entities & summary_entities)
 return preserved / len(source_entities)

# Usage
nlp = spacy.load("en_core_web_sm")
score = entity_preservation_score(source, summary, nlp)
print(f"Entity preservation: {score:.1%}")
# 0.8 = 80% of key entities made it to summary

Why this matters: A summary might read smoothly but lose critical actors, amounts, or dates. If your earnings summary drops revenue figures, entity preservation catches it immediately. Target: 85%+ preservation for most domains.

3. Numerical Accuracy Auditing

This is surprisingly overlooked. Summarizers hallucinate numbers constantly. They'll generate contextually plausible but factually wrong figures.

import re

def extract_numbers_with_context(text):
 """Extract numbers and their nearby context"""
 pattern = r'(\b\w+\s+)?[\$€£]?([\d, .]+(?:M|B|T|K)?)\b'
 matches = re.finditer(pattern, text, re.IGNORECASE)

 results = []
 for match in matches:
 start = max(0, match.start() - 30)
 end = min(len(text), match.end() + 30)
 context = text[start:end].strip()
 results.append({
 'number': match.group(2),
 'context': context,
 'position': match.start()
 })
 return results

source_nums = extract_numbers_with_context(source_text)
summary_nums = extract_numbers_with_context(summary_text)

# Compare: Are the numeric values actually in the source?
summary_values = {n['number'] for n in summary_nums}
source_values = {n['number'] for n in source_nums}

hallucinated = summary_values - source_values
if hallucinated:
 print(f"WARNING: Summary contains numbers not in source: {hallucinated}")

This is crude but effective. For financial, medical, or technical content, run this as a guardrail on every summary. Hallucinated numbers are one of the easiest sins to catch.

Building a Practical Evaluation Framework

Test	What It Measures	Failure Mode	Target Score
Entailment Score	% of claims in summary supported by source	Fabricated claims, misquotes	90%+
Entity Preservation	% of key entities (people, orgs, amounts) preserved	Dropped details, lost actors	85%+
Numerical Accuracy	% of numbers in summary present in source	Hallucinated figures	100%
ROUGE-L	Longest common subsequence overlap	Unrelated text	0.35+
Compression Ratio	Summary length / source length	Too verbose or too sparse	0.25-0.40

Real-World Implementation

one team integrated this. They were summarizing support tickets to create knowledge base articles. Initially, they trusted ROUGE scores. Their KBA had high coverage but wrong information.

They deployed this test suite:

class SummarizationTester:
 def __init__(self, model, nlp, entailment_model):
 self.model = model
 self.nlp = nlp
 self.entailment = entailment_model

 def test_batch(self, documents, reference_summaries):
 results = []

 for doc, reference in zip(documents, reference_summaries):
 generated = self.model.generate(doc)

 tests = {
 'entailment': self.score_entailment(doc, generated),
 'entities': self.entity_preservation(doc, generated),
 'numbers': self.numerical_accuracy(doc, generated),
 'rouge_l': self.rouge_score(reference, generated),
 }

 # Flag failures
 if tests['entailment'] < 0.85:
 tests['status'] = 'FAIL_HALLUCINATION'
 elif tests['entities'] < 0.80:
 tests['status'] = 'FAIL_COMPRESSION'
 elif tests['numbers'] < 1.0:
 tests['status'] = 'FAIL_ACCURACY'
 else:
 tests['status'] = 'PASS'

 results.append(tests)

 return results

Result: They caught a model that looked good on ROUGE but was systematically dropping crucial context. By implementing these three tests, they achieved actual reliability.

When Metrics Still Fail You

All automated metrics are proxies. Here's what they still miss:

• Nuance loss: A summary can be technically accurate while stripping away important caveats and context
• Causal relationships: Hard to test whether "A caused B" is preserved accurately
• Domain-specific claims: Legal, medical, or technical assertions need domain expertise to verify
• Implicit contradictions: A summary might be internally consistent but contradict information elsewhere

Solution: Combine automated tests with sampling. For high-stakes use (legal, medical, financial), do human validation on 5-10% of summaries. Automated metrics are guardrails, not gospel.

The Production Checklist

Before deploying an AI summarizer:

1. Test 200+ examples from your actual domain and distribution
2. Run entailment, entity preservation, and numerical accuracy tests
3. Compare results to your reference summaries (use multiple references if possible)
4. Set hard thresholds: Anything below 85% entailment gets flagged for review
5. Sample 20 "passing" summaries and read them yourself
6. Monitor in production: re-run tests weekly on a sample
7. Track failure modes (hallucination vs. compression vs. drop-outs)