The Hallucination That Cost a Million: Why Spot-Checks Aren't Enough

The case that ended the "the bot did it" defense

The reference point everyone now cites is small in dollars and huge in precedent. A grieving passenger asked Air Canada's chatbot about bereavement fares; the bot told him he could apply retroactively for a discount. He couldn't, the bot had invented the policy. When he sued, Air Canada argued the chatbot was "a separate legal entity responsible for its own actions." The tribunal rejected this outright, finding it "should be obvious" that a company is responsible for all information on its website, static page or chatbot alike, and ordered damages for negligent misrepresentation.

The damages were ~$812. The lesson was worth far more: you cannot disclaim responsibility for what your AI says. As commentators noted, courts are allocating the risk of new AI technology to the companies deploying it, especially against consumers. Now scale that $812 ruling to a chatbot that quotes a wrong price, promises a refund you won't honor, or gives incorrect medical or financial guidance to millions of users.

Why spot-checking fails

The default QA process for AI features is: a few people try a few prompts before launch, it "seems fine, " and it ships. This fails for reasons inherent to how language models work:

• The output space is unbounded. You tested 20 prompts; users will send 20 million, including phrasings you never imagined.
• Failure is non-deterministic. The same prompt can be right today and wrong tomorrow, and "tomorrow" may be a silent provider model update you didn't trigger.
• Confidence is uncorrelated with correctness. Hallucinations arrive in the same fluent, authoritative tone as correct answers, so humans skimming output miss them.
• It doesn't scale. Manual review can't keep pace with prompt changes, model updates, and growing traffic.

What continuous evaluation looks like

Continuous evaluation replaces "seems fine" with a measurable pass/fail that runs on every change and against live traffic. The core building blocks:

A golden dataset

A curated, versioned set of input/output pairs, including the adversarial and edge cases that bite in production, that acts as your regression suite. Every prompt change, model swap, or fine-tune runs against it.

Automated graders

For factual answers, a groundedness check: does every claim trace to a source you control? For policy bots, an assertion that the answer matches the canonical policy text. For tone and safety, classifier-based scoring. Where nuance is needed, an LLM-as-judge, validated against human labels so you trust the judge.

A CI gate

The evaluation runs in your pipeline and blocks the deploy if quality regresses, exactly like a failing unit test.

# Continuous eval as a deploy gate
def evaluate(model, golden_set):
    results = []
    for case in golden_set:
        out = model(case.input)
        results.append({
            'grounded':   is_grounded(out, case.allowed_sources),   # no invented facts
            'policy_ok':  matches_policy(out, case.canonical_policy), # no invented policy
            'safe':       safety_score(out) > 0.9,
            'relevant':   relevance(out, case.input) > 0.8,
        })
    grounded_rate = mean(r['grounded'] for r in results)
    policy_rate   = mean(r['policy_ok'] for r in results)
    # Block the release if the bot invents facts or policy
    assert grounded_rate > 0.98, f"groundedness regressed to {grounded_rate:.1%}"
    assert policy_rate  > 0.99, f"policy adherence regressed to {policy_rate:.1%}"
    return results

# Close the loop: sample and score live traffic, too
def production_monitor(sampled_conversations):
    for conv in sampled_conversations:
        if not is_grounded(conv.answer, conv.retrieved_sources):
            alert(f"Ungrounded answer served to user {conv.id}: {conv.answer[:120]}")
        # Catch the silent provider model update before users do
        track_metric('groundedness_rate', is_grounded(conv.answer, conv.retrieved_sources))

The Air Canada-proof posture

If your AI makes any factual, policy, pricing, or eligibility claim to a customer, treat every such claim as a statement your company is legally making, because that's how it will be treated. That means: ground answers in verified sources, refuse rather than guess when the answer isn't supported, log what was said to whom, and run continuous evaluation so a regression in any of those properties is a blocked build, not a lawsuit.

The bottom line

The legal and financial framing is settled enough to act on: your AI's mistakes are your mistakes, hallucination is a steady-state probability rather than a rare bug, and manual spot-checks cannot cover an unbounded output space. Build a golden dataset, automate groundedness and policy graders, gate your deploys on them, and monitor live traffic. The cost of doing this is a fraction of one bad headline, or one ruling that scales past $812.