Your Eval Coverage Is a Rounding Error

The demo set is not the test set

Here is the pattern in nearly every team that ships an LLM feature. Someone writes a handful of prompts that exercise the happy path. The model answers them well. Those prompts become the de facto regression suite, they get rerun before each release, they keep passing, and everyone feels covered. They are not covered. They have a demo script that the model was implicitly tuned to pass, masquerading as a quality bar.

The gap is one of basic statistics. As the practitioner literature puts it bluntly, the quality of an evaluation depends on the quality of the test dataset, which must be representative of real-world usage, diverse enough to cover different scenarios, and explicitly include the edge cases where the model might fail (ByteByteGo, A Guide to LLM Evals). A set that only contains the cases you already know work is structurally incapable of finding new failures.

Coverage means spanning the input distribution, not counting prompts

The instinct when told “you need more coverage” is to write more prompts that look like the ones you have. That doesn’t help. Adding fifty more happy-path variations to a happy-path set increases the count and not the coverage. Real coverage is a property of the distribution: does your eval set contain inputs from every region of the space your users occupy, including the sparse, weird, and hostile regions?

This is the same insight the test-generation research community formalizes as fault coverage versus fault exposure. The 2025 TestCase-Eval benchmark (full paper), built on 500 algorithm problems and 100,000 human-crafted solutions from Codeforces, separates two questions a good test set must answer: does the set probe diverse input scenarios and cover potential failure modes (fault coverage), and can it craft tailored inputs that actually reveal a specific wrong implementation (fault exposure)? An eval set can have a thousand cases and still have terrible fault coverage if all thousand cluster in one corner of the space.

The regions everyone skips

Map your input space and you will find systematic blind spots. The ones that most reliably produce production incidents:

• Long-tail entities. The common product names work; the obscure SKU, the newly-launched plan, the regional variant nobody tested do not.
• Adversarial and off-topic framings. Users ask things you never intended the system to handle, and the model answers anyway, confidently and wrongly.
• Format and length extremes. The 4,000-token paste, the single-word query, the message with three languages mixed in.
• Temporal edges. Questions that hinge on “as of today” or a date range the model silently ignores.
• Negation and constraint. “Everything except X, ” “only if not Y”, phrasings where dropping one word flips the correct answer.

Build coverage deliberately, then measure it

The fix is not heroics; it’s a process that mines real traffic, augments it with targeted adversarial cases, and then quantifies the spread so you can see your own blind spots instead of guessing at them.

# Don't count prompts. Measure distributional coverage of your eval set.
def coverage_report(eval_set, production_sample):
    # Bucket both sets by behavioral dimensions, not by surface text
    dims = ['intent', 'entity_rarity', 'input_length_band',
            'language', 'has_negation', 'is_adversarial']

    prod_dist = distribution(production_sample, dims)
    eval_dist = distribution(eval_set,          dims)

    gaps = []
    for bucket, prod_share in prod_dist.items():
        eval_share = eval_dist.get(bucket, 0.0)
        # A bucket users hit often but the eval set barely touches = blind spot
        if prod_share > 0.02 and eval_share < prod_share / 3:
            gaps.append({'bucket': bucket,
                         'prod_share': prod_share,
                         'eval_share': eval_share})
    return sorted(gaps, key=lambda g: -g['prod_share'])

That report turns “we feel under-tested” into a ranked list of exactly which behaviors your users hit that your suite ignores. Fill the top gaps first, those are the behaviors that will fail first in production.

# Mine real failures into permanent cases, and stabilize against run-to-run variance
def harden_eval_set(eval_set, production_logs):
    # Every production complaint or low-score sample becomes a regression case
    for incident in production_logs.flagged():
        eval_set.add(Case(input=incident.input,
                          expected=incident.correct_answer,
                          tags=['from_incident', incident.category]))

    # Run each case N times; a case that passes 3/5 is NOT passing
    for case in eval_set:
        results = [grade(model(case.input), case.expected) for _ in range(5)]
        case.pass_rate = mean(results)
        assert case.pass_rate >= 0.8, \
            f"flaky behavior on {case.id}: {case.pass_rate:.0%} pass rate"
    return eval_set

How big does the set need to be?

There is no magic number, but the direction is unambiguous: dozens of examples provide weak confidence, thousands provide strong confidence. The honest framing is to think in terms of the precision you need. If you want to detect a 2-percentage-point regression in a pass rate, you need enough samples that 2 points is outside your confidence interval, which pushes you toward the high hundreds or low thousands per critical behavior, not per system. A 30-case suite cannot distinguish a real 5% regression from sampling noise, which means it cannot do the one job a regression suite exists to do.

The pragmatic structure most mature teams converge on is a layered set: a lean, deterministic core that gates every deploy (fast, cheap, high-signal), plus a larger, periodically-run comprehensive set that spans the full distribution and catches the slow drift the core misses (Techment, LLM Regression Testing strategies). The core keeps CI fast; the comprehensive set keeps you honest about coverage.

The bottom line

“It passed our evals” means nothing until you know what those evals covered. Ten hand-picked prompts is a demo, not a suite, and a suite that only contains cases you already know pass is structurally blind to new failures. Stop counting prompts and start measuring distributional coverage against real traffic; mine every production incident into a permanent case; run each case multiple times to defeat run-to-run variance; and size each critical behavior’s set to the regression you actually need to detect. The behaviors that break in production are the ones no one wrote a case for, so the entire job of coverage is writing the cases no one wanted to.