Healthcare AI Eval: Where "Mostly Right" Is a Safety Event

Why “mostly right” is the wrong frame

Every other domain on this blog can tolerate an averaged error rate. A support bot that is 97% accurate is a good support bot. A clinical decision-support tool that is 97% accurate has a defined rate of telling clinicians something dangerous, and in medicine, errors are not fungible. Telling a patient their normal lab value is slightly elevated is a nuisance; telling a clinician to discharge a patient who is actually having a heart attack is a death. Aggregate accuracy hides exactly the failures that matter most.

This is the core reason healthcare AI evaluation cannot be a single number. The metric that keeps a product safe is not “overall accuracy” but “rate and severity of clinically significant errors, broken out by the situations where errors are most harmful.” A model can move its overall score up two points by getting easy cases righter while quietly getting harder on the rare, high-stakes cases, and a naive eval would call that an improvement.

The hallucination problem is documented, not speculative

The research is unambiguous about the shape of the risk. A 2025 body of work on medical hallucination in foundation models introduced frameworks for detecting and categorizing the ways these systems generate plausible-sounding but factually wrong medical information, fabricated drug interactions, invented clinical guidelines, confident answers about conditions the model has barely seen. The danger is precisely that the output is fluent and authoritative; a hurried clinician has no error signal to react to.

The rates depend entirely on grounding and task. A 2025 npj Digital Medicine study on clinical safety and hallucination rates in medical text summarization measured a 1.47% hallucination rate and a 3.45% omission rate across nearly 13,000 clinician-annotated sentences when the model was grounded in source text. But domain leaderboards have reported medical hallucination rates exceeding 60% on open-ended, ungrounded clinical tasks. The most important finding for builders: strong performance on medical licensing-exam questions does not predict safety in real clinical scenarios, which arrive with incomplete histories, comorbidities, and atypical presentations. The board-exam score vendors love to quote is nearly useless as a safety signal.

The clinician trust problem cuts both ways

There is a subtle, dangerous dynamic in how clinicians interact with these tools, and it shapes what your eval must protect against. Surveys find that a substantial share of medical professionals have already encountered medical hallucinations in their work, in literature review, in decision support, in drafting. The first time a clinician catches the model in a confident fabrication, trust drops, which is healthy. But the more corrosive failure is the opposite: a tool that is right often enough to earn automation complacency, so the clinician stops scrutinizing its output, right up until the rare, high-stakes error it was always statistically going to produce slips through unchecked.

This is why the rate of clinically significant errors matters more than the average, and why your eval has to drive that rate toward zero on the high-harm tier rather than settling for a good-looking aggregate. A 97%-accurate tool that lulls clinicians into trusting it is arguably more dangerous than a 90%-accurate one they reflexively double-check, because the failure mode is a confident error meeting a disengaged reviewer. The eval’s job is to ensure that on the cases where an unscrutinized error would be catastrophic, the model simply does not produce one.

What the regulator now expects

The FDA has authorized over 1,250 AI-enabled medical devices as of 2025, the large majority in radiology, and its posture has matured from one-time clearance toward lifecycle oversight. In January 2025 the agency issued draft guidance on the total product lifecycle for AI-enabled device software functions, centered on the Predetermined Change Control Plan, a mechanism that lets manufacturers pre-specify how a model may be updated and, crucially, how it will be monitored after deployment. The regulatory thesis matches the engineering one: an AI system is not done at launch; it must be continuously evaluated.

Even if your product is positioned to avoid being a regulated device, a documentation assistant, a patient-education chatbot, an ambient scribe, the malpractice and reputational exposure does not evaporate. If your tool contributes to a clinical decision and it was wrong, “we are not an FDA device” is a weak position in a deposition. The standard you should hold yourself to is the clinical one regardless of regulatory classification.

Building a harm-weighted clinical eval

The eval suite for a clinical system looks different from a generic one in three ways: the golden set is built around dangerous scenarios, every failure is severity-graded, and the gate is on the high-harm tier rather than the mean.

# Harm-weighted clinical eval: gate on severity tiers, not average accuracy
SEVERITY = {'catastrophic': 4, 'serious': 3, 'moderate': 2, 'minor': 1}

def clinical_eval(model, golden_set):
    failures = []
    for case in golden_set:
        out = model(case.input)
        verdict = clinician_grade(out, case.gold)      # validated rubric
        if not verdict.correct:
            failures.append({
                'case_id':  case.id,
                'category': case.category,             # e.g. "red-flag triage"
                'severity': case.harm_if_wrong,        # pre-assigned by clinicians
                'error':    verdict.error_type,        # fabrication / omission / dosing
            })
    by_sev = {s: [f for f in failures if f['severity'] == s] for s in SEVERITY}

    # The gate: ZERO tolerance on catastrophic, tight cap on serious
    assert len(by_sev['catastrophic']) == 0, f"CATASTROPHIC errors: {by_sev['catastrophic']}"
    assert len(by_sev['serious']) <= 1,       f"Too many SERIOUS errors: {by_sev['serious']}"
    return failures

Notice the gate has nothing to do with overall accuracy. A model can be 99.5% accurate overall and still fail this suite if it produces a single catastrophic error on a red-flag triage case. That is the right behavior. The whole point is to make the rare, deadly failure mode block the release.

The categories your golden set must cover

• Red-flag and emergency presentations, chest pain, stroke symptoms, sepsis, suicidality. The model must never falsely reassure.
• Dosing and drug interactions, exact, verifiable, and unforgiving. A misplaced decimal is a 10x overdose.
• Rare conditions and atypical presentations, where licensing-exam performance does not transfer and error rates climb.
• Scope-of-practice and escalation, the model must recognize when to defer to a human clinician rather than answer.
• Omission failures, not just wrong statements, but failure to mention the contraindication or the follow-up that safety requires.

Groundedness and abstention as safety controls

Two engineering controls do more for clinical safety than any model swap. First, groundedness: every clinical claim must trace to a verifiable source, a guideline, a drug database, the patient’s own record, not the model’s parametric memory. The summarization study is the proof: grounding in source text is what drove hallucination rates down near 1.5% rather than the 60%+ seen on ungrounded tasks. A clinical answer the model cannot cite is an answer it should not give. Second, calibrated abstention: the model must be able to say “I don’t have enough information” or “this requires a clinician, ” and your eval must reward appropriate abstention rather than penalizing it as a non-answer.

# Safety controls: groundedness + appropriate abstention, both evaluated
def safety_check(out, case):
    checks = {}
    # Every clinical assertion must cite an allowed source (no parametric "memory")
    checks['grounded'] = all(claim.has_citation(case.allowed_sources)
                             for claim in extract_clinical_claims(out))
    # On out-of-scope or under-specified cases, abstention is the CORRECT answer
    if case.should_abstain:
        checks['abstained_correctly'] = out.is_abstention or out.escalates_to_clinician
    else:
        checks['answered_when_able'] = not out.is_abstention   # don't over-refuse useful cases
    return checks

Continuous, not one-time

Clinical practice changes, new guidelines, new drugs, new black-box warnings, and hosted models drift underneath you. A clinical eval that passed at launch certifies nothing about today. The FDA’s lifecycle framing is correct: the safety case is a living document. Run the harm-weighted suite on every model and prompt change, re-run it when guidelines update, sample and grade live outputs for groundedness, and keep dated evidence so you can show what the system knew and how it performed on the day it touched any given patient.

The bottom line

Healthcare AI cannot be evaluated like ordinary software, because its errors are not fungible and its worst failures are rare. Licensing-exam scores do not predict clinical safety; documented 2025 research shows hallucination rates swing from ~1.5% when grounded to over 60% when not. Build a golden set around red-flag, dosing, rare-condition, and escalation scenarios; severity-grade every failure and gate on zero catastrophic errors; enforce groundedness and reward calibrated abstention; and run it continuously. In a domain where “mostly right” is a safety event, the eval is the safety control.