AI Testing for Healthcare: The FDA Won't Accept 'It Usually Works'

Your diagnostic AI model has 94% sensitivity and 96% specificity on your test set. Your team is excited to deploy it. Your regulatory affairs team says: "Not yet. Show us validation on external cohorts. Show us performance across racial and ethnic groups. Prove you're monitoring for drift. Document your entire development pipeline. And get ready for FDA review."

Welcome to healthcare AI. The rules are different here, and they exist for reasons that aren't obvious until a patient sues you because your system missed a cancer diagnosis.

The FDA Framework: What You Need to Know

The FDA doesn't regulate software generally. But it does regulate software that:

• Makes or supports a diagnosis
• Recommends treatment
• Monitors patient health
• Assesses risk of disease progression

In 2023, the FDA released guidance specifically on "Modification to Predetermined Changes to an Artificial Intelligence/Machine Learning In Vitro Diagnostic Devices." Key points:

Your AI/ML system is a medical device. It requires submission and clearance. You can't just deploy it because it works on your data.

You must have a Clinical Validation Plan. Before you submit, you need evidence that your model works on real patients from populations representative of intended use. This isn't your training data. This is new data, prospectively collected or obtained from external sources, validated against ground truth (usually radiologist consensus, pathology results, or other gold standards).

You need a Software Bill of Materials (SBOM). Every model version, training data version, dependency, and hyperparameter configuration must be documented. This isn't optional. This is audit trail material.

You must demonstrate the model doesn't "wander." Continuous monitoring showing that performance doesn't degrade over time. If your model performed at 94% specificity at launch and 88% at 6 months, you have a problem.

The FDA's position: if a human radiologist's accuracy varies by 5% over time, we want to know why. The same applies to your AI system.

Clinical Validation: It's Different From Tech Validation

Your ML engineer trained a model on 50,000 chest X-rays and tested on a held-out set of 5,000. Validation set sensitivity: 92%. You're done.

The FDA says: that's internal validation. Now show us external validation. Take your model to three hospitals you've never worked with. Have their radiologists independently label 500 new chest X-rays using their own protocols (not yours). Run your model on those images. Compare to consensus labels. What's the sensitivity now?

Often external validation shows lower performance than internal validation. That's normal. But if the drop is large (94% → 84%), you have a generalization problem. Your model works on your data but doesn't transfer to other institutions.

This is clinically important. Your training data came from one hospital system. That system has certain CT protocols, patient demographics, and disease prevalence. Another system is different. A model that works perfectly in a wealthy urban hospital might perform terribly in a rural clinic with older equipment.

Prospective validation is even better than external retrospective validation. You prospectively identify patients for whom a clinical decision is needed. You collect their images/tests. Your model makes predictions. Then you follow clinical workflow, doctors make decisions based on standard protocols, not your model. Later, you measure your model's performance against actual clinical outcomes.

Prospective validation takes months and is expensive. But it's the gold standard. Many FDA submissions include prospective validation because it's the strongest evidence that your system actually improves clinical outcomes.

Demographic Parity and Bias: This Isn't Optional

Your diagnostic model has 94% accuracy overall. Sounds good. Then you stratify by race and ethnicity:

• White patients: 95% accuracy
• Black patients: 78% accuracy
• Hispanic patients: 82% accuracy
• Asian patients: 96% accuracy

Congratulations, you just built a system that diagnoses some patients well and misses disease in others based on race. This is why bias testing is mandatory in healthcare AI, not optional.

The FDA expects you to test for performance disparities across protected groups. This means:

Demographic stratification: Segment your validation set by race, ethnicity, gender, age group, and socioeconomic status (if available). Calculate sensitivity, specificity, and other key metrics separately for each group.

Reporting disparities: If Model performance differs by >5% between groups, flag it. Document why (is it training data imbalance? Different disease presentation across populations? Measurement bias in your labels?). Plan mitigation.

Mitigation strategies: If your model performs worse on a subgroup, you have options: (1) retrain on balanced data, (2) implement group-specific thresholds, (3) escalate uncertain cases for human review, (4) disclose the limitation to clinicians. Option 4 is often the right one: "This model works well for X population but has not been validated for Y population."

Documentation: The FDA wants to see your bias testing methodology, results, and decisions. Write it down. Keep evidence.

This is where many teams fail. They build a model on data that's not representative (too much of one demographic group), achieve good aggregate performance, and miss systematic biases. The FDA review process catches this.

Continuous Monitoring: Post-Market Surveillance

You got FDA clearance. You deployed your diagnostic model. Now what?

The FDA expects you to monitor performance in production. Not manually. Automatically. Continuously. You need:

Baseline performance metrics from validation: At launch, your model had 93% sensitivity on external validation data. That's your baseline.

Real-world performance tracking: Every patient who uses your system, you log the model's output and (eventually, when ground truth becomes available) the true outcome. Monthly, you calculate sensitivity and specificity on the current patient cohort. If sensitivity drops to 88%, you have a signal.

Drift detection and alerting: Implement the drift detection methods from earlier posts. If performance degrades by >5%, escalate. If you can't explain why, consider rollback.

Stratified monitoring: Monitor performance separately for each demographic group. If the model starts performing worse for one group, you detect it. This is non-negotiable.

Complaint handling: If a patient or clinician reports an issue (model made wrong diagnosis, missed a cancer, etc.), you have a process to investigate. You log it. You determine root cause. If it's a systematic problem, you address it company-wide.

Many companies treat post-market surveillance as a checkbox. The FDA treats it as accountability. You're selling a diagnostic tool that affects people's health. You're responsible for making sure it doesn't silently degrade.

HIPAA Compliance: It's About Data, Not Just Privacy Policy

Your diagnostic model trains on patient imaging data. That's Protected Health Information (PHI). HIPAA rules apply. But this gets tricky with AI.

Data minimization: Train only on data necessary for the task. If your model works with imaging data alone, don't include names, medical record numbers, or other identifiers. Easier said than done (you need identifiers for auditing), so use pseudonymization: replace patient identifiers with random IDs that can be linked back but aren't obvious.

Data use agreements: If you source data from hospitals, you need Data Use Agreements that specify what you can use the data for, who can access it, how long you keep it, and what happens when you're done. Write them explicitly. A one-liner ("yes, you can use this for AI development") isn't enough.

Audit trails: Log who accessed what data, when, and why. If an employee queries all imaging from a particular hospital, that's logged. If someone downloads a cohort of patient records, you have a record. This isn't paranoia, it's required for HIPAA audits.

Encryption and access control: Data in transit (uploading to servers) and at rest (stored on disk) must be encrypted. Access controls limit who can view raw data. Your model can see the data for training, but not your sales engineer. Document this.

Breach notification: If you have a data breach (someone unauthorized downloads patient data, a researcher leaves the company with datasets, a vendor you work with gets hacked), you have 60 days to notify affected individuals and HHS. Have a breach response plan in place before you need it.

The intersection of HIPAA and AI testing is where things get messy. You want to share model performance data with stakeholders. But "our model achieved 94% sensitivity on imaging from Hospital X" might be considered protected information if it could be linked back to patients. Be conservative.

Documentation: This Is Not Optional

Your FDA submission includes mountains of documentation. This isn't bureaucracy, it's evidence. Here's what you need:

Clinical validation report: Methodology, patient population demographics, gold standard for labels, performance metrics stratified by demographics, external validation results, prospective validation results (if applicable).

Algorithm development history: Which training algorithms did you try? Why did you choose this one? What hyperparameters did you tune? What was the validation curve (did adding more training data help?). What's the final model architecture?

Training data provenance: Where did your training data come from? How many patients? What population demographics? Are there known biases in the source data? Did you apply any corrections (e.g., oversampling minority groups)?

Test data reports: The data you used to validate your model. Demographics, collection protocol, ground truth methodology, any disagreements between annotators.

Software architecture: How does the model get deployed? What are dependencies? What happens if a dependency breaks? How do updates get pushed? This matters because deployment bugs kill patients too.

Failure mode analysis: What are the ways your model could fail? False negatives (missing disease)? False positives (flagging healthy patients)? Performing worse on certain populations? For each failure mode, what's the likelihood and severity?

Post-market plan: How will you monitor for drift? How will you collect real-world performance data? What triggers an investigation? What's the escalation path if something goes wrong?

This documentation isn't for the FDA. It's for you. It forces you to think through all the ways your system could be wrong. You'll find bugs and biases in the documentation phase before they reach patients.

Real Examples: Where Healthcare AI Fails

A sepsis prediction model trained on data from urban hospitals with diverse populations. Performance looked great. Deployed to a rural hospital with a predominantly white elderly patient population. Performance dropped 12%. The model had learned patterns specific to diverse urban patients and didn't transfer.

A dermatology classification model trained on images from fair-skinned patients (easier to photograph, dataset had more of them). Black patients presented. Performance dropped from 94% to 71%. The model had learned color patterns that didn't generalize. This was a real incident that got significant media coverage.

An ICU mortality prediction model that optimized for accuracy but introduced a perverse incentive: the model learned to assign lower mortality risk to Black patients (because they had fewer documented comorbidities, not because they were actually at lower risk, but because implicit bias in medical documentation meant their conditions were less likely to be recorded). When deployed, it systematically under-triaged Black patients. The bug was caught in bias testing, but it was close.

Healthcare AI failures don't show up in a test set. They show up when a patient gets misdiagnosed and their family sues.

Practical Next Steps

Month 1-2: Plan clinical validation. Decide whether external retrospective, prospective, or both. Identify hospital partners. Get IRB approval (usually needed for research involving human subjects, even if you're using de-identified data).

Month 2-4: Conduct bias testing on current validation set. Stratify by race, gender, age. Identify disparities. Decide on mitigation.

Month 4-6: Execute clinical validation. Collect external data or set up prospective study. Measure performance.

Month 6-8: Prepare FDA submission materials. Document everything. Get ready for questions (FDA will ask for more evidence).

Month 8-12: FDA review cycle. Answer their questions. Provide more data if needed.

Month 12+: Post-market surveillance setup. Deploy monitoring, drift detection, bias tracking. Establish complaint handling process.

This timeline is realistic for a serious healthcare AI product. If your team says they'll do it in 2 months, they're not taking it seriously.

The Why Behind the Regulation

It's easy to resent healthcare regulation as bureaucratic friction. But consider: an AI system that diagnoses cancer sits between hope and despair for a patient. If it misses a cancer that's detectable, someone dies who didn't need to. If it falsely diagnoses cancer, someone goes through chemotherapy unnecessarily.

The FDA's framework exists to minimize these risks. It's not perfect, but it's better than the alternative: shipping whatever works and figuring out the damage later.

This is why testing in healthcare isn't about validation metrics. It's about responsibility.