TL;DR
The right amount of AI testing depends on three numbers: cost of a bad output, release frequency, and blast radius (how many users see each release). Multiply them and you get an annual quality risk; allocate 5-15% of that to an eval program. Most teams underspend by 5-10x. Diminishing returns kick in around 80% category coverage, past that, marginal cases are not worth it.
"How much testing do we actually need?" is the most common question we get from teams setting up an AI quality program in 2026. It's also the wrong question.
The right question is: "what is the cost of a bad output × the rate at which we ship × the number of users exposed, and how much of that risk are we willing to insure with testing?" That framing turns testing from a vague cost center into a calculable investment.
The risk equation
For any AI feature, expected annual quality risk is roughly:
Risk_$ = (P_bad_output) × (Cost_per_bad_output) × (Releases_per_year) × (Users_per_release)
Worked example for a customer-support chatbot at a mid-sized SaaS:
- P_bad_output (without testing): 5%
- Cost_per_bad_output: $40 (support escalation, churn risk)
- Releases_per_year: 24 (every two weeks)
- Users_per_release exposed: 50,000
Risk_$ = 0.05 × $40 × 24 × 50,000 = $2.4M / year
If a $200K/year eval program reduces P_bad_output from 5% to 2%, you've eliminated $1.4M in risk for a $200K spend. ~7x ROI. The math justifies the investment.
The hard part is honestly estimating P_bad_output and Cost_per_bad_output. Most teams underestimate both by 2-3x.
Right-sizing the eval set
Eval set size is a coverage and confidence problem.
Coverage: cover every category of input that occurs in production at >0.5% frequency. For a typical product, that's 20-50 categories. Each category needs at least 20 cases to be statistically meaningful (CI widths get acceptable around 20).
Confidence: how precisely do you need to measure quality? Detecting a 1-point regression on a 10-point scale needs ~100 cases per segment. Detecting a 0.5-point regression needs ~400.
Pragmatic ladder:
| Product stage | Eval set size | Minimum tier |
|---|---|---|
| Internal tool / low-risk | 50-200 | Manual eval pre-release |
| External beta | 500-1500 | Automated CI gate |
| Production, B2B | 2k-8k | Continuous + shadow deploys |
| Mission-critical / regulated | 10k-50k+ | Adversarial + governance |
Diminishing returns
Doubling your eval set from 100 to 200 cases catches roughly 30% more regressions. Doubling from 1,000 to 2,000 catches maybe 5%. Doubling from 10,000 to 20,000 catches under 1%.
The reason: the failure space is a long tail. The first thousand cases cover the dense head of common categories. Each additional thousand catches a steadily smaller fraction of remaining failures. At some point, the marginal case adds zero value because your model is no longer regressing in that area.
Empirical rule from the alt.qa customer base: hit 80% category coverage and stop. Past 80%, invest in quality of cases (better adversarial probes, harder edge cases) rather than quantity.
The four budget allocations
For a $200K/year quality program, here's a defensible allocation:
- 40%, eval set curation: $80K. Engineering + domain-expert hours building, refreshing, and pruning the eval set quarterly.
- 25%, tooling + infra: $50K. CI runners, eval platform license (or self-hosted ops), inference cost for evals.
- 20%, judging + scoring: $40K. LLM-judge costs, human reviewer panels for calibration, periodic re-calibration.
- 15%, production observability: $30K. Trace logging, drift monitors, online evaluators, dashboards.
Most teams over-spend on tooling and under-spend on case curation. The eval set is the asset; the platform is just the wrench.
Common over- and under-investment patterns
Under-investment indicators:
- Eval set hasn't been refreshed in >6 months
- No segmented metrics (only aggregate scores)
- No adversarial probes
- Production drift not monitored
- No cost regression tests
- Same regression keeps shipping
Over-investment indicators:
- Eval suite takes >1 hour to run on a PR (engineers ignore it)
- 50+ category breakdowns nobody actually uses
- Custom in-house eval platform built from scratch
- Duplicate eval sets for tiny variations
- Quality engineers outnumber ML engineers
The minimum viable AI eval
If your budget is zero and you need to start tomorrow:
- Spend one day curating 50 cases that cover your 8-10 highest-frequency input categories.
- Define a 3-category rubric (good / acceptable / bad).
- Run the model against the cases, score by hand. Track the score in a Google Sheet.
- Re-run before every deploy. Block deploys that drop the score by >5%.
This is level-2 maturity. It catches roughly 40% of the regressions that would otherwise reach production. It costs almost nothing. Most teams who skip this step regret it within a quarter.
The maximum useful AI eval
For a regulated, mission-critical product (healthcare, finance, government):
- 20k+ case eval set covering all input categories at >0.1% frequency
- 5+ orthogonal metrics (functional accuracy, factuality, fairness, safety, latency, cost)
- Adversarial / red-team eval run weekly
- Continuous online eval at 5-10% sampling
- Drift monitors at hourly cadence with paged alerts
- Model cards, eval cards, and audit trails for every release
- External auditor review at minimum annually
- Dedicated quality and compliance engineering team
Past this point, additional spend is symbolic, not functional.
The honest answer
The honest answer to "how much AI testing is enough?" is: enough to bring your production failure rate to a level your business can tolerate, given the cost of failures and the rate of release. For most B2B SaaS products, that's a 1k-5k case eval set, run on every PR, plus continuous monitoring and shadow deploys. For most consumer products it's smaller. For most regulated products it's bigger.
The wrong answer is "we'll figure it out." Without an explicit risk number, every quality investment looks like overspending until the day a customer-visible failure makes it look like underspending. Calculate the risk; size the program; defend the budget.