Ask a clinical AI study one question first: how sure are you? Most headline results answer with a single number — 92% accuracy, an area under the curve of 0.89 — and stop there. That number is half a result. The other half is the interval around it, the range the figure would wander across if you ran the study again on a fresh sample of patients. On the small samples common in this field, that interval is wide enough to swallow the conclusion. This guide explains why so many AI studies are underpowered, shows with published figures how wide the real uncertainty is, and gives you the sample-size standards that separate a solid claim from a hopeful one. As of July 2026.
A point estimate is half a result
A confidence interval (CI) is the honest companion to any performance number. It says: given this sample size, the true value plausibly lies in this range. Two properties matter. The interval narrows as the sample grows, and it widens as the outcome gets rarer — because rare events, not raw patient counts, carry the statistical information. A model tested on 5,000 patients where only 50 had the disease is, for the purpose of precision, closer to a 50-patient study than a 5,000-patient one.
The practical consequence is that two models can look different and be statistically indistinguishable. If model A reports 90% sensitivity and model B reports 86%, but each interval spans eight or ten points, the four-point gap is inside the noise. Reporting the point estimate without the interval hides exactly this. It is why the glossary entry on sensitivity, specificity, and AUROC insists the operating-point numbers arrive with their uncertainty attached, and why our checklist for reading a validation study treats a missing interval as a finding in itself.
How wide is wide?
The uncomfortable answer is: wider than most authors admit. A methodological analysis of cross-validation showed that the sample sizes common in imaging and neuroimaging studies inherently produce large error bars — on the order of plus or minus 10 accuracy points for 100 samples — and, worse, that the standard error computed across cross-validation folds strongly underestimates the true uncertainty 1. In other words, the intervals studies do report are often too narrow, and the real ones are wider still.
You can see honest intervals when a rigorous meta-analysis reports them. Pooling the strongest studies comparing deep learning with clinicians, one review found a pooled sensitivity of 87.0% with a 95% CI of 83.0% to 90.2% for the models and 86.4% (79.9% to 91.0%) for the clinicians — overlapping ranges that support a verdict of rough equivalence rather than dominance. The same review concluded that few studies presented externally validated results and that poor reporting is prevalent, limiting reliable interpretation of the accuracy figures on offer 8.
| Source of the number | Reported precision | What it tells you |
|---|---|---|
| Cross-validation at n=100 1 | ~±10 accuracy points | Small samples give wide error bars |
| Across-folds standard error 1 | Understates the truth | Reported intervals are often too narrow |
| Pooled deep learning sensitivity 8 | 87.0% (95% CI 83.0-90.2) | Overlaps the clinician interval |
The events-per-predictor problem
Width is a symptom; the disease underneath is often too little data for the model's complexity. The standard measure is events per predictor (also called events per variable) — the number of outcome events divided by the number of candidate predictors the model weighs. When it is low, the model has too few real examples relative to its degrees of freedom, so it memorises noise and reports optimistic performance that will shrink on new patients.
Two systematic reviews put numbers on how common this is. Across 152 machine-learning prediction-model studies spanning every specialty, 148 of 171 analyses (87%, 95% CI 81% to 91%) were at high risk of bias, and 85 of 152 models (56%) were developed with an inadequate number of events per candidate predictor 3. A review focused on oncology found the problem sharpest in the most complex models: the median events per predictor was 8 for regression-based models but fell to 3.4 for other machine-learning methods and 1.7 for ensembles — and the required sample size was justified in only 5 of 62 studies (8%) 4. A median events-per-predictor of 1.7 means the typical ensemble model in that review had fewer than two real events for every variable it considered.
| Review | Metric | Finding |
|---|---|---|
| ML prediction models, all specialties 3 | High risk of bias | 87% of analyses |
| ML prediction models, all specialties 3 | Inadequate events per predictor | 56% of models |
| Oncology ML models 4 | Median events per predictor (ensembles) | 1.7 |
| Oncology ML models 4 | Sample size justified | 8% of studies |
A concrete picture helps. Imagine a model built to predict a complication that occurs in 40 of 1,000 admitted patients, using 20 candidate variables. That is 40 events across 20 predictors — two events per predictor — and a flexible machine-learning method will happily fit the 40 events almost perfectly, learning the idiosyncrasies of those particular patients rather than a transportable signal. The apparent accuracy looks excellent; the model has essentially memorised. Add more variables, or a more complex method, without adding events and the ratio falls further. This is why the events count, rather than the impressive total patient count, is the number to hunt for first — and why a review finding a median of 1.7 events per predictor for ensemble models is describing a field-wide tendency to overfit 4.
When bias masquerades as capability
Underpowering does more than widen intervals — it lets bias impersonate performance. The most revealing evidence comes from a review that extracted 282 head-to-head comparisons of machine learning against ordinary logistic regression. Among the 145 comparisons at low risk of bias, the difference in logit(AUROC) was 0.00, with a 95% CI of -0.18 to 0.18 — a flat tie. Among the 137 comparisons at high risk of bias, the same figure jumped to 0.34 in machine learning's favour 2. The advantage, in other words, lived almost entirely in the poorly controlled studies. Where the design was sound, the fancy model matched the simple one; the median study in this set carried about 8 events per predictor, and many far fewer 2.
The lesson generalises well beyond this one comparison: when you see a small study reporting a large, clean advantage, the advantage and the smallness are often the same fact. A tiny, optimistically analysed sample is precisely the condition under which a flattering gap appears and then fails to replicate. This is the numerical cousin of the design failures covered in our guide on prospective versus retrospective evaluation.
Small trials, big claims
Even the prospective end of the field — randomised trials, the strongest design — runs small. A systematic review of randomised trials of machine-learning interventions found 41 trials with a median of just 294 participants and a range from 17 to 2,488 7. A trial of a few hundred patients can be perfectly valid, but it cannot support a confident claim about a rare outcome or a small subgroup, because the interval around any such estimate will be broad. Sample size and confidence interval are the same coin: a study with 17 participants is telling you almost nothing with precision, whatever its point estimate says.
The spread is enormous, too, which makes a single median deceptive. In the logistic-regression comparison above, sample sizes ranged from 72 to nearly four million, and events per predictor from 0.3 to several thousand 2. A reader who sees only the reassuring high end can be lulled into trusting a literature whose low end is severely underpowered. Precision and size are not the same as correctness, either: a very large study can be precisely wrong if it is biased, delivering a tight interval around a distorted number — the flip side of the bias effect shown earlier, where poorly controlled small studies invented an advantage that vanished under scrutiny. Width and bias are separate problems, and a study can escape one while falling straight into the other.
What adequate actually looks like
The remedy is to plan sample size rather than inherit it. For developing a model, formal methods now compute the size needed to limit overfitting and to estimate the outcome's overall risk precisely, replacing the old habit of grabbing whatever data existed 6. For external validation, a dedicated calculation targets the precision of calibration, discrimination, and clinical utility. Its worked example is sobering: for a model predicting an outcome with a 1.8% event rate, achieving precise estimates required at least 9,835 participants (177 events) — with the calibration-slope criterion, rather than the headline sample count, doing the driving 5. Set that against validation studies that routinely enrol a few hundred, and the mismatch between what is claimed and what the data can bear becomes plain.
| Task | Standard | What it targets |
|---|---|---|
| Developing a model 6 | Sample-size calculation | Limit overfitting, precise risk estimate |
| External validation 5 | Minimum-sample calculation | Precise calibration, discrimination, utility |
| Worked validation example 5 | ≥9,835 participants (177 events) | Precision at a 1.8% event rate |
A model whose study reports its sample-size reasoning, its events per predictor, and a confidence interval on every headline number has done the arithmetic that most have skipped. One that reports a bare percentage has left you to guess how much of it is real.
How to read this
Four cautions travel with these numbers. First, a wide interval is a statement of honesty, never a defect to hide — be more suspicious of a suspiciously tight one on a small sample, which usually signals the across-folds understatement described above 1. Second, precision and correctness are different: a large study can be precisely wrong if it is biased, so read sample size alongside the risk-of-bias questions in our validation checklist. Third, the event count, not the patient count, sets the precision, so always find the number of outcome events before trusting an interval. Fourth, these base rates come from prediction and imaging models, the most-audited corners of the field; a modality without such a review is unmeasured rather than exempt. For where the field's numbers stand overall, see our clinical AI trial results tracker, and for how a well-calibrated model reports its uncertainty, see model calibration and AUROC explained for clinicians.
Sources and method
This guide synthesises one methodological analysis of cross-validation uncertainty 1, three systematic reviews that quantify sample-size and events-per-predictor adequacy across machine-learning prediction models 234, two sample-size standards for development and external validation 56, one review of the size and reporting of machine-learning randomised trials 7, and one meta-analysis reporting pooled accuracy with its intervals 8. Every figure is drawn from the primary source cited beside it. We revisit this page on a 180-day cycle and whenever a new methodological review or reporting standard lands. Sample-size adequacy is a statistical judgement rather than a regulatory one; for decisions that turn on clearance or reimbursement, confirm the current evidentiary requirements with your compliance or regulatory counsel before acting.