Two AI models can report the same headline accuracy and be worlds apart in how much you should trust them. The difference is often invisible in the abstract: it lives in one word — validation — and specifically in whether the number was earned on data the model had effectively already seen, or on data from a place it had never been. Internal versus external validation is the single distinction that most reliably separates a result that will survive contact with your patients from one that will quietly fall apart. This guide explains what each measures, shows how far performance typically drops between them, and pins every figure to its primary source. As of July 2026.
The two questions they answer
Internal validation asks: is this result reproducible within the setting it came from? You hold back part of the development data — a random split, or the folds of cross-validation — and test on that. It is a real and necessary check against one kind of self-deception, but it shares the training data's every quirk: the same scanners, the same coding habits, the same patient mix, the same disease prevalence.
External validation asks a harder question: does this result travel? Here the test data comes from a genuinely different source — another hospital, another country, a later time period, a different device. It is the step that estimates generalizability rather than reproducibility, and it is where inflated numbers tend to collapse 5. The framing from the methodological literature is a graded sequence — internal, internal-external, and external validation — because each rung answers a distinct question about how far a model can be trusted to move 6.
The trouble is that the easy rung is the one most studies stop at. Among 516 studies of diagnostic imaging AI published in early 2018, only 6% (31 studies) performed any external validation at all, and none of those 31 combined the three design features a robust test would need — a diagnostic-cohort design, multiple institutions, and prospective data collection 1. A separate review of AI-versus-clinician imaging studies reached the same verdict from another angle: few prospective studies and randomised trials existed, and most non-randomised studies were at high risk of bias 8. The prior you should carry into any new paper follows directly: rigorous external evaluation is the exception, so the burden is on the study to prove it is one of the exceptions.
How far performance actually falls
When external validation does happen, the news is rarely good. The most direct measurement comes from a systematic review that gathered every published external-validation study of deep-learning radiology algorithms it could find and compared each algorithm's internal score with its external one. Of 86 algorithms, 70 (81%) reported at least some decrease in external performance; 42 (49%) reported a modest decrease of 0.05 or more on the unit scale; and 21 (24%) reported a substantial decrease of 0.10 or more 2. Tellingly, no study characteristic reliably predicted which models would hold and which would crumble — meaning you cannot read the drop off the abstract in advance.
| Evidence | What was measured | Result |
|---|---|---|
| 516 imaging AI studies 1 | Share with any external validation | 6% (31 studies) |
| 86 radiology algorithms 2 | Share with any external drop | 81% (70) |
| 86 radiology algorithms 2 | Share with a substantial drop (≥0.10) | 24% (21) |
| Proprietary sepsis model 3 | External AUROC vs missed cases at threshold | 0.63; missed 67% |
| Pneumonia CNNs, 3 hospitals 4 | Internal beat external | 3 of 5 comparisons |
Read the table as a rough dose-response. A number reported without external validation (row one) is an unknown; a number reported with it (rows two onward) tends to shrink. The practical takeaway is to treat every internally validated figure as an upper bound, and to ask, for any externally validated one, how different was the external site?
Why healthcare punishes this so hard
Healthcare is unusually cruel to models that travel because the things a model silently learns are the things that differ most between hospitals. The clearest demonstration comes from a cross-site study of pneumonia detection on chest radiographs, run across three hospital systems. The models achieved better internal than external performance in 3 of 5 natural comparisons — but the sharper finding was why. The networks could robustly identify which hospital system and even which department produced a given image, and the sites differed enormously in disease burden: pneumonia prevalence was 34.2% at one hospital against 1.2% and 1.0% at the other two. That gap was large enough that merely sorting images by hospital system reached an AUROC of 0.861 — the model was partly detecting the hospital, not the disease 4.
This is the mechanism behind most external-validation collapses. A model rewarded for accuracy on its home data will happily exploit any signal correlated with the outcome there — a scanner's noise signature, a portable-versus-fixed radiograph marker, a site's baseline prevalence. None of those signals mean the same thing at the next hospital, so the borrowed accuracy evaporates. The related failure of carrying a stale model into a shifting population is covered in our guide on dataset shift and model decay; here the point is spatial rather than temporal, but the root cause — a model leaning on signals that do not transport — is the same.
The worked case: a sepsis model at scale
The most consequential external-validation story concerns a proprietary sepsis prediction model embedded in a widely used electronic health record and switched on at hundreds of hospitals largely on the strength of the vendor's own internal figures. When independent researchers validated it externally on more than 27,000 patients, it predicted the onset of sepsis with an area under the curve of 0.63 — far below the developer-reported range — and, at the alerting threshold hospitals actually ran, it failed to identify 1,709 patients with sepsis, or 67% of them 3. It also generated frequent alerts on patients who never developed sepsis, imposing real alert burden for little gain.
Two lessons sit inside that single study. First, an internally impressive model was deployed at population scale before anyone external had checked whether the number held — the exact ordering this guide argues against. Second, the summary figure (AUROC 0.63) and the number that governs bedside behaviour (67% of cases missed at the deployed threshold) were both bad, but the second was worse and more concrete. A summary curve can hide an operating point; see our glossary entry on sensitivity, specificity, and AUROC for why the threshold, not the curve, is what fires or fails to fire an alert.
The rungs of validation
It helps to hold the whole ladder in view rather than treating validation as a single yes-or-no box.
| Rung | What it does | What it proves |
|---|---|---|
| Internal (split / cross-validation) | Tests on held-out data from the same source | Reproducibility within one setting |
| Internal-external (leave-one-site-out) | Rotates whole sites in and out of training | Early signal of transportability 6 |
| External (independent site / era) | Tests on a genuinely different source | Generalizability to a new population 5 |
| Local evaluation before go-live | Tests on your data and workflow | Fitness for your specific setting |
Internal-external validation — sometimes run as leave-one-site-out cross-validation — is a useful middle rung: it reuses multi-site development data to estimate how the model behaves at a held-out site without waiting for a wholly new cohort 6. But it does not retire the top rung. Even a model that transported well to one external site can miscalibrate at yours, because performance is a property of the model paired with a population, its data pipelines, and its workflows 5. This is why the reporting standards now insist on it: TRIPOD+AI, the 2024 guidance for reporting prediction models built with regression or machine learning, requires studies to state clearly how and where a model was validated, including performance on data external to the development sample 7.
External validation also comes in two flavours that a careful reader keeps separate. Geographic external validation moves the model across space — a new hospital, region, or country — and is what most of the figures above measure. Temporal external validation moves it across time, testing on patients from a later period than the development data. The two probe different weaknesses: the geographic test catches a model leaning on site-specific quirks, while the temporal test catches one leaning on practices that have since changed. A model can pass one and fail the other, which is why the strongest evidence packages report both, and why a single "externally validated" label should prompt the question validated how — across which axis? The temporal axis is the entry point to the decay problem covered separately; the point here is that a study claiming external validation has told you very little until it says which move it survived.
How to read these numbers
Four cautions travel with everything above. First, the direction of the drop is reliable but its size is not predictable from study features 2, so resist the temptation to assume a small, safe decrement — some models lose very little and some lose a great deal. Second, "externally validated" is a floor, not a ceiling: a single external site tells you the model survived one move, not that it will survive yours. Third, external validation can look fine on discrimination while calibration silently fails, handing you probabilities that are systematically off; our guide on how to read an AI validation study walks through checking both. Fourth, these base rates are from imaging and prediction models — the fields with the most audits — and other modalities may be better or worse studied, so absence of a drop figure is absence of evidence, never evidence of transportability.
For where the field's numbers stand overall, see our clinical AI trial results tracker; for the timing question that sits alongside this spatial one, see our companion guide on prospective versus retrospective evaluation. A model that is well calibrated and externally validated at several sites has earned real trust; one validated only internally has earned a footnote.
Sources and method
This guide synthesises two systematic reviews that quantify how often clinical AI is externally validated and how far performance falls when it is 12, one external-validation study used as a worked cautionary example 3, one cross-site study that isolates the mechanism behind transportability failure 4, two methodological references that frame the rungs of validation 56, the current reporting standard for prediction models 7, and one review of design and reporting quality across the field 8. Every figure is drawn from the primary source cited beside it. We revisit this page on a 180-day cycle and whenever a governing standard or a large new audit lands. For decisions that turn on regulatory clearance or reimbursement, confirm the current evidentiary requirements with your compliance or regulatory counsel before acting — clearance status is a separate question from validation quality.