The most common way an AI result misleads is timing. A model is trained and then tested on data that already exists — historical, tidy, often hand-curated for the study — and it posts a number that looks like proof. What it has actually earned is a promising retrospective result, which is the lowest bar in clinical evaluation, never the last one. Whether that promise survives depends on a different kind of study: a prospective one, run forward on new patients inside the real workflow. This guide sets out what each design can and cannot show, walks through documented cases where retrospective promise did not hold prospectively, and ends with one where it did. As of July 2026.
What each design can prove
Retrospective evaluation is fast, cheap, and repeatable. You take existing data, apply the model, and compute accuracy. Its weakness is baked into its convenience: historical datasets are usually curated. Poor-quality images are excluded, labels are cleaned, the case mix is whatever was easy to assemble, and the model never has to cope with an input it cannot handle. Every one of those conveniences flatters the result.
Prospective evaluation runs the model forward on patients as they arrive, in the setting and workflow where it will be used. It is slower and costlier, and it exposes everything the curated study hid: ungradable inputs, workflow friction, shifting prevalence, and the human response to the model's output. This is the timing counterpart to the place question covered in our guide on internal versus external validation — and a model needs to clear both. The reporting standards treat the move from retrospective, "silent" testing to live prospective use as its own stage of evidence: DECIDE-AI exists specifically as the reporting guideline for the early stage clinical evaluation of decision-support systems driven by artificial intelligence 6.
The mammography ledger
The cleanest illustration that retrospective and prospective evidence can point in opposite directions comes from breast-cancer screening. A systematic review of test-accuracy studies gathered every eligible study of AI in mammography and sorted them by design. Its first finding was an absence: no prospective studies measuring the test accuracy of AI in screening practice were found, and the studies that did exist were of poor methodological quality 1.
The contrast inside the review is the lesson. Five smaller laboratory studies — AI reading a test set of images in isolation, at high risk of bias — reported that all five evaluated AI systems were more accurate than a single radiologist. But the studies that compared AI against radiologists' real clinical decisions on screened women told the opposite story: 34 (94%) of 36 AI systems were less accurate than a single radiologist, and every one was less accurate than the consensus of two or more 1. The laboratory setup — clean images, no workflow, a handicapped human comparator — produced exactly the flattering result retrospective designs are prone to. The evaluation closer to practice reversed it.
| Design | Setting | Verdict on AI |
|---|---|---|
| Laboratory test set (5 studies, high bias) 1 | Curated images, isolated read | All 5 systems beat a single radiologist |
| Comparison with real radiologist decisions 1 | Screening records | 94% of 36 systems worse than a single radiologist |
| Prospective accuracy in screening practice 1 | Live programme | No studies existed |
Why did the laboratory studies flatter the AI so badly? Two mechanisms, both characteristic of retrospective design. First, the comparator was handicapped: a single radiologist reading an isolated image on a test set has none of the prior notes, priors, and consultation a working radiologist draws on, so the human baseline is artificially low. Second, the image sets were curated — enriched with clearer cases and cleaned of the ambiguous, poor-exposure films that fill a real programme — which inflates any reader's apparent accuracy and shifts what a given threshold means. Strip both artefacts away by comparing against real clinical decisions, and the ranking flips. The retrospective number was measuring the study design as much as the model.
The field-deployment gap
Numbers on a page also hide the physical reality of a clinic. A deep-learning system for diabetic eye disease — the kind of tool that scores at specialist level on curated retinal images — was studied prospectively across eleven clinics in Thailand, with researchers observing nurses and patients rather than only counting correct classifications. The evaluation surfaced a gap that no retrospective run could: in the authors' words, there were "tensions between the model's thresholds for data quality, and the quality of data that arise from an imperfect, resource-constrained environment" 2. Images that a curated dataset would have quietly excluded were the everyday output of real clinics with variable lighting and equipment, and the system's insistence on high-quality inputs collided with the pace and constraints of frontline work. The model was capable; the deployment was where capability met friction.
The sepsis model
Sometimes the prospective reckoning arrives after the tool is already switched on at scale. A proprietary sepsis prediction model, embedded in a widely used electronic health record and enabled at hundreds of hospitals largely on internal figures, was validated by independent researchers on more than 27,000 real patients. It predicted sepsis onset with an area under the curve of 0.63 and, at the alert threshold hospitals actually ran, missed 1,709 patients with sepsis — 67% of cases — while firing frequently on patients who never developed it 3. The ordering was backwards: population-scale deployment came first, independent forward-looking evaluation second. Post-deployment monitoring of exactly this kind is the discipline our glossary entry on algorithmovigilance describes.
When prospective evidence confirms
None of this means retrospective promise always evaporates. The point of prospective evaluation is to find out, and sometimes the answer is yes. In a prospective, randomised, controlled trial, 1,058 patients undergoing colonoscopy were assigned to a standard procedure or to one assisted by a real-time deep-learning polyp-detection system. The AI arm raised the adenoma detection rate from 20.3% to 29.1% — a large, statistically significant improvement in the measure that matters for preventing colorectal cancer 8. The study had limits its authors named — it was unblinded, single-country, and the extra detections were weighted toward small lesions — but it did the essential thing: it tested the tool forward, on real patients, against a randomised control, and the benefit held. A retrospective claim graduated into prospective evidence.
The difference between this case and the failures above is instructive. The colonoscopy trial measured a clinically meaningful endpoint prospectively with a control group; the mammography laboratory studies measured accuracy on curated images with a handicapped comparator. Same technology family, opposite evidentiary weight.
How thin the prospective base rate is
Step back from individual cases and the structural problem appears: prospective, randomised evidence for clinical AI is scarce. A systematic review searching through mid-2019 found only 10 registrations for deep-learning randomised clinical trials, of which just two had been published 4. A later review that focused specifically on randomised trials of machine-learning interventions found 41 RCTs, with a median of 294 participants (range 17 to 2,488); 51% were single-site, no trial adhered to all CONSORT-AI reporting standards, and 38 (93%) did not assess poor-quality or unavailable input data — the very failure mode the Thai clinic study made visible 5. A review of translation challenges put the situation plainly: there are still limited examples of AI techniques successfully deployed into clinical practice, and randomised controlled trials should be viewed as the gold standard for evidence generation, even where running them is hard 7.
| Evidence | What it counted | Finding |
|---|---|---|
| DL RCT registrations to mid-2019 4 | Registered / published | 10 registered, 2 published |
| ML intervention RCTs 5 | Trials / median size | 41 RCTs, median 294 participants |
| ML intervention RCTs 5 | Adhered to all CONSORT-AI | None |
| ML intervention RCTs 5 | Did not check input-data quality | 93% |
The shape of that RCT evidence is as telling as its scarcity. In the same review, just over half the trials ran at a single site, and more than a third clustered in a single specialty — endoscopy — where a real-time detection task is unusually well suited to a trial. Overall risk of bias was high in about one in six trials, and none reported everything the CONSORT-AI standard asks for 5. So even the prospective evidence that exists is concentrated, single-site, and incompletely reported. A field cannot be judged mature on the strength of a few dozen small trials huddled around the one task that was easiest to test.
The bridge from silent to live
The way to avoid both the mammography trap and the sepsis ordering error is to treat evaluation as a staged path, and to know which stage a study sits on. Retrospective accuracy comes first. Then silent prospective testing, where the model runs live but no one acts on its output, so you can watch it meet real inputs without risk. Then live prospective evaluation, where clinicians see and respond to the output — the stage DECIDE-AI was written to structure and report 6. Most tools that fail do so at the transition into live use, because that is where human trust, workflow, and messy data all arrive at once. A model evaluated only retrospectively has yet to meet any of them.
How to read this
Three cautions travel with everything above. First, retrospective is a legitimate and necessary starting point — the failure is treating it as a verdict, never running the study at all. Second, prospective is stronger evidence but carries its own weaknesses: a prospective study can be small, single-site, unblinded, or measure a proxy rather than patient benefit, so read its design with the same scepticism, using our checklist for reading a validation study. Third, small prospective samples produce wide uncertainty; a median of 294 participants leaves confidence intervals broad enough to matter, which is the subject of our companion guide on sample size and confidence intervals. For where the field's prospective numbers stand overall, see our clinical AI trial results tracker, and for the related question of whether a result holds at a different place, see internal versus external validation and dataset shift and model decay.
Sources and method
This guide synthesises one systematic review that separates laboratory from real-decision breast-screening evidence 1, one prospective human-centered field study 2, one external-validation study used as a worked example 3, two reviews that quantify how scarce prospective randomised evidence is 45, one reporting guideline for early live evaluation 6, one review of translation challenges 7, and one prospective randomised trial used as a confirming counter-example 8. Every figure is drawn from the primary source cited beside it. We revisit this page on a 180-day cycle and whenever a governing guideline or a major new trial lands. For decisions that turn on regulatory clearance or reimbursement, confirm the current evidentiary requirements with your compliance or regulatory counsel before acting — a retrospective clearance record is a separate question from prospective clinical benefit.