A clinical AI model is validated at one moment, on one population. Then the moment passes and the population changes. New treatments alter who is admitted, a coding update relabels a diagnosis, a scanner is swapped, a pandemic rewrites the case mix overnight. When live data drifts away from the data a model learned from, the model gets quietly worse — a phenomenon called model decay, driven by dataset shift. This guide sorts the published failures into the three shapes decay actually takes, names the signal that warns you first, and reviews what the evidence says slows it down. As of July 2026. For the one-line definition and its place in the glossary, start with model drift; this guide is the longer, case-by-case treatment.
Three geometries of failure
"The model got worse" is too coarse to act on. Decay arrives through three distinct geometries, and each is diagnosed and fixed differently. Naming which one you face is the first move.
| Geometry | What changes | Documented case |
|---|---|---|
| Spatial (site-to-site) | The deployment site differs from the development site | Sepsis model scored AUC 0.63 at new hospitals 2 |
| Temporal (drift over time) | The world moves while the model stays fixed | Kidney-injury models over-predict across nine years 3; COVID reshapes admissions 4 |
| Deployment-environment | The clinic produces data the model refuses | Retinopathy model versus real-clinic image quality 5 |
Formally, one clinician-facing account describes the underlying problem plainly: a system "underperforms because of a mismatch between the data set with which it was developed and the data on which it is deployed" 1. That single mismatch expresses itself as covariate shift (the inputs move), label shift (the outcome's base rate moves), or concept shift (the relationship between inputs and outcome moves). The three geometries above are where those abstract shifts show up in a real deployment.
Spatial shift: the same model, a new site
The most immediate failure is a model that worked where it was built and falters where it is installed. When a widely implemented proprietary sepsis model was evaluated outside its origin system, it "predicted the onset of sepsis with an area under the curve of 0.63" and, at its live alerting threshold, "did not identify 1709 patients with sepsis (67%)" 2. Nothing had drifted over time; the gap was present from the first day the model met a different population, coding practice, and workflow. This is why external validation is the precondition for trusting any transported model — a strong internal result is an upper bound, and the drop on arrival can be severe.
Spatial shift is also where bias and decay entangle. A study across radiology, dermatology, and ophthalmology showed that models leaning on demographic shortcuts can be algorithmically corrected to look fair on their home distribution, yet "this optimality is not true in new test settings" 7. A fairness fix tuned to one site is itself a site-specific artifact that need not survive the move. The practical reading: fairness, like accuracy, is a claim about a population, and both have to be re-earned at each new deployment.
Temporal shift: the world drifts while the model holds still
Even inside a single site, a frozen model ages against a moving world. The clearest long-run evidence comes from acute kidney injury: using Veterans Affairs admissions, seven parallel models were validated over nine subsequent years. "Discrimination was maintained for all models," yet "calibration declined as all models increasingly overpredicted risk," with the drift tracking a genuine change in the event rate 3. The model's ability to rank patients held; its ability to state a trustworthy probability did not. A monitoring dashboard watching only an AUROC would have seen nothing wrong for the better part of a decade.
Temporal shift can also arrive as a shock rather than a slow slide. During COVID-19, emergency-department admission models faced both kinds of change at once, because "the underlying data distributions characterising episodes can change with time (data drift), and so can the relationship between episode characteristics and associated clinical outcomes" 4 — covariate and concept shift together. The pandemic is the vivid version of a routine truth: any event that changes who presents, or what their features mean, is a shift the model cannot see from the inside.
Deployment-environment shift: the data the model will not accept
The third geometry is easy to miss because it hides in the gap between a benchmark and a clinic. A deep-learning system for diabetic retinopathy, accurate in the lab, was deployed across eleven clinics. The field study found "tensions between the model's thresholds for data quality, and the quality of data that arise from an imperfect, resource-constrained environment" 5: retinal images taken in ordinary clinic lighting frequently failed the model's strict quality bar and were rejected, sending patients and staff back around the loop. The model had not decayed in any statistical sense. The environment simply produced inputs it was built to refuse — a shift in the conditions of use rather than in the data distribution, and one that only a human-centered evaluation surfaces.
Why calibration usually breaks first
A pattern runs through the temporal cases: discrimination is durable, calibration is fragile. A systematic review of temporal-shift studies found that "calibration deterioration was more common (n = 11) than discrimination deterioration (n = 3)" 6, echoing the kidney-injury result exactly. The reason is intuitive — a model can keep ranking sicker patients above healthier ones long after the actual risk attached to a given score has moved. That makes calibration the earliest warning light on the dashboard, and it is precisely the metric most deployment monitoring omits. If you track one thing beyond raw volume, track the gap between predicted risk and observed event rate over time.
What actually slows decay
Naming the failure is half the job; the evidence also points to what helps.
- Update and recalibrate on a schedule tied to drift, not the calendar. In the temporal-shift review, the common mitigations were model-level: refitting, probability recalibration, and model updating. All of them "were successful at preserving calibration" though "not uniformly successful in preserving discrimination" 6. Recalibration is the cheap, high-yield first move when probabilities drift.
- Prefer designs that decay more slowly. Foundation models pretrained on large electronic-health-record histories were more robust than conventional count-based models: "temporal distribution shift negatively impacts the performance of clinical prediction models over time," but the pretrained representations "often exhibited less decay" in discrimination 8. Architecture is one lever on how fast the clock runs.
- Plan the updates before you need them. A model that must be frozen to stay compliant will decay by default. A predetermined change control plan is the regulatory mechanism for authorising defined updates — including recalibration — in advance, so maintenance does not require restarting review from zero.
Detecting decay: monitoring as a discipline
None of the above works without ongoing measurement, because decay is invisible from inside the model. Continuous, equity-aware surveillance of a live model has become a named practice — algorithmovigilance — framed as "advancing methods to analyze and monitor artificial intelligence–driven health care for effectiveness and equity" 9. In operational terms that means tracking the input distributions the model receives, comparing predicted risks against observed outcomes, watching calibration alongside discrimination, and breaking every one of those out by subgroup so a fair model does not decay unevenly. The validation checklist that earns a model its first deployment is described in our guide on how to read an AI validation study; monitoring is that checklist, repeated forever.
Because updating a cleared or deployed model can itself trigger regulatory review, confirm your change-control and monitoring obligations with regulatory or compliance counsel before modifying a model in production — the safe engineering move and the compliant one are not always identical.
How to read this
Three cautions travel with the cases above. First, the specific figures — the AUC of 0.63, the nine-year calibration slide — describe their own settings and are evidence that a failure mode is real, rather than a forecast of your model's decay rate. Second, the three geometries co-occur: a transported model can face spatial shift on day one and temporal shift thereafter, and monitoring has to watch for both. Third, the absence of a detected drop is only as trustworthy as the monitoring behind it; a model that is not being measured is not "stable," it is unobserved.
Sources and method
This guide synthesises one clinician-facing account of dataset shift and its taxonomy 1, four primary decay and shift studies spanning spatial 2, temporal 34, and deployment-environment 5 failure, a systematic review quantifying which metric decays and what mitigations help 6, evidence on shift-robust model design 78, and the framing paper for post-deployment monitoring 9. Every figure is drawn from the primary source cited beside it. We revisit this page on a 180-day cycle and whenever lifecycle-monitoring guidance is revised. For the field-level picture of how far clinical AI has spread and where it strains, see our AI in healthcare statistics.