Ask most people how good a risk model is and they will quote its AUROC. That number answers one question — does the model rank sicker patients above healthier ones — and leaves a second, equally important one untouched: when the model says a patient's risk is 20%, is it really 20%? That second question is calibration, and a calibration curve is how you read it. This guide takes the curve apart by the ways it fails, pairing each shape with a real published model that failed that way and the decision the failure would distort. As of July 2026.
What a calibration curve shows
A calibration curve plots the risk a model predicts (on the horizontal axis) against the event rate actually observed (on the vertical axis), for groups of patients grouped by their predicted risk. Perfect calibration is the 45-degree diagonal: among patients told they have a 10% risk, 10 in 100 go on to have the event. Where the curve sags below the diagonal, the model is overpredicting — promising more disease than occurs. Where it rises above, the model is underpredicting. This is a different axis of performance from discrimination, and a model can score well on one while failing the other. The stakes are not academic: the analysis that named calibration the "Achilles heel" of predictive analytics warns plainly that "poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making" 1.
The points on the curve are formed by grouping patients — either into bins of similar predicted risk or, better, by fitting a smooth line through the whole range — and then comparing predicted with observed rates in each group. That grouping is why a curve reads as a shape rather than a single score: the failure can live at one end of the risk range and nowhere else, and a model that is accurate on average can still mislead badly among the highest-risk patients, who are often the ones a decision hangs on.
Four levels of calibration
It helps to know how strict a claim of "well calibrated" is. Models "can be calibrated in the mean, weak, moderate, or strong sense" 1, each level more demanding than the last.
| Level | What it requires | How it is checked |
|---|---|---|
| Mean | Average predicted risk equals overall event rate | Calibration-in-the-large / intercept |
| Weak | No general over- or under-prediction, and risks not too extreme | Calibration intercept and slope |
| Moderate | Predicted risks match observed proportions across the range | Flexible calibration curve |
| Strong | Correct for every combination of patient characteristics | A "utopic goal," rarely attainable |
Strong calibration is described as "a utopic goal" 1; moderate calibration — the curve tracking the diagonal across the whole range — is the realistic target you should look for in a validation study.
How calibration is measured
Two summary numbers do most of the work. The calibration intercept, or calibration-in-the-large, asks whether the average predicted risk matches the overall event rate; a value that pushes predictions systematically high or low is the signature of the overprediction problem below. The calibration slope asks whether the spread of predictions is right, with a target value of 1; a slope under 1 flags risks that are too extreme 3. Above these, a flexible calibration curve — a smoothed line fitted through the data rather than a few coarse bins — shows where along the risk range a model drifts off the diagonal. A dedicated clinician tutorial exists precisely because these measures are unevenly taught and easy to compute badly 6; one practical lesson from that literature is to distrust a single pass-or-fail goodness-of-fit test, which can wave through clinically important miscalibration that a curve would make obvious at a glance.
Reading the shapes
Systematic overprediction
The most common and most consequential failure is a curve that sits below the diagonal everywhere: the model overstates risk across the board. Its danger is easiest to see when two models with near-identical discrimination diverge on calibration alone. Take two cardiovascular-risk models validated in two million UK patients. Their ranking ability was almost the same, yet one was well calibrated and the other overestimated risk — and at the 20% treatment threshold that difference reshaped who got treated 1:
| Model | AUC (discrimination) | Flagged for treatment at 20% |
|---|---|---|
| Well-calibrated model | 0.771 | 110 per 1000 men |
| Overestimating model | 0.776 | 206 per 1000 men |
The overestimating model, despite a marginally higher AUC, "would select almost twice as many (206 per 1000 men) because a predicted risk of 20% based on this model actually corresponded to a lower event rate" 1. Nearly twice the treatment rate, driven entirely by miscalibration that AUROC could not see. That is over-treatment — cost, side effects, and anxiety — flowing directly from a curve nobody checked.
Risks that are too extreme
A second shape is a curve steeper or shallower than the diagonal, captured by the calibration slope. A slope below 1 means the model's risks are too extreme — high predictions too high, low predictions too low — which is the classic signature of an overfit model carried to new data. The standard performance framework treats the calibration slope and intercept as core measures precisely because they catch this, and concludes that "reporting discrimination and calibration will always be important for a prediction model" 3. A slope problem quietly widens the gap between the confident-looking extremes a model reports and the muted reality.
Drift after deployment
The third failure is temporal: a model that was well calibrated on the day it launched slowly stops being so. In a study that developed kidney-injury models and validated them over nine subsequent years, "discrimination was maintained for all models," but "calibration declined as all models increasingly overpredicted risk" 2. The ranking stayed fine; the numbers drifted upward until they no longer matched reality. The authors' framing is the one to carry: "models must be well calibrated to support decision-making, yet calibration deteriorates over time" 2. This is the calibration face of model drift, and it is why a good result at launch buys you monitoring, never a permanent guarantee.
The practical response those authors draw is worth carrying: because the rate and pattern of drift differed by modelling method, they argue that updating protocols should respond to periods of rapid drift rather than wait for a fixed annual or biannual review 2. Calibration, in other words, is a maintenance commitment, and a study that reports it only once — at launch, on the development population — has told you the least demanding version of the story.
Why discrimination is not enough
Each shape above shares a root cause: discrimination and calibration are independent, and a model audited only on AUROC can hide any of them. This is the central message of the calibration literature — that a model with excellent ranking can still report probabilities that are wrong in a direction that changes care. Because clinical decisions are made against risk thresholds, the size and direction of a calibration error translate directly into how many patients are treated or not. A curve is the fastest way to see that translation before it reaches a patient.
From curve to decision
Calibration matters because decisions are made on the numbers, and there is a tool that makes the link explicit. Decision curve analysis plots the net benefit of acting on a model across the range of thresholds a clinician might use 5. Crucially, miscalibration erodes that net benefit: an analysis of how calibration affects decision-analytic performance shows that a poorly calibrated model can be worse than useless at some thresholds even when its discrimination is respectable 7. So the honest test of a model is whether acting on its probabilities does more good than harm at the threshold you would actually use — ranking ability alone cannot answer that.
This reframes what "good enough" calibration means. A model that is imperfectly calibrated but errs in the same direction as the truth across your threshold can still add net benefit; one whose miscalibration crosses the threshold — telling you to treat when you should wait — can subtract it. Calibration is therefore judged against the decision it feeds, rather than in the abstract, and a small error near an important threshold can matter more than a large error far away from one 7.
How to read a calibration curve
A short routine handles most studies. Look first for whether a calibration curve or plot is reported at all — its absence is itself a finding. When it is present, read three things: the intercept (is the model systematically over- or underpredicting?), the slope (are its risks too extreme?), and how closely the flexible curve tracks the diagonal across the full range you care about, including away from the average. Then ask when and where it was measured, because calibration is the first thing to break on new data and over time.
Two cautions. Calibration is assessed in groups, so a curve can look reassuring on average while individual estimates remain uncertain — the reason strong calibration stays out of reach. And a curve is only as trustworthy as the sample behind it; small validation sets produce noisy curves that neither confirm nor condemn a model. Where a study reports the intercept and slope but shows no plot, you can still learn a great deal from those two numbers alone — and their absence, alongside a headline AUROC, is a reason to hold the result at arm's length. These interpretive points are stable, but reporting conventions continue to develop, so treat any single convention as current as of July 2026.
Sources and method
The framework and the four levels of calibration are taken from the "Achilles heel" analysis 1, whose cardiovascular worked example is quoted directly; the drift example uses the reported findings of a kidney-injury study 2; the calibration slope and intercept come from the standard performance framework 3; and the decision-analytic link draws on decision curve analysis 5 and the study of calibration's effect on net benefit 7. That the topic still needs a dedicated clinician tutorial 6, and that a systematic review of covid-19 models found nearly all at high or unclear risk of bias with calibration frequently unreported 4, is the reason this guide exists. Every figure is the number reported in its primary source. We revisit this guide on a 180-day cycle. For the wider reading discipline, see how to read an AI validation study and, for the ranking side, AUROC explained; for the definitions, see model calibration and external validation; for the numbers, the clinical AI trial results tracker.