A model can post an excellent overall accuracy and still be unsafe for the patients who most need it to work. Aggregate metrics average across everyone; they say nothing about whether the model fails a particular group — by race, sex, skin tone, age, or the intersection of several. Subgroup performance is the question those averages hide, and a bias audit is the procedure that answers it. This guide gathers the published failures that make the case, explains why they happen, and sets out the checks that would have surfaced each one before it reached a patient. As of July 2026.
Why the average is not enough
Overall accuracy, and even a strong AUROC, is a weighted average over the tested population. If a subgroup makes up a small share of that population, a model can fail it badly while the headline barely moves. The gap becomes visible only when performance is measured within each group and, just as importantly, at the intersections where groups overlap. This is the same discipline that separates a robust validation study from a fragile one — the subject of our guide on how to read an AI validation study — turned on the question of for whom the number holds. An audit answers three things a single figure cannot: does the model discriminate as well inside each group, are its probabilities equally trustworthy across groups (a matter of calibration), and does either property hold up on external populations it was never trained on.
The documented failures
The case for subgroup auditing rests on a decade of specific, peer-reviewed failures. Each one below cleared publication in a high-impact venue, and each would have been caught earlier by a metric the original development did not report.
| Subgroup axis | Documented failure | Where measured |
|---|---|---|
| Race (measurement) | Pulse oximeters missed low oxygen far more often in Black patients | 12 |
| Prediction target | A cost-as-need algorithm under-referred Black patients | 3 |
| Race / intersection (imaging) | Chest X-ray classifiers underdiagnosed under-served groups, worst at intersections | 4 |
| Skin tone (dermatology) | AI performed substantially worse on dark skin tones | 5 |
| Sex (dataset balance) | Sex-imbalanced training degraded accuracy for the underrepresented sex | 6 |
A signal that fails by race
The starkest example predates modern AI but governs every model built on top of it. Pulse oximetry — the fingertip oxygen reading feeding countless alerts and risk scores — systematically overestimates blood oxygen in patients with darker skin. In a two-cohort analysis, occult hypoxemia, meaning a true arterial saturation below 88% despite a reassuring reading of 92 to 96%, occurred in 11.7% of Black patients versus 3.6% of White patients: "Black patients had nearly three times the frequency of occult hypoxemia that was not detected by pulse oximetry as White patients" 1. The consequence is measurable, not hypothetical. In a later cohort of patients with COVID-19, occult hypoxemia occurred in 28.5% of Black, 29.8% of non-Black Hispanic, and 30.2% of Asian patients versus 17.2% of White patients, and was associated with delayed recognition of eligibility for oxygen-threshold-specific therapy 2. Any model that consumes an oxygen reading inherits this bias whether or not its own code ever touches race.
An algorithm biased by its target
A model can be skewed by nothing more than the quantity it was trained to predict. A widely used commercial algorithm meant to flag patients for extra care management predicted future health costs as a stand-in for health needs. Because the system spends less on Black patients at the same level of illness, the algorithm rated them lower-risk than equally sick White patients: at a given risk score, Black patients were considerably sicker. Correcting the target so it tracked illness rather than cost would have raised the share of Black patients flagged for additional help "from 17.7 to 46.5%" 3. No subgroup slice of accuracy would have caught this on its own, because the model was accurate at predicting the thing it was asked to predict. The audit that exposed it was an audit of the label.
Imaging models that underdiagnose
In chest radiography, classifiers built with standard techniques "consistently and selectively underdiagnosed under-served patient populations" — labelling genuinely ill patients as healthy — and the miss rate was highest at the intersections, "for example, Hispanic female patients" 4. Underdiagnosis is the most dangerous direction of error, because a false reassurance delays care rather than prompting a second look. A separate large study traced one mechanism directly to the data: when a training set was deliberately imbalanced by sex, performance dropped consistently for the underrepresented sex 6. A dataset's composition is a design choice with clinical consequences, whether or not anyone made it deliberately.
Dermatology and skin tone
Dermatology AI shows the same pattern along skin tone. On the Diverse Dermatology Images set — assembled precisely because most training data skews toward lighter skin — state-of-the-art models "exhibit substantial limitations... particularly on dark skin tones and uncommon diseases" 5. The encouraging half of the finding is that fine-tuning on the diverse images narrowed the gap between light and dark skin tones, which tells you the deficit lived in the data rather than in the task. That is the recurring shape of these stories: the model is doing exactly what its data taught it, and the data under-represented someone.
Why subgroup failures happen
Four mechanisms account for most of the documented cases, and a good audit probes for all four.
- Biased measurement in the inputs. When a signal itself is less accurate for a group — as pulse oximetry is for darker skin — every downstream model absorbs the error 12.
- A biased prediction target. When the label is a convenient proxy rather than the outcome of interest, the model faithfully reproduces the proxy's bias, as with cost standing in for need 3.
- Unrepresentative training data. When a group is under-represented in the training set, accuracy for that group suffers, whether the axis is sex 6 or skin tone 5.
- Corrections encoded by design. Several clinical algorithms adjust their output up or down by a patient's race. One analysis reconsidered the use of race correction across specialties and argued that these adjustments can shift care by race on assumptions that often lack a sound basis 7. Whatever the intent, a race term is an input an audit must inspect explicitly.
These mechanisms sit at different points of the development pipeline, which is why a one-time check at the end is insufficient. One review frames the ethics of clinical machine learning as a pipeline problem, warning that "models can amplify existing health inequities" from problem selection through post-deployment monitoring 8. Equity is a property of the whole process, earned across it rather than ticked off at validation.
How to run a bias audit
An audit turns those mechanisms into a checklist. The aim is to make the model prove it works for whom, using the specific metric that would surface each failure mode.
| Step | What to check | The failure it catches |
|---|---|---|
| 1. Stratify performance | Sensitivity, specificity, and calibration within each subgroup | Group-specific misses hidden by the average 46 |
| 2. Test intersections | Metrics at overlaps (e.g. sex × race), not one axis at a time | Compounded failure at intersections 4 |
| 3. Interrogate the label | Whether the prediction target is the outcome or a biased proxy | Target-driven bias 3 |
| 4. Inspect the inputs | Whether any predictor encodes race or a race correction | Encoded adjustments 7 |
| 5. Check the data mix | Subgroup representation in training and test sets | Under-representation 56 |
| 6. Trace the signal | Whether any input is measured less accurately for a group | Measurement bias 12 |
Two operational notes make the checklist usable. First, subgroup metrics need adequate numbers in each cell; a group too small to measure is a group you cannot claim the model serves, and that limitation belongs in the write-up. Second, an audit is a standing commitment, never a one-off — populations, coding, and instruments drift, so subgroup performance is monitored after deployment as part of algorithmovigilance, rather than certified once and filed.
How to read these numbers
Three cautions travel with everything above. The failures collected here are the ones that were found; the base rate of unaudited models in the field is unknown, and absence of a reported subgroup gap is not evidence of its absence. The studies also span different tasks, datasets, and eras, so a specific percentage — the 11.7% occult-hypoxemia rate, the 17.7-to-46.5% correction — describes its own setting and should not be transplanted onto yours; treat them as proof that the failure mode is real, not as your expected effect size. And a clean subgroup audit is necessary but insufficient: a model can be equitable across groups and still be poorly calibrated, externally invalid, or measuring the wrong outcome, which is why this guide sits alongside the wider validation checklist rather than replacing it.
Bias findings can also carry legal, regulatory, and accreditation weight that varies by jurisdiction; confirm your obligations with compliance or counsel before acting on an audit result.
Sources and method
This guide synthesises seven peer-reviewed subgroup-failure studies spanning measurement bias 12, a biased prediction target 3, imaging underdiagnosis and dataset imbalance 46, dermatology skin-tone disparity 5, and encoded race corrections 7, together with one framework paper that places equity across the development pipeline 8. Every figure is drawn from the primary source cited beside it. We revisit this page on a 180-day cycle and whenever a governing equity-reporting standard is updated. For the numbers behind the wider field, see our clinical AI trial results tracker; for the discipline of monitoring these gaps after launch, see algorithmovigilance.