Automation bias

Automation bias is the tendency to over-rely on an automated system's output, accepting its advice with less scrutiny than you would give the same information from another source 1. It causes two kinds of error: commission, when a clinician follows a wrong suggestion, and omission, when a clinician misses a problem because the system did not flag it 2.

Why does automation bias matter in healthcare?

Clinical AI is often deployed as an assistant, with a clinician expected to catch its mistakes. That design assumes the reviewer stays independent of the tool. Automation bias is the evidence that the assumption fails in predictable ways. A systematic review of 74 studies found it across clinical decision support, and identified what makes it worse: trust in the system, heavy workload, task complexity, and time pressure 1.

The EU AI Act writes the term into law. Article 14 requires high-risk AI systems to be provided in a way that lets the people overseeing them "remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)" 5. A human in the loop is only a safeguard if that human still checks.

How large is the effect in clinical studies?

Two experiments give a sense of scale.

In mammography, 27 radiologists read cases while a purported AI system suggested a BI-RADS category. When the suggestion was correct, readers rated 79.7% to 82.3% of mammograms correctly, depending on experience. When it was wrong, accuracy fell to 19.8% for inexperienced readers, 24.8% for moderately experienced readers, and 45.5% for very experienced readers 3. Experience softened the effect. It did not remove it.

In inpatient care, a randomized vignette study enrolled 457 hospitalist physicians, nurse practitioners, and physician assistants across 13 US states. Their baseline diagnostic accuracy for pneumonia, heart failure, and COPD was 73.0%. A standard AI model raised it by 2.9 percentage points. A systematically biased model lowered it by 11.3 points 4.

The asymmetry matters for deployment. The gain from a correct model was small; the loss from a flawed one was nearly four times larger.

Common misunderstandings

Experts are immune. The most experienced mammography readers still lost 36.8 percentage points of accuracy on cases with a wrong suggestion 3. A human-factors review found automation bias in naive and expert participants alike 2.

Explanations fix it. In the hospital study, adding image-based explanations to the biased model improved accuracy by 2.3 points, a difference that did not reach statistical significance 4.

Training fixes it. Goddard's review lists training and emphasizing user accountability among mitigators 1. The human-factors review was blunter: automation bias "cannot be prevented by training or instructions" 2. Training helps at the margin. Interface design and workload carry more weight.

It only happens when clinicians multitask. A review focused on verification complexity found automation bias in single tasks, typically diagnosis, where checking the machine's answer is hard work 6.

A related effect, deskilling, is what happens to unaided performance after long use of a tool. The first real-world signal came from colonoscopy, covered in Briefing 003.

What to ask a vendor

• What happens to clinician accuracy when your tool is wrong? Ask for a reader study that includes deliberately incorrect outputs, as the mammography study did 3.
• Does the interface present information or a recommendation? Goddard found that choice, the position of advice on screen, and displayed confidence levels all changed how much users over-relied 1.
• Is the displayed confidence calibrated?
• How often does the tool fire when nothing is wrong? A low positive predictive value trains users to dismiss alerts, which is the omission side of the same problem.
• Do you report how often clinicians change or reject the output in deployment?

Related terms

• Human-in-the-loop: the oversight design automation bias undermines.
• Clinical decision support system: where most of the evidence comes from.
• Designing human oversight for clinical agents: oversight patterns that account for over-reliance.