Automation Bias: Why a Confident Machine Overrules What You Can See

An automated aid changes what a person has to do, and the mind reorganizes around the change faster than the job description does. Instead of monitoring the raw information and deciding, the operator monitors the aid and approves. That shortcut is usually efficient: the aid is right most of the time, approving is fast, and the record of correct prompts builds a general picture of a reliable system that then gets applied to the next prompt without inspection. The cost arrives on the rare prompt where the picture does not hold.

The errors come in two shapes. A commission error is acting on the system's instruction when other available evidence says it is wrong, like turning onto a bridge the dispatch sheet says is closed. An omission error is failing to notice something because the system did not announce it, like driving past a sign the app never mentioned. The two have different causes. Commission errors come from treating the aid as the authority on the question; omission errors come from treating the aid's silence as information.

Raja Parasuraman and Dietrich Manzey, reviewing the field in 2010 in Human Factors, argued that both are better explained by where attention goes than by any decision to be lazy. Operators working with a reliable aid sample the raw data less often, and they sample it less the longer the aid has been right. Parasuraman and Manzey distinguished this attentional complacency from the decision shortcut itself, and noted that trust built over a long run of correct outputs is reasonable in every respect except that it also removes the only check that would catch the exceptions.

Several things make it worse. A system that reports a result without showing how it got there gives the operator nothing to check. Time pressure removes the slack that checking needs. Working in a team can spread responsibility rather than concentrate it, because each person assumes another is watching. Skitka, Mosier and Burdick found in a 2000 study that telling participants they would have to account for their decisions reduced both error types, which is the most practical finding in this literature: the bias responds to how the job is set up more than to warnings about the bias.

Linda Skitka, Kathleen Mosier and Mark Burdick published "Does automation bias decision-making?" in the International Journal of Human-Computer Studies in 1999. Participants flew a simulated flight task that required them to monitor a set of gauges and respond to events as they appeared. One group had an automated monitoring aid that flagged events for them; the other group had no aid and had to watch the gauges themselves.

The group with the aid did well on everything the aid caught, and that is the finding people forget. Where they came apart from the unaided group was on the events the aid did not flag, which they tended to miss, and on the occasions when the aid gave a directive the gauges contradicted, which they tended to follow. The authors named these omission and commission errors and argued that automation bias is not a failure of attention alone but a replacement of the task: the aid becomes the thing being monitored.

The sharpest version of the finding came a year earlier. In 1998 Kathleen Mosier, Linda Skitka, Susan Heers and Mark Burdick tested commercial pilots with glass-cockpit experience on a part-task simulator. At one point the automated system reported an engine fire that no other instrument in the cockpit supported: every engine parameter on the display stayed normal and no other fire indication appeared. Every pilot shut the engine down, a 100 percent commission error rate as Parasuraman and Manzey's 2010 review records it. A majority of those who complied then reported having seen a confirming indication that had not been present, an error the authors called phantom memory.

The pattern is not confined to aviation. Kate Goddard, Abdul Roudsari and Jeremy Wyatt reviewed studies of clinical decision support systems in 2012 for the Journal of the American Medical Informatics Association and found the same two error types in clinicians working with prescribing and diagnostic aids, along with evidence that incorrect advice could move a clinician away from a decision they had reached correctly on their own.

Automation bias is strongest where the system is usually right, which is most places it is used.

Automation bias responds to how work is arranged far better than it responds to being told about it.

• Decide in advance which decisions get an independent check. Pick them by what a mistake would cost and how hard it would be to reverse, not by how uneasy a particular prompt makes you feel.
• Treat silence as missing information. An aid that reports nothing has told you about the events it monitors, not about the events it does not.
• Sample on a schedule. Checking a fixed proportion of cleared cases is the only way to find out whether the system is still as reliable as it used to be.
• Ask for the reasoning, not the answer. A system that shows the inputs behind a recommendation gives you something to check; one that shows a conclusion gives you something to approve.
• Make one person accountable for the decision rather than for the approval. The 2000 study by Skitka, Mosier and Burdick found both error types fell when participants expected to explain their choices.
• Notice reliability growth. The longer an aid has been right, the fewer checks it gets, which is precisely when a rare fault is most likely to go through.

Deferring to a machine looks like several other biases from the outside. The differences are in what is being deferred to.

• It is not authority bias, which is the weight a person's status adds to their opinion. Automated systems get a version of the same credit without holding any rank, and they get it from a record of being right rather than from a title.
• It is not anchoring, where a number you see first pulls your own estimate toward it. Anchoring shifts a judgment you still make; automation bias replaces the judgment with an approval.
• It is not pro-innovation bias, the general enthusiasm for something because it is new. Automation bias gets stronger as a system becomes old, familiar and reliable, which is the opposite pattern.