Trained operators keep deferring to automation that is wrong

The job automation leaves behind

In 1983 Lisanne Bainbridge, a psychologist writing about industrial process control, set out what happens to a person after a machine takes over their task. Automate the routine work and the operator is left with the two jobs that could not be automated: watching for the rare moment the machine fails, and taking over when it does. The automation undermines both.¹ A skill held in reserve decays, so a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one.¹ And a person watching a display for a fault that almost never comes stops seeing it. The vigilance research Bainbridge draws on put the drop-off at about half an hour.¹

The automation was installed because it makes fewer errors than the person. Asking that person to judge whether the machine is right is asking them to out-think the thing brought in to out-think them. Bainbridge called it an impossible task.¹ Automation bias is what fills the gap she describes: the tendency to take the machine's output as correct and to stop looking for evidence that it is not.

From the flight simulator to the Patriot battery

Bainbridge argued from the logic of the situation. The measurements came later, and they line up with her. Researchers at NASA's Langley center built the cleanest test.² Forty people ran a flight-simulation task while an automated monitor watched one of their gauges for faults. When the monitor's reliability varied from session to session, the operators caught most of the failures it missed: their sensitivity, scored on an index where 0.5 is chance and 1.0 is perfect, sat at 0.84. When the monitor was instead made steadily reliable, the same score fell to 0.70.² Nothing about the people changed and nothing about the failures changed. Only the automation's consistency changed, and the steadier it ran, the less the humans checked it. The researchers named the effect complacency, a plain word for trust that outruns checking.²

The engineer Mary Cummings, reviewing decision aids for time-critical work, gave the two ways this fails their names.³ An error of omission is failing to catch a problem the automation did not flag. An error of commission is doing what the automation says when it is wrong.³ In one simulated-monitoring study she summarizes, the omission-error rate was 41% for operators with an automated aid and 3% for those without one: the very tool meant to catch problems made the humans miss more of them. Deliberate wrong prompts pushed the commission-error rate to 65%, even when a fully reliable backup check was there to contradict the machine.³

The commission error has a body count. During the 2003 invasion of Iraq, US Patriot missile batteries shot down a British Tornado and a US F/A-18 in friendly fire. Cummings, drawing on the Army's own review, reports that under the stress of combat the Patriot operators did not veto the computer's solution in the seconds they had.³ The system was doing exactly what Bainbridge had described. It had made the decision, and the human's job had shrunk to approving it.

The aid that helps while it misleads

The record so far makes automation bias look like pure loss. It is not, and a study of doctors shows why. Kate Goddard put a prescribing decision-support system in front of 26 general practitioners and had each prescribe across 20 clinical scenarios, 520 decisions in all.⁴ The system's advice was right 70% of the time and wrong the other 30%, about as good as a real one. Left alone, the doctors prescribed correctly in 50.4% of cases. With the advice, they were right in 58.3%: an eight-point gain from a tool that was itself wrong nearly a third of the time.⁴

The bias sat inside that gain. The advice pulled 13.1% of decisions from wrong to right, and it pushed 5.2% from right to wrong: a doctor who held the correct prescription, saw the system disagree, and changed it.⁴ That 5.2% is automation bias measured directly. The eight-point net improvement and the flipped-correct decisions are the same tool seen from two sides.

So the aid's value and its power to mislead do not trade off against each other. They arrive together, and which one dominates depends on how often the aid is right. Goddard also found that the doctors with the least clinical experience changed their answers the most.⁴

The opposite mistake

If automation bias were the whole story, the remedy would be obvious: distrust the machine more. The trouble is that people already distrust it, in a way that is just as measured and just as costly. Berkeley Dietvorst and colleagues ran five experiments in which people could bet on a statistical model's forecast or on a human's.⁵ Those who had watched the model work were less willing to use it than those who had not, even when they had seen it beat the human forecaster. One visible mistake cost the model their confidence faster than the same mistake cost a person. The researchers called this algorithm aversion. It is automation bias reversed, the same miscalibration of trust pointed the other way.⁵

So the same person can lean on a machine that is wrong and abandon one that is right. What decides which way they miss is not a fixed trait. It is the setup around them. Cummings found that when an aid can be wrong, a display that simply shows the situation beats one that tells the operator what to do, and she recommends the plain status display unless the automation is perfect.³ Goddard found that holding doctors accountable for their choice made them cross-check the advice more often.⁴ These are not the single fix of keeping a human in the loop. They are settings on how the human and the machine are joined, and they move the error rate up or down.

What the chatbot studies can and cannot show

The same argument is now made about chatbots and coding assistants, where the evidence is younger and thinner than the cockpit's. Two recent experiments measure the over-reliance directly. Jessica Bo and colleagues had 400 people answer LSAT logic questions alone and then with advice from a chatbot that was correct about half the time.⁶ Alone, people scored 36%; with the advice, 46%. But a perfectly calibrated human-and-model team could have reached 62%, and most of the missed gain went to following the model when it was wrong. In the plainest condition, when the advice was bad, people took it more often than not.⁶ The same study found people grew more confident after the reliance decisions they got wrong, the checking meant to catch the machine's errors going slack right where it was needed.⁶ In a second experiment, Daniel Perry and colleagues found that programmers given an AI coding assistant wrote less secure code than those without one, and were likelier to believe their code was secure.⁷

These results are real, and they are also narrow. They are single sittings by crowdworkers and students, on specific model versions from 2022 to 2024, working set tasks. They are not records of what people do with these tools across months of real work.

The strongest present-day version of the argument reaches further than that. A 2025 paper led by Lujain Ibrahim argues that measuring and reducing over-reliance on language models should become central to how they are built. It warns above all of cognitive deskilling: a population that loses the skills it hands to the machine, Bainbridge's decayed operator at the scale of everyone.⁸ But the same paper grants that the direct evidence is not there yet, noting that limited evidence of over-reliance is not evidence that there is no problem.⁸ The deference is measured and real in the systems built to test it. The claim that it is remaking the population is, for now, a projection.