Automation Bias

A driver follows the GPS onto a closed road despite the barrier in front of them. A loan officer approves every application the scoring tool flags green without reading the file. A developer merges AI-written code because the tests passed and the explanation sounded confident. Each is automation bias: the system’s answer gets more trust than the evidence supports, and errors slip past the person who was meant to catch them.

Research on clinical decision support and aviation found the pattern long before chatbots. It includes errors of commission, following a wrong suggestion, and errors of omission, missing a problem because the system didn’t flag it. It tends to get worse when people are busy, when the system is usually right, and when its output looks authoritative.

That matters for any plan that relies on a person reviewing AI output. A human-in-the-loop step only adds safety if the reviewer has the time, information and authority to disagree. The EU AI Act names the risk directly: Article 14 requires that people overseeing high-risk AI systems be able to stay aware of automation bias and to override or ignore the system. Practical countermeasures include showing sources and confidence, asking reviewers to decide before seeing the AI’s suggestion in high-stakes cases, auditing a sample of approved outputs, and tracking how often reviewers actually change anything. Automation bias is a human-factors problem, distinct from bias in a model’s training data, such as selection bias.