"A human in the loop" is often proposed as a safeguard. It only works if the human can genuinely understand and change the outcome.
When Oversight Is Needed
The more consequential and irreversible a decision, the more human involvement it needs: hiring, credit, benefits, medical and legal decisions, content that could harm people, and actions agents take in the world.
Models of Oversight
- Human in the loop: a person approves each decision.
- Human on the loop: the system acts, and people monitor and can intervene.
- Human in command: people decide whether and how the system is used at all.
Avoiding Automation Bias
People tend to accept automated recommendations, especially under time pressure. To make oversight meaningful:
- give reviewers the information and time to judge;
- show why the system made its recommendation and how confident it is;
- make disagreeing easy, and don't penalise it;
- occasionally include cases where the right answer is known, to check reviewers are engaged;
- track override rates — near-zero overrides may signal rubber-stamping.
Appeals and Contestability
People affected by decisions should be able to ask for an explanation and a human review, with a clear, accessible process.
Measure the System as a Whole
Evaluate the combined human-plus-AI outcome, not just the model. Sometimes a model helps most by triaging cases rather than recommending decisions.