A Human in the Loop Is Not Enough
“Human in the loop” has become one of the standard answers to AI risk.
The logic sounds reassuring.
AI recommends. A human reviews. The human approves or overrides.
Problem solved.
Except maybe not.
A recent Harvard Business Review article, AI Is Undermining Leaders’ Judgment. Here’s What to Do About It., highlights a troubling possibility: putting a human after an AI recommendation does not necessarily preserve independent human judgment.
Sometimes it may do the opposite.
In one field experiment cited by the authors, 228 experienced evaluators reviewed submissions to an MIT innovation challenge. When AI recommended rejecting submissions that an expert panel had approved, human evaluators often followed the AI anyway.
More surprisingly, they were even less likely to override an incorrect AI recommendation when the system provided a narrative explanation.
The explanation made the AI more persuasive. It did not make the decision better.
That should make anyone designing human-AI decision systems uncomfortable.
Human review is not human oversight
The usual governance architecture looks something like this:
Technically, a person remains involved.
But what if that person routinely accepts the recommendation? What if the AI's explanation is so coherent that challenging it begins to feel unreasonable? What if reviewers gradually stop forming an independent view at all?
The human is still in the loop. Human judgment may not be.
That gives us an important distinction:
Human review ≠ human oversight
The difference is not whether a person clicked “approve.” The difference is whether that person exercised independent judgment.
More intelligence can create more deference
The HBR authors describe what they call the original judgment paradox.
Organizations have spent years building better decision-support systems. Yet as those systems become more capable, leaders may increasingly defer to them rather than originate their own interpretations.
The authors define original judgment partly through two capacities: noticing signals beyond the obvious and forming an independent interpretation rather than simply adopting the dominant model or group view.
They describe a possible erosion process:
That last stage is especially interesting.
Imagine an organization where executives increasingly rely on the same models, similar prompts, similar summaries, similar recommendations, and similar explanations.
The organization may have more intelligence than ever. It may also have fewer genuinely independent interpretations.
That is not obviously safer.
Explanation is not understanding
AI explainability is often treated as part of the solution. Make the model explain itself and people can judge whether the answer makes sense.
Reasonable.
But the experiment cited by HBR complicates that assumption. A good explanation can increase understanding. It can also increase trust. And those are not necessarily the same thing.
A plausible narrative may make an incorrect recommendation feel more legitimate.
So perhaps another distinction is needed:
Explanation ≠ understanding
The question should not simply be: Can the AI explain its recommendation?
It should also be: Does the explanation help the human reason independently—or make disagreement psychologically harder?
That is a much tougher design problem.
What if AI's job were to challenge the decision?
This suggests a different role for AI inside consequential decision systems.
Instead of asking another model: Do you agree with this recommendation?
Ask: What is the strongest reason this recommendation could be wrong?
That changes the architecture. A decision process might look more like:
The AI does not receive more authority. It receives a different responsibility.
Challenge the assumptions. Find contradictory evidence. Surface the edge case. Identify the threshold that could flip the decision. Offer the strongest alternative interpretation.
In other words:
Do not use AI merely to produce better answers. Use AI to produce better challenge.
Disagreement may be useful data
The HBR authors recommend structured curiosity and intentional dissent as ways to preserve judgment.
Their structured-curiosity framework deliberately collects unusual signals, compares disparate evidence, delays premature closure, and triggers alternative interpretations.
Their approach to intentional dissent similarly emphasizes independent perspectives and structured challenge rather than hoping someone feels courageous enough to object during a high-stakes meeting.
That has an intriguing implication for AI systems. Perhaps disagreement should not be treated as noise to eliminate. Perhaps it should be measured.
Imagine tracking:
- the system recommendation
- the AI counterargument
- the human's initial view
- the final decision
- whether an override occurred
- why it occurred
- what eventually happened
Now the organization can learn something much more interesting than whether humans approved the AI. It can learn:
- Where do humans catch errors?
- Where do humans introduce errors?
- Where does the system repeatedly miss context?
- Where does AI-generated dissent reveal something useful?
- Where do everyone agree—and still get the outcome wrong?
That is not merely human-in-the-loop. That is decision-quality feedback.
The real goal is judgment integrity
AI governance often concentrates on the model:
- Is it accurate?
- Is it secure?
- Is it explainable?
- Is it compliant?
Those questions matter. But consequential AI systems also need another layer: Is the decision process preserving independent judgment?
A human reviewer should not become ceremonial governance for an increasingly persuasive machine.
The goal is not to make humans override AI more often. Nor is it to make AI defer to humans automatically.
The goal is to design a process in which both can be challenged.
Because the future risk may not simply be that AI gets decisions wrong. It may be that AI becomes persuasive enough that humans stop noticing when it does.
A human in the loop is a start.
Independent human judgment in the loop is the real objective.