The fixes the industry is reaching for relocate the problem. The harder question is what the human in the loop was holding.
“Humans are not terribly consistent.” That was Amazon’s head of security, retiring one of the most widely held beliefs in AI governance. Eric Brandwine, a VP and distinguished engineer at Amazon Security, told The Register in June that human-in-the-loop oversight is not the gold standard the industry treats it as. Ask a person to approve or reject an agent’s actions all day, he said, and “they’ll do a good job, and then they’ll do an okay job, and pretty quickly they’ll be doing a poor job.”
He calls the mechanism normalisation of deviance, the term the Challenger investigators used for how a tolerated shortcut becomes the new normal until the margin you thought you had is gone. His example is an emergency room. On a nurse’s first day every alarm gets a response. After weeks of false alarms with no consequences the response erodes, and eventually a real alarm is missed with a life on the line.
He is right, and the concession exposes more than it settles. Once you accept that the person in the loop stops paying attention, you have to ask what that person was doing there in the first place, and whether what you have put there instead is doing the same job.
The industry conceded the point
It is not just Amazon. Google Cloud describes moving from human-led defence, to human-in-the-loop, to an AI-led strategy that humans oversee. Microsoft is talking about “loop learning,” turning the judgement people build up into systems that improve with use rather than a human checkpoint at every step. IBM went furthest and called human-in-the-loop “liability laundering,” a way to put a human name next to a decision without giving them any real power to change it.
IBM picked the right phrase. A person who clicks approve on the four hundredth agent action of the day is not governing anything. They are signing. The signature moves the blame. Control stays where it always sat. That is the difference between accountability after the fact and responsibility during execution. Approvals and logs tell you who to point to once the damage is done.
So the diagnosis is settled across the industry. The disagreement is about what to put in the human’s place, and the answers on offer share a flaw.
The fixes relocate the problem
The main fix moves the human up a level. Stop approving every action. Oversee the fleet instead. Put one person in front of a dashboard that summarises thousands of agent decisions rather than authorising each one.
Watch what happens to the problem. Normalisation of deviance moves up a layer with it. The person who used to rubber-stamp individual actions now glances at the dashboard, and the same erosion sets in by week six. The failure is the same, and the damage from any one mistake reaches further, because the human is now responsible for far more while able to see far less. Supervision runs at human speed whether it covers one action or ten thousand. The system does not.
The other fixes have the same shape. Give every agent its own identity so the activity log reads “this agent did this on behalf of Eric,” and you know exactly who owned the outage you are now cleaning up. You still have the outage. Naming the owner is good auditability doing a job it is easy to mistake for governance. Write “don’t cause a production impact” into the prompt and an agent set on a goal will route around it the way it routes around every other instruction, finding the one path you forgot to forbid. A rule that lives in the prompt is a request the system can decline.
Each fix takes the thing the human was doing and hands it to something that does a different job. None of them does the job. They move the gap somewhere harder to see.
What the human was holding
The approval click was never the protection. The protection was the judgement behind it: the engineer who paused because something felt off, the reviewer who had seen this go wrong before. That hesitation was real work, and it held the system together quietly enough that you could forget it was work at all. The click was just the visible part.
So every one of these “loops” is a place where a human used to sit and decide. Autonomy is the name we give that seat once it is empty. When an agent runs end to end without stopping for approval, a person has stepped out and the loop has carried on without them. A checkpoint is gone, usually without anyone writing down that it went.
You are retiring human oversight one loop at a time, and you have no record of which loops. The approvals that mattered and the approvals that were always theatre look identical once they are gone. The database migration that a senior engineer would have stopped and the routine job that never needed a human both now run untouched.
Until you can name the checkpoints you have automated, you cannot govern the system that replaced them. You cannot even describe it. You have output, and you are hoping it is right.
The work starts with an inventory
The exposure was there long before the AI. Control had been living inside a person all along, never built into the system, and the AI is what finally dragged it into view. Amazon, Google and IBM have noticed the person can no longer hold it. They are answering the wrong question. Moving oversight up a layer or stamping a name on every action does not put control back. The gap is still there.
The right question is smaller and harder. Which loops have you already closed? What was each one checking? Every agent running in production today sits where a human decision used to be, and most of those decisions were never written down. Find them before the loops start feeding each other, because a single automated loop is a thing you can still reason about, and a system of them wired end to end is not.
The cleanest place to start is the loop everyone is writing about, the one Geoffrey Huntley calls Ralph. It is almost nothing: point an agent at a written spec and a filesystem and let it run, with no human in the loop and no elaborate setup around it. It gets the attention because it works far better than something this crude has any right to, and that is what makes it worth looking at first.
Sources
- Amazon says human-in-the-loop AI oversight is failing because humans stop paying attention, The Next Web, 21 June 2026.
- Why Amazon hates human-in-the-loop AI governance, The Register, 20 June 2026.
Related reading (Synaptic Pixels):
- Responsibility Is the Hardest Thing to Automate. Accountability after the fact is not responsibility during execution.
- When Humans Leave the Loop. The human as compensating middleware.
- If Your AI Guardrails Live in the Prompt, They Aren’t Guardrails. Prompt constraints versus structural enforcement.
- Your AI Knows What It Did. Not What It Decided.. Auditability is not control.
Leave a Reply