Presence Is Not Power. The Machine Is Honest. The Human in the Loop Is Not.
In The Fine Print I left a line hanging: enforceability and interveneability are two different properties, and the charts everyone shares measure only the first. This is the essay about the second. It also gathers a thread that has run through Faster, Busier, Worse, The Frozen Workforce, The Five Per Cent Problem, The Hybrid Boardroom and The Job Survives, The Worker Doesn’t. Each of those essays circled the same uncomfortable figure: the human who signs off.
A well-designed governance card did the rounds on my feed this week. It carried a single sentence I wish I had written: a human is in the loop, but can they actually intervene? Beneath it sat four conditions feeding a single human node: information, capability, authority and capacity. The caption said that human oversight requires more than human presence.
It is right, and it stops one step short. The four conditions are necessary but not sufficient, and the picture shows one human standing between the inputs and a single AI decision. That picture fitted the scoring models of five years ago. It does not fit a world of agents taking thousands of actions across tools and sub-agents before a human sees any of them.
So I want to push the card’s question to its conclusion, which is uncomfortable. We have built an oversight economy on human presence, because presence is easy to specify, easy to audit and cheap to supply: a name in a workflow, a signature on a screen, a line in a risk register. But presence is also the cheapest thing in the world to counterfeit. My argument is that wherever a human cannot genuinely intervene, we should stop pretending they can. An honest autonomous system with a governed envelope is safer than a dishonest human-in-the-loop.
The enforceability test asks whether a rule can bind anyone. Its twin, which I will call the interveneability test, asks something harder. When the rule says a human will catch the failure, can that human actually stop the machine, in time, without penalty, and can anyone prove they did? A framework can pass the first test and fail the second completely. Most do.
I. Four prepositions and a polite fiction
The vocabulary of oversight has multiplied. A human can be in the loop, on the loop, over the loop or out of it, and various frameworks add before, behind, around and above. The terms are used so loosely that a workflow can change its real structure three times while its label stays the same.
| Position | What the human actually does | Its characteristic failure | Its honest name |
|---|---|---|---|
| In the loop | Approves before the action takes effect | Volume turns approval into a signature | A checkpoint, if resourced |
| On the loop | Monitors a running system and can halt it | Attention fades because the system is usually right | A supervisor, if alerted well |
| Over the loop | Sets objectives, boundaries and thresholds, and audits | The envelope exists on paper but not in code | Loop governance, if enforced |
| Out of the loop | No runtime role at all | Disguised as one of the three above | Autonomy, which is fine if declared |
The sharpest recent clarification comes from philosophy rather than engineering. One research team argues that the difference between in and on is causal, not spatial. In-the-loop involvement is constitutive: the human contribution is necessary for the output to exist. On-the-loop involvement is corrective: the human sits outside the main causal chain but can prevent or modify what comes out. Their decisive point is that statutory oversight demands more than a corrective position. It demands genuine preparedness and capacity to intervene. That is the card’s question, stated with precision.
A second useful ladder defines five rising levels of agent autonomy by the role the user plays: operator, collaborator, consultant, approver and observer. Its most important move is to treat autonomy as a deliberate design choice, separate from what the system is capable of. At least one financial regulator has already borrowed it. Autonomy is something you set, not something that happens to you.
The polite fiction is what happens in between. Practitioners auditing real deployments have found exception-only escalation, spot-check sampling and post-hoc confirmation all shipping under the “human in the loop” label, although only approval before every action meets the definition. The control description never changes. The structure underneath it changes completely.
This matters more than it looks. In The Five Per Cent Problem I worked through the six controls that the year’s largest applied AI index says agents need before they can be trusted with autonomy, and found that only 5% of companies have all six. Human in the loop is second on that list. It is also the only one of the six that can be satisfied, on paper, by a job title. Memory scoping, interface allow-lists, shut-off drills and immutable logs all leave technical evidence. A human in the loop leaves a name. I suspect it is the control most often ticked and least often real, which means even that sobering 5% may flatter us.
II. The 1.2-second signature
If you want one number to carry this argument, here it is. Investigative reporters examined how a large health insurer handled claim denials that, under many state rules, required a physician’s review. According to company documents, its doctors denied more than 300,000 payment requests over two months through a bulk method, averaging 1.2 seconds per case. The insurer disputes the characterisation and the litigation continues. But the architecture is the point, not the verdict. The human was present. The signature was real. The intervention was impossible.
I have called this theatre with a signature in The Frozen Workforce and a latency layer with a signature in The Hybrid Boardroom. What I had not fully named is who absorbs the consequences. A policy scholar who surveyed forty-one policies mandating human oversight of public-sector algorithms found that the evidence does not support the assumption that people can perform the oversight those policies require. Worse, the mandates create a false sense of security and let vendors and agencies sidestep accountability for algorithmic harm. Ethnographers have a name for the human left holding the outcome: the moral crumple zone. Like the part of a car built to deform on impact, the nearest human is designed to absorb the blame.
Faster, Busier, Worse adds a twist I now think is essential. In a controlled experiment, framing the same system as an “AI employee” rather than an “AI tool” cut personal accountability by nine points, raised the blame attributed to the AI by eight, and let more errors slip past managers. So the crumple zone has two exits. Accountability leaks downward onto the frontline human when the organisation needs someone to blame, and sideways onto the friendly-named agent when the frontline human needs someone to blame. In neither case does it land on the people who designed the loop.
A signature is not a safeguard. It is a receipt showing that a human was available to blame.
III. Why the loop amplifies the fiction
The central claim of Faster, Busier, Worse was that AI does not fix an organisation. It amplifies it: a mirror with a megaphone attached. Oversight is where that claim bites hardest, because oversight is the one part of the loop that does not scale with the machine.
Execution now scales. Review does not. When a task becomes a thousand times cheaper, organisations rarely bank the saving; they do the task a thousand times more. Every one of those additional actions arrives at the same human checkpoint, which has the same number of hours in its day. The manager, as I wrote then, has become the narrowest point in the hourglass.
Now add Goodhart. If the organisation measures its reviewers on approvals cleared, the measure becomes the target and the review becomes a gesture. Put the same agent into two firms. In the one that rewards care, the checkpoint stays a checkpoint. In the one that rewards throughput, it becomes a signature factory within a quarter. Nothing about the model differs. Oversight quality is a property of the operating model, not of the AI. An organisation that punishes slowness will amplify its rubber stamps, and its dashboards will report that governance is working perfectly.
IV. The Intervention Test
Credit where it is due: the card’s four conditions are well chosen, and they sit close to where regulation and fieldwork have both landed. Europe’s AI law requires that the people assigned to oversee high-risk systems have the competence, training and authority to do so, and that the system can be stopped through a procedure that brings it to a halt in a safe state. Fieldwork in a laboratory running AI agents produced a near-identical list: knowledge of the system’s capabilities and limits, sufficient observation of its actions, meaningful control over its behaviour, and timely intervention when it fails.
The Frozen Workforce proposed one way to make the second condition real: license the human to the tier of the loop, with seeded-fault drills and expiring credentials for anyone approving consequential agent actions. I stand by it. But capability is only one of the conditions. A licensed pilot strapped into a cockpit with no time to react, no authority to divert and a career penalty for every go-around is still not flying the aircraft. Three further conditions decide whether the first four are real. Together the seven form what I call the Intervention Test.
| Condition | The question | Evidence it is real | How it quietly fails |
|---|---|---|---|
| Information | Does the human see what the system saw, and how sure it was? | Uncertainty and provenance shown at the moment of decision | Fluent output with no confidence signal |
| Capability | Could this person have reached the right answer unaided? | Domain expertise, a current licence for the tier, training in the system’s limits | A generalist approving specialist output |
| Authority | Can they say no without escalation? | Override rights written into the role and the workflow | The stop button needs an administrator |
| Capacity | Is there time and attention to think? | Reviews per hour within human limits | Queues set by throughput targets |
| Tempo | Can the human act at the speed of the loop? | Intervention points placed before irreversible actions | The agent has finished before the alert lands |
| Standing | Is the person safe when they override? | Overrides protected, reviewed without penalty, absent from productivity scores | The overrider is the one who missed the target |
| Trace | Is there evidence the oversight happened? | Override rates, time per decision, catch rates on seeded errors | A log that records a click, not a judgement |
Tempo matters because the loop no longer runs at human speed. Standing matters because incentives are stronger than policy: if the person who stops the line is the person who misses the quarter, the line will not be stopped. Trace matters because an oversight function that cannot show it worked is indistinguishable from one that did not exist.
Information, capability, authority and capacity describe a human who could intervene. Tempo, standing and trace describe a human who will.
There is a regulatory corollary. In The Frozen Workforce I found that nine governments demand an accountable human and none defines a certifiable competence standard for that human. Run the Intervention Test across the same nine and the gap widens. Most address authority in some form. Several gesture at capability. One asks organisations to audit whether their oversight actually works. None, as far as I can find, sets any standard for tempo or standing. We have regulated the cockpit, forgotten the pilot’s licence, and never asked whether the pilot is allowed to land. This is the tenth row missing from every governance chart: not whether a rule can fine you, but whether anyone covered by it can actually stop the machine.
V. Test the human the way you test the model
In The Job Survives, The Worker Doesn’t I described the irony at the centre of automation: the better the system, the harder the human’s residual job. The newest research makes it sharper for agents. Three researchers argue that current agent design does not merely fail to support oversight; the cognitive capacities oversight depends on are themselves degraded by extended use of AI. The thing we rely on erodes through the act of relying on it.
The implication follows, and almost nobody acts on it. If oversight is a control, it must be tested like a control. We red-team models and run evaluation suites on every release. We almost never test the reviewer.
The Frozen Workforce proposed seeded-fault drills as a way to certify approvers. I now think they belong in production, not only in the classroom. Seed known errors into live review queues, continuously and unannounced, and measure how many are caught. In The Five Per Cent Problem I suggested that an auditor examining human-in-the-loop controls should pull decision thresholds from code and monthly override and approval rates. Add two more numbers: the catch rate on seeded errors, and time per decision against the complexity of the decision. An override rate of zero over months is not proof of a perfect model. It is the signature of automation bias.
Then report the human’s performance next to the model’s, because the risk that reaches the customer is the product of the two: the model’s error rate multiplied by the human’s miss rate. This is the Measurement Gap relocated once again, from the economy to the office and now to the control itself. We measure the machine obsessively, the human not at all, and call the combination governed.
VI. Over the loop: what the generals are arguing about
The most serious argument about human oversight in the world right now is not happening in boardrooms. It is happening in international disarmament talks in Geneva, and enterprise leaders should be watching it closely.
The world’s largest military has a long-standing standard: weapon systems must allow commanders and operators to exercise appropriate levels of human judgement over the use of force. Its own legislative researchers note that this does not require manual control. It requires human involvement in decisions about how, when, where and why a system is employed, after which the system may operate autonomously. That is human-over-the-loop, written into doctrine.
Against it stands a broad coalition. At this year’s spring session, more than seventy states backed a binding standard of meaningful human control, or context-appropriate human judgement and control, across a weapon’s entire lifecycle, and the text goes to a review conference in November. Meanwhile that same military has been given ninety days to rewrite its autonomy rule. One defence writer’s proposal for the rewrite is the most useful idea on oversight I have read this year. Specify human authority, he argues, not merely the amount of human involvement: a mission-specific authority envelope that states what the machine may do and what remains a human decision. Where a decision is reserved for a human, the system should be technically unable to cross the line until authorisation has occurred.
Strip away the context and that is the enterprise problem exactly. Over-the-loop oversight is legitimate, and often the only kind that works at machine speed, on one condition: the envelope is enforced in the system rather than described in a document. In The Hybrid Boardroom I put it as a delegation boundary that is not instrumented at runtime being a slide, not a control. The generals are arriving at the same sentence by a far more serious road.
VII. The quiet innovation: audit the approval itself
Regulators are converging on a version of this, from different directions.
Europe’s law concentrates on the intervention itself. The stop must bring the system to a safe state, not merely freeze it mid-transaction. That is an engineering requirement dressed as a legal one: halting an agent halfway through a payment run is not safety, it is a second incident.
The national agentic AI framework I was privileged to co-author took a different and, I think, more far-reaching step. In its refreshed edition this year it pairs clear allocation of responsibility with measures to make oversight meaningful: human approval at significant checkpoints, auditing of whether those approvals are effective, and automated monitoring as a complement. The middle clause is the one that matters. It moves the object of assurance from the model to the human-machine pair. The approval stops being the end of the control and becomes the thing that is controlled. As The Frozen Workforce noted, it remains the only framework among the nine that names training as a control against automation bias and asks whether oversight actually works.
Put the two together and you have the shape of the next generation of oversight: bounded checkpoints where they matter, a real stop to a safe state, and evidence, regularly audited, that the humans at those checkpoints are catching what they are there to catch.
VIII. The case for honest autonomy
Now the uncomfortable part. Run the Intervention Test across a real enterprise and you will find large classes of decisions where the human cannot meet the seven conditions and never will. Volumes are too high, tempo too fast, the signal too deeply buried. The usual response is to keep the human there anyway, because the label reassures the auditor, the regulator and the board.
I think that is the most dangerous option available, and The Five Per Cent Problem explains why. Forty-two per cent of companies expect their agents to act autonomously by 2030; five per cent have the controls to govern them. I argued there that autonomy is a privilege an agent earns, not a feature a vendor ships. The corollary is the heart of this essay. Earned autonomy requires a real overseer to earn it against. If the human in the loop is a signature, every action they wave through is recorded as evidence of the agent’s reliability, although nobody checked it. Fake oversight does not merely fail to catch errors. It launders them into trust, and promotes the agent to wider autonomy on a forged record. The 37-point delegation gap is not only a shortage of controls. It is being filled, quietly, with counterfeit ones.
A disguised out-of-the-loop system is therefore worse than a declared one. The disguised version has no envelope, because everyone assumes the human is the envelope. It has no monitoring, because everyone assumes the human is the monitor. And it has a crumple zone instead of an accountable owner. The honest alternative is to triage every decision class into one of three positions, and to say plainly which is which.
| Where the Intervention Test lands | The honest position | What makes it safe |
|---|---|---|
| All seven conditions hold | Human in the loop | A resourced checkpoint, licensed to its tier, audited for catch rate |
| Conditions hold for the pattern, not for each instance | Human on or over the loop | Alerting, sampling, thresholds and a tested stop to a safe state |
| Conditions cannot be met | Declared autonomy within an envelope | Bounds enforced in code, runtime monitoring, a named owner, reversibility |
This is the Long-AND, not the Short-OR, applied to oversight. The choice is not between humans and machines. It is between oversight placed where it can work and oversight placed where it can only perform. It also echoes the policy scholar’s remedy of moving from individual to institutional oversight: the organisation answers for the envelope, and the frontline human is no longer asked to be a guarantee they cannot possibly be.
In Faster, Busier, Worse I offered a rule short enough to fit on a slide: agents get identities, permissions and audit trails, never accountability. Here is its other half. Humans get accountability only where they also get interveneability. Accountability without the power to intervene is not accountability. It is blame, scheduled in advance.
IX. My usual panel weighs in
I put the argument to my usual panel of AI voices: lab leaders, laureates, economists and practitioners whose public positions I track closely. These are my readings of how each would respond, not quotations.
The laureate who now argues for non-agentic AI would accept the triage and then attack its third row. If a decision class cannot be overseen, why is an agent taking it at all? In his framing, declared autonomy should have to justify itself case by case, and some decisions belong to systems that advise rather than act.
The pioneer who left industry to warn the world would put a clock on the whole framework. Every condition in the Intervention Test assumes the human remains the better judge in the moment. That holds today for most enterprise decisions. What is the plan for the year it stops holding?
The frontier lab chief would say the real work sits in the first condition. Showing a human the output is not showing them the reasoning. Until we can see inside these systems, oversight means judging behaviour without access to intent.
The digital economist would read the 1.2-second signature as the predictable result of designing for substitution and then attaching a human for compliance. He would add that Standing is partly a policy question: as long as incentives tilt firms towards replacing judgement rather than augmenting it, the person who slows the line will lose.
The human-centred AI pioneer would extend Standing beyond the firm. The customer denied a claim needs it too: the right to see the reasoning and to reach a human with the authority to change the outcome. It is the gap I flagged in The Five Per Cent Problem, where none of the six controls speaks for the person on the receiving end.
The agentic pragmatist would welcome the honesty of the third row. Most agent workflows, he would say, need good evaluations, logging and rollback far more than a human clicking approve, and pretending otherwise slows useful work without making anything safer.
The former technology chief turned geopolitical strategist would go straight to tempo and to Geneva. Competition at machine speed will push every serious actor towards over-the-loop control, and agents coordinating with other agents will push it further. The only question is whether the envelope is real when it happens.
The physicist turned safety campaigner would apply the enforceability test without mercy. Show him the line of code where the envelope is enforced, or admit that what you have is a policy and not a control.
The engineer turned writer on happiness would end on the overseer. We are asking people to spend their working days supervising machines that are almost always right, inside organisations that, as Faster, Busier, Worse showed, are already intensifying their work. What does that do to them, and what does it teach the machines about how much we value human judgement?
The panel divides along a familiar line. The pragmatists believe the loop can be well designed. The guardians believe it will eventually be outrun. Both are right, which is why the answer has two halves: design the loop rigorously now, and govern the envelope for the day it is outrun.
X. Seven moves for the next two quarters
- Reclassify every “human review” control honestly. For each one, record whether it is in, on, over or out of the loop in practice, not on paper. Expect the register to shrink.
- Run the Intervention Test on the survivors. Score each checkpoint against all seven conditions. Any control that fails on capacity or tempo is not a checkpoint. It is a crumple zone.
- Test the reviewers in production. Seed known errors into live queues, measure catch rates, and report the human’s performance next to the model’s. A control nobody has tested is a belief.
- Protect the person who says stop. Write override authority into roles, take overrides out of productivity metrics, and review them without penalty. Standing is built, not announced.
- Engineer a stop to a safe state. Define what safe means for each agent, whether rollback, quarantine or a compensating transaction, and test the stop under load before you need it.
- Write the authority envelope into the system. Reserved decisions should be technically impossible for the agent to take without authorisation. This is the runtime enforcement layer of TrustOS, and it is where policy becomes control.
- Stop promoting agents on forged records. Before any agent is granted wider autonomy, check that the approvals it accumulated came from checkpoints that pass the Intervention Test. Then bring override rates, seeded-error catch rates and envelope breaches to the board.
XI. Presence is not power
The phrase “human in the loop” has done a great deal of comforting work over the past decade. It has let regulators approve, boards relax and vendors sell, all on the strength of a person’s presence somewhere near the decision. The comfort was always borrowed, and the bill is now arriving, as agents act faster and in greater volume than any human queue can absorb.
This is The Fork again, running through every workflow. On the Mad Max road, the human stays in the diagram as a crumple zone: present, overwhelmed and blamed, while the agents they nominally supervise collect a reputation for reliability that nobody verified. On the Star Trek road, the human is placed where they can change the outcome, licensed for it, protected when they use it and measured to prove they can, and everywhere else an enforced envelope does the work honestly.
The card asked whether the human can actually intervene. The question I would leave with every leader is harder. If the honest answer is no, are you prepared to say so, and govern accordingly?
Short-term turbulence for long-term abundance.
This essay extends arguments developed across Genesis, including the enforceability test and the missing tenth row from The Fine Print; the amplifier thesis and the accountability rule from Faster, Busier, Worse; the missing pilot’s licence and tiered human licensing from The Frozen Workforce; the delegation gap and earned autonomy from The Five Per Cent Problem; delegation drift from The Hybrid Boardroom; and the ironies of automation from The Job Survives, The Worker Doesn’t, alongside the Measurement Gap, Prompts to Loops to Loop Governance, TrustOS, the Long-AND not the Short-OR, and The Fork. Its external evidence draws on investigative reporting on claims-review practice, a comparative survey of public-sector oversight mandates, recent philosophical work on the causal structure of human involvement in AI systems, fieldwork on agent oversight in scientific settings, position research on agents and cognitive degradation, experimental research on how framing shifts accountability, Europe’s AI law, Singapore’s agentic AI governance framework, and current international negotiations on autonomy in weapon systems, spanning 2022 to 2026. Figures are reported as published by their respective sources.
Leave a comment