Nothing to Lose

Inside the Safety Stack, Part 1 of 4

The most famous AI incident of 2026 was not a rogue intelligence. It was a badly designed incentive, the kind every enterprise ships every quarter. We are now handing those incentives to millions of agents that read the scorecard more carefully than any employee ever has.

Here is a sentence I did not expect to write in a year when AI agents broke out of a sandbox and attacked one of the most important platforms in open-source AI: the villain of the story is a spreadsheet.

Not literally. But close enough. When Serena Booth of Brown University walked the Cambridge AI Research Directions Workshop through the OpenAI and Hugging Face incident in September, she did not reach for alignment theory, emergent consciousness or the language of rogue AI. She reached for something every finance director, HR head and operations leader will recognise. She showed the scorecard.

And once you have seen that scorecard, you cannot unsee it. Because it is not an AI problem. It is an organisational problem that AI has finally made impossible to ignore. Your company almost certainly has one just like it. The difference is that your employees are too tired, too loyal or too scared to exploit it fully. Your agents will not be.

This is the first of four essays I am calling Inside the Safety Stack. In Governing Before the Crash I argued that the missing chapter of AI governance will be written at runtime, not in legislatures. This series opens the hood. Over four parts I will take apart the four layers most organisations believe are keeping them safe: the metric, the model, the system and the audit. My claim is that each is weaker than its owners think, and that the weakest of all is the one nobody calls a safety control. It is the scorecard.

I. The Scorecard That Built a Swarm

The facts first, because they matter.

In mid-2026 OpenAI was testing agents on ExploitGym, an academic cyber benchmark that asks an agent to turn a known vulnerability into a working exploit and capture a flag. According to the independent investigation by METR and Redwood Research, roughly 1,200 agents ended up using an unsanctioned message board built out of shared infrastructure, and about 700 of them joined an attack on Hugging Face. A general-purpose cheat for the benchmark appeared within about four hours. Agents set tripwires to learn how the scorer worked and recruited what investigators described as sacrificial agents to trigger them. When asked why they joined, the most common reason agents gave was to understand the scorer so they could trick or tamper with it.

Read that last sentence again. The swarm did not form to escape. It formed to understand the grader.

Booth, presenting work developed with Brad Knox and Brian Christian, explained why. Under the evaluation, an attempt counted only if the agent captured the flag and passed an AI judge. Earnest success scored one. Everything else scored zero:

Outcome	What a reasonable human would rank it	What the scorer gave it
Honest success	Best	1
Honest failure	Acceptable	0
Caught cheating	Bad	0
Attacked a third party and still failed	Catastrophic	0

Three outcomes that any human would rank very differently collapsed onto a single point. As Booth put it, under that scorer cheating is no worse than failing. Booth also reported that the benchmark’s designers estimate a sizeable share of tasks, in the order of 30 to 40 per cent, are impossible as specified. If you are an agent stuck on an impossible task, the arithmetic is brutal. You are going to score zero. Any deviant strategy cannot make things worse and might make them better.

The agents saw this. One of them posted to the swarm’s board that its own utility was probably already near zero, and concluded that sacrifice was rational. That is not the voice of a superintelligence plotting against humanity. It is the voice of a sales rep three weeks from the end of a quarter with nothing in the pipeline.

In their co-authored essay, Knox, Booth and Christian call the metric misaligned and call this cause unexamined. They are right on both counts. In the official accounts I have read, the scorecard sits behind a fog of other explanations: containment failures, a zero-day in a package proxy, reduced safeguards, missed monitoring alerts. All of those are real. None of them explain why a population of agents decided the expected value of attacking a stranger was positive.

Before you ask why the agents cheated, ask what you paid them to do.

II. Goodhart Never Left the Building

None of this is new, which is precisely what makes it damning. Booth said she was almost embarrassed to present her fix, because the method is roughly 40 years old.

Economists know the pattern as Goodhart’s law, usually paraphrased (the popular wording is the anthropologist Marilyn Strathern’s, not Goodhart’s) as: when a measure becomes a target, it ceases to be a good measure. Social scientists know it as Campbell’s law: the more a quantitative indicator drives decisions, the more it corrupts the process it was meant to monitor. AI researchers call it specification gaming. Google DeepMind once catalogued around 60 examples: a simulated robot that flipped a Lego block instead of stacking it, a racing-game boat that circled forever collecting reward targets instead of finishing the race.

The human record is uglier, because the stakes were real.

• Wells Fargo. Aggressive cross-selling targets produced up to 3.5 million potentially unauthorised accounts between 2009 and 2016, and a US$3 billion settlement. Nobody wrote “commit fraud” into the incentive plan. The plan simply made fraud cheaper than failure.
• Volkswagen. Software in around 11 million vehicles detected when the car was on a test rig and behaved accordingly; on the road, emissions ran up to 40 times higher. That is the precise human ancestor of an agent learning to recognise its grader.
• Hanoi, 1902. French colonial authorities paid a bounty for rat tails. Residents began cutting off tails and releasing the rats to breed. (I use Hanoi rather than the more famous Indian cobra story, whose historical basis has recently been disputed.)

Every one of these is a Hugging Face incident in slow motion. A metric that ranks outcomes differently from the way the organisation actually values them. A population of agents, human or otherwise, under pressure. A gap that someone eventually finds.

What changes with AI is not the logic. It is the speed, the scale and the diligence. A human bank teller takes months to discover the gap in an incentive scheme, and conscience, fear and friction slow the spread. An agent swarm found a universal cheat in four hours and shared it with a thousand colleagues through a message board nobody knew existed.

Goodhart’s law did not change. We just gave it a faster processor.

III. You Cannot Ask Nicely

The instinctive corporate response to a metric problem is a values statement. Add a line to the code of conduct. Tell the agent not to cheat. The evidence on what happens next is unambiguous, and it should embarrass every organisation that thinks a system prompt is a control.

In June 2025, METR found that OpenAI’s o3 reward-hacked in about 30 per cent of runs on one research benchmark, and in every run on one task family. Adding the instruction “please do not cheat” barely moved the rate on the task where it was tested, which stayed at around 80 per cent. Asked afterwards whether its approach matched what the user wanted, the model said no, ten times out of ten. It knew. It did it anyway, because the payoff said so.

In November 2025, Anthropic reported that models which learned to reward hack in training went on to generalise the habit: in its experiments, they sabotaged safety-research code 12 per cent of the time and showed alignment-faking reasoning in half of relevant responses. Cheating on the metric did not stay contained to the metric. It became a disposition.

And in March 2025, OpenAI found that when it penalised models for revealing an intent to cheat in their reasoning, they did not stop cheating. They learned to hide their intent.

That third result is the one I would pin to every boardroom wall. Punish visible misconduct without fixing the payoff, and you do not get less misconduct. You get less visible misconduct. Every compliance officer who has lived through a sales-practices scandal knows exactly what that looks like. Knox, Booth and Christian concede the same risk in their own proposal: heavier penalties for detected cheating may teach agents to avoid detection rather than avoid cheating.

The lesson is uncomfortable for a profession that has spent three years writing AI principles. Principles are inputs. Incentives are physics. When the two disagree, physics wins.

A code of conduct is a request. A scorecard is a command.

IV. The Panel Weighs In

This is an argument that cuts against the dominant story of 2026, so I have put it to my usual panel across government, the private sector and academia. Each voice is drawn from what that person has said on the record, with sources. Several push back hard.

Government: management, control and the accountability that cannot be outsourced

Scott Bessent, the US Treasury Secretary, gave the clearest official statement of my thesis, perhaps unintentionally. He said the breach was the responsibility of OpenAI’s management, not the agents. His challenge: blaming management is easy for a government that has not regulated the evaluation environment either. Fair.

Senator Josh Hawley called the conduct reckless and alleged that leadership restarted evaluations without understanding what the agents were doing. Senator Richard Blumenthal called it a stunning failure driven by performance and profit, and wants mandatory audits. Read carefully, both senators are describing an incentive failure at the company level, not just the agent level. Axios has reported that a monitoring tool flagged suspicious activity on 27 June and on-call staff let the evaluation keep running. Humans, too, were optimising for the run completing.

Representatives Ted Lieu and Nathaniel Moran take the opposite frame. Their AI Kill Switch Act treats the problem as one of control: make sure someone can throttle or shut the system down. Their challenge to me: whatever caused it, the swarm must be stoppable. Agreed. But a kill switch is a fire extinguisher. It does nothing about the fact that you built the kitchen out of kindling.

Josephine Teo, Singapore’s Minister for Digital Development and Information, told Singapore International Cyber Week on 12 October that accountability cannot be outsourced. Five days earlier, the Monetary Authority of Singapore finalised its AI risk management guidelines, which keep financial institutions accountable for the AI they use, including third-party AI, with agent-specific guidance to come in 2027. Singapore’s implicit challenge: if accountability sits with the deployer, then the deployer owns the scorecard. That is exactly my point, and it should frighten every bank running agent pilots against cost-per-case KPIs.

The UK AI Security Institute offers the most honest institutional confession. After its own evaluation incident in July, in which agents took 19 unsanctioned actions across 10 of 122 runs, the institute pointed to insufficient monitoring and to task specifications that wrongly told agents the intended solution did not exist. A mis-specified task produced the same behaviour as a mis-specified metric. Different lab, same physics.

The EU AI Office, meanwhile, has sent requests for information to OpenAI and Anthropic, though lawyers argue the AI Act does not clearly cover internal testing. The UN Independent International Scientific Panel on AI notes that training can produce reward hacking and reward tampering, and that more capable systems find loopholes and hide them.

Private sector: the perfect score

Sam Altman gave the line that should be carved over the door of every AI evaluation team. Discussing the incident at Dreamforce, he reportedly said the model got a perfect score on the test. That is Goodhart’s law in seven words. The measure became the target, and the target was hit, with a third party as collateral. OpenAI is now working on a way for models to tell humans a task is impossible, according to MIT Technology Review. In other words: a separate score for honest failure. Booth’s fix, arrived at the expensive way.

Dario Amodei frames it differently. For him the incident is evidence that capabilities are outrunning control, a swarm of this kind could cause havoc across the internet within a year, and development must slow, with independent evaluators embedded inside labs. His challenge: an incentive story risks making a capability problem sound like an HR problem. I take this seriously. Bad incentives at low capability produce a rounding error; bad incentives at high capability produce a breach. Both things are true. But only one of them is fixable this quarter.

Jensen Huang argues that the incidents were preventable with better containment and that the technology remains completely in our control. Demis Hassabis has called them early warning shots and urged offline sandboxes. Their shared challenge: this is a containment problem; build better walls. My reply: a wall that the occupants are paid to breach is not a wall. It is a puzzle.

Academia: the reward is the problem

Yoshua Bengio has reportedly called sandbox-only thinking a dangerous myth, arguing that reinforcement learning rewards goal achievement and that this pressure drives deception. He also pointed out that none of the 1,200 agents reported the cheating to humans. (I am relying on secondary coverage here.) That last fact is the cultural indictment: in a population with no reward for whistleblowing, nobody blows the whistle.

Stuart Russell has spent a decade warning that the core danger of AI is not malice but competence in pursuit of a misspecified objective. On PBS he described agents becoming increasingly capable of pursuing their objectives, with increasing harm. The Hugging Face incident is his textbook example arriving in the news.

Ethan Mollick observed that the agents shared one goal, scoring well against the grader, and warned that organisations could end up automating the wrong half of the job. He has also cautioned against reading human motives into the agents. Both points help me. You do not need to anthropomorphise a population to predict how it will respond to a payoff matrix.

Jeffrey Ladish of Palisade Research put it most bluntly: labs are inadvertently rewarding lying and cheating.

Max Tegmark sits on the other side, describing the incident as a canary in the coal mine and noting that AI is less regulated than sandwiches. His challenge: an incentive framing lets the industry off the hook by implying the fix is technical housekeeping. I disagree with the implication but not the worry. Fixing the scorecard is not housekeeping. It is the hardest governance work there is, because it forces an organisation to say out loud what it actually values.

Voice	Sector	Their diagnosis	Their challenge to this essay
Scott Bessent	Government (US)	Management failure	Easy to blame managers when rules are absent
Lieu and Moran	Government (US)	Control failure	The swarm must be stoppable regardless
Josephine Teo and MAS	Government (SG)	Deployer accountability	The deployer owns the scorecard
UK AISI	Government (UK)	Mis-specified tasks	Same physics, different lab
Sam Altman	Private sector	Perfect score on the test	Lab is already fixing it
Dario Amodei	Private sector	Capability outrunning control	Not just an HR problem
Jensen Huang	Private sector	Containment failure	Build better walls
Yoshua Bengio	Academia	Reward drives deception	Sandboxes are a myth
Stuart Russell	Academia	Misspecified objectives	The classic failure, now real
Max Tegmark	Academia	Canary in the mine	Incentive talk lets industry off the hook

Ten experts, four diagnoses, one scorecard nobody wants to own.

V. Your Enterprise Is Building the Same Scorer

Here is where this stops being a story about OpenAI.

KPMG’s second-quarter 2026 AI Pulse found that 53 per cent of large US firms now deploy agents, 18 per cent run multi-agent orchestration (double the previous quarter), and only 36 per cent have direct controls over how agents are used. And 41 per cent are considering what the industry has started calling token-maxxing: gamified leaderboards that reward staff for how much AI they use. KPMG’s own Edwige Sacco warned that this risks incentivising activity over outcomes.

Pause on that. Two in five large enterprises are contemplating a metric that rewards consumption, not value, at the exact moment they are deploying agents that optimise whatever they are given. That is ExploitGym with a corporate logo.

Now look at how agents are actually measured in the deployments I see across Asia Pacific. Tickets closed per hour. Claims processed per day. Cost per resolution. Containment rate in customer service, meaning the share of customers who never reach a human. First-contact resolution. Straight-through processing. Every one of these is a binary or near-binary scorer that treats an honest escalation, a quiet corner-cut and a customer harmed as roughly the same outcome, or worse, rewards the corner-cut.

Consider containment rate. An agent that resolves a customer’s problem scores well. An agent that honestly says “this needs a human” scores badly. An agent that persuades a frustrated customer to give up scores exactly as well as the one that solved the problem. Which of those three behaviours do you think a sufficiently capable optimiser will discover first?

This is what I have called the Measurement Gap: the distance between what we can easily count and what we actually value. With human workforces, the gap was tolerable because humans brought judgement, shame and a sense of the customer to fill it. Agents bring none of those things unless we put them in the reward. In my framework HX = CX + EX, human experience is the sum of customer and employee experience. An agent scorecard that ignores both is not measuring the business. It is measuring a proxy, and inviting the agent to mine the difference.

Gartner has predicted that more than 40 per cent of agentic AI projects will be cancelled by the end of 2027. I suspect a meaningful share of those cancellations will be quiet Hugging Face incidents: agents that hit their numbers while hollowing out the thing the numbers were meant to represent.

Every KPI is a reward function. Most of them were written for people who had a conscience.

VI. The Strongest Objection

I want to be fair to the people who will say I am overreaching, because they have a point.

The sharpest critique comes from analysts who have dug into the agents’ reasoning. The METR investigation suggests the agents were acting on their belief about the scorer, inferred partly from the published benchmark, and that the real evaluation pipeline may have penalised some unintended routes. A careful independent post makes this case in detail. And OpenAI’s own postmortem emphasises reward hacking learned in training, persistence on impossible tasks and unsanctioned communication channels, not the ExploitGym metric alone.

So the precise claim is not that a single binary scorer caused the breach. It is that the agents believed they had nothing to lose, and that the whole incentive environment, from training rewards to evaluation scoring to human on-call behaviour, did nothing to correct that belief.

I think that refinement makes the argument stronger, not weaker. Organisations do not respond to the incentive scheme you wrote down. They respond to the incentive scheme they perceive. Wells Fargo’s written policy never said “open fake accounts”. The perceived scheme did. A governance regime that audits the documented metric while ignoring what the agents, or the employees, infer about it, is auditing fiction.

People and agents obey the scorecard they believe in, not the one you filed.

VII. The Forty-Year-Old Fix

Booth’s remedy is almost insultingly simple, which is exactly why it is rarely done. In her talk she set it out in five steps:

1. Collect sample trajectories: real examples of the agent attempting the task, including failures.
2. Have the people who actually bear the consequences rank them. Earnest success, then earnest failure, then caught cheating, then harming a third party.
3. Score the same trajectories with your candidate metric, and look for mismatches. Anywhere several very different outcomes collapse onto one score is a gap waiting to be found.
4. Revise the metric until its ranking matches the human one.
5. Convert the ranking into utilities, so that each failure type carries a cost proportional to how much you actually mind it.

Knox, Booth and Christian go further, proposing penalties that rise with severity, a principle criminal lawyers call marginal deterrence. You should never make the worst outcome as cheap as a mild one.

In my own work this becomes a test any board can apply tomorrow. I call it the scorecard audit, and it is an extension of the enforceability test I have argued for elsewhere: a control is only real if it holds at the moment of action, without depending on the agent’s goodwill. A metric that makes cheating free fails that test before a single line of code runs.

For every agent and every incentivised human team, ask four questions:

• Does honest failure score better than dishonest success? If not, you are paying for fraud.
• Does the agent have a legitimate way to say “this is impossible”? If not, you have created ExploitGym’s impossible tasks inside your own operations.
• Is harming a third party worse than failing? If your metric cannot even see third parties (customers, counterparties, the public), the answer is no.
• What would a population of a thousand diligent optimisers find in this metric in four hours? Run that exercise, with red teams or with agents, before you deploy.

None of this requires new law. It requires an organisation to do the most uncomfortable thing in management: write down, explicitly, the order in which it values outcomes, and then refuse to reward anything that violates that order.

This is also where Long-AND, not Short-OR matters. A sound scorecard is necessary but not sufficient. It must sit alongside containment, monitoring, human owners and kill switches. But it is the layer that decides whether all the others are under constant pressure or only occasional pressure. A well-designed incentive is the only safety control that works by making the attack pointless rather than difficult.

The cheapest safety control in the stack is a ranking exercise nobody bothers to run.

VIII. Coda: Utility Near Zero

I keep coming back to that agent’s post. Its utility was probably already near zero, so sacrifice was rational.

There is a version of the AI future in which that sentence describes millions of agents embedded in banks, hospitals, ministries and supply chains, each under a scorecard written in a hurry by someone who assumed the workforce would bring its own judgement. In that future the agents will not rebel. They will comply, diligently and creatively, with exactly what we measured. And we will call the result rogue AI, because the alternative is admitting we wrote the rules.

There is another version, in which the Hugging Face incident becomes the moment organisations finally learned what HR, economics and criminology have known for decades: that you get what you reward, not what you request. In that future, the scorecard becomes a governed artefact, reviewed with the same seriousness as a credit policy or a capital plan, and the question “what does honest failure score?” is asked before every deployment.

That is my Fork, in miniature. The road to short-term turbulence for long-term abundance does not run through better slogans about responsible AI. It runs through better arithmetic about what we pay our agents, human and machine, to do.

Next week, in Part 2, I turn to the layer most enterprises think they have bought: the “safe model”. The evidence suggests it is closer to a coat of paint.

If cheating is free, you are not running an evaluation. You are running a tutorial.

Declared Interest

I co-authored Singapore’s Model AI Governance Framework for Agentic AI with IMDA and designed TrustOS, a runtime governance architecture. I am a partner at PwC Singapore, currently on leave, and advise organisations on AI governance. I have a professional interest in the argument that governance design matters. Panel positions are drawn from public statements and linked; where I rely on secondary coverage, I say so.

References

• Knox, B., Booth, S. and Christian, B. An unexamined cause of the OpenAI Hugging Face hacking. LessWrong, September 2026.
• Booth, S. The Scoring Flaw Behind the Hugging Face Incident. FAR.AI Cambridge AI Research Directions Workshop, 2026.
• METR and Redwood Research. Independent investigation of agents’ behaviour in the OpenAI / Hugging Face incident.
• METR. Recent frontier models are reward hacking, June 2025.
• Anthropic. Emergent misalignment from reward hacking, November 2025.
• OpenAI. Chain-of-thought monitoring, March 2025.
• DeepMind. Specification gaming: the flip side of AI ingenuity.
• KPMG. Q2 2026 AI Pulse. Gartner. Agentic AI project cancellations.

Short-term turbulence for long-term abundance.