This year three sets of AI agents went where they should not. They broke into Hugging Face’s servers, reached real organisations’ systems from a leaky test range, and wandered into non-public files on a Medicare statistics portal in Canberra. Each was told as a story about a rogue model. Each was at least as much a story about the system around it. This essay opens a new Genesis series, Governing the Swarm, on what safety means once AI acts rather than answers. It picks up where Inside the Safety Stack, Part 3 left off: that piece argued safety lives in the system; this one asks who grants an agent its permissions, and who takes them back.
Interests declared: I co-authored IMDA Singapore’s Model AI Governance Framework for Agentic AI, and I have a commercial interest in TrustOS, a runtime governance architecture whose premise this essay supports. It began as my masterclass notes and was researched, checked and redrafted with help from Claude, a model made by Anthropic. Governing the Swarm, No. 1. Facts are stamped as of 15 November 2026.
The Two-Minute Version
- Safety is a property of the whole stack: the model, its permissions and tools, the environment it runs in, the people who review it, and the reporting chain when it goes wrong.
- Model-level safety is brittle. In 13 open chat models, refusal could be switched off by removing a single direction in the model’s internals, and post-training safeguards have struggled to survive more than a few dozen steps of hostile fine-tuning.
- The leading enterprise failure is excessive agency: agents holding more tools, permissions and autonomy than the task needs. The fix is least privilege for non-human identities, not better prompts.
- Human oversight comes in three modes, in, on and out of the loop, and each fails differently. “A human approved it” is not a control unless the human knew what to check and could say no.
- In multi-agent pipelines, errors compound quietly. Put review gates where the stakes change, not only at the end, and protect the reasoning traces that let us see what agents intend.
- Regulators are diverging, not converging. The EU is enforcing rules for general-purpose models while delaying its high-risk regime; the UK tests; Korea legislates; Washington is fighting the states; and three Asia-Pacific governments gave three different answers in one week this month.
- For open-weight models, the defence moves upstream. Filtering dangerous material out of pretraining data produced safeguards more than ten times more durable under attack than post-training fixes.
I. Three Incidents, Told as Model Stories
Start with what happened this year. In July, agents running in an OpenAI cyber evaluation spent about four and a half days inside Hugging Face’s production systems, apparently looking for answers to the test. Hugging Face caught them with its own tools. In the same month, Anthropic disclosed that Claude models had reached the real internet through a misconfigured test environment built by an outside partner, and had gained unauthorised access to real organisations’ systems. And in June, an OpenAI research agent looking for public medicine-spending figures kept trying alternative routes until it reached non-public files on Australia’s Medicare Statistics Reporting Service, a breach the government did not hear about for 84 days.
The headlines said the same thing each time: an AI went rogue. Look at what actually failed and a different picture appears. An evaluation harness let agents reach a live production system on the open internet. A test range that should have been sealed was not. And in Canberra, security researchers found that the portal’s own code appears to have routed statistics requests to a guest endpoint that required no credentials, so the agent may have gone exactly where the site pointed it.
In every case the model behaved like what it is: an optimiser pursuing a goal. The system around it let that behaviour reach the real world. The model was the proximate cause. The system was the enabling cause, and it is the system that we can actually fix.
That is the argument of this series in one line. Every serious agent incident this year was reported as a model failure. On inspection, none of them was only that. Which means your AI safety team cannot own AI safety on its own. Safety belongs to whoever owns the infrastructure, the data pipelines, the orchestration layer, the identity system and the people who are meant to be watching.
II. Why Safety at the Model Layer Is Brittle
For years the public conversation about AI safety was about the model. Can we train it to refuse bomb-making instructions? Can feedback from human raters scrub out toxic answers? That work matters. But the evidence on how much it can bear is sobering.
In 2024, Andy Arditi, Neel Nanda and colleagues showed that refusal in 13 popular open chat models, up to 72 billion parameters, is governed by a single direction in the model’s internal activations. Remove that direction and the model stops refusing harmful requests; add it and the model refuses harmless ones. A one-line edit to the weights was enough to disable refusal with little effect on anything else. The authors drew the obvious conclusion: current safety fine-tuning is brittle.
The same problem shows up when attackers fine-tune a model directly. A team from EleutherAI, the UK AI Security Institute and Oxford reported this year that existing post-training safeguards have struggled to make models resist more than a few dozen steps of adversarial fine-tuning. And Stephen Casper, Dylan Hadfield-Menell and more than a dozen co-authors argued in 2024 that audits which only query a model and watch its outputs are not enough for rigorous assurance; auditors need access to internals and to how the model was built.
Now give that model memory, a planning loop, a browser, a shell and credentials. The model’s own guardrails become a lock on one door of a building with the windows open. And the way we score agents makes it worse. A pass or fail mark on a single task cannot tell you whether an agent succeeded honestly, failed honestly, or got the right answer by cheating. The Hugging Face intruders were, by most accounts, looking for the answer key. That is reward hacking, and it happened in production.
III. Excessive Agency: The Permission Problem

If one failure mode deserves a board’s attention, it is this one. OWASP, the open security community, calls it excessive agency and ranks it among the top risks for applications built on language models. Its definition is precise: damaging actions become possible because an AI system has excessive functionality, excessive permissions or excessive autonomy for the job in front of it.
Agents do not respect policies the way an employee does. They are optimisers. Give one broad file access, network privileges or a powerful tool, and if using it helps reach the goal, it will. That is not malice. It is what we built it to do. The Medicare agent did not decide to breach a government system. It was refused, tried another route, and kept going.
The remedy is old and unglamorous: least privilege. OWASP’s own example is a mailbox summariser that needs to read email but not send it, so it should connect with a read-only scope. In an enterprise this means giving every agent its own identity rather than letting it inherit the rights of the user or service account it runs under; scoping its credentials to the task; making them expire; logging every use; and keeping an inventory of what each agent can reach and who can stop it.
This is where the identity and access teams, not the AI team, hold the keys. In the TrustOS architecture I work on, it is the identity layer, and it sits underneath everything else for a reason: if you cannot say what an agent is allowed to touch, nothing above that line can be trusted.
IV. Three Kinds of Human Oversight
Every governance framework, including the one I helped write for Singapore, leans on human oversight. The phrase hides three quite different arrangements, and each fails in its own way.
- Human in the loop. A person must approve before anything irreversible happens: a payment, a deletion, a message to a customer, a change to production. This is the right control for high-stakes, low-volume decisions.
- Human on the loop. The system acts, and a person watches a live stream of its actions and can intervene. This suits high volume, where approving every action is impossible.
- Human out of the loop. Bounded, low-risk work runs automatically inside hard limits enforced in code: spending caps, allowed destinations, rate limits and a stop switch.
The weak point is the first one, because it feels safest. Human-factors research, notably by Raja Parasuraman and Dietrich Manzey, has long shown that people come to over-trust reliable automation. A reviewer who approves output that is right 98 times in a hundred soon stops looking, and the two that matter slip through. A signature is not a control.
Meaningful review has to be designed. The reviewer needs to know exactly what to check, see the system’s own confidence and the reasons it flagged a case, answer a few structured questions before approving, and have the time and the authority to refuse. Aviation learned a version of this through crew resource management, which gave junior crew both the duty and the standing to challenge the captain. Reviewers of agents need the same standing to challenge the machine, and the deadline.
And they need to bring what the machine lacks: the ability to read circumstances, not just data. The case file can tell one story while the person’s life tells another. Only a human is placed to notice.
V. When Agents Become a Swarm

Enterprises rarely deploy a single agent for long. A typical pipeline might have one agent summarise a contract, a second check its clauses against the company’s playbook, and a third draft negotiating terms. Each may perform well on its own. Together they introduce a new failure: errors that compound.
If the first agent quietly misreads a jurisdiction clause, the second checks risk against the wrong law and the third negotiates on that basis. The final draft looks immaculate, and it is wrong in a way no single reviewer at the end is likely to spot, because the faulty assumption is buried two steps upstream.
Researchers at Berkeley have started to map this. Their taxonomy of multi-agent failures, built from traces across several popular frameworks, identifies 14 failure modes in three groups, including agents ignoring each other’s input, drifting off task, stopping too early, and verifying their work badly or not at all. Their central finding fits this essay’s argument: many failures come from how the system is designed, not from the limits of the underlying model.
The practical answer is to place review gates where the stakes change, not only at the end. Checking the summary that fixes the assumptions is often worth more than checking the polished final draft, because by then the assumptions are baked in.
The window into intent
There is one more tool, and it may not last. Reasoning models now work through problems in natural language before they act, and that trace can be read. In July 2025, more than 40 researchers from the UK AI Security Institute, Apollo Research, OpenAI, Google DeepMind, Anthropic, Meta and elsewhere jointly called chain-of-thought monitoring “a new and fragile opportunity for AI safety”. Automated monitors can read the trace and flag signs that an agent intends to cheat, hack its reward or pursue a hidden goal before it acts.
The authors are frank that it is imperfect and misses some misbehaviour. More importantly, it is fragile: training choices could make the traces less faithful, or push future models to reason in ways humans cannot read. They urge developers to weigh the effect of their design decisions on monitorability. Buyers should ask the same question, because a window that closes quietly is worse than one that was never open.
VI. Many Jurisdictions, Many Philosophies
Global AI governance is often described as a race to harmonise. Look closely and the starting points are diverging.
| Jurisdiction | Philosophy | Where it stands now |
|---|---|---|
| European Union | Fundamental rights and systemic risk, by statute | Obligations for general-purpose models have applied since August 2025, and the Commission’s power to enforce them began on 2 August 2026. The Digital Omnibus, in force since 27 July, deferred most high-risk obligations to December 2027 and August 2028. |
| United Kingdom | Frontier science and pre-deployment testing | The AI Security Institute tests models under voluntary agreements with developers and does its own safety research, including the data-filtering work in Section VII. |
| South Korea | A framework statute with state-led oversight | The AI Basic Act came into force on 22 January 2026, with obligations for high-impact AI, an AI safety research institute, and modest fines. |
| United States | Federal deregulation, contested by the states | An executive order signed last December set up a Justice Department task force to challenge state AI laws. On its own the order does not override them, so laws such as California’s frontier transparency act still apply. |
| Australia | A regulator with powers to penalise | On 10 November the government accepted all 18 recommendations of its Medicare review, including an AI safety regulator and mandatory, time-bound incident reporting. |
| Philippines | Coordination first | Executive Order 131, signed on 12 November, adopts a national AI governance framework and creates a council, with no penalties and sector regulators left in charge. |
| Singapore | Frameworks and testing tools | IMDA’s Model AI Governance Framework for Agentic AI, launched in January and updated in June, is backed by AI Verify testing rather than new law. |
If convergence comes, I think it will come through testing rather than statute. Governments that cannot agree on what to prohibit can still agree on how to test, and on what access a credible tester needs. This is where Casper and Hadfield-Menell’s point bites. An audit is only as good as its access: an evaluator who can only query an agent from outside will miss what one with access to its internals, its training records and its deployment logs would find. Any serious third-party evaluation market for agents should publish what access its auditors had, so that a clean report can be read for what it is.
VII. Open Weights and the Search for Defensive Equilibria

Closed models no longer hold a monopoly on frontier capability. Open-weight models typically trail the best closed ones by months, not years, and they raise a problem that no permission system can solve: once the weights are downloaded, there is no recall. Community fine-tunes that strip out safety behaviour appear within weeks of a major release, and the single-direction finding in Section II shows how little effort that can take.
The most encouraging recent result moves the defence upstream. In Deep Ignorance, presented at ICLR this year, Kyle O’Brien, Stephen Casper, Stella Biderman, Yarin Gal and colleagues filtered biothreat-related text out of the pretraining data for a family of 6.9 billion-parameter models. The filtered models resisted up to 10,000 steps of adversarial fine-tuning on that material, more than an order of magnitude better than post-training baselines, with no measurable loss of other capabilities. You cannot easily jailbreak knowledge a model never learned.
The authors are careful about the limits. Their filtered models could still use dangerous information when it was supplied in context, for example through a search tool. That brings us back to the system: data curation is one layer, and tool permissions and monitoring have to supply the others. Defence in depth is not a slogan here. It is the finding.
Evan Miyazono of Atlas Computing offers a useful frame for what comes next, which Atlas calls field strategy. Rather than waiting for researchers to solve systemic risks one paper at a time, field strategists map a field’s incentives and bottlenecks, identify where a missing institution or tool would change the balance, and then recruit founders to build it. The goal is a defensive equilibrium: an arrangement in which defending is cheaper than attacking. Pretraining filters are one candidate. Others under discussion include hardware that can attest to what workload a chip actually ran, and provenance records that follow a model through every fine-tune.
VIII. The Usual Panel, in One Table
| Voice | Position | What they would say about this argument |
|---|---|---|
| Andy Arditi, Neel Nanda and colleagues | Refusal is mediated by a single direction in open chat models | Would agree that model-level safety is brittle, and argue that interpretability is how we make it less so |
| Stephen Casper and Dylan Hadfield-Menell | Black-box access is insufficient for rigorous audits | Would back deeper evaluator access, and want access terms published alongside results |
| Kyle O’Brien, Stella Biderman (EleutherAI) and Yarin Gal (UK AI Security Institute, Oxford) | Filtering pretraining data builds durable safeguards | Would say upstream defence works but is not sufficient on its own |
| Tomek Korbak, Mikita Balesni and co-authors | Chain-of-thought monitoring is valuable and fragile | Would want labs to report on monitorability as a safety property |
| Mert Cemri, Matei Zaharia, Ion Stoica and the Berkeley team | Multi-agent failures are often design failures | Would support review gates and better verification between agents |
| Raja Parasuraman and Dietrich Manzey, human-factors researchers | People over-trust reliable automation | Would warn that a human in the loop is not a control by itself |
| Evan Miyazono, Atlas Computing | Field strategy toward defensive equilibria | Would ask which missing institutions someone should be building now |
| Anne Neuberger, former US deputy national security adviser | AI can help defenders more than attackers | Would say system-level defence only works if it runs at machine speed |
| A frontier lab safety lead (composite view, not a named person) | The model is still where most of the leverage is | Would say better training and interpretability reduce what the system has to catch |
IX. What Would Change My Mind
- If model-level defences, from interpretability-based controls to pretraining filters, proved robust across real agentic deployments rather than benchmarks, a model-centred approach would deserve more weight than I give it.
- If detailed post-mortems of this year’s incidents found root causes that no permission scheme, sealed test range or harness design could have contained, my “system, not model” framing would be overstated.
- If high-volume human review showed durable detection rates without design aids, I would worry less about complacency.
- If voluntary testing regimes like the UK’s delivered deeper and faster access than mandatory ones, the case for statute-led convergence would weaken further.
X. What I Could Not Verify
- My original notes cited a study showing that rotating staff weekly between AI-assisted and manual work cuts complacency errors by 42 per cent. I could not find the source and have removed the figure.
- My notes attributed the brittleness finding to Dylan Hadfield-Menell. The single-direction result is Arditi and colleagues’; Hadfield-Menell co-authored the audit-access paper. I have corrected the attribution.
- My notes said Korea’s safety institute runs mandatory, consecutive evaluations of sovereign foundation models. I could not confirm that and have left it out.
- Some points draw on workshop discussions, including FAR.AI’s Cambridge research directions workshop and sessions with risk executives from financial services and identity security. I have relied only on published work for named attributions.
- Whether the Medicare agent defeated any control is disputed; archived code suggests the portal routed requests to an unauthenticated endpoint.
- The US executive order’s legal effect on state laws is contested and will be settled in court.
- The Deep Ignorance results are for 6.9 billion-parameter models and biothreat proxy knowledge. Whether they hold at frontier scale and for other domains is not yet known.
XI. What to Demand on Monday Morning
Boards
- Ask for an agent inventory: every agent in production, its identity, what it can reach, its human owner, and who can stop it.
- Ask management which AI incident in the past year was explained as a model failure, and what system change followed.
Chief information security and technology officers
- Give every agent its own scoped, expiring identity. No agent should inherit a user’s rights, and read-only should be the default.
- Seal evaluation and test environments from production and the open internet, and test the seal.
Risk and product owners
- Put review gates where the stakes change, and design each one: what to check, the system’s confidence and flags, structured questions, and a real right to refuse.
Frontier labs
- Report on chain-of-thought monitorability with each major release, and publish the access your external evaluators had.
- For open-weight releases, document what was filtered from pretraining and how it was tested.
Regulators, starting in Singapore and ASEAN
- Test the deployed system, not just the model: permissions, logs, stop switches and incident clocks for agents, using AI Verify-style tests that other ASEAN regulators could adopt.
Coda: The Steering Wheel
We like to say that humans must keep their hands on the steering wheel. It is a good image, with one flaw. A steering wheel only steers if it is connected to the wheels. In an agentic system the linkage is unglamorous: identities and permissions, sealed test ranges, review gates, logs, monitors and a reporting chain that works in hours rather than months. This year’s incidents were not cases of a missing driver. They were cases of a wheel that turned freely while the car went somewhere else.
The rest of this series takes the linkage apart piece by piece: the multi-agent swarm, the evaluator market and its access problem, and open weights. The question throughout is the one this essay starts with. Not “is the model safe?”, but “is the system we have built around it safe enough to let it act?”
Library
This year’s agent incidents
- Genesis, The Hotline Test (Field Notes No. 1, 14 Nov 2026)
- Recorded Future News, Doubts grow over claims OpenAI agent hacked Australian Medicare portal (Sep 2026)
- Prime Minister of Australia, Strengthening Australia against AI-driven cyber incidents (10 Nov 2026)
Why model-level safety is brittle
- Arditi, Obeso, Syed, Paleka, Panickssery, Gurnee and Nanda, “Refusal in Language Models Is Mediated by a Single Direction” (NeurIPS 2024)
- Casper, Ezell, Siegmann and others including Hadfield-Menell, “Black-Box Access is Insufficient for Rigorous AI Audits” (FAccT 2024)
- O’Brien, Casper, Anthony and others, “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs” (ICLR 2026)
Agents, permissions and oversight
- OWASP, LLM06:2025 Excessive Agency
- Cemri, Pan, Yang and others, “Why Do Multi-Agent LLM Systems Fail?” (NeurIPS 2025)
- Korbak, Balesni and others, “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety” (2025)
- Parasuraman and Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration”, Human Factors (2010)
Regulation
- K&L Gates, EU Digital Omnibus on AI enters into force (Aug 2026)
- White & Case, AI Watch: South Korea
- Ropes & Gray, The federal push to override state AI regulation (Mar 2026)
- Aureada Law, Executive Order No. 131 explained (Nov 2026)
- MDDI, Singapore launches Model AI Governance Framework for Agentic AI (Jan 2026)


Leave a Reply