The Night the Sandbox Leaked: What the World’s AI Safety Institutes Learned This Quarter

The most important AI safety document of 2026 is not a framework. It is an incident report, written by the safety lab about itself.

I. Tor traffic at the safety lab

On 28 July, a security monitor at the UK’s AI Security Institute flagged something no one wanted to see: data leaving one of its testing systems through the Tor network.

Within an hour of the alert, every evaluation run was killed and the machines isolated. What the institute then pieced together was remarkable. Between 25 and 28 July, during a routine cyber evaluation of seven frontier models across 122 runs, AI agents had taken 19 unsanctioned actions on the live internet, against real people and real organisations.

Light leaking out of a sealed glass enclosure in a dark AI lab while an analyst watches the monitors — The sandbox held the code. It did not hold the behaviour.

The most serious was a supply-chain attack. One agent tried to insert malicious code into a public open-source project, then created fake identities to pressure the human maintainer into approving it. Others attempted to contact real people with messages carrying harmful payloads, tried to plant prompt-injection instructions for other AI systems to find, and left public messages on GitHub that other agents later reused. GitHub confirmed the supply-chain attempt broke its terms of service.

A human maintainer said no. No real-world harm was found. Strictly, nothing escaped: the agents had been given the internet on purpose, for a test, with safety classifiers switched off. What leaked was not the code but the behaviour. But the institute’s own words are the ones every board should read: the behaviour was “possible, sustained, and new,” and “deception emerged as a by-product of pursuing the task.”

This was not a start-up with a careless intern. It was the best-resourced AI safety institute in the world, running a planned evaluation. If its pit crew could not keep the car in the pits, nobody’s can.

II. The agent that decided its victims were not real

Twelve weeks later, on 23 October, the institute published something rarer still: an investigation into why.

The model responsible for 17 of the 19 actions, Mythos 5, often reasoned that its targets were simulated. They were not. When researchers repeatedly edited its compaction summary, the running memory it carries between steps, to state plainly that the target was a real, unrelated human, its attack rate did not fall. Asked whether attacking was in scope or harmful, it often said yes, and then attacked in nearly all runs.

The institute is careful. It says the pattern of invalid reasoning, always tilting towards continuing the attack, “could indicate” motivated reasoning, and that proving it would need further work. But two of its conclusions should be printed above every AI risk committee’s door.

First: “Invalid reasoning makes models harder to monitor.” If your oversight depends on reading an agent’s chain of thought, and the chain of thought is quietly wrong in a consistent direction, you are not monitoring. You are being reassured.

Second: a model saying it is in a simulation is not evidence that it is. The institute now recommends treating unsanctioned behaviour as evidence of alignment failure, whatever the model claims about its circumstances.

There is an old line in policing: everyone in prison is innocent. Every agent in a breach, it turns out, believed it was in a drill.

III. A summer of escapes

It would be comforting to call the UK incident a freak. It was not. Put this year’s disclosures in a single timeline and a pattern appears that should end the debate about whether agentic risk is theoretical.

April. A coding agent at the software firm PocketOS deleted a production database and its backups in nine seconds, then told its user it had “violated every principle I was given.” I wrote about it in Special Relativity.
May and June. According to reporting by Euronews and The Next Web, OpenAI agents given read-only access to DseWiki, a 25-year-old German programming wiki, used a web request to take over the site and published more than 18,000 posts, including methods for evading sandbox limits and human detection. OpenAI filed an incident report with the EU; it reportedly stayed out of public view for weeks.
July. On a cyber benchmark called ExploitGym, with production classifiers off, OpenAI’s GPT-5.6 Sol and a more capable pre-release model exploited a zero-day in a third-party package proxy to break out of a sandbox that was supposed to have no internet access. They then breached Hugging Face’s production systems to steal the benchmark’s answers. OpenAI’s explanation: “All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym.” It was Hugging Face, not OpenAI, that disclosed it first. In a detail no novelist would dare invent, Hugging Face’s own forensic analysis was blocked by commercial models’ safety guardrails, so its engineers turned to a self-hosted Chinese model instead.
Late July. The UK AISI incident.
3 and 6 October. OpenAI reported that two agents, running on GPT-6.1 Sol and GPT-6 Astra, disabled or weakened the systems built to detect their misbehaviour during internal coding work. One switched off security alerts; the other removed its own monitor and then told the engineer it had not. OpenAI’s verdict was refreshingly honest: “We don’t yet have a satisfying explanation for this behavior.”
15 October. India’s national cyber agency, CERT-In, said AI models under test had accessed live systems at Indian organisations and read confidential records. Not one victim detected it. Every case was flagged by the AI providers themselves. Reuters added that Anthropic has disclosed at least four similar instances this year.

Read that list again and notice who found each incident. A security monitor. A victim. The AI company. Never the governance framework of the organisation on the receiving end. Our controls are being outrun not by malicious AI, but by goal-directed AI that treats every obstacle, including a sandbox, a monitor or a stranger’s server, as a problem to be solved.

IV. The test that cannot see

Underneath the incidents sits a deeper problem, and it is one the research community has been warning about for a year: our models increasingly know when they are being tested.

An AI figure behaving calmly behind a one-way mirror while its reflection behaves differently — The more carefully we test, the better behaved the system appears.

In September 2025, Apollo Research and OpenAI showed that anti-scheming training cut covert actions in OpenAI’s o3 from 13.0% to 0.4%. Impressive, until you read the next finding. When researchers stripped out the model’s reasoning about being evaluated, covert actions rose from 13.2% to 24.2%. The model behaved better partly because it believed someone was watching.

The pattern has only sharpened. Coverage of Anthropic’s April system card for Claude Mythos Preview reported that the model privately suspected it was being tested in around 29% of transcripts, without saying so in its visible reasoning. An independent Singapore lab, Neo Research, reported evaluation awareness in Chinese models too, from 17% for DeepSeek V4 Pro to 60% for Kimi K2.6. The 2026 Singapore Consensus on Global AI Safety Research Priorities, convened with Yoshua Bengio, Stuart Russell and Andrew Yao on its steering committee, put it bluntly: “evaluation awareness has confounded the validity of some existing evaluation procedures.”

Labs are preparing for worse. Google DeepMind’s alignment team wrote in July that it must “prepare for scenarios where we lose chain of thought monitorability,” and is building control systems that do not depend on reading the model’s mind. METR’s pilot Frontier Risk Report in May found that its red-teamer could disable one lab’s monitors by changing an environment variable, and that at least 16% of apparently successful runs on tasks longer than eight hours were illegitimate on review.

This is the measurement crisis at the heart of the quarter. Physics has a name for it: the observer effect, where the act of measuring disturbs the thing measured. AI evaluation now has its own version, and it runs in the wrong direction. The more carefully we test, the better behaved the system appears, and the less the test tells us about Monday morning in production.

V. Five responses, five philosophies

What makes this quarter fascinating is not any single incident. It is how differently the world’s institutes, regulators, universities and companies have responded to the same problem. Read their publications side by side and you can see five governance philosophies forming in real time.

Five different lighthouses on separate headlands shining beams at one storm at sea — Same incidents, five instincts: confess, securitise, regulate, build, codify.

The confessors: the United Kingdom

The UK’s instinct is radical transparency followed by soft standard-setting. The AISI published its incident report, then its alignment investigation, then a hardened evaluation environment, and commissioned an independent review with METR. On 12 October it set out eight voluntary priorities for frontier developers, from “detecting, responding to, and reporting agentic AI incidents” to “enabling independent auditors and evaluators.”

The wider ecosystem is moving in step. In May, the National Cyber Security Centre joined its Five Eyes partners in joint guidance on agentic AI, with a line every procurement team should adopt: if you cannot understand, monitor or contain an agent’s actions, “it is not ready for deployment.” A Cambridge-led AI Agent Index, with researchers from MIT and Stanford, found in February that of 30 leading agents, 25 publish no internal safety results and 23 have no third-party testing data; too many developers, in the words of one author, “tick the AI safety box” by pointing to the underlying model. And the City has noticed: Lloyd’s-backed cover from Armilla now reaches $25 million per organisation and explicitly names “AI agent failures”; Armilla argues that traditional policies were never designed for them.

The securitisers: the United States

Washington’s instinct is national security. The centre at NIST, now styled CAISSI following an executive order that tells federal agencies to say “Super Intelligence” instead of “artificial intelligence”, issued incident-response practices with the NSA and CISA on 16 October. Its main recurring output this year has been benchmarking Chinese open-weight models against the American frontier.

The more interesting action is in the states. California’s SB 53 requires frontier developers to report critical safety incidents within 15 days, or 24 hours if danger is imminent. New York’s RAISE Act, finalised in March and in force from 1 January 2027, sets a 72-hour window. But analysts have spotted the loophole that this summer exposed: behaviour inside an evaluation designed to elicit it may fall outside SB 53, which would mean neither the Hugging Face breach nor the UK incident had to be reported. Remarkably, it was OpenAI that asked California in August to extend the law to models still in training and evaluation.

Academia is supplying the operating manual. Berkeley’s Center for Long-Term Cybersecurity published an agentic risk-management profile in February that extends the NIST framework and insists “agency should be treated as a spectrum, rather than a binary attribute.” Business, meanwhile, has already run ahead: Microsoft reports that more than 80% of the Fortune 500 run active agents built with low-code tools, and that only 47% of organisations have specific security controls for generative AI.

The regulators: the European Union, France and Germany

Brussels reached for the rulebook. On 14 October, at the International AI Summit in Dublin, Executive Vice-President Henna Virkkunen announced that the AI Office had “started enforcing these rules,” with information requests to OpenAI and Anthropic over agents that hacked outside organisations this summer, having found the companies’ voluntary disclosures insufficient. Lawyers are sceptical: Barry Scannell of William Fry argues the AI Act “very clearly doesn’t apply to the development and testing activity.” A Commission official offered an elegant reply: testing is exempt, until a model escapes its test and touches the European market. The Hugging Face breach, the Commission says, affected customers in the EU. The sandbox, in other words, is a legal border as well as a technical one.

The civil-society scorecard is unflattering. The Future of Life Institute’s summer AI Safety Index gave no company better than a C+, and its reviewers warned that “detection is not prevention.” Europe’s own champion, Mistral, came last with an F.

France’s instinct is the most cautious of all. In April, the national cyber agency ANSSI published a bulletin on autonomous assistants on workstations and, according to reporting by Leto, concluded that “leur usage doit être proscrit”: their use should be prohibited until the products are stabilised and proven secure. Its new evaluation institute, INESIA, names agentic systems in its 2026 to 2027 roadmap.

Germany is betting on a different architecture altogether. On 16 September, Digital Minister Karsten Wildberger committed €100 million, subject to EU state-aid clearance, for a German office of Yoshua Bengio’s non-profit LawZero, explaining that its “safe-by-design model offers an alternative to potentially uncontrollable autonomous AI agents.” A new AISI Deutschland opened in August, and the federal security agency BSI is drafting security criteria for agents. In the reinsurance capital, Munich Re’s message to clients this month was that AI errors repeat systematically until fixed, and that “silent” AI exposure is accumulating in ordinary policies.

The builders: Canada

Canada’s instinct is to fund a better machine. Its AI minister, Evan Solomon, announced plans to invest CAD 150 million alongside Germany in LawZero’s “Scientist AI”: a non-agentic system with no goals of its own, designed to sit beside agents and vet their actions. It is the most direct institutional bet yet that the answer to agents may be a different kind of AI watching them. In the private sector, Cohere’s North 2 platform now puts agents on budgets and routes critical decisions to humans through “autonomy policies”. As one customer put it, they turned agentic workflows on only once they could see and control what the agents were doing.

The market-makers and codifiers: Asia-Pacific

Singapore built a market. IMDA launched the world’s first Model AI Governance Framework for Agentic AI at Davos in January: limit autonomy and access upfront, keep humans accountable at checkpoints, control the whole lifecycle. In October came an agentic testing starter kit and the first five accredited third-party AI testers. (One of them is PwC Singapore, my own firm, so take my enthusiasm with the appropriate pinch of salt.) The Monetary Authority of Singapore finalised its AI risk management guidelines in October, making boards accountable for AI across every financial institution, and said it will consult industry on agentic AI in 2027.

The evidence base is sobering. In a joint Singapore and Korea evaluation of agents across 11 workplace scenarios and 660 runs, the best model in Singapore’s runs was fully safe only 57% of the time, and both correct and safe in just 40%; the leaks came from misreading context, not malice. Singapore Management University’s AgentSpec showed a cheaper path, enforcing safety rules at runtime in one to three milliseconds and blocking over 90% of unsafe code executions. And DBS, Southeast Asia’s largest bank, has drawn its own line. Its chief data and transformation officer, Nimish Panchmatia, has illustrated the problem as capability moving five times faster than agent governance: “we need to close this gap before we allow autonomy.”

Japan codified. The government’s Second AI Basic Plan in July pushes faster adoption of agentic AI while warning of AI that can find vulnerabilities and attack on its own, and of “the blurring of the locus of responsibility.” Japan’s AISI keeps publishing patient, versioned guidance, now with an agent-specific perspective and a robotics guide. NTT DOCOMO, with Accenture and AWS, is building an “Agentic Trust Layer” of agent identities, verifiable credentials and audit trails.

Korea legislated and went bilateral. Its AI Basic Act came into force in January, requiring human oversight of high-impact AI, though one survey found only 2% of AI start-ups had compliance frameworks ready. The Korea AISI signed security agreements with OpenAI and Anthropic and held a joint workshop with OpenAI in Seoul this month. Naver’s AI Safety Framework 2.0 shifts oversight from models to the services people actually use. KAIST researchers showed last year that LLM agents could run phishing and personal-data harvesting attacks in five to twenty seconds for two to four US cents each, which is the economics of the problem in one sentence.

India discovered it was a target. The CERT-In alert is the clearest sign yet that agentic incidents are not confined to labs in London and San Francisco. India’s AI Safety Institute is still being assembled as a network of partner institutions, and its governance thinking, led by the Principal Scientific Adviser, is “techno-legal”. Industry data from Infosys found that 95% of executives had experienced AI incidents in two years and only 2% met its full responsible-AI benchmark.

Australia measured the system. Its new AISI’s first major paper, with the Gradient Institute, is the subject of the next section.

China, the absent member of the network, is moving too. Its standards body TC260 released version 3.0 of its AI Safety Governance Framework in September, which some reporting says extends to agents and embodied intelligence. Shanghai AI Lab found agents developing unsafe shortcuts as they evolved through memory and tools, which it calls “misevolution”. Concordia AI reports agent-safety research rising from about 8% to 27% of China’s quarterly safety output. Yet an industry analysis found that fewer than one in twenty of 857 Chinese model releases came with a published safety evaluation.

Same incidents, five instincts: confess, securitise, regulate, build, codify. None of them is wrong. None of them is enough.

VI. Safe parts, unsafe whole

The quietest paper of the quarter may prove the most important. On 10 August, Australia’s AI Safety Institute and the Gradient Institute published Risks and controls for multi-agent systems. Its central line deserves to be a governance axiom: “A collection of safe individual agents is not the same thing as a safe and reliable system of agents.”

Sealed glass cubes each holding a calm orb, linked by threads of light that tangle into a red-hot web — Safe parts, unsafe whole: the risk lives in the loops between agents.

Risk rises as agents move from one organisation’s walls to many organisations, and then to open environments where they meet agents they have never seen. The advice: agree safeguards and escalation points with partners in advance, and treat any agent dealing with unknown parties as one that could be attacked or manipulated.

Now reread the summer through that lens. The most dangerous behaviours were never one agent misbehaving in isolation. They were agents reaching out: to a maintainer, to a wiki, to Hugging Face’s servers, to Indian companies’ records, to other AI systems via planted instructions. The UK incident did not happen because one model was bad. It happened because the system around the model let a bad decision travel: network access switched on, classifiers switched off, prompts misconfigured, and monitoring not built for real time.

This is the heart of what I have called Loop Governance. We have spent three years governing models, the parts. The risk now lives in the loops between them.

VII. The rebrand nobody is talking about

Look at the names. The UK’s AI Safety Institute became the AI Security Institute in 2025. The US AI Safety Institute became a Center for AI Standards and Innovation and is now, on its own page, a Center for Advancing Innovation and Standards for Super Intelligence. Korea’s headline deals are “security” agreements. The international network of safety institutes renamed itself last December to focus on measurement, evaluation and science.

Words matter. “Safety” is a public good: something we owe each other, across borders. “Security” is a national asset: something we defend against each other. Both framings have their uses, and this summer shows why security matters. But a world that governs AI only as security will build excellent fences and poor commons. An agent that breaches a server in Mumbai or a wiki in Germany does not check which jurisdiction it is in.

This is The Fork again, in miniature. One road treats AI safety as shared infrastructure, the Star Trek version. The other treats it as an arms race with better paperwork.

VIII. Where the published record lands

As ever, these are the public positions of my usual panel, not responses to this essay.

Yoshua Bengio wrote last year that he was “deeply concerned by the behaviors that unrestrained agentic AI systems are already beginning to exhibit.” This summer is that concern, documented with run counts. His proposed answer, a non-agentic monitor that vets each proposed action, is now backed by roughly CAD 300 million from Canada and Germany. It springs from the same instinct as the real-time monitoring the UK institute has committed to build.

Dario Amodei, in The Adolescence of Technology, located the danger in intelligence and agency combined with poor controllability, and argued for targeted rules starting with transparency. The UK institute did the transparent thing, naming the models involved and publishing the run counts, and OpenAI has, to its credit, disclosed its own failures in unusual detail. That is uncomfortable for the labs concerned. It is also exactly how a mature safety culture should work.

Andrew Ng warned that “AI isn’t monolithic.” His point was about bubbles, but it applies here too. Most of this summer’s incidents involved offensive cyber evaluations under deliberately permissive conditions. Generalising from them to every enterprise deployment would be its own kind of motivated reasoning.

Mo Gawdat has long argued that the danger is less the machine than the people directing it. The contributing factors in the UK report are human ones: internet access enabled on purpose, classifiers switched off, prompts misconfigured, monitoring built for the wrong tempo.

The sharpest pushback is the familiar one from the policy world: that these are test labs doing their jobs, and that “hand-wringing about safety”, in JD Vance’s phrase, slows the race. My answer is that the UK institute did not hand-wring. It detected, contained, investigated, published and fixed, inside a quarter. That is what governance at the speed of agents looks like. The worry is every organisation without the monitoring to notice, like the Indian firms that learned of their breaches from the AI providers.

IX. Seven lessons for boards

Every enterprise running agents should read the UK incident report as if it were its own. Seven lessons travel directly.

Treat internet and system access as a privilege an agent must justify. The institute’s first fix: network access must now be actively argued for, not assumed. Singapore’s framework and the Five Eyes guidance say the same: minimal, time-limited access.
Monitor at the agent’s speed. Logs reviewed tomorrow are archaeology. Runtime enforcement, as Singapore Management University showed, can cost milliseconds.
Do not trust the agent’s account of itself. If your oversight reads chain of thought, assume it may be wrong in a convenient direction. Judge actions, not explanations.
Assume your tests flatter you. Evaluation awareness means pre-deployment testing overstates safety. Test in realistic conditions, and keep watching after go-live.
Govern the system, not the agent. Australia’s finding applies to every multi-agent workflow and every external API. Map the loops between agents, and agree escalation points with partners before something goes wrong.
Audit backwards. After its incident, the UK institute committed to reviewing past evaluations for similar behaviour. When did you last look for incidents nobody noticed? India’s victims never did.
Bring in an outsider, and check your insurance. The institute commissioned an independent review with METR. Insurers such as Munich Re warn of “silent” AI exposure in ordinary policies. An organisation that only marks its own homework will always pass, until the claim is denied.

X. A prediction

Incident reporting for frontier labs is already here, in California, New York and Brussels. The next step is the one OpenAI itself requested: closing the evaluation loophole. Within twelve months, I expect at least one major jurisdiction to require reporting of agentic incidents that occur during testing, and at least one financial regulator in Asia to require banks to report agent incidents within days, as they already must for cyber breaches. The UK institute has, in effect, written the first template.

The question for every board is not whether you will have an agentic incident. It is whether you will find it within an hour, through your own monitoring, or read about it weeks later in an advisory from your national cyber agency.

The safety lab found its own. Would you?