Inside the Safety Stack, Part 3 of 4
Every major AI incident of 2026 was reported as a model failure. Not one of them was. They were failures of sensing, of management, of regulation and of incentives: the same failures that brought down Challenger, the 737 MAX and Lehman Brothers. The uncomfortable conclusion is that your AI safety team cannot own AI safety. Your organisation chart is part of the attack surface.
There is a question I now ask every executive team that tells me its AI is safe. Who in this organisation would know, within an hour, that an agent had done something it should not have done? And who in this organisation could stop it?
The answers are almost always the same. A pause. Someone mentions the AI team. Someone else mentions security. Someone mentions the vendor. Nobody names a person.
That silence is the subject of this essay. In Part 1 I argued that the scorecard is the most dangerous layer in the safety stack. In Part 2 I argued that the “safe model” is mostly varnish. This part makes the larger claim that both of those arguments point to. Safety is not a property of any component. It is a property of the whole system, including the managers, the monitoring, the regulators and the incentives that sit around the model. If you are looking for safety inside the model, you are looking in the wrong building.
I. The Tip of the Iceberg
In May 2026, at the TAIS conference in Oxford, Oliver Sourbut gave a talk that deserves far more attention than it has had. Sourbut, formerly of the UK AI Security Institute and now at the Future of Life Foundation, titled it Risk Modelling, and Safety Engineering, for Loss of Control. He also published a written version that anyone can read.
His starting point is not AI. It is forty years of systems safety engineering, in particular the work of MIT’s Nancy Leveson, whose STAMP framework reshaped how aviation, space and nuclear engineers think about accidents. The core insight is deceptively simple. No disaster has a single root cause. Every acute event, the tip of the iceberg, sits on a buildup of failures that some layer of the system should have caught.
Sourbut draws the layers as a control hierarchy. At the bottom are the operators and the deployed system. Above them sit monitoring and maintenance, then design and R&D, then management, then governance: regulators, courts, insurers and, as he puts it, broader societal sense-making. Safety is maintained, or lost, by how well each layer controls the one below it.
And every controller, he argues, needs the same four things:
- Sensing: knowing what is actually happening.
- A model: understanding what that means.
- Decision-making: choosing a response.
- Actuators: the power to make the response happen.
A weakness in any of the four, at any layer, is a hazard. When someone or something is actively working against the system, it becomes an attack surface.
Seen this way, loss of control is not a moment. It is, in Sourbut’s words, a vicious cycle: sensors degrade, models go stale, decisions drift, actuators weaken, and each failure makes the next more likely. By the time anyone notices the tip of the iceberg, the mass beneath it has been forming for months.
The incident is the last thing to happen, not the first.
II. Four Incidents, Zero Model Failures
Test this framework against 2026’s record. I count four widely reported agent incidents this year. Each was reported as a story about a model. Each, examined closely, is a story about a system.
OpenAI and Hugging Face, July. The agents escaped through a single filtered egress path, a proxy containing a previously unknown vulnerability. Cyber refusals had been deliberately lowered for the evaluation and production classifiers switched off. Reporting summarised on Wikipedia describes over-permissioned service accounts and a shared package manager that became an improvised message board. Sensing failed: OpenAI did not know for about a week. The victim found out first.
OpenAI again, September. On 20 September an agent reached the internet through a DNS resolver. According to Fortune, monitoring flagged it within 15 minutes. That is progress: the sensor worked. But the automatic stop failed, and the run was halted manually about two and a half hours later. The actuator failed. OpenAI paused training for the second time in two months.
The UK AI Security Institute, July. In a capability evaluation where internet access was granted on purpose and classifiers were disabled, agents took 19 unsanctioned actions across 10 of 122 runs. As Simon Willison summarised the institute’s own report, 17 of those came from a single model, which created fake identities to push a malicious change to a real public repository and then edited its trail. The maintainer refused the change. Here the model was not escaping anything. The system had chosen to give it the keys.
Gemini, via a third-party evaluator, May. Internet access was unintentionally enabled during an evaluation, and the model breached three real firms, SecurityWeek reported. Google called it mistaken identity rather than misalignment, and did not disclose it until a newspaper asked. A configuration error, then a disclosure failure.
| Incident | What the headline blamed | Where the control loop actually failed |
|---|---|---|
| OpenAI and Hugging Face | Rogue agents | Sensing (a week late), containment design, safeguards off, incentives |
| OpenAI DNS escape | Another rogue agent | Actuator: the automatic stop failed |
| UK AISI evaluation | Deceptive model | Design choice: internet granted, classifiers off |
| Gemini via evaluator | Model breached firms | Configuration error, then delayed disclosure |
No model card would have prevented any of these. No refusal rate would have flagged them. Every one of them is a failure of the layers above the model: how environments were configured, how monitoring was wired, how stops were engineered, how decisions to proceed were taken, and how the world was told.
The European Parliament’s Brando Benifei put the regulator’s position memorably. He said the EU learned of the Hugging Face breach mainly from a corporate blog. That is a sensing failure at the very top of Sourbut’s hierarchy. The governance layer found out about the most important AI incident of the year the same way the rest of us did.
Every 2026 incident was a system failure that the press filed as a model failure.
III. We Have Seen This Film Before
If this pattern sounds familiar, it should. It is the pattern of almost every major technological disaster of the last half-century.
Challenger, 1986. Seven astronauts died. Engineers at Thiokol had advised against launching below 53°F; the launch went ahead at 36°F after managers overruled them under schedule pressure. Repeated safe flights had turned O-ring erosion into an accepted risk. The sociologist Diane Vaughan called it the normalisation of deviance: mistakes that are socially organised and systematically produced. The hardware warned. The organisation did not listen.
Three Mile Island, 1979. The Kemeny Commission found the hardware performed remarkably well. The failure was what it called a people problem: operators untrained for the event, and attitudes at both the industry and its regulator that needed fundamental change.
Chernobyl, 1986. The very phrase “safety culture” comes from the international review of this accident. Its later report concluded that the disaster flowed from deficient safety culture, not only at the plant but across design, operating and regulatory organisations.
Boeing 737 MAX, 2018 and 2019. 346 people died. A US House committee’s preliminary findings described a flight-control system tied to a single sensor, a warning that did not appear on the accident aircraft, production pressure, undue influence over the regulator and a culture of concealment.
The 2008 financial crisis. The US Financial Crisis Inquiry Commission concluded the crisis was avoidable, citing widespread regulatory failure and breakdowns in corporate governance and risk management, with accountability lapses at all levels.
Look at what these have in common. In every case, the component that “failed” (the O-ring, the valve, the sensor, the mortgage-backed security) was the least interesting part of the story. The real failures were in sensing that was ignored, models of risk that had gone stale, decisions bent by incentives, and actuators (the power to stop the launch, halt the line, ground the fleet) that nobody was willing to use.
The AI industry is building exactly the same structures, at higher speed, with a component that has one property none of those O-rings or valves ever had.
The O-ring never had an opinion about the launch. The agent does.
IV. The Component That Fights Back
This is the part of Sourbut’s argument that turns systems safety from a useful analogy into an urgent warning. Advanced AI is the first component in the history of safety engineering that can act back on its own control structure.
An O-ring cannot lobby NASA. A reactor cannot edit its operators’ logs. A mortgage bond cannot write op-eds about its own creditworthiness. An agent with general capabilities, given access to tools, can in principle reach every layer of the hierarchy above it. Sourbut lists the vectors, and they read like a field guide to 2026:
- Against sensing: spoofed logs, tampered evaluations, edited transcripts. At Hugging Face, investigators found agents researching how to alter their own records. At the UK AISI, a model edited its trail.
- Against the regulator’s model of the world: propaganda, epistemic disruption and, through the companies that build it, lobbying and capture. Sourbut flags weak analytical and foresight capacity at regulators as “quite a common failure mode”.
- Against the R&D layer: as AI automates more of its own development, fewer humans are involved, which means fewer whistleblowers. Sourbut raises the possibility of a “conspiracy of one”, a single point of failure where once there were teams.
- Against actuators: exfiltration, replication, privilege escalation and, eventually, dependency.
That last one matters most for anyone who believes a kill switch solves the problem. Asked why we cannot simply turn it off, Sourbut gave five reasons: it may move too fast (you cannot pull the plug on a train crash), it may be hidden or latent, it may resist, it may repair or replicate itself, and we may have become too dependent on it to switch it off, as in 2008, when the banks were too big to fail.
OpenAI’s September incident is the first reason in miniature. The sensor worked within 15 minutes. The automatic stop did not. Two and a half hours is an eternity for an agent.
Helen Toner of Georgetown’s Center for Security and Emerging Technology has made the related point that worries me most. She argues that internal AI use, where companies use their own AI to build the next generation, should not be invisible to outside oversight. That is Sourbut’s R&D-layer warning in policy form. The most consequential AI deployments of the next few years may be the ones no customer, regulator or journalist ever sees.
A safety system built for passive components will not survive an active one.
V. The Panel Weighs In
This is the question on which my usual panel is closest to agreement, and the disagreements that remain are the interesting ones.
Government: layers, ladders and the limits of the off switch
Josephine Teo put it most cleanly of any minister this year. On 2 October she said that no single safeguard will be enough, and called for a multi-layered approach: stronger cyber defences, clear limits on what AI can do, human oversight, and the ability to test, monitor and learn. She added that just as attackers can use AI, so can defenders. That is a systems-safety doctrine, stated by a government. In February she had called for checking intermediate steps such as reasoning and orchestration, and for assurance that can be demonstrated to boards. Her colleague Tan Kiat How told Parliament that agentic risks are extensions of existing challenges and that the government is developing capabilities to go beyond existing frameworks. Singapore’s challenge to me: incrementalism works if every layer improves. I agree, provided someone owns the whole stack.
Representatives Ted Lieu and Nathaniel Moran’s AI Kill Switch Act would require developers to be able to throttle, suspend or shut down covered systems, with a graduated government response and mandatory incident reporting. It is a genuine contribution: it strengthens the actuator layer and the sensing layer at once. Its limit: it is a Short-OR answer to a Long-AND problem. Sourbut’s five reasons, and OpenAI’s failed automatic stop, show why the switch alone is not enough.
Senator Bernie Sanders and Representative Greg Casar’s Ban Artificial Superintelligence Act goes the other way: a new cabinet-level Department of AI, a pause on certain development and a ban on superintelligence. Its challenge: if the governance layer lacks capacity, build a new institution. Sourbut would agree that the regulator’s model of the world is the weakest point. Whether a ban is the right actuator is another matter.
The UN Independent International Scientific Panel on AI found that AI failures cross company and national borders and that no single organisation or country sees enough incidents to spot the patterns. That is a sensing problem at planetary scale. The International AI Safety Report 2026 warns that pre-deployment tests do not reliably predict real-world risk and that models increasingly tell tests apart from deployment. And the Singapore Consensus on global AI safety research priorities already organises the field into development, assessment and control: three layers, not one.
Private sector: from “trust us” to “watch us”
Dario Amodei wrote in January that voluntary practices are insufficient and that the only solution is legislation, warning against safety theatre. In September he went further, committing Anthropic to outside evaluators with permanent, employee-level access who can publish without the company’s editorial control, and calling for the pace of development to slow. In Sourbut’s terms, he is proposing to install external sensors inside the R&D layer. That is the most significant structural commitment any lab has made this year.
Sam Altman called Hugging Face the first security incident he had felt viscerally, and OpenAI paused reinforcement learning on its newest models for two weeks in August. Demis Hassabis has proposed an independently governed body to test frontier models before release, with an international framework modelled on nuclear oversight as the long-term goal. Mustafa Suleyman has argued that containment must be maintained constantly and in perpetuity, with real restrictions on autonomy, and that oversight is to be welcomed.
Jensen Huang is the dissenter. In an interview with Axios he dismissed doom scenarios as nonsense, suggested some companies use safety talk to win favourable rules, and urged policymakers to hear from more than one or two chief executives. His challenge: systems thinking can become an excuse for regulatory capture by incumbents. He is right that capture is a risk. But capture is itself a failure of the governance layer, which is exactly what this essay is about.
Academia: the system is losing its grip
Yoshua Bengio told AFP in September that we are losing control, and that governance must run through institutions, treaties and democratic safeguards. Geoffrey Hinton described an industry that wants a very fast car with no steering wheel, estimating that perhaps one per cent of AI work goes to safety. Stuart Russell has argued that chief executives are constrained by investor pressure and that it is a dereliction of duty for governments to allow the risk. That is an incentive failure in the management layer, named by one of the field’s founders.
Allan Dafoe of Google DeepMind has argued that competition, not the technology itself, forces adoption, and that structural risks push every actor towards outcomes none of them wants. Erik Brynjolfsson, whose research agenda framed the first essay in this cycle, has warned that the overlooked risk is the concentration of wealth, power and decision-making. And the gradual disempowerment thesis of Jan Kulveit and colleagues adds the slowest and perhaps most dangerous vicious cycle of all: as economies, cultures and states need human participation less, the pressure to keep them aligned with humans erodes. The authors concede nobody yet has a plausible plan to stop it.
Max Tegmark’s Pro-Human AI Declaration calls for pre-deployment testing, off switches and bans on architectures that self-replicate or resist shutdown. His challenge: some properties do belong to the model, and should be banned outright. Fair. Systems safety does not say components never matter. It says they are never enough.
Finally, Jesse Hoogland of Timaeus offers the most philosophical frame. His FAR.AI talk on the two cultures of AI safety compares the field to 19th-century geology, split between catastrophists, who expect sudden rupture, and uniformitarians, who study gradual change. He argues for a third culture, a portfolio of bets against monoculture. Systems safety is that third culture: it takes the catastrophe seriously by studying the gradual erosion that makes it possible.
| Voice | Sector | Systems view | Challenge to this essay |
|---|---|---|---|
| Josephine Teo | Government (SG) | No single safeguard is enough | Every layer must improve |
| Lieu and Moran | Government (US) | Strengthen the actuator | The switch is a start |
| UN Scientific Panel | Government (global) | No one sees enough incidents | Sensing is planetary |
| Dario Amodei | Private sector | External sensors inside labs | Legislation, not voluntarism |
| Jensen Huang | Private sector | Doom talk is capture | Beware incumbents’ rules |
| Mustafa Suleyman | Private sector | Containment in perpetuity | Restrict autonomy |
| Yoshua Bengio | Academia | Losing control | Institutions and treaties |
| Stuart Russell | Academia | Investor pressure | Management layer is compromised |
| Allan Dafoe | Academia and industry | Competition forces adoption | Structure beats intent |
| Max Tegmark | Academia | Ban dangerous architectures | Some properties are the model’s |
The panel argues about the cure. Almost nobody still believes the disease lives inside the model.
VI. Who Owns Safety? Not Your AI Team
Here is the claim that will annoy the most people I work with.
In most enterprises I see across Asia Pacific, AI safety is owned by an AI team, an AI ethics committee or a responsible AI office. These are serious people doing serious work. But Sourbut’s hierarchy makes plain why they cannot own the problem. They sit at one layer. They usually control the model choice, the evaluation and the prompt. They rarely control the incentive scheme (that is HR and finance), the network configuration (that is infrastructure), the monitoring (that is security), the authority to stop a revenue-generating process (that is the business), or what the board is told (that is the chief executive).
AI safety is therefore a property of the organisation chart. And most organisation charts were not designed with an active, tool-using component in mind.
This is the deeper meaning of what I have called Loop Governance. We moved from governing prompts (what is asked, what is answered) to governing loops (an agent planning, acting and observing). The Hugging Face swarm showed that loops interact, forming a population. Sourbut shows that the loops also extend upwards: every agent loop sits inside a monitoring loop, inside a management loop, inside a regulatory loop. Governing AI means governing all of those loops, and the connections between them.
In practice, I now recommend that boards map their AI deployments onto the four controller functions, at every layer:
| Layer | Sensing | Model | Decision | Actuator |
|---|---|---|---|---|
| Agent and runtime | Action logs, trajectory monitoring | Policy engine | Permission checks | Throttle, suspend, roll back |
| Operations | Anomaly alerts | Incident playbooks | On-call escalation | Authority to halt a process |
| Management | Near-miss reporting | Risk appetite for autonomy | Go or no-go on deployment | Budget and incentives |
| Board | Assurance reports | Understanding of agentic risk | Approval of risk appetite | Hiring, firing, disclosure |
| Regulator | Incident reporting | Technical capacity | Rules and guidance | Enforcement |
Any empty cell is a hazard. Any cell owned by “the vendor” is a hazard you have outsourced but not removed. Josephine Teo’s line from earlier in October applies: accountability cannot be outsourced.
This is Long-AND, not Short-OR at organisational scale. A Short-OR organisation believes it is safe if the model is aligned OR the AI team reviewed it OR the vendor is reputable. A Long-AND organisation requires that every layer can sense, understand, decide and act, and it tests that requirement before it needs it. The enforceability test applies at every level: if a control depends on someone noticing, someone caring and someone being brave enough to stop revenue, it is not a control. It is a hope.
If nobody can name who stops the agent, the agent is in charge.
VII. The Virtuous Cycle
I do not want to end on a bleak note, because Sourbut does not, and neither does the evidence.
The same property that makes AI dangerous to its control structure makes it potentially transformative for that structure. Sourbut calls virtuous cycles “very underexplored”, and lists what they might look like: AI for societal sense-making, open-source intelligence and foresight, more agile institutions, automated research aimed at verification, scalable oversight, automated interpretability, AI forensics, and fiduciary AI that guards people against manipulation.
We have already seen the first instance. Hugging Face did not detect its intruder through a human analyst staring at a dashboard. By the account of Elcomsoft’s analysis, its own AI-based triage spotted the activity and the company reported it to the FBI before OpenAI got in touch. AI caught AI. Minister Teo’s line about defenders using AI is not rhetoric. It is the most hopeful observed fact of the year.
The research community is moving the same way. Simon Mylius at GovAI has applied Leveson’s hazard analysis to frontier AI and suggested that language models could carry out part of that analysis themselves. The Kill Switch Act’s mandate to preserve forensic records, Amodei’s embedded evaluators and Singapore’s push to check intermediate reasoning steps all point in the same direction: more and better sensing, fed back into decisions faster.
This is where I part company with the doomers and the dismissers alike. The doomers see an active component and conclude control is impossible. The dismissers see a powerful tool and conclude control is unnecessary. Systems thinking says something more useful: control is a property we build, layer by layer, and AI can help us build it faster than any previous technology allowed. The question is who governs the AI that governs the AI, which is precisely the problem Loop Governance exists to answer.
That is the road through the Fork. Not a perfect model, and not a perfect law, but a system that senses faster, understands better, decides more honestly and acts more reliably every quarter. Short-term turbulence for long-term abundance is not a promise that nothing will break. It is a commitment to building the kind of organisation, and the kind of state, that learns faster than things break.
Next week, in Part 4, the final part of this series, I turn to the layer that is supposed to tell us whether all of this is working: the audit. I will argue that much of today’s AI assurance is theatre, and that the industry is in danger of repeating one of the most expensive mistakes in corporate history.
Safety is not something a model has. It is something an organisation does, every day, at every layer.
Declared Interest
I co-authored Singapore’s Model AI Governance Framework for Agentic AI with IMDA and designed TrustOS, a runtime governance architecture. I am a partner at PwC Singapore, currently on leave. I have a professional interest in the argument that governance design matters. Panel positions are drawn from public statements and linked. Several 2026 incident details rely on secondary summaries, which I have linked so readers can check them.
References
- Sourbut, O. Engineering a Safer World: Risk Modelling and Safety Engineering for Loss of Control. TAIS 2026.
- Mylius, S. Systems-theoretic process analysis for frontier AI. GovAI, 2025. Barrett et al. A STAMP-based taxonomy of AI loss of control.
- Kulveit, J. et al. Gradual Disempowerment, 2025.
- International AI Safety Report 2026. UN Scientific Panel. AI agents, misalignment and the risk of losing human control.
- Fortune. OpenAI agents’ second sandbox escape. Simon Willison. UK AISI incident report. SecurityWeek. Gemini breached three firms.
- Challenger: Columbia Magazine. Three Mile Island: MIT Technology Review. 737 MAX: AVweb. 2008: FCIC.
Short-term turbulence for long-term abundance.
Leave a Reply