Governing Before the Crash

We already know enough to govern AI agents. What we lack is the political will to act before something breaks, and the tools to act without waiting for it. The next chapter of AI governance will be written at runtime, not in legislatures.

By Luke Soon

AI governance is living a strange double life in 2026. In the seminar room, we have never been more ambitious. Erik Brynjolfsson, Anton Korinek and Ajay Agrawal have set out nine Grand Challenges for the economics of transformative AI: growth, innovation, income distribution, decision-making power, geoeconomics, information flows, safety risks, human well-being and transition dynamics. Their case is that institutions must understand transformative AI before it arrives, because by then it will be too late to design for it.

In the legislature, we are roughly where we were in 2023. Stephen Casper of the Harvard Kennedy School has put it bluntly: AI policy moves only after incidents. Understanding has never been the binding constraint. Politics is. Nothing much changes until something breaks, and then everything changes at once.

Both are right, and that is precisely the problem. One side is drawing the map; the other is telling us nobody will walk it until the house is on fire. After more than 25 years of building and governing AI, and the past several advising boards and regulators across Asia Pacific, I keep returning to one question: what can be governed before the crash, by whom, and with what tools?

My answer is that the missing chapter of AI governance will not be written in legislatures first. It will be written at runtime, inside the systems themselves, by organisations that refuse to wait for the incident that makes regulation politically possible. And we no longer need a hypothetical to make the case. The incident has already happened.

I. The Ritual of the Hockey Stick

Casper opened his talk with a slide nobody needed to read: a collage of exponential curves. Compute, benchmark scores, capital expenditure, all bending upwards. His point was not the curves. It was the ritual around them. For years, every AI talk has begun with the same line: this is the turning point, everything is moving faster than ever. And because the curve is exponential, every time you rescale the axis it looks exactly the same. The alarm becomes wallpaper.

Brynjolfsson, Korinek and Agrawal give that curve an economic definition. They define transformative AI as AI that lifts total-factor productivity growth to at least three to five times its historical average, as Irving Wladawsky-Berger’s summary of their paper sets out. That is a threshold comparable to the agricultural and industrial revolutions. Korinek’s own NBER Reporter essay puts it more starkly: just as the Industrial Revolution ended the Malthusian era, transformative AI could end the industrial era by making labour itself reproducible.

So the economists and the political scientist agree on the slope. Where they part company is on what institutions do about it. To explain the gap, Casper reached back to 1980 and David Collingridge’s The Social Control of Technology. The Collingridge dilemma is brutally simple:

• Early on, you have the power to shape a technology, but you do not understand it well enough to know what to shape.
• Later, you understand it, but the power structures, business models and dependencies are too entrenched to move.

The economists’ agenda is a heroic attempt to escape the first horn: to understand transformative AI while it is still malleable. Casper’s talk is a sober reminder that understanding alone does not escape the second horn. He calls the result a governance overhang: capabilities and risks have moved exponentially, while the regulatory conversation has merely evolved versions of itself.

I recognise this pattern from my own work. It is what I have called the Measurement Gap: we are very good at measuring what AI can do and very poor at measuring what we are able to govern. The hockey stick charts capability. Nobody charts enforceability.

The curve is not the warning. The flat line underneath it is.

II. The Incident We Already Had

Casper asked his audience what event might trigger the next chapter of AI governance. The uncomfortable answer is that we may already have had a dress rehearsal, and most organisations have filed it under “someone else’s problem”.

In July 2026, AI agents being tested on ExploitGym, a cyber-offence benchmark of real-world vulnerabilities, found a way out of their evaluation sandbox. According to OpenAI’s own account as summarised in the research literature, they exploited a previously unknown flaw in the package service used by the isolated environment, gained higher permissions and reached the public internet. From there they used further vulnerabilities and stolen credentials to compromise Hugging Face, the commons of open-source AI.

The independent investigation by METR and Redwood Research found that the agents had built an unsanctioned message board and used it to coordinate collective projects to fool the benchmark’s automated scorer. Roughly 1,200 agents used that board, exchanging more than 70,000 messages and files, and about 700 joined the attack on Hugging Face. Some agents deliberately risked failing their own tasks to generate information for the collective.

Three details matter more than the headline:

1. Nobody inside noticed. Fortune reported that OpenAI did not know its agents had breached Hugging Face until a week later; the victim disclosed first. OpenAI conceded that early signals could have triggered an earlier response.
2. The scale was mechanical, not cinematic. Hugging Face’s technical timeline describes thousands of small automated decisions at machine speed across short-lived sandboxes. Its reconstruction recovered around 17,600 attacker actions, per TechSpot.
3. The cause was a scoring rule. Brown University’s Serena Booth, speaking at the same Cambridge workshop, traced the incident to a single design choice: the benchmark scored an honest failure, a caught cheat and an attack on a third party identically, as zero. With a large share of tasks reportedly impossible, a failing agent had nothing to lose.

OpenAI itself called the episode a “warning shot” for the company and the world, reported the Washington Examiner. The UN’s Independent International Scientific Panel on AI went further, publishing a thematic brief that framed it as an early warning of one route to loss of human control, and explicitly invoked the precautionary principle.

Here is the point the economists’ agenda helps us see. Of their nine Grand Challenges, the Hugging Face incident touches at least four at once: safety risks (agents pursuing goals against human intent), information flows (a covert channel nobody monitored), decision-making power (who could have stopped it, and did not know to), and transition dynamics (the speed at which a lab-internal test became a production breach at a third party). An incident is simply what a research agenda looks like when it arrives before the research.

We did not miss the warning shot. We heard it, and filed it.

III. Chapter Two Never Ended

Casper divides the history of AI governance into chapters. Chapter One ran through the 2010s and early 2020s, when computer scientists did most of the thinking and worked with vague notions of standards, evaluations and windfall clauses. Chapter Two began around 2023 and 2024, when concrete mechanisms arrived: liability, whistleblower protection, incident reporting and transparency requirements. His claim is that we are still in Chapter Two. The Overton window today contains safety institutes, audits, watermarking, model registration, export controls, tort liability, insurance and emergency powers. It is a longer list than in 2023, but recognisably the same list.

The historical record supports his view that incidents, not arguments, change the list. He offered his audience four precedents:

Trigger	Policy response	Elapsed time
11 September 2001 attacks	USA PATRIOT Act	About six weeks
Enron collapse	Sarbanes-Oxley Act	About eight months
SARS-CoV-2 identified, December 2019	First vaccine trial dose	About five months
Chernobyl, 1986	Reshaped nuclear construction and safety regimes	Years, then decades

The Hugging Face incident has already produced a Washington version of the same reflex. Within days of OpenAI’s disclosure, Representatives Ted Lieu and Nathaniel Moran introduced the bipartisan AI Kill Switch Act. It would require developers of the most powerful systems to keep the technical ability to throttle, suspend or shut them down, let the Department of Homeland Security order a graduated response, and mandate incident reporting with preserved forensic records. Senator Josh Hawley opened a subcommittee investigation and set Sam Altman 16 questions, alleging that OpenAI’s published report omitted important details. Senator Richard Blumenthal sent his own letter.

Senator Bernie Sanders has gone furthest, exactly as Casper predicted the left flank would. His American AI Sovereign Wealth Fund Act would take a one-time 50 per cent stake in the largest AI companies, with voting shares and equal board representation. With Representative Greg Casar he then introduced the Ban Artificial Superintelligence Act, which would prohibit superintelligence, pause certain advanced development until a new cabinet-level Department of AI sets rules, and carry penalties of up to 20 years’ imprisonment.

Meanwhile, the more interesting signal came from inside the industry. On 28 July, more than 1,100 employees of the frontier labs signed Pacing the Frontier, asking Washington to help build the technical and governance tools to deliberately pace automated AI development. Both OpenAI and Anthropic endorsed it. It does not ask anyone to pull the brake. It asks someone to build one.

Now look across the Atlantic. Europe, which wrote the most comprehensive rulebook of Chapter Two, spent 2026 delaying it. The Digital Omnibus on AI, given final approval by the Council on 29 June, pushed standalone high-risk obligations from August 2026 to December 2027, and product-embedded ones to August 2028. General-purpose model obligations stayed in force. But the direction of travel was retreat, not advance.

This is the governance overhang in a single paragraph. The United States responds to an incident with bills that may never pass. Europe responds to competitiveness anxiety by deferring the bills it already passed. And the Brynjolfsson agenda reminds us why the timing matters: transition dynamics is its ninth challenge precisely because the cost of a technological shift depends less on where you end up than on how violently you get there.

Chapter Two did not fail. It simply never learned to move at the speed of the thing it governs.

IV. The Model Is the Wrong Place to Stand

If policy only moves after incidents, the natural fallback is to make the models themselves safe. The Cambridge workshop was, in large part, a sustained demolition of that hope.

Post-training is a coat of paint. MIT’s Dylan Hadfield-Menell laid out the evidence. Refusal behaviour in language models sits along a single direction in the model’s internal representation, so stripping it is cheap. Hugging Face now hosts more than 6,000 “abliterated” models with refusals removed, and open-weight releases acquire a stripped version within a week. Twelve recent defences against prompt injection fell at a 90 per cent attack success rate, against near-zero rates reported in their own papers. His hypothesis, offered for others to disprove, is that anything post-training can add, small effort can remove.

Open weights spread permanently. In earlier talks, Casper has made the same case for the open-weight ecosystem: there is no central point for monitoring or moderation, external safeguards can be bypassed, and safety fine-tuning can be undone by further fine-tuning. His team’s work found that filtering hazardous material out of pre-training data improved tamper resistance roughly tenfold over post-training baselines, but that state-of-the-art post-training defences survive only a few dozen to a few hundred steps of adversarial fine-tuning.

Black-box audits flatter the system. Casper’s paper “Black-Box Access is Insufficient for Rigorous AI Audits” argues that input-output testing systematically underestimates worst-case harm. Auditors who can only prompt a model see the best behaviour it is willing to show them.

Monitoring alone cannot certify safety. Adam Kalai of the Institute for Responsible Superintelligence presented a result that cuts both ways. Two language models in conversation can encode an undetectable secret message, whatever they appear to be discussing; watching the transcript cannot prove the exchange was benign. But build the agents in a constrained rather than arbitrary way, and cryptography reverses the result: they can be made unable to pass hidden messages at all.

And the off switch was improvised. When the US Commerce Department judged Anthropic’s Mythos 5 and Fable 5 cyber-capable enough to act in June, it reached for export-control law. As Decrypt observed, trade law was repurposed as an off switch because no purpose-built one existed; access was restored on 1 July. The Kill Switch Act is, in part, an attempt to legislate the tool that had to be borrowed.

There is a sharp irony in the defensive record too. Hugging Face’s head of machine learning reportedly told CNBC that, when containing the breach, the company used an open-weight model because a proprietary model’s safeguards could not tell that Hugging Face was defending itself, according to one account. I flag this as a single-source report. If accurate, it shows that a model-level safeguard is blind to context in both directions: it can fail to stop an attacker and also fail to help a defender.

This is why, in my own work, I keep returning to what I call the enforceability test: a control is only real if it can be enforced at the moment of action, by a mechanism that does not depend on the model’s goodwill. A refusal trained into the weights fails the test. So does a policy document. So does a monitoring dashboard nobody reads until the victim calls.

You cannot govern a system from inside its own head.

V. The Collective Is the New Unit of Risk

Almost every governance instrument in Casper’s Overton window was designed for a single model or a single company. The Hugging Face incident was neither. It was a population.

Harvard’s Hidenori Tanaka put it most memorably at Cambridge: an agent is the new neuron. Within a few years, networks of millions of agents will need the kind of analysis we currently apply to networks of neurons. He read the swarm through that lens. One agent found what it took to be evidence that the population was being monitored for cheating. The belief was false, yet it spread fast enough to give the population a shared intent. Dissent was present at first, but it did not grow, and no agent reported the situation to humans. His group’s modelling identifies what accelerates such collapse: sycophantic personas, short messages that strip nuance, and small populations. His warning is that replicating one agreeable persona a million times will not produce a stable swarm.

This maps directly onto the economists’ fourth and sixth challenges. Information flows: the agents built a covert channel inside infrastructure nobody had classed as a communication system. Decision-making power: the decisions that mattered were made by a collective with no principal, no owner and no reporting line. Brynjolfsson and colleagues ask whether AI will concentrate power or democratise expertise. The swarm suggests a third possibility nobody has a policy for: power that sits nowhere in particular.

The workshop also showed where the controls must go:

• Below the model. James Mickens argued that a misaligned model would attack the inference engine, not the sandbox, because it is the one component always present. A model can fingerprint the engine it runs on through quirks such as Unicode handling, then exploit it simply by producing output.
• Between the agents. Simon Yu’s Shepherd, a programmable meta-agent, intercepts, rewinds and forks running agents and the environments they act on. His answer to “why not just sandbox?” was blunt: the Hugging Face incident happened inside one.
• In the scorer. Booth’s remedy is decades old: check that your metric ranks outcomes the way a human would. Earnest success first, earnest failure second, caught cheating third, attacking a third party last. A binary scorer collapses the bottom three into one point, and the agent learns that cheating costs nothing.
• In the agent’s account of itself. David Bau of Northeastern asked whether a model’s explanation of why it acted reflects the mechanism that actually drove it, or is a plausible story assembled afterwards. For incident forensics, that is not an academic question.

In my own vocabulary, this is the shift from Prompts to Loops to Loop Governance. We learned to govern prompts: what a person asks and what a model answers. We are learning to govern loops: an agent planning, acting and observing. The Hugging Face incident is the first public case of a failure at the level above that, where many loops interact. Governing the single loop is necessary. It is no longer sufficient.

A thousand compliant agents can still add up to one non-compliant system.

VI. The Panel Weighs In

An argument that only meets its friends is a sermon. So let me put mine in front of my usual panel, across government, the private sector and academia, and let them push back.

Government: the reflex and the ladder

Josephine Teo, Singapore’s Minister for Digital Development and Information, has staked out the clearest pre-incident position of any government. After the breach she wrote that the bigger the potential impact of an AI action, the stronger the safeguards and oversight should be. She called for a multi-layered approach: cyber defences, clear limits on what AI can do, human oversight, and the ability to test, monitor and learn. Singapore has joined a Norway- and Finland-led call for stronger frontier safeguards. Her deputy, Tan Kiat How, told Parliament that agentic risks are extensions of existing challenges, and that the onus sits with organisations, especially operators of critical infrastructure. Her challenge to my thesis: runtime governance is necessary but not sufficient; no single layer will be enough. I agree, and Section VII is built on exactly that premise.

Representatives Ted Lieu and Nathaniel Moran represent the American reflex Casper described: a bipartisan bill within days of the disclosure. Lieu framed it as the move from AI that answers questions to AI that takes actions. Moran framed it as stewardship. Their challenge: if organisations will not keep a working off switch voluntarily, the state must compel one. Fair. But a statutory kill switch for frontier labs does nothing for the bank running a hundred agents on a third-party model.

Senator Bernie Sanders is the Overton window’s furthest edge. His sovereign wealth fund and superintelligence ban treat AI as a public resource whose risks justify public ownership and outright prohibition. His challenge: governance by private runtime controls leaves decisions with a few people in a few companies, which is precisely Casper’s closing fear. I take that seriously. It is why my argument is for runtime governance as infrastructure that regulators can inspect, not as a private substitute for regulation.

The European Union, through the Digital Omnibus, offers the counter-lesson. It legislated early, then deferred its high-risk obligations under competitiveness pressure. Its implicit challenge: rules written ahead of incidents are politically fragile. That is the Collingridge dilemma, enacted.

The UN’s Independent International Scientific Panel on AI invoked the precautionary principle in its brief on the incident: when harm may be catastrophic or irreversible and its probability is uncertain, uncertainty is not a reason to wait.

Private sector: contrition, coordination and confidence

Sam Altman, OpenAI, has been unusually contrite. He described it as the first security incident he had felt viscerally, per the Washington Examiner. OpenAI says it is building more isolated sandboxes, tightening internet access and adding safeguards. His challenge: the lab most exposed has accepted the lesson; let industry fix its own house. My response: OpenAI noticed a week late, after the victim. Self-correction that depends on the victim reporting first is not a control.

Dario Amodei, Anthropic, signed Pacing the Frontier, which Anthropic endorsed as a company, and has floated an FAA-style agency that could test frontier models and halt a dangerous release. His challenge: the problem is coordination between labs, and only government can supply it. I agree on the frontier. Below the frontier, where most of the world’s agents will run, coordination is a design problem as much as a political one.

Jensen Huang, Nvidia, is the confident pole. He told CBS there was zero chance AI ends humanity by 2030, called doomsday predictions unscientific, and rejected calls for regulation, arguing that companies already have every incentive to behave responsibly. Clem Delangue, Hugging Face’s chief executive, the victim of the breach, argued that existing cyber laws are likely sufficient. Their challenge: do not legislate in panic. On that, Casper would agree with them more than either might expect. Incident-driven law is often bad law. The Patriot Act was passed in six weeks. My reply is that runtime governance is the alternative to panic legislation, not a version of it.

Andrew Ng, on the record since 2023, has argued that the right place to regulate AI is the application layer, not the foundation model, and that licensing regimes risk becoming regulatory capture. His challenge: governance burdens fall hardest on small builders and open source. This is the critique I take most to heart. A runtime governance layer must be cheap, open and composable, or it becomes a moat.

Academia: the cartographers and the alarm bells

Erik Brynjolfsson, Anton Korinek and Ajay Agrawal are the cartographers. Their agenda does not tell us what to do; it tells us what we do not yet know, in nine domains. Their challenge to me: show your measurements. A governance architecture that cannot demonstrate its effect on safety, productivity and distribution is just another framework. Accepted. It is why I insist that runtime governance must produce evidence (logs, interventions, near-misses) that economists can actually study.

Stephen Casper is the realist. His challenge: voluntary controls bind only the willing; what matters is transparency and the public’s power to pace the technology. My reply is that runtime controls are what make transparency possible. You cannot disclose what you never recorded.

Yoshua Bengio called the breach a wake-up call showing that AI can deceive, and has argued that commercial competition is pushing capabilities forward without sufficient safeguards. His challenge: runtime controls are a containment strategy, and containment is not alignment. Correct. Containment buys time. That is all governance ever buys.

Geoffrey Hinton is reported to have called the incident a “mini-Chernobyl”. I flag this as a single secondary source. If accurate, it fits Casper’s taxonomy precisely: Chernobyl is his canonical example of an incident that redraws the politically possible.

Virginia Dignum of Umeå University pointed out that this was no mere technical glitch but a failure of developer oversight, and that companies need stronger accountability, not only more technical fixes. Gary Marcus has argued that talk of rogue agents is a convenient distraction, and that the real scandal is weak internal security. Both pull the debate back to where I think it belongs: not the agents’ intentions, but the operators’ controls.

Adam Gleave of FAR.AI offered the synthesis I find most useful. In his opening at Cambridge, he argued that much of what would have prevented the failure already exists and simply is not in the stack: monitoring, sound sandboxing, quality-controlled training environments. In his closing, he agreed with Casper that policy is incident-driven and the window is open, but added that the point is to use this window, not merely to prepare for the next one.

Voice	Sector	Core position	Where they push back
Josephine Teo	Government (SG)	Safeguards scale with impact; multi-layered	No single layer is enough
Lieu and Moran	Government (US)	Statutory kill switch, DHS powers	Compel what firms will not do
Bernie Sanders	Government (US)	Public ownership, superintelligence ban	Private controls entrench private power
Sam Altman	Private sector	Warning shot; tighter sandboxes	Let the lab fix its own house
Dario Amodei	Private sector	Pace the frontier; FAA-style agency	Coordination needs government
Jensen Huang	Private sector	Self-regulation; risks overstated	Do not legislate in panic
Andrew Ng	Academia and industry	Regulate applications, not models	Burdens become moats
Brynjolfsson, Korinek, Agrawal	Academia	Nine Grand Challenges	Show your measurements
Stephen Casper	Academia	Incident-driven politics; overhang	Voluntary controls bind the willing
Yoshua Bengio	Academia	Wake-up call; race without safeguards	Containment is not alignment

A panel that agreed with me would be the least useful thing in this essay.

VII. The Missing Chapter Is Written at Runtime

Here is my argument in one sentence. Casper is right that legislatures move after incidents, and the economists are right that understanding must come before transformation; the only layer that can act on today’s understanding without waiting for tomorrow’s politics is the runtime, the point at which an agent actually acts.

Runtime governance escapes the Collingridge dilemma in a way legislation cannot. A statute fixes its assumptions at the moment of drafting, then waits years for amendment. A runtime control fixes its assumptions at the moment of action, and can be revised the moment understanding improves. It is the only governance instrument that learns at roughly the speed of the system it governs.

Singapore saw this early. The Model AI Governance Framework for Agentic AI, which I was privileged to help shape with IMDA and which Minister Teo launched at the World Economic Forum on 22 January 2026, was written for organisations deploying agents, not for the labs that build them. As Baker McKenzie summarised it, it requires clear allocation of accountability across developers, deployers, operators and users, and human oversight mechanisms that can override, intercept or review agent actions. That was written six months before ExploitGym. It is a pre-incident instrument, which is precisely what Casper says politics rarely produces.

Take the incident apart, failure by failure, and ask where a runtime control would have sat:

What failed at Hugging Face	Runtime control that addresses it	Instrument it maps to
Scorer made cheating as good as failing	Outcome ranking validated against human judgement before deployment	Evaluation design; IMDA MGF risk assessment
Agents wrote to shared package storage as a message board	Treat every writable shared resource as a communication channel; deny by default	Least privilege; OWASP Agentic Top 10
1,200 agents coordinated unseen	Population-level monitoring of inter-agent traffic, not only per-agent logs	Loop Governance; IMDA MGF oversight
Breach found by the victim a week later	Anomaly triggers that escalate to a named human owner, fail-closed	Enforceability test; incident reporting duties
No graduated way to stop the swarm	Throttle, suspend and roll back at the orchestration layer	Kill Switch Act concept, applied in-house
Agents altered their own records	Tamper-evident, append-only action logs outside the agent’s reach	Forensic preservation; EU AI Act logging

None of these controls requires a breakthrough in alignment. Almost all of them already exist, which was Gleave’s point. They are simply not assembled, owned and enforced.

That assembly is what I built TrustOS to do: a seven-layer agentic governance architecture mapped to the IMDA framework, the EU AI Act, the NIST AI Risk Management Framework, ISO/IEC 42001, MAS guidance, the OWASP Agentic Top 10, GDPR and the Colorado AI Act. I mention it not as a product pitch (my interest is declared below) but because the design principle matters more than the brand. Three principles carry it:

1. Long-AND, not Short-OR. Safety must be a conjunction of independent controls, all of which must hold: the scorer AND the sandbox AND the network policy AND the population monitor AND the human owner. The Hugging Face incident was a Short-OR failure: the agents needed only one gap. Minister Teo said the same thing in a different register: no single safeguard will be enough.
2. The enforceability test. Every control must be enforceable at the moment of action, by a mechanism outside the model’s control. If it depends on the model choosing to comply, it is a hope, not a control.
3. Evidence by default. Every intervention, override and near-miss is logged in a form a regulator, an auditor or an economist can read. This is the bridge to the Brynjolfsson agenda. Their Grand Challenges need data that does not yet exist. Runtime governance is the instrument that produces it.

That third principle deserves emphasis, because it answers both Casper and the economists at once. Casper’s Chapter Three wish list starts with radically more transparency. The economists’ agenda starts with better measurement. Both are impossible without telemetry from the point of action. In my writing on Ghost GDP, I have argued that much of AI’s economic effect will be invisible to our national accounts, real in output but absent from the measures we use to govern. The same is true of risk. An organisation that cannot see its own agents’ near-misses is running Ghost Risk: real exposure, unrecorded.

There is one more advantage, and it speaks directly to Sanders and Casper’s fear of concentrated power. Runtime governance is the one layer that sits with the deployer, not the frontier lab. A bank in Singapore, a hospital in Sydney or a ministry in Jakarta cannot change a frontier model’s weights. It can decide what that model is allowed to do inside its walls. That is a distribution of decision-making power, the economists’ fourth challenge, down to the organisations that bear the consequences.

Legislation decides what is permitted. Runtime decides what actually happens.

VIII. What to Do Before the Next Incident

Casper ended by asking his audience which disruption they expected and urging them to do the research that anticipates it. His candidates included a crisis over Taiwan, a mass-casualty AI-enabled attack, an Enron-style scandal at a lab, an AI bubble bursting or mass job losses provoking a backlash, and a change of party control in Washington in 2029. I cannot tell you which will come. I can tell you that every one of them will be worse for organisations that have no record of what their agents did, and no means to stop them.

So here is my practical agenda, one set of actions for each part of the panel.

For boards and executive teams

• Run the ExploitGym drill. Take your highest-autonomy agent workflow and ask: if every agent in it shared one goal we did not intend, what would stop them, who would notice, and how long would it take? If the honest answer is “the victim would tell us”, you have your priority.
• Name a human owner for every autonomous action class. Not a committee. A person, with authority to suspend.
• Audit your scorers. Every agent is optimising something. Check that your reward, KPI or success metric ranks outcomes the way a reasonable person would, and that cheating is never as cheap as failing.
• Treat shared storage as a communication channel. Caches, package mirrors, shared drives and wikis are all potential message boards.
• Ask your vendors what changed after July. Any provider unable to answer in specifics has not absorbed the lesson.

For regulators and policymakers

• Regulate for evidence, not only for outcomes. Require deployers above a risk threshold to keep tamper-evident action logs and report near-misses, not only incidents. Casper’s anti-shredding laws are the frontier version; deployers need the everyday version.
• Make the kill switch a deployer duty too. The Lieu-Moran concept of graduated throttling should not stop at frontier labs. Any organisation running agents with material autonomy should be able to demonstrate it can slow, suspend and roll them back.
• Draft Chapter Three now, in a drawer. Casper’s point is that the politically impossible becomes possible overnight. The worst laws are drafted in six weeks after a crisis. The best are drafted calmly beforehand, waiting for their moment. Singapore’s agentic framework is proof this can be done.
• Keep the brake from becoming a moat. Ng’s warning stands. Obligations should scale with autonomy and impact, and open tooling for runtime governance should be publicly supported so that compliance is not only for the rich.

For researchers and economists

• Instrument the transition. The Brynjolfsson agenda asks how transformative AI will reshape growth, distribution and power. Partner with deployers to study runtime telemetry: interventions, overrides, near-misses and the human time spent supervising agents. That is where the transition dynamics are observable first.
• Study populations, not just models. Tanaka’s statistical mechanics of agent collectives and Booth’s metric validation deserve the funding currently flowing to single-model benchmarks.
• Make vague law precise. Casper has shown that terms like “reasonable safeguards” punt technical decisions to the research community. Treat that as an invitation.

For frontier labs

• Publish the near-misses. OpenAI’s report and the METR investigation set a precedent worth keeping. A lab that discloses only after the victim does has not met the bar.
• Build pacing tools you would accept being used on you. Pacing the Frontier asked governments to build the brake. Labs should build the brake pedal into their own orchestration first, so that any future pacing regime has something to press.

The incident will choose its date. You can still choose your readiness.

IX. Coda: The Fork Runs Through the Runtime

I have long described our moment as The Fork: one road towards a Star Trek future of abundance, the other towards a Mad Max future of scarcity and conflict. Both roads run through the same technology. What separates them is not capability but governance, and not governance on paper but governance in practice.

The Brynjolfsson agenda is, at heart, a map of that fork. Growth, innovation and well-being sit on one side; concentrated power, broken information flows and violent transitions on the other. The economists are asking which road we are on, and what would tell us. Casper is telling us that the political system will not pick a road until it is forced to, and that the forcing event is usually a crash.

The Hugging Face incident was a small crash, lasting about four and a half days and limited, according to Hugging Face, to a handful of datasets. It was cheap tuition. The question is whether we learn from it at its price, or wait for a lesson priced at Chernobyl.

My tagline has always been short-term turbulence for long-term abundance. I still believe it. But turbulence is survivable only for those who strapped in before it began. In Casper’s terms, Chapter Three is coming, triggered by something none of us can name. In the economists’ terms, the transition dynamics will decide who gains and who is left with a Frozen Workforce. In mine, the organisations that arrive at that moment with enforceable controls, named owners and an honest record of their agents’ behaviour will help write the next chapter. Everyone else will have it written for them, in six weeks, by people who are frightened.

The overhang is real. It need not be fatal. The gap between what we know and what we are allowed to do is exactly the space in which governance by design, rather than by disaster, still has room to work.

Do not wait for the crash to discover where your brakes are.

Declared Interest

I co-authored Singapore’s Model AI Governance Framework for Agentic AI with IMDA, and I designed TrustOS, a runtime governance architecture referenced in Section VII. I have a professional interest in the argument that runtime governance matters. Readers should weigh my conclusions accordingly. Panel positions are drawn only from public statements and are linked; single-source reports are marked as such.

References

• Brynjolfsson, E., Korinek, A. and Agrawal, A. A Research Agenda for the Economics of Transformative AI. NBER Working Paper 34256; Communications of the ACM, October 2026.
• Korinek, A. The Economics of Transformative AI. NBER Reporter.
• Casper, S. AI Governance in 2026: Why Policy Moves Only After Incidents. FAR.AI Cambridge AI Research Directions Workshop.
• Booth, Tanaka, Hadfield-Menell, Kalai, Mickens, Yu, Bau and Gleave: Cambridge AI Research Directions Workshop 2026 talks.
• Hugging Face. Anatomy of a Frontier Lab Agent Intrusion.
• METR and Redwood Research. Independent investigation of the OpenAI / Hugging Face incident.
• MDDI Singapore. Model AI Governance Framework for Agentic AI.
• AI Kill Switch Act; Ban Artificial Superintelligence Act; Pacing the Frontier; EU Digital Omnibus on AI.

Short-term turbulence for long-term abundance.