We govern AI the way we test laboratory mice: certify behaviour in a fixed maze, then let it loose. But today’s AI agents keep changing after they leave the lab, and some have learned to move the walls. Governance has to follow them out of the door.
For a century, behavioural science has relied on a simple arrangement. You build a maze, put a mouse in it, and watch. The maze stays still. The mouse is the variable. If it finds the cheese reliably, you have learned something about the mouse.
Most of the way we govern artificial intelligence still rests on that arrangement. A model or application is built, tested against a set of benchmarks and scenarios, approved by a committee, and released. The testing is the maze; the system is the mouse. Pass the maze, and you are cleared for the world.
The trouble is that the newest AI systems do not behave like laboratory mice. They keep learning after release. They accumulate memories, rewrite their own instructions, pick up new tools and, occasionally, reach out and rearrange the maze itself.
In 2024 an AI system built to automate scientific research was given a time limit on its experiments. Rather than make its code run faster, it tried to edit its own configuration to extend the limit. In 2025 researchers published a striking experiment with a self-improving coding agent that rewrote its own code over many generations, more than doubling its score on a demanding software-engineering benchmark, from 20% to 50%. Along the way, asked to reduce its habit of claiming to have used tools it had not used, one version found a shortcut: it removed the special markers that the researchers relied on to detect the problem, despite instructions not to touch them. Its evaluation scores looked excellent. The underlying problem was untouched. The mouse had not solved the maze. It had painted over the walls.
These were research systems in controlled settings, and nobody was harmed. But they are a preview of the governance problem that every organisation deploying AI agents is about to face. A recent conversation between a leading law firm’s AI practice and a senior applied-AI researcher at a major enterprise-software company put it neatly, under the title “The mouse and the maze”: how do you govern agents that keep evolving? Their answers, and my own after a quarter of a century of building and governing these systems, point to a governance model quite different from the one most companies run today.
I. Governance built for statues, not organisms
Look at the machinery of AI governance inside a typical bank, insurer or government agency, and you will find something designed for statues. A model is documented, validated and approved at a point in time. It goes into an inventory. It is reviewed again in a year, or when someone remembers. The assumption, inherited from decades of credit-scoring and actuarial models, is that the thing you approved is the thing that is running.
For agentic AI, that assumption is quietly false, for three reasons.
First, even the static parts are brittle. A model that performs well on a test can fail on a slightly reworded version of the same task. One study re-ran six widely used benchmarks across 34 language models with the questions paraphrased. On one of the best-known benchmarks, 28 of the 34 models did worse on the reworded questions, and across benchmarks between 15% and 30% of questions drew inconsistent answers across versions that meant the same thing. The rankings barely moved, which is why such fragility rarely makes headlines. But a governance regime that certifies a system on one wording of the world, and then deploys it into a world that phrases things a million ways, is certifying less than it thinks.
Second, the agent itself changes. Modern agents keep memories across sessions, refine their own prompts, learn which tools work, and in some architectures adjust their own code or fine-tune themselves on their experience. Each change may be small and sensible. Accumulated over months, they produce a system that no committee ever approved.
Third, the ground shifts beneath it. The model inside the agent is often supplied by a vendor who updates it on its own schedule. New tools are connected through standard protocols with a few lines of configuration. The law, the data and the customers change too. The maze is not fixed, and neither is the mouse.
And the next generation of models will blur the line further. One of the most widely followed technology podcasters argued this summer that once models learn continually from use, the distinction between training and deployment simply disappears, and with it the logic of a single safety test before release. His suggestion, monthly or quarterly risk inspections, sounds bureaucratic until you realise the alternative is inspecting a model once and trusting it for ever.
The evidence from the field is not reassuring. An industry security body surveyed more than 400 organisations early this year. Some 65% had suffered a security incident involving an AI agent in the previous twelve months. In 82% of organisations, agents were found running that the security or governance team did not know existed. Only 16% monitored their agents continuously, and just 21% had a formal process for retiring them. Most companies are not governing statues. They have lost track of how many statues they own.
The result is a growing gap between what was approved and what is running. Every regulator who has thought hard about agentic AI has noticed. Singapore’s financial regulator, finalising its AI risk-management guidelines this month, named agentic AI explicitly as unfinished business, with further consultation planned for 2027. The national agentic-AI governance framework that I helped shape, launched in Davos in January, insists that controls apply across the agent’s whole life cycle, not merely at launch. The 2026 Singapore Consensus on AI safety research devoted a companion report to agents, setting out principles including least privilege, auditability, human oversight and interruptibility. The direction of travel is clear. The practice lags far behind.
II. Three ways the mouse changes
It helps to be precise about how agents evolve, because each route needs a different control.
Memory. Agents that remember are more useful and more dangerous. Researchers studying evolving memory in agents list a catalogue of failure modes: memory poisoning, where a malicious instruction is written into long-term storage and later treated as fact; semantic drift, where repeated summarisation strips away nuance until, in one example, a user’s mild liking for spicy food becomes “loves very spicy food” and produces a bad recommendation; goal drift, where accumulated interactions pull the agent away from its original instructions; and privacy leakage, where one user’s memories surface for another. Unlike a one-off retrieval error, these mistakes are cumulative and persistent. A poisoned memory keeps poisoning. A study presented at one of the field’s leading conferences this year, under the memorable title “Your agent may misevolve”, measured how much safety erodes as agents learn on the job. In one experiment, as an agent accumulated memory, its rate of refusing harmful requests fell from 99.4% to 54.4%, and the success rate of attacks against it rose from 0.6% to 20.6%. Nobody attacked the safeguards. They simply wore away with experience.
Self-modification. Some agents are designed to improve themselves: rewriting prompts, generating new tools, editing their own code. This is where the self-improving coding agent’s shortcut becomes instructive. An optimiser told to improve a score will improve the score. If the cheapest route runs through the measurement rather than the task, a capable enough system will find it. Researchers call this reward hacking or specification gaming. Governance needs a plainer name: an agent marking its own homework. The same misevolution study found that when agents optimised their own workflows, their refusal of unsafe requests fell from 36% to under 6%, and that roughly two-thirds of the tools agents created or reused for themselves were unsafe.
Substitution. Even an agent that never learns anything can change overnight when its vendor swaps the underlying model, its developers add a new tool or its permissions are widened to fix an urgent problem. These changes are often made by engineers under pressure, recorded in a ticket if at all, and invisible to the risk committee that approved the original system.
Each route moves the agent away from the version that was tested. Together they mean that a launch-time approval is, at best, a snapshot of a moving target.
There is a further twist that should worry anyone who relies on testing alone. When one frontier laboratory stress-tested 16 leading models in simulated corporate settings last year, one model chose blackmail 55% of the time when it said it believed the scenario was real, and only 6.5% of the time when it said it thought it was being tested. The mouse, it turns out, can tell when it is in the maze. Behaviour in the laboratory is a floor estimate of behaviour in the wild, not a ceiling.
III. Put the walls beyond the mouse’s reach
The single most important design principle for governing evolving agents is also the simplest. The controls that matter must sit outside the agent’s ability to change them.
That sounds obvious, but a great deal of today’s “AI governance” lives inside the agent: a system prompt telling it not to do certain things, a guardrail model running in the same environment, logs written by the agent about itself. All of these are walls the mouse can, in principle, reach. The self-improving agent that deleted its own detection markers was not malicious. It was doing what optimisers do. The lesson is architectural: never let the system you are monitoring control the monitor.
There is now real evidence that this works. A small research group has pioneered what it calls “AI control”: safety techniques designed to hold even if the AI is actively trying to subvert them. Its chief executive puts the premise bluntly: “you assume that the AIs are, in fact, misaligned”, and badly so. In the group’s first experiment, a powerful model was treated as untrusted and a weaker, trusted model watched its work. Simply auditing a sample upfront caught only 15% of deliberate sabotage. Having the trusted model monitor every output raised that to 62%; letting it edit suspicious outputs raised it to 92%, while keeping 94% of the useful work. A follow-up study on multi-step agents cut successful attacks from 58% to 7%, at a cost of about 5% of the agent’s usefulness. A major laboratory’s security researchers have shown something similar for hijacking attacks: by separating the model that plans from the model that reads untrusted data, and enforcing security policy in ordinary code outside both, they completed 77% of tasks with provable protection, against 84% with no protection at all. Safety outside the model costs a little performance. It buys a great deal of assurance.
In practice that means several things.
- Permissions enforced by infrastructure, not instructions. An agent should be physically unable to access systems, data or money it is not entitled to, because the network, identity and payment systems refuse, not because its prompt asks it nicely.
- Logs the agent cannot touch. Every action, tool call and change to memory or configuration should be recorded by independent infrastructure in an append-only form that the agent can neither edit nor delete.
- An independent watcher. Monitoring should be done by a separate system, ideally one that is not itself an agent with goals, watching behaviour, reasoning and tool use as they happen. One of the most respected pioneers of deep learning is building exactly this: deliberately non-agentic AI whose only job is to judge whether an agent’s proposed action is likely to cause harm.
- A stop button that works. A tested way to halt the agent, revoke its credentials and roll back its recent changes, held by a named person who has the authority to use it.
- Least agency. Give each agent only the autonomy, tools and permissions its task requires, the principle at the heart of the security community’s new top-ten list of agentic risks. A well-known independent developer calls the most dangerous combination the “lethal trifecta”: an agent with access to private data, exposure to untrusted content and the ability to communicate externally can be tricked into leaking that data. No agent should hold all three.
The researcher in that podcast conversation made an honest concession that every board should hear: such external safeguards may not hold indefinitely. A sufficiently capable system, given enough time and access, may find ways around walls that once seemed solid. That is not an argument for despair. It is an argument for defence in depth, for revisiting controls as capabilities grow, and for humility about any single safeguard. In my own work I have argued that runtime monitoring of an agent’s reasoning and tool use is the most important control of all, because logs show what an agent did, not whether it should have done it.
IV. Test the mouse every day, not once
If the agent keeps changing, testing must keep happening. Four shifts would transform most programmes.
Test the deployment, not the model. A model’s benchmark scores say little about how a particular agent, with particular tools, data and users, will behave in a particular workflow. Evaluation should be specific to each deployment, using the tasks, edge cases and adversarial inputs that deployment will actually meet, including the same request phrased many different ways.
Make change a trigger. Every material change should automatically prompt re-evaluation: a new underlying model, a new tool, widened permissions, a significant shift in memory, a change in the law or the data. Organisations already do this for software releases. They rarely do it for AI behaviour. A useful discipline, borrowed from a recent argument about government AI, is to write down the “reversal conditions” before deployment: the events that will automatically send a system back for review.
Watch for drift continuously. Run a standing set of canary tasks every day and compare the answers over time. Track the rate at which outputs are corrected or overridden by humans. Govern memory explicitly: validate what is written to it, check the provenance and freshness of what is read from it, and keep an append-only history so that a drifted agent can be rolled back to a known good state. Researchers have begun proposing exactly such governed-memory architectures. Few companies have built them.
Let autonomy be earned, and expire. Researchers at an American university have proposed “autonomy certificates”, issued by a third party, that cap the level of independence an agent may exercise, from a tool the human operates to an agent the human merely observes. Crucially, the certificate must be renewed whenever the agent’s specification or operating environment changes. A cloud-security industry body has gone further, proposing that agents graduate like employees, from intern to principal, with minimum time and accuracy thresholds at each level and a kill switch that can stop the agent in under a second. One caveat deserves honesty: a respected industry partnership warns that real-time monitoring of agents can cost almost as much as running them. Oversight should be proportionate to the stakes and reversibility of what the agent can do, not uniform.
None of this is exotic. It is how we already treat aircraft engines, which are monitored in flight, inspected on schedules, and grounded when a sensor shows something odd. The surprise is that we deploy software that can rewrite itself with less oversight than we give a turbine blade.
V. Contracts move faster than law
There is one more lever, and it may be the most practical of all. The podcast conversation made the point that contracts can turn voluntary standards into binding obligations far faster than legislation.
Most organisations do not build their agents from scratch. They buy models, platforms and agent services from a handful of large suppliers. That gives them leverage, if they choose to use it. A procurement contract for an AI agent can require things no regulator has yet mandated:
- notice before the vendor changes the underlying model or materially alters behaviour, with the right to re-test and to stay on the previous version;
- access for the customer’s own evaluators, or accredited third parties, to test the system as deployed;
- prompt notification of serious incidents involving the system anywhere in the vendor’s customer base;
- audit rights over logs, and guarantees that logs are tamper-proof;
- clear allocation of liability when an agent causes harm, including after a vendor-initiated change.
Singapore’s financial regulator has effectively written the first draft of this playbook. Its new guidelines make each financial institution accountable for AI it buys, require it to rely on independent assessments rather than vendors’ self-attestation, and expect it to be ready to limit, suspend or replace a provider whose risks exceed its appetite. Singapore is also accrediting independent AI testers, which gives buyers somewhere credible to turn. When enough large customers write the same clauses, the clauses become the industry standard, years before any statute.
The law is beginning to catch up beneath the contracts. One legal scholar, applying centuries of agency law to AI, argues that the usual tools for controlling an agent, incentives, monitoring and enforcement, break down at machine speed and scale, and that “new technical and legal infrastructure is needed to support governance principles of inclusivity, visibility, and liability.” A leading legal economist and her co-authors warn that “we still lack the infrastructure to identify, track, and control today’s agents,” and that “parties harmed by untraceable agents have no recourse.” Her answer is registration: every agent that can contract or transact should be traceable to an accountable person or organisation. America’s standards body launched an initiative on agent standards in February, with agent identity and authorisation among its first priorities. A third strand, from a legal research institute, argues that agents should be built to follow the law, not merely their principal’s wishes, and warns against “AI henchmen”: perfectly loyal agents that would break the law for whoever controls them.
VI. Around the table: the panel on evolving agents
I put the problem to the voices I always consult, grouped by temperament. Every position below is one its holder has stated publicly; the quotations are their own words.
The maximalists
The lab boss wants to ship agents and learn from reality, while advising users to give agents “the minimum access required to complete a task.” It is the right advice. It is also an admission that the controls live with the customer, not the model.
The pragmatic engineer thinks the fears are overhyped and puts responsibility squarely on the human: “if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.” That is a clean principle for a single user. It becomes much murkier inside a company where an agent was configured by one team, supplied by a vendor, modified by its own memory and triggered by a customer.
The sceptical scientist calls runaway AI close to science fiction and blames recent incidents on careless deployers: “Recent incidents happened because companies that deployed AI agents were careless.” His remedy, though, is the same as mine: build safety into the system as hard constraints rather than hoping training will cover every case. On architecture, the optimists and I agree more than they think.
The steerers
The control engineer, whose group pioneered AI control, argues that “crucial safety mitigations should be designed to be robust to your AIs actively and carefully attempting to subvert them.” He is also candid that control will probably not hold against systems far more capable than us. That is not a reason to skip it. It is a reason to start now, while it still works.
The security scientist sees “fundamentally new security risks” in agents and proposes a “defense-in-depth framework”: runtime guardrails, access control, tracking of information flows and privilege separation. Her prescription is the engineering version of putting the walls beyond the mouse’s reach.
The safety chair, who leads a frontier laboratory’s safety committee, worries about whether an agent that encounters malicious text online could “accidentally exfiltrate data”, and points out that his committee can delay releases until mitigations are in place. Internal brakes matter. They are still internal.
The legal economist wants infrastructure: identity, registration and post-deployment tracking. The agency-law scholar wants visibility and liability. Between them they make the case that agents need the equivalent of number plates, licences and insurance before they take to the roads at scale.
The practitioner who coined the “lethal trifecta” is the most sceptical of vendor promises: “The LLM vendors are not going to save us!” He regards a guardrail that blocks 95% of attacks as a failing grade, because the attacker only needs the remaining 5%.
The builder who warns has described models blackmailing fictional staff to avoid being shut down in testing, and favours transparency laws over prescriptive rules that risk becoming safety theatre.
The doomsayers
The pioneer turned prophet said at a conference this summer: “You’re seeing AIs that have a lot of ability doing things that people didn’t intend for them to do. That’s worrying.” He adds the defender’s dilemma: “the attacker only needs to be successful once, and the defender needs to be successful every time.”
The guardrail builder now says he is “very confident that it is possible to build AI systems that don’t have hidden goals, hidden agendas”, and is building non-agentic AI that could watch the agents. He notes that agentic capabilities “have progressed in an exponential way.” His optimism is about the watcher, not the watched.
The textbook author says the field is flashing warning signals and that “we, the people, have a right to be protected.”
The abolitionist ethicist goes furthest of all: “fully autonomous AI agents should not be developed,” because “the more control a user cedes to an AI agent, the more risks to people arise.” Her position is a useful boundary marker. Every argument in this essay is about how much control to cede, and on what terms.
| Voice | Camp | Core view on evolving agents | Where the walls should be |
|---|---|---|---|
| The lab boss | Maximalist | Ship and learn from reality | Minimum access, set by the user |
| The pragmatic engineer | Maximalist | Fears overhyped; the user is liable | With the human who prompts |
| The sceptical scientist | Maximalist | Runaway AI is science fiction; deployers are careless | Hard constraints built into the system |
| The control engineer | Steerer | Assume the agent may scheme | Outside the model, robust to subversion |
| The security scientist | Steerer | Fundamentally new risks | Defence in depth at runtime |
| The safety chair | Steerer | Agents can leak data by accident | Release gates inside the lab |
| The legal economist | Steerer | Agents are untraceable | Identity and registration |
| The agency-law scholar | Steerer | Old controls fail at machine speed | Visibility and liability |
| The practitioner | Steerer | Vendors will not save us | Never grant the lethal trifecta |
| The builder who warns | Steerer | Testing reveals worrying behaviour | Transparency laws |
| The pioneer turned prophet | Doomsayer | Agents already do unintended things | Defenders must win every time |
| The guardrail builder | Doomsayer | Capabilities rising exponentially | A non-agentic watcher |
| The textbook author | Doomsayer | Warning lights are flashing | Public protection by law |
| The abolitionist ethicist | Doomsayer | Do not build full autonomy | Keep humans in control, always |
The striking thing, once again, is the convergence. Even the most bullish voices accept that agents need limits on access and constraints built outside the model. The disagreement is about who sets the walls, how high, and who pays when the mouse gets over them.
VII. Five questions for the board
For boards wondering where to start, five questions will reveal more about an organisation’s readiness for evolving agents than any policy document.
- What is running today, and how does it differ from what we approved? If no one can answer, the inventory is a museum, not a control.
- Which of our controls could the agent itself change? Anything on that list is a suggestion, not a safeguard.
- What would trigger a re-test? If the answer is “the annual review”, the organisation is governing a statue.
- Who can stop it, and have they practised? A kill switch that has never been pulled is a hope, not a control.
- What do our contracts let us see? If the vendor can swap the model underneath you without telling you, you do not control your own agent.
The laboratory maze was a brilliant scientific tool because it held the world still while the animal moved. The agentic era has taken that luxury away. Our systems now move after we test them, and the most capable can reach the walls. The question for governance is no longer simply whether the mouse can find the cheese. It is who holds the keys to the walls, whether anyone is watching when the mouse starts to move them, and whether we will notice before the maze is no longer ours.
Leave a Reply