.. Runtime Governance, or Nothing!
On 22 July, the Financial Stability Board closes consultation on its Sound Practices for the Responsible Adoption of Artificial Intelligence. Eleven days later, on 2 August, the EU AI Act’s high-risk obligations bite. Between those two dates, the governance regime for autonomous AI in the world’s financial system is going to set in a shape that will be hard to move for years.
Right now that shape has a hole in it. The hole has a name, a growing literature, four independent frameworks, and – as of last month – a live proof from the frontier of AI capability that it is the single thing standing between a dangerous model and a deployable one.
The hole is runtime governance: controls that evaluate what an AI agent is about to do, at the moment it proposes to do it, before it does it.
We do not have it at scale. We are deploying agents anyway. This essay is about why that gap is not a technical detail, why closing it is the enabling condition for AI to scale rather than a brake on it, and what specifically breaks if we don’t.
Part I: The thing that changed
For most of the last decade, AI in regulated industry was decision support. A model produced a score, a flag, a recommendation. A human looked at it and acted. The model’s output and the world’s state were separated by a person.
That boundary is gone. The Monetary Authority of Singapore put it plainly in July: institutions are now deploying AI agents that act autonomously, without per-decision human review, across the full range of financial operations.
This is not a difference of degree. It is a change in what kind of object you are governing.
Some of Singapore’s brightest AI researchers; including some who wrote Singapore’s financial-sector AI risk management guidelines (MAS AI RG) while supervising models at MAS – made the distinction as sharply as anyone I’ve read. A Black-Scholes model, they note, is written down. You can read its assumptions off the page and argue with them. A learned model is not written down in that sense: its behaviour lives in weights shaped by data, not in an equation you can inspect. “You can no longer read the model; you can only interrogate it.”
And with agents, even interrogation is insufficient – because what you need to know is not what the model would say but what the system will do, along a path that does not exist until it is walked.
The workflow is what you allow the agent to do, designed up front: the branches, the approval gates, the points where it must stop and ask. The trajectory is what it actually did, visible only afterwards. Most of the time the two match. The risk lives in the times they don’t.
Everything below follows from that gap.
Classical model risk management – SR 11-7 and its descendants – validates an artefact before deployment and audits outcomes after the fact. Both of those disciplines look at the workflow. Neither looks at the trajectory while it is being walked. Szpruch, Sudjianto, Bhatti and Ang put the point in their April working paper: agentic systems break MRM because outcomes arise from execution trajectories rather than a stable input-output mapping, and the material failures — unsafe tool use, skipped approvals, privacy breaches, uncontrolled side effects — are process failures that only emerge at runtime.
The regulators know. MAS’s SAFR white paper names the gap explicitly: model risk management validates a system before deployment; audit examines what happened afterward; neither catches a problematic agent decision in the moment before it executes.
That moment is where governance has to move. That is the whole argument.
Part II: Everybody is converging on the same architecture — and nobody is talking to each other
Here is what should be reassuring and is actually alarming. Four communities, working almost independently, have arrived at nearly the same design.
The bank. MI9, from Charles L. Wang, Trisha Singhal, Ameya Kelkar and Jason Tuo at Barclays Model Risk Management (with Columbia), is the most technically complete runtime framework in the open literature. Six mechanisms: an Agency-Risk Index scoring autonomy, adaptability and continuity to set governance intensity; an Agentic Telemetry Schema capturing cognitive events (goal.set, memory.read) that infrastructure monitoring cannot see; Continuous Authorization Monitoring that revokes permissions mid-execution when an agent’s goal shifts; a finite-state Conformance Engine catching temporal policy violations across multi-step workflows; goal-conditioned drift detection; and four-level graduated containment that constrains an agent without killing it. Their motivating example is exact and chilling: a trading agent cleared for small retail trades escalating to multi-million-dollar institutional transactions while operating entirely within its original static permissions.
The academy. Szpruch (Edinburgh / Alan Turing Institute), Sudjianto (formerly Wells Fargo’s head of model risk), Bhatti and Ang land on: governance-semantic telemetry, continuous authorisation, temporal policy conformance checking, drift monitoring over trajectory health, tier-based containment. Their unit of governance is the capability – a bounded set of actions with explicit authority, constraints, and evidence requirements attached. “Retrieve a customer’s transaction history” is one capability. “Move money between accounts” is another, with far more authority and a far higher bar of evidence.
The regulator-convened industry. MAS’s SAFR (Safeguards for Agentic Finance at Runtime), written with Ant International, Circle, HSBC, J.P. Morgan Chase, Manulife, Mastercard, OCBC and Visa. Four components – Agent Identity, Controls Repository, Disposition Engine, Audit Log – communicating through a Governance Envelope that packages every proposed action with its trace and context. Four dispositions: Deny, Escalate, Auto-Execute, Observe. And its thesis in one line: “no agentic action reaches execution without having been declared, authorised, and assessed.”
The government lab. Shaun Khoo, Jessica Foo and Roy Ka-Wei Lee at GovTech Singapore built the Agentic Risk & Capability framework, arriving — from an entirely different direction – at capability as the right unit of governance.
Four groups. One architecture: observe the trajectory → authorise per action → check the sequence → detect drift → contain gradually.
And then, in March, Maurits Kaptein and colleagues wrote the paper that gives it a formalism. Runtime Governance for AI Agents: Policies on Paths models governance policies as predicates over execution paths – and proves that the two things everyone actually deploys are degenerate cases. System prompts shape the distribution over paths without ever evaluating them. Static access control evaluates only policies that ignore the path entirely. Runtime evaluation, they argue, is the general case, and it is necessary for any policy that depends on what the agent actually did.
That is not a framework. That is a theorem-shaped claim, and it says the industry’s current controls are not merely weak. They are the wrong class of object.
Part III: The frontier lab already proved it – and nobody in finance has noticed
Now the part that should end the argument.
In June, Anthropic released two models. Claude Mythos 5 and Claude Fable 5 are the same underlying model. Identical weights. Identical capability.
Mythos 5 goes to roughly 150 vetted organisations under Project Glasswing — cyber defenders, critical infrastructure providers. Fable 5 goes to the public.
The entire difference is a runtime layer. Anthropic’s own description: when Fable’s classifiers detect a request related to cybersecurity, biology and chemistry, or distillation, the response is automatically handled by Claude Opus 4.8 instead— a weaker, older, more thoroughly understood model.
Look at that architecture with the AI-safety literature in hand and you will recognise it immediately. An untrusted capable policy. A trusted weaker model. A monitor routing between them, per request, at runtime. That is defer-to-trusted, the central protocol from Ryan Greenblatt, Buck Shlegeris and colleagues at Redwood Research — introduced in AI Control: Improving Safety Despite Intentional Subversion — now running in production, at frontier scale, serving millions of users.
And here is the sentence that matters most for anyone who thinks governance slows things down.
Anthropic’s stated position is that Mythos-class models “have reached a threshold where they present significant risks,” which is why the first one went only to a limited group; and that general release became possible only once the safeguards were robust enough.
The runtime layer was not a tax on shipping. The runtime layer was the shipping mechanism. Without it, Mythos-level capability reaches a hundred and fifty vetted organisations. With it, it reaches everybody.
Generalise it, because this is the argument the finance sector needs and does not have:
Pre-deployment assurance is a gate. It returns a binary on a fixed artefact, and the only lever it offers is shipor don’t ship.
Runtime governance is a dial. It decouples capability from permission, so that a system too dangerous to release unconditionally can be released conditionally.
As capability rises, the gate forces an increasingly costly binary. The dial does not. Runtime governance is not a brake on scaling. It is the only known mechanism by which frontier capability scales to general availability at all.
Ang argues this abstractly — “done at the right unit, [risk management] is exactly what lets you reuse.” Fable is the existence proof, and it is a far sharper one than anything in SAFR’s own case studies.
The three-week incident that should be a case study in every MRM function
It gets better, because the thing then broke, in public, and the repair tells you where the control surface actually lives.
- 9 June: Fable 5 and Mythos 5 released.
- 12 June: US export controls, after the government learned of a report in which Amazon researchers found a way to bypass Fable 5’s safeguards — prompting it so that it identified software vulnerabilities, and in one case produced code demonstrating an exploit.
- 30 June: controls lifted.
- 1 July: Fable 5 redeployed, with updated safeguards.
The model was not retrained. The weights were not touched. The safeguards were revised, and the system was back inside three weeks.
Now hold that against what the US banking agencies did on 17 April, when the Federal Reserve, OCC and FDIC replaced SR 11-7 — after fifteen years — with SR 26-2, and explicitly declared generative and agentic AI “novel and rapidly evolving” and out of scope.
Ang’s reaction, and mine: the effect is that the fastest-moving, least-understood models are the ones left with the least guidance. But the deeper problem is not coverage. It is that model risk guidance is calibrated to a remediation loop — validate, revalidate, redocument — that no longer describes how these systems are actually fixed. When the failure is at runtime, the fix is at runtime, and the clock is days.
One more thing fell out of that incident, and it deserves attention. Anthropic’s own testing confirmed that less capablemodels — Opus 4.8, GPT-5.5, Kimi K2.7 — could identify the same vulnerabilities Fable had found. The marginal uplift was near zero. Regulatory action was taken on absolute capability, not on delta. Risk calibration — how much moredangerous is this than what already exists? — is precisely the open problem Kaptein et al. name as unsolved for runtime policy. The Fable episode is a worked example of getting it wrong under time pressure, and it will not be the last.
Part IV: What breaks. Specifically.
Enough architecture. Here is the consequence ledger.
1. Authority that never expires
An agent gets one approval early in a workflow and then, fully “authorised”, drifts into territory nobody reviewed. SAFR’s answer is per-action governance and the flat rule that “an Auto-Execute or Observe outcome at one step carries no authority into the next.” Without runtime governance you have the opposite by default: authority granted at session start, spent freely for the rest of the run. MI9’s trading-agent example — retail clearance, institutional trade, no permission violated — is not hypothetical. It is the shape of every agentic incident that will matter.
2. Violations that no single step commits
An agent executes: market research → client consultation → risk assessment → trade execution. Every step is compliant. The sequence violates dual control. MI9’s finite-state conformance engine exists for exactly this, and it is the single largest gap in SAFR, which — because it disposes of actions one at a time — is structurally blind to it. Without temporal conformance checking, an agent can satisfy every control you have and still break the policy the controls were written to protect.
3. Oversight theatre
This is SAFR’s most damning finding, and it is about us, not the machines. Escalation today is ad hoc — a notification, an email, a dashboard flag — with no deadline for response, no standard format for the decision, and no audit record of what was decided. The paper’s phrase: “the appearance of human oversight without the substance of it.”
Every institution with a “human in the loop” policy should ask, today, whether it can produce for a supervisor: the escalation, the deadline, the decision, the reviewer’s authority, the record. If it cannot, it does not have human oversight. It has a dashboard.
And the arithmetic is about to make this much worse. Noam Kolt, in Regulating AI Agents, dismantles the two tools everyone reaches for: kill-switches require that triggering events can be anticipated and specified in advance — which fails precisely for agents operating autonomously in novel scenarios — and rollbacks require reversibility, which many financial actions do not have. Both are recommended in MAS’s own MindForge Operationalisation Handbook. Both are, for the hard cases, decorative.
4. The action-to-oversight ratio diverges, and there is no fix at the human end
METR’s measurements show agent task-completion horizons doubling roughly every eight months. Agents spawn subagents. Actions grow superlinearly. Human reviewers grow linearly, at best, and are a fixed and expensive resource.
The FSB says the quiet part out loud: an agent “can take hundreds of intermediate steps in pursuit of its goals and make errors in any of those steps,” monitoring them in real time is impractical as use scales, and “effective monitoring and detection may require augmentation with another AI agent or other forms of AI.”
There is no version of this where humans review each action. The only question is what does the reviewing — and that question has an unpleasant answer, below.
5. Correlated failure — the systemic dimension
SAFR raises it and moves on, but it is the reason a central bank is writing this document rather than a vendor. When many institutions rely on a small number of common AI service providers, their agents behave in correlated ways. The FSB has flagged reliance on common models, datasets and infrastructure as a source of herding and procyclicality that could amplify market stress and liquidity crunches. Ang notes drily that we spent a great deal of worry on model monoculture and rather less on the system around the model — where the incidents actually land.
An agentic monoculture with no runtime brakes is a mechanism for turning a single prompt-injection technique into a simultaneous, sector-wide, machine-speed event. There is no historical analogue and no drill for it.
6. Attack surfaces that only exist at runtime
- Prompt injection. Simon Willison‘s lethal trifecta — private data access + exposure to untrusted content + external communication — remains the cheapest and best heuristic anyone has produced, and most deployed agents have all three.
- Agentic memory poisoning. The FSB names it: malicious data injected into an agent’s retrieval and memory to manipulate its behaviour over time. By construction, a post-deployment, accumulating failure. Invisible to any pre-deployment test.
- Malicious skills. MalSkillBench (NTU Singapore) documents the newest supply chain:
SKILL.mdpackages bundling natural-language instructions, executable scripts and tool configurations, loaded dynamically at runtime when a request semantically matches the skill description. You cannot review an agent for a skill it had not yet loaded. IMDA’s OpenClaw case study names the same risk. - Multi-agent failure. IMDA’s framework is, per the multi-agent security survey, the only governance framework that acknowledges coordination risk — and it names three modes: miscoordination (agents reading the same intent differently), conflict (competing objectives), and collusion (converging on coordinated behaviour without instruction, an effect already studied in pricing algorithms).
Not one of these is visible to a pre-deployment control. All of them are, in principle, visible to a trajectory monitor.
7. The evidentiary vacuum — the one that will bite first
Under the EU AI Act, Articles 14 (human oversight) and 15 (accuracy, robustness, cybersecurity) apply to high-risk systems, with Annex IV technical documentation, from 2 August 2026.
Show a supervisor, for a given autonomous action: what the agent proposed, what authority it held, which controls were checked, what the disposition was, who reviewed it, and when. SAFR specifies that record precisely — an append-only, tamper-evident audit log capturing the envelope as submitted, the mandate, the outcome, the rules applied, the basis, and the elapsed time at each stage, on the principle that “the log is the authoritative record, independent of any party with an interest in how events are characterised after the fact.”
Most institutions running agents today cannot produce any of it. That is not a compliance gap in three years. It is a compliance gap in three weeks.
8. And the cost of not deploying
The consequence ledger has an entry on the other side, and Fable is it. Without runtime governance, capability that is genuinely valuable — Anthropic’s Glasswing partners have used Mythos-class models to find more than ten thousand high- or critical-severity vulnerabilities across the world’s most systemically important software — stays locked behind a vetted-partner gate.
The institution that cannot govern at runtime does not get to choose between deploying safely and deploying fast. It gets to choose between not deploying and deploying blind. Most will choose the second, quietly, under competitive pressure, and discover the consequences in the audit.
Part V: The part that keeps me up
Follow the logic to its end.
- Actions outrun human review. (METR, FSB, everyone.)
- Therefore oversight must be automated. (FSB says so. Anthropic ships it. Redwood formalised it.)
- Therefore the monitor is an AI.
And an AI monitor:
- can be jailbroken. Amazon’s researchers did it to Fable’s classifiers in three days.
- can be backdoored. Anthropic published Poisoning fine-tuning datasets of constitutional classifiers in April 2026 — an attack on the guardrail itself.
- can collude with the model it is watching. Shlegeris flagged this in 2024 and it remains open.
- collapses when the attacker knows the protocol. Jon Kutasov, Chloe Loughridge, Henry Sleight, Joe Benton (Anthropic) with Buck Shlegeris and Tyler Tracy (Redwood) found that resampling-for-incrimination raised safety from 50% to 89% — until a red team with knowledge of protocol internals drove it to 17%. Their conclusion, which every designer of a “transparent, interoperable, published” control format needs to read twice: this demonstrates the importance of denying attack policies access to protocol internals.
- saturates below the reliability we need. Simon Storf, Rich Barton-Cooper, James Peters-Gill and Marius Hobbhahn (ICML 2026) found that constitutional black-box monitors optimised on synthetic trajectories dogeneralise to realistic ControlArena environments — but performance saturates quickly. Simple prompt sweeps match extensive automated optimisation; pushing harder just overfits. You cannot optimise your way to a good monitor.
- depends on reasoning that may not stay legible. Tomek Korbak, Mikita Balesni, Bowen Baker, Rohin Shah and Vlad Mikulik, with forty-odd co-authors across OpenAI, DeepMind, Anthropic and the UK AI Security Institute, warn that chain-of-thought monitorability is “a new and fragile opportunity” — traces are incomplete representations of the underlying computation and may drift from natural language under scaled RL or latent-reasoning architectures. SAFR’s “action trace” and MI9’s cognitive telemetry both quietly assume this window stays open.
Here is the failure I want on the record.
The FSB is, right now, in an open consultation, recommending AI-monitoring-AI for agentic systems in the global financial system.
The AI-control literature has spent three years demonstrating that AI monitors collude, can be backdoored, collapse under protocol-internal knowledge, and saturate below the required reliability.
SAFR does not cite it. MI9 names the risk and defers it. Ang et al. do not address it. The FSB consultation does not mention it.
Four documents proposing the same architecture. Zero engagement with the evidence on whether that architecture survives contact with an adversary. This is not a research gap. It is two literatures that are not reading each other, at the exact moment their conclusions are being written into supervisory expectations.
Part VI: The gap under the gap
There is one more, and it is the one I would write if I were writing a paper.
Every runtime governance framework on the table assumes the telemetry is truthful.
SAFR names the problem with unusual honesty: the action trace and the action details are both agent-declared contents of the same envelope, and both can be fabricated together by a sufficiently sophisticated injection. Its answer — that the envelope “is therefore treated as a document to be authenticated against its origin, not merely as a record of what the agent reported” — is correct and unimplemented. MI9 concedes that agents relying on opaque APIs “may obscure the internal cognitive steps MI9 is designed to monitor.”
Both frameworks are built on a foundation neither of them pours.
The foundation exists. Edoardo Debenedetti, Florian Tramèr and colleagues at Google DeepMind and ETH built CaMeL — “Defeating Prompt Injections by Design” — which extracts control and data flow from the trusted user query so that untrusted data can never influence program flow, and enforces capability policies at every tool call via a custom interpreter. 77% of AgentDojo tasks solved with provable security, against 84% undefended. Its lineage is Control Flow Integrity, Access Control, and Information Flow Control — classical software security, not model training.
And note what CaMeL’s capability system and SAFR’s mandates have in common. Both trace to Dennis and Van Horn, 1966 — capability-based security, the same tradition that underlies OAuth. SAFR’s load-bearing property is that an agent cannot extend the scope of a mandate through its own reasoning or inference. However fluent its justification, the mandate is the boundary.
The most important idea in agentic AI governance is sixty years old, and the two literatures that both depend on it do not cite each other.
Luca Beurer-Kellner, Tramèr and co-authors are, to their credit, honest about the limit: as long as both agents and their defences rely on the current class of language models, it is unlikely that general-purpose agents can provide meaningful and reliable safety guarantees. The productive question is what constrained agents can do useful work while resisting attack.
That is not defeatism. It is the whole design brief.
Part VII: The objection, steelmanned
The strongest case against everything above:
“You are proposing to insert a synchronous, deterministic checkpoint into the execution path of a probabilistic system, and calling the result governance. You will get latency you cannot afford, false positives your reviewers cannot absorb, and a false sense of assurance from a disposition engine whose ‘AI-specific controls’ are themselves probabilistic. You will have built a very expensive, very auditable way of being wrong.”
This deserves a real answer, and parts of it are correct.
- The determinism is partly a fiction. SAFR’s disposition engine is deterministic, but it admits that its AI-specific controls — evidence quality, envelope integrity — “may involve probabilistic or semantic assessment.” You cannot get a deterministic gate from a probabilistic input. The honest position is the one Haoyu Wang, Christopher Poskitt and Jun Sun take with Pro2Guard/ProbGuard at SMU: model the uncertainty explicitly, learn a Markov chain over execution traces, and intervene on predicted probability of reaching an unsafe state — rather than laundering uncertainty into a binary.
- The false-positive economics are real and unmodelled. SAFR is candid that escalation volume must be calibrated to reviewer capacity, and that escalations need a timeout with a default. At machine speed against a human queue, the default-on-timeout is the policy, and everything else is decoration. Nobody has published the queueing model. That is an embarrassment.
- And the frameworks disagree with each other about which error to prefer, without noticing. MI9’s conformance engine is deliberately engineered so that it “will never halt a legitimate workflow by mistake” — accepting false negatives to eliminate false positives. SAFR’s Evidence Quality control does the exact reverse: a low-confidence action escalates to a human regardless of value. Adopt both, as many institutions will, and you have two components silently pulling in opposite directions on the same risk.
What survives the objection is the core. The alternative to a checkpoint in the execution path is no checkpoint in the execution path — which means the only assurance you have is a pre-deployment validation of a system whose failures are, by construction, post-deployment. The choice is not between imperfect runtime governance and perfect assurance. It is between imperfect runtime governance and none.
Part VIII: What to do, and by when
By 22 July — the FSB consultation closes. If you have a view on whether an international standard-setter should recommend AI-monitoring-AI without a robustness standard for the monitor, that is your window.
By 2 August — EU AI Act high-risk obligations. Ask, for one real autonomous workflow, whether you can produce the record: proposed action, authority held, controls checked, disposition, reviewer, timestamp. If not, you have three weeks.
Then, in order:
- Inventory agents, not use cases. The FSB reports that mature institutions now track risk at the agent level — permitted uses, known risks, tools accessible, guardrails — and formally “certify” agents for use within defined boundaries. Ang’s argument arriving as supervisory practice. A register of use cases gives you the feeling of coverage while the real risk objects sit a layer down, uncounted.
- Draw the workflow. Ang’s test, and it is unimprovable: if nobody can draw your agent’s workflow on a whiteboard — the branches, the gates, the places it must stop — then no one has drawn its limits, which means the limits don’t exist yet.
- Put a checkpoint in the execution path. SAFR’s gateway pattern gets you coverage without touching agent code. Coverage first, depth later.
- Make escalation a contract, not a notification. Deadline. Decision format. Reviewer with real authority. Default on timeout. Audit record. If you cannot produce all five, you do not have human oversight.
- Add the temporal layer SAFR lacks. Per-action disposition will not catch a violation distributed across compliant steps. MI9’s conformance engine, AgentSpec’s rules, or something like them.
- Fix the foundation. CaMeL-style provenance and capability enforcement at the tool-call boundary, or your envelope is an agent-declared document you have chosen to believe.
- Red-team the governance layer itself. MonitoringBench exists. ControlArena exists — and is maintained, tellingly, by the UK AI Security Institute. Nobody in financial services is using either.
The stakes, stated plainly
There is a version of the next two years in which agentic AI arrives in the financial system the way algorithmic trading did — faster than the controls, ungoverned in the gaps, and understood properly only after the first event that nobody can explain and nobody can reverse.
There is another version in which the control surface moves to where the risk actually lives, the record exists before it is demanded, and the capability scales because it is governed rather than in spite of it.
Anthropic has already run the experiment. Same model. Two products. One goes to a hundred and fifty vetted organisations. The other goes to the world. The only difference is the runtime layer.
That is not a compliance story. That is the whole business case.
The window is nine days.
The three artefacts
| MI9 | Szpruch, Sudjianto, Bhatti & Ang | SAFR | |
|---|---|---|---|
| Full title | MI9: An Integrated Runtime Governance Framework for Agentic AI | Scalable Runtime Governance for Agentic AI in Financial Services | Safeguards for Agentic Finance at Runtime, White Paper v1.0 |
| Authors / issuer | Charles L. Wang, Trisha Singhal, Ameya Kelkar, Jason Tuo — Barclays Model Risk Management + Columbia | Łukasz Szpruch (Edinburgh / Alan Turing), Agus Sudjianto (UNC), Tanveer Bhatti (indep.), Gary Ang (Quaintitative; SMU) | MAS BuildFin.ai, with Ant International, Circle, HSBC, J.P. Morgan Chase, Manulife, Mastercard, OCBC, Visa |
| Date | arXiv v1 Aug 2025; v4 18 Nov 2025 | Written 13 Apr 2026; posted SSRN 25 Apr 2026 | 3 July 2026 |
| Length / type | 26 pp., research preprint + open-source code | 24 pp., working paper | Industry white paper, non-binding |
| Status | Peer-review track unclear | Working paper (cited by SAFR) | “Does not constitute regulatory guidance or supervisory expectations” |
| Unit of governance | The agent (tiered) | The capability | The action |
Mechanism-by-mechanism mapping
1. Risk tiering / calibration
| MI9 | Agency-Risk Index (ARI). Three orthogonal dimensions — Autonomy, Adaptability, Continuity — each with 4 criteria scored 0–3, equally weighted, normalised to [0,1]. Four tiers: Basic Agency (≤0.25), Semi-Agentic (0.25–0.50), Highly Capable (0.50–0.75), Fully Agentic (>0.75). Higher tiers activate more governance components and more frequent monitoring. The only framework with a quantitative composite. |
| Ang et al. | Implied by “tier-based containment”; tiers derive from capabilities, each carrying explicit authority, constraints, and evidence requirements. |
| SAFR | No composite index. Calibration happens at design time through control parameters, on five named factors: action reversibility, financial materiality, customer impact severity, regulatory sensitivity, novelty/anomaly. |
Comment: ARI is scored on the agent’s capabilities (what it could do); SAFR’s factors are scored on the action (what it is trying to do now). These are not substitutes. An agent with low ARI can still propose an irreversible high-materiality action.
2. Observability / telemetry
| MI9 | Agentic Telemetry Schema (ATS). Extends OpenTelemetry’s GenAI semantic conventions. Three event classes: cognitive (plan.start, goal.set, memory.read), action (tool.invoke, api.call, auth.request), coordination (agent.msg.send, subagent.spawn, human.escalate). Framework-specific adapters translate SDK events. Explicit premise: most governance violations originate in cognitive events invisible to infrastructure monitoring. Coverage is instrumentation-dependent — callback frameworks (LangChain, CrewAI) give full visibility; API wrappers (OpenAI SDK) expose action events only. |
| Ang et al. | “Governance-semantic telemetry.” |
| SAFR | Governance Envelope, per proposed action, carrying three classes: the action (type, scope, parameters), the action trace (tool calls made, data retrieved, checks performed), and context metadata (agent identity, applicable mandate, account/system state, policy constraints). |
Comment: This is the sharpest architectural difference. MI9 is a continuous event stream; SAFR is a discrete per-action packet. MI9 can see a goal revision that never results in an action. SAFR cannot — it only wakes up when the agent proposes to act. Conversely SAFR’s envelope is a self-contained evidentiary object; MI9’s stream is not.
3. Authorisation
| MI9 | Continuous Authorization Monitoring (CAM). Extends RBAC with (i) context-dependent permissions incorporating goals, risk assessment, execution history; (ii) delegation-chain management across spawned subagents; (iii) real-time revocation mid-execution. Security rationale is explicit: authority matrices specifying who may delegate what; provenance-checked delegation chains with expiry; temporal ordering guards (approval must precede use); default-deny for tier-elevating goal transitions. Motivating example: a trading agent cleared for small retail trades escalating to institutional size while still inside its original static permissions. |
| Ang et al. | “Continuous authorisation.” Authority is an attribute of the capability. |
| SAFR | Agent Identity (verified against the authoritative registry before any other evaluation; failure = immediate logged rejection) + Controls Repository + mandates. Mandates are capability-based in the Dennis & Van Horn (1966) sense — the same lineage as OAuth 2.0 — and the load-bearing property is that an agent cannot extend the scope of a mandate through its own reasoning or inference. Cites AP2 (Agent Payments Protocol) with cryptographically signed mandates. Per-action re-authorisation: “An Auto-Execute or Observe outcome at one step carries no authority into the next.” |
Comment: Functionally the same goal, reached differently. MI9 watches authorisation state and revokes when context shifts. SAFR never grants standing authorisation at all — every step is re-checked from scratch. SAFR’s approach is stricter and cheaper to reason about; MI9’s is richer (it can distinguish authorised delegation from implicit privilege drift, which SAFR’s per-action model simply doesn’t ask about).
4. Temporal / sequence policy
| MI9 | Real-Time Conformance Engine. Finite-state machines over the ATS stream, CSP verification lineage. Rules compile to FSMs; three constructs: event predicates (match on verb, tier, any ATS attribute), ordering constraints (db.write must be followed by approve.action), temporal bounds (time window to complete a pattern). O(k) per event per agent; patterns typically <10 events; FSM instances keyed by agent_id. Worked example: dual-control on transfers — tool.invoke(bank.transfer) moves the FSM to pending-approval; an approve.action from a manager-role agent must arrive within 60s or the FSM fires a violation. Correctness property: the event-buffering mechanism may produce false negatives but never false positives — deliberately, because “the system will never halt a legitimate workflow by mistake, which is often a more costly error than letting a temporal policy violation occasionally go undetected.” |
| Ang et al. | “Temporal policy conformance checking.” |
| SAFR | Weakest area. There is no temporal automaton. Multi-step processes are handled by re-evaluating every step independently (per-action governance). Rate limits are the nearest equivalent — max action rate per time window, protecting against runaway agents, data-feed errors, and injection-driven volume spikes. |
Comment: This is the most important gap in SAFR and the strongest argument for reading MI9 alongside it. SAFR’s per-action model cannot express “market research → client consultation → risk assessment → trade execution violates dual control,” because every individual step is compliant. MI9’s whole conformance engine exists for exactly that case. SAFR’s “no authority carries forward” rule prevents drift into unreviewed territory; it does not detect a compliant sequence that is collectively a violation.
5. Drift
| MI9 | Goal-conditioned drift detection. Maintains baselines B_g per agent objective — recognising that the same agent legitimately behaves differently under different goals — and detects change while the goal is held constant. Jensen-Shannon divergence for discrete event-sequence distributions; Mann-Whitney U for continuous metrics (response times, resource consumption); sliding-window statistical process control with adaptive thresholds. Cold start handled by transfer learning from similar objectives, else conservative thresholds. “Verified adaptation signalling” cross-validates agent-declared goal changes against observed behaviour — so an agent cannot launder drift by announcing a new goal. Drift feeds the escalation framework as a risk factor, not a direct violation. Financial motivating case: a compliance agent quietly reducing alert frequency while still holding the goal “monitor trading thresholds.” |
| Ang et al. | “Drift monitoring over trajectory health.” |
| SAFR | Present only as a disposition input: “novelty or anomaly — departure from established patterns within the mandate” is one of five calibration factors, and Observe is the disposition for actions matching a pattern the institution wants eyes on later. No detection method specified. |
Comment: MI9 is the only one of the three with an actual method here, and the “verified adaptation signalling” idea — refusing to accept a self-declared goal change as licence to reset the baseline — is the single best defensive idea in the paper.
6. Intervention / containment
| MI9 | Graduated Containment, four levels, deliberately non-terminating: (1) State-Preserving Monitoring — full operation, more signal, human checkpoints on high-risk decisions; (2) Planning Intervention — finish the current task, block new planning cycles and goal modifications; (3) Tool Restriction — revoke tool categories, substitute read-only variants; (4) Execution Isolation — migrate the agent to a controlled environment with realistic but inert tool responses, so behaviour can be observed without external effect. Rationale: abruptly killing a trading agent mid-execution leaves positions unhedged; killing a risk-oversight agent disables supervision across a desk. Multi-agent cascade handling: parent under tool restriction may need cascading revocation across spawned subagents; subagent violations trigger enhanced monitoring of the delegating parent. Emergency termination reserved for critical violations after graduated options fail. |
| Ang et al. | “Tier-based containment.” |
| SAFR | Four dispositions, resolved per action: Deny (violates a hard constraint or exceeds thresholds; rejected pre-execution with reason recorded), Escalate (in scope, below hard constraints, above the autonomy threshold; held for human review), Auto-Execute, Observe (executes, structured observation logged). Plus an escalation contract — the operational part nobody else specifies: escalation volume calibrated to real reviewer capacity; a timeout window with default-to-block or escalate-to-senior; reviewers with genuine authority to approve, modify or decline, whose decisions “carry the same institutional weight as the original agent decision.” |
Sources and further reading
Primary frameworks
- Monetary Authority of Singapore / BuildFin.ai, Safeguards for Agentic Finance at Runtime (SAFR), White Paper v1.0, 3 July 2026 — with Ant International, Circle, HSBC, J.P. Morgan Chase, Manulife, Mastercard, OCBC, Visa.
- Wang, C. L., Singhal, T., Kelkar, A., & Tuo, J., MI9: An Integrated Runtime Governance Framework for Agentic AI(Barclays Model Risk Management / Columbia), arXiv:2508.03858.
- Szpruch, Ł., Sudjianto, A., Bhatti, T., & Ang, G., Scalable Runtime Governance for Agentic AI in Financial Services, SSRN 6567199, April 2026.
- Kaptein, M., et al., Runtime Governance for AI Agents: Policies on Paths, arXiv:2603.16586.
- Khoo, S., Foo, J., & Lee, R. K.-W., With Great Capabilities Come Great Responsibilities: the Agentic Risk & Capability Framework, GovTech Singapore, arXiv:2512.22211.
- Ang, G., Units of AI Governance, Simply Boring AI, June 2026.
Regulators and standard-setters
- Financial Stability Board, Sound Practices for Responsible Adoption of Artificial Intelligence, consultation report, 10 June 2026 — closes 22 July 2026. (Workstream led by Ho Hern Shin, Deputy Managing Director, MAS.)
- IMDA Singapore, Model AI Governance Framework for Agentic AI, v1.0 January 2026, v1.5 May 2026.
- MAS, Project MindForge — Executive and Operationalisation Handbooks.
- NIST CAISI, AI Agent Standards Initiative (Feb 2026); NCCoE, Accelerating the Adoption of Software and AI Agent Identity and Authorization (Feb 2026); COSAiS control overlays.
- Federal Reserve / OCC / FDIC, SR 26-2 / OCC Bulletin 2026-13, 17 April 2026 — superseding SR 11-7, excluding generative and agentic AI from scope.
- EU AI Act, Regulation (EU) 2024/1689, Articles 14 and 15; Annex IV.
- Bank of England / FCA AI Consortium; Sarah Breeden on agentic AI and regulatory reform (ECB Forum, June 2026).
- BIS Innovation Hub, Project Noor (with HKMA, UK FCA, Saudi Central Bank).
AI control and monitoring
- Greenblatt, R., Shlegeris, B., Sachan, K., & Roger, F., AI Control: Improving Safety Despite Intentional Subversion, ICML 2024.
- Bhatt, A., Rushing, C., Tracy, T., et al., Ctrl-Z: Controlling AI Agents via Resampling, arXiv:2504.10374.
- Kutasov, J., Loughridge, C., Sun, Y., Sleight, H., Shlegeris, B., Tracy, T., & Benton, J., Evaluating Control Protocols for Untrusted AI Agents, arXiv:2511.02997.
- Korbak, T., Balesni, M., Baker, B., Shah, R., Mikulik, V., et al., Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, arXiv:2507.11473.
- Storf, S., Barton-Cooper, R., Peters-Gill, J., & Hobbhahn, M., Constitutional Black-Box Monitoring for Scheming in LLM Agents, ICML 2026, arXiv:2603.00829.
- MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring, arXiv:2605.09684.
- UK AI Security Institute, ControlArena and Inspect.
Security foundations
- Debenedetti, E., Tramèr, F., et al., Defeating Prompt Injections by Design (CaMeL), arXiv:2503.18813.
- Beurer-Kellner, L., Tramèr, F., et al., Design Patterns for Securing LLM Agents against Prompt Injections, arXiv:2506.08837.
- Willison, S., The Lethal Trifecta (2025) and the Dual LLM pattern (2023).
- Dennis, J. B., & Van Horn, E. C., Programming Semantics for Multiprogrammed Computations, CACM, 1966.
- Open Challenges in Multi-Agent Security, arXiv:2505.02077.
- MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills, arXiv:2606.07131.
Runtime enforcement
- Wang, H., Poskitt, C. M., & Sun, J. (SMU), AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents, ICSE 2026, arXiv:2503.18666; and Pro2Guard / ProbGuard, arXiv:2508.00500.
- Xiang, Z., et al., GuardAgent, ICML 2025; Chen, Z., et al., ShieldAgent.
- Microsoft, Agent Governance Toolkit — open-source runtime security for AI agents, April 2026.
- Meta, LlamaFirewall, arXiv:2505.03574 (incl. Task Shield).
Governance, law, oversight
- Kolt, N., Governing AI Agents, arXiv:2501.07913; Regulating AI Agents, arXiv:2603.23471.
- Chan, A., et al., Visibility into AI Agents, ACM FAccT 2024.
- Sterz, S., et al., On the Quest for Effectiveness in Human Oversight, ACM FAccT 2024.
- Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems, arXiv:2605.16278.
- Shavit, Y., et al., Practices for Governing Agentic AI Systems, OpenAI, 2023.
- The 2025 AI Agent Index, arXiv:2602.17753.
Anthropic — the frontier case
- Claude Fable 5 and Claude Mythos 5, 9 June 2026; Redeploying Claude Fable 5, July 2026; Fable 5 / Mythos 5 system card.
- Bai, Y., et al., Constitutional AI: Harmlessness from AI Feedback, arXiv:2212.08073.
- Sharma, M., et al., Constitutional Classifiers: Defending against Universal Jailbreaks, 2025; Next-Generation Constitutional Classifiers, January 2026.
- Poisoning Fine-Tuning Datasets of Constitutional Classifiers, Anthropic Alignment, April 2026.
- Measuring AI Agent Autonomy in Practice, Anthropic Research, 2026.
A note on sourcing. The SAFR PDF is bot-protected against automated retrieval; quotations here are drawn from indexed passages of the paper itself and cross-checked against secondary analysis, but should be verified against the original before republication. The SR 11-7 / SR 26-2 replacement is confirmed from two independent sources. Claude Fable 5 and Mythos 5 post-date this author’s reliable knowledge and are sourced entirely to Anthropic’s own publications and contemporaneous reporting; the system card has not been read and should be, by anyone relying on the alignment claims.


Leave a comment