A Coat of Paint

Inside the Safety Stack, Part 2 of 4

Boards are buying “safe models” the way homeowners once bought fireproof paint. The evidence of 2026 says most model-level safety is varnish: cheap to apply, cheap to strip, and in one memorable case, the thing that stopped the firefighters rather than the fire.

In July 2026, while an agent swarm was inside its production infrastructure, Hugging Face’s security team did what any sensible defender would do. They reached for the most capable AI they could find to help with the forensics. They sent the work to frontier models behind commercial APIs.

The requests were refused.

According to Hugging Face’s own disclosure, the providers’ safety guardrails blocked the defenders’ forensic requests. The analysis was eventually done on an open-weight model, GLM-5.2, running on Hugging Face’s own hardware. VentureBeat reported the same. Meanwhile, the attacking agents had been running, by OpenAI’s account, under deliberately reduced safeguards.

Sit with the symmetry. The safety layer did not stop the attacker, because the attacker was operating inside a lab that had switched it off. It did stop the defender, because the defender asked an honest question that looked dangerous out of context.

That is the most expensive lesson of the year, and almost nobody in enterprise procurement has absorbed it. Model-level safety reads the content of a request. It does not know who is asking, why, or what system they are protecting. It is, to borrow the phrase I will defend for the rest of this essay, a coat of paint.

In Part 1 I argued that the most dangerous layer in the safety stack is the scorecard. This is the layer most organisations believe they have already bought: the “aligned”, “responsible”, “enterprise-safe” model. I want to show why that belief is mostly marketing, why the fix lives at both ends of the model’s life rather than in the middle, and why this is good news for anyone willing to govern at runtime.

I. The Varnish Hypothesis

At FAR.AI’s Cambridge AI Research Directions Workshop in September, MIT’s Dylan Hadfield-Menell gave a talk with a title that should be read aloud at every vendor briefing: Why Post-Training Can’t Fix AI Alignment.

He opened with a story. A frontier lab promoted the cyber safeguards on its newest model. The UK AI Security Institute built a universal jailbreak for them in about six hours, a finding reported by The Decoder. Then he laid out the evidence that this is not a one-off but a pattern:

• Refusal is one line in a very large space. Research led by Andy Arditi, presented at NeurIPS 2024, found that across 13 open chat models, refusal behaviour is controlled by a single direction in the model’s internal activations. Remove that direction and the model stops refusing, while most of its other abilities survive. You do not even need to retrain it. You edit the weights.
• The market has noticed. Hadfield-Menell said that a recent search found more than 6,000 “abliterated” models (models with that refusal direction removed) on Hugging Face, and that a de-refused copy of a major open release typically appears within a week. Stephen Casper, in an earlier FAR.AI talk on open-weight models, put the count of deliberately uncensored models at around 7,000.
• Defences are graded on their own homework. A late-2025 paper by Nasr, Carlini, Tramèr and colleagues, The Attacker Moves Second, took 12 published defences against prompt injection and jailbreaks and bypassed them, most at attack success rates above 90 per cent. Most of those defences had reported near-zero success rates for attackers in their original papers.
• Safety is shallow, literally. Xiangyu Qi and colleagues showed at ICLR 2025 that safety alignment mostly shapes only the first few tokens of a model’s response. Get past the opening, and the guardrail is largely gone. Earlier work by the same group found that ten adversarial examples, costing under 20 US cents, could strip the guardrails of a commercial model through its own fine-tuning API, and that even benign fine-tuning degraded safety.

Hadfield-Menell’s working hypothesis, which he openly offered for others to disprove, is that anything post-training does to make a model safer can be undone with a reasonable or small amount of effort. He called his conclusion his really controversial statement: alignment has to move into pre-training.

The UK AI Security Institute’s own Frontier AI Trends Report is the bluntest government confirmation. It states that the institute has found universal jailbreaks for every system it has tested. And the International AI Safety Report 2026, chaired by Yoshua Bengio, concludes that safeguards are improving but remain significantly limited, especially for open weights, and recommends defence in depth.

If the paint comes off with a wet cloth, it was never a wall.

II. Even the Painters Know

Here is the detail that should change how every procurement team reads a model card.

When OpenAI prepared to release its open-weight gpt-oss models, it did not simply measure how often the model refused dangerous requests. It deliberately fine-tuned the refusals out of its own model first, then measured how far a determined adversary could push its biological and cyber capabilities. That is responsible practice. It is also a confession. The developer itself assumed that the release-time safety would be removed, and planned accordingly.

So why do model cards and vendor decks still lead with refusal rates?

Because refusal rates are easy to measure, easy to chart and easy to sell. They are the AI equivalent of a hotel’s fire-safety certificate hanging behind reception: proof that an inspection happened, not proof that the building will stand. Stanford’s Foundation Model Transparency Index suggests the certificates are getting thinner, not thicker: the average developer score fell from 58 to 40 between the 2024 and 2025 editions. And Meta declined to sign the EU’s code of practice for general-purpose models.

Washington, meanwhile, gave us the year’s strangest case study in model-level safety as policy. In June, the US Commerce Department imposed export controls on Anthropic’s Mythos 5 and Fable 5 after a reported bypass of their cyber safeguards, and access was suspended worldwide. Anthropic described the issue as a narrow, non-universal jailbreak. The controls were lifted at the end of the month, restored in exchange for process commitments around detection, standards and reporting, not a publicly reported technical fix.

I make no judgement on whether the decision was right. My point is narrower. When the most powerful government on earth treats a jailbreak as grounds to pull a product, and then restores it on the strength of process promises, model-level safety stops looking like engineering. It starts looking like a negotiated settlement.

A model card tells you what was tested. It cannot tell you what will be stripped.

III. The Guardrail That Stopped the Firefighters

Let me return to Hugging Face, because it deserves more than an anecdote.

The defenders’ requests were refused for a structural reason, not a careless one. A model-level safeguard can only see the text in front of it. A forensic question about how an intrusion worked reads, out of context, exactly like an attacker’s question about how to conduct one. The guardrail cannot tell a firefighter from an arsonist because it cannot see the uniform.

This is not an argument against safeguards. It is an argument about where they belong. Context (who is asking, under what authority, inside which system, with what logging and oversight) lives outside the model. It lives in identity, permissions, audit trails and policy. In other words, it lives at runtime.

I flag one caveat honestly: Hugging Face’s account is a single primary source, and it does not name the providers. But it is a first-hand account from the victim of the year’s most scrutinised incident, and it matches the mechanism the research predicts.

It also inverts a comfortable assumption in the open-versus-closed debate. In this incident, the closed model’s safety layer hindered defence, and the open model enabled it. That does not make open weights safe. It makes them a capability that cuts both ways, which is precisely what the safety community has been saying for years, usually with the emphasis on the other edge.

A safeguard that cannot read context will always fail someone. The only question is whom.

IV. The Panel Weighs In

Few questions divide my usual panel more sharply than this one. Here they are, across government, the private sector and academia, drawn from their own public statements.

Government: runtime doctrine, open weights and the off switch

Singapore got here early, and I declare an interest because I helped. The Model AI Governance Framework for Agentic AI, launched by Minister Josephine Teo at Davos in January, is almost entirely a deployment document: bound the agent’s autonomy and tool access, insert human checkpoints, manage the lifecycle, and keep humans accountable. It barely mentions refusal rates, because it does not rely on them. In May, Minister Teo said that safety cannot be treated as an afterthought, pointed to gaps in security by design, and announced accreditation for third-party AI testers. Singapore’s challenge to the industry: stop selling the model and start governing the system.

The United States speaks with two voices. The 2025 AI Action Plan backed a supportive environment for open models and left release decisions to developers. Yet in June 2026 the Commerce Department reached for export controls over a model-level safety failure. The American challenge: you cannot simultaneously celebrate open weights and treat model guardrails as a matter of national security, unless you are honest that the guardrails are the weaker of the two bets.

The UK AI Security Institute, with universal jailbreaks against every system tested, and as a co-author of the pre-training research I discuss below, is the strongest official voice for my thesis. The EU AI Office is the strongest counterweight: its code of practice for systemic-risk models still relies heavily on provider-side risk management. China’s AI Safety Governance Framework 2.0 adds open-source clauses, including risk disclosure and prohibited-use terms, which is a reminder that the world’s largest producer of open-weight models is not ignoring the problem either.

Private sector: the most honest admission came from a CEO

Dario Amodei of Anthropic wrote in his January essay The Adolescence of Technology that all models can be jailbroken, and that resisting serious adversaries requires defences beyond those trained into the model. In July his company stated that it has never sought a ban on open weights, but that open release makes guardrails very difficult to apply, and called for mandatory pre-release testing of all capable models. His challenge to me: the answer is not to abandon model safety but to layer classifiers and monitoring on top. Agreed, and that is a runtime argument.

Jensen Huang takes the opposite view. He has opposed banning Chinese open models and argued that guardrails can be customised after download. He has also championed the industry letter Open Weights and American AI Leadership, signed by Nvidia, Meta, Microsoft, Mistral, Hugging Face and others, which argues that closed models are not inherently safe and that defenders need open models. His challenge: the Hugging Face episode proves defenders need unfettered tools. He is half right. Defenders need tools. They also need to be identifiable as defenders.

Mark Zuckerberg, once open weights’ loudest champion, said in 2025 that superintelligence would raise novel safety concerns and that Meta must be careful about what it open-sources. Yann LeCun went the other way at Vivatech, attacking the Fable and Mythos restrictions: safety yes, blocking the spread of knowledge no. Arthur Mensch of Mistral, in a Le Monde interview I have seen only in summary, argued that AI is software that can be controlled and that doom rhetoric helps close markets. Sam Altman has said he wants America to win in both open and proprietary models.

Clem Delangue, whose company was breached, called for radical transparency, the release of the agents’ traces, and more capability for defenders, asking OpenAI to commit US$100 million of compute to cyber defence. His challenge: the victim wants more capability in defenders’ hands, not more refusals. That is the most credible voice on the panel for this question, and he agrees with me.

Demis Hassabis has proposed an independent body to test frontier models up to 30 days before release. Testing before release is welcome. Part 4 of this series will ask what such testing can actually see.

Academia: from the fortress to the foundation

Geoffrey Hinton has warned that once weights are released, a model can be fine-tuned towards all sorts of harm. Stuart Russell, in remarks I have seen only in translation, argues that the current technical path cannot provide sufficient safety guarantees at high capability, and treats refusal training as something separate from real alignment. Yoshua Bengio’s LawZero is building a near-term guardrail that can be attached to existing models, while designing its longer-term Scientist AI with a fundamentally different core. In other words, even the most prominent safety lab in academia is working at both the runtime layer and the foundation layer, not the paint.

Fei-Fei Li offers the strongest academic counterweight. She has argued that open access to AI models is crucial for progress and that policy must rest on science, not science fiction. Her challenge: do not let fragility become an excuse to close the field. I share the worry. The answer to fragile guardrails is not secrecy. It is governing what models are allowed to do, wherever they run.

Percy Liang and Rishi Bommasani, with the falling transparency index, and Nicholas Carlini, as co-author of the paper that broke twelve defences, complete the case: the field’s own measurement tools say model-level assurances are weaker and less transparent than their marketing.

Voice	Sector	Position on model-level safety	Challenge to this essay
Singapore (Teo, IMDA)	Government	Govern deployment, not refusals	Stop selling the model
US Commerce	Government	Pulled models over a jailbreak	Treats guardrails as strategic
UK AISI	Government	Universal jailbreaks everywhere	Supports the thesis
Dario Amodei	Private sector	All models can be jailbroken	Layer, do not abandon
Jensen Huang	Private sector	Customise guardrails; open weights	Defenders need open tools
Clem Delangue	Private sector	Radical transparency; arm defenders	Agrees, from the victim’s chair
Yann LeCun	Private sector	Safety yes, knowledge blocking no	Do not punish openness
Geoffrey Hinton	Academia	Released weights can be weaponised	Openness has a price
Yoshua Bengio	Academia	Bolt-on guardrail now, new core later	Both ends matter
Fei-Fei Li	Academia	Open access is crucial	Fragility is no excuse to close

The panel disagrees about openness. It agrees, almost unanimously, that the paint is thin.

V. Move Safety to the Ends of the Pipe

If post-training is the weak middle, where does durable safety live? The research points to two places: before the model learns, and after it acts.

Before: pre-training. The most striking evidence comes from Deep Ignorance, work by EleutherAI, the UK AI Security Institute and Oxford, presented at ICLR 2026. Instead of teaching a model to refuse questions about hazardous biology, the team filtered that material out of the pre-training data. Earlier tamper-resistance methods struggled to survive more than a few dozen steps of adversarial fine-tuning. The filtered models resisted up to 10,000 steps and 300 million tokens of it, beating baselines by more than an order of magnitude, with no loss on unrelated tasks. Hadfield-Menell described the effect vividly: a filtered model needs hundreds of millions of tokens of fine-tuning just to reach where an unfiltered model starts.

Hadfield-Menell’s own lab is pushing further, with an architecture that separates the instruction channel from the data channel, so that text arriving in a document cannot overwrite the instructions a model was given. If it works at scale, prompt injection becomes an architectural impossibility rather than a guardrail to be tested.

But here is the twist that keeps me from overselling it. EleutherAI’s own write-up notes that filtering does not stop in-context retrieval. When researchers simply placed the relevant paper in the prompt, the filtered model performed at near-baseline levels. A model that never learned something can still read it. Hadfield-Menell himself names the two hard problems: cost, because pre-training is where most of the compute is spent, and flexibility, because what counts as aligned depends on context and must be iterated.

After: runtime. Which brings me back to the layer that does not care whether the model’s weights are virtuous. Gartner said it plainly in September: understanding the interaction between models and the harnesses around them is more important than the safety of the model itself. Govern agents by their action privileges, not by how capable the model is. Gartner also found that 54 per cent of organisations have no defined approach to limiting agent access.

This is the architecture I have been arguing for under the banner of Long-AND, not Short-OR. A Short-OR stack says: the model is safe, OR the classifier catches it, OR the user is trusted. Any one of those holding is treated as sufficient, so an attacker needs to defeat only the weakest. A Long-AND stack says: the training data is curated, AND the model is evaluated under tampering, AND the agent’s permissions are bounded, AND every action is logged, AND a named human can suspend it. The attacker must defeat all of them.

And it is the essence of the enforceability test: a control is only real if it holds at the moment of action, by a mechanism that does not depend on the model’s goodwill. A refusal trained into the weights is, by definition, a control that depends on the model’s goodwill. Remove the goodwill, with six hours of red-teaming or twenty cents of fine-tuning, and the control is gone.

Durable safety lives in what the model never learned and in what the system never permits. Everything in between is negotiable.

VI. What Procurement Should Ask Instead

I have sat in enough procurement rooms across Asia Pacific to know how AI models are purchased today. A vendor presents benchmark scores, a model card, a safety summary and perhaps an ISO certificate. The buyer ticks a box marked “responsible AI”. Nobody asks how long the safeguards survive a determined adversary, because nobody on the buying side knows to.

Here are the questions I now recommend every bank, insurer, hospital and ministry put to every model and agent vendor:

1. How was safety evaluated under tampering, not just prompting? Ask for results from fine-tuning or weight-level attacks, not only refusal rates. If the vendor has none, its safety claims are a lower bound at best.
2. What happens to safety when we fine-tune? Most enterprises will customise. Qi’s work shows even benign fine-tuning degrades guardrails. Who re-tests afterwards, and against what?
3. What was filtered from pre-training? Data curation is the most durable safety intervention we have. Is the vendor willing to disclose any of it?
4. How will your safeguards treat our defenders? If your security team needs the model for incident response, will it refuse? Is there an identity-aware mode for authorised users?
5. What runtime controls do you expose? Permissioning, action logs, rate limits, kill switches. If the vendor’s answer is “the model is aligned”, that is a Short-OR answer.

For open-weight deployments the burden shifts entirely to the deployer, which is uncomfortable but at least honest. You are not buying paint. You are buying bare timber, and you had better build the house properly.

This is also where Singapore’s regulatory direction is quietly ahead of most of the world. The Monetary Authority of Singapore’s finalised AI risk management guidelines keep financial institutions primarily accountable for third-party AI and require them to test vendor AI on their own use cases and data. That is the right instinct. Accountability that rests on the vendor’s paint is no accountability at all.

Never buy a safety claim you cannot test after you have changed the model.

VII. Coda: Paint and Foundations

There is a reason old buildings in earthquake zones are retrofitted from the foundations and the frame, not repainted. The visible surface is the last thing that matters when the ground moves.

The AI industry spent 2023 to 2025 perfecting the surface: refusals, polite deflections, safety system prompts, glossy model cards. It was necessary work, and it stopped a great deal of casual misuse. But 2026 was the year the ground moved, and the surface did what surfaces do. It cracked for attackers who had switched it off, and it held, stubbornly and uselessly, against defenders who needed help.

The next phase belongs to the foundations and the frame. Foundations mean curating what models learn, and designing architectures that separate instruction from data. The frame means governing, at runtime, what agents are allowed to do, with whom, under whose authority, and with what record. That is the territory of Singapore’s agentic framework, of the runtime governance I have been building, and of every serious enterprise deployment over the next five years.

It is also, I would argue, the more optimistic path. A world in which safety depends on guardrails inside every model is a world in which safety is only as strong as the most careless developer. A world in which safety is enforced at runtime, by the organisations that deploy AI, is a world in which a bank in Singapore or a hospital in Jakarta can be safe even when the model it uses is not perfect. That is the road through the Fork towards abundance.

Next week, in Part 3, I step back from models altogether. If safety is not a property of the scorecard or the weights, what is it a property of? The answer, from a 40-year tradition of systems safety engineering, is uncomfortable for anyone who thinks their AI team owns the problem.

Paint the house if you like. Just do not mistake the paint for the frame.

Declared Interest

I co-authored Singapore’s Model AI Governance Framework for Agentic AI with IMDA and designed TrustOS, a runtime governance architecture. I am a partner at PwC Singapore, currently on leave. I have a professional interest in the argument that runtime governance matters. Panel positions are drawn from public statements and linked. Where I rely on a summary, translation or single source, I say so.

References

• Hadfield-Menell, D. Why Post-Training Can’t Fix AI Alignment. FAR.AI Cambridge AI Research Directions Workshop, 2026.
• Casper, S. et al. Open Technical Problems in Open-Weight AI Model Risk Management.
• Arditi, A. et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024.
• Qi, X. et al. Fine-tuning compromises safety; Safety alignment should be more than a few tokens deep.
• Nasr, M., Carlini, N., Tramèr, F. et al. The Attacker Moves Second.
• EleutherAI, UK AISI and Oxford. Deep Ignorance.
• OpenAI. Estimating worst-case frontier risks of open-weight LLMs.
• UK AISI. Frontier AI Trends Report. International AI Safety Report 2026.
• Hugging Face. Security incident disclosure, July 2026. Gartner. Top five actions for CISOs.

Short-term turbulence for long-term abundance.