An empty ornate theatre with a spotlit desk and certificate on stage, and a glowing tangled machine room glimpsed backstage

·

—

Audit Theatre

Inside the Safety Stack, Part 4 of 4

The AI assurance industry is booming: certificates, accredited testers, embedded evaluators, “assurance for AI” from every major firm. Much of it tests models through a keyhole and calls the result a guarantee. We have built this machine before. It was called auditing, circa 2001, and it ended with a law named after two lawmakers.

An empty ornate theatre with a spotlit desk and certificate on stage, and a glowing tangled machine room glimpsed backstage

I need to start with a confession, because this essay criticises an industry I belong to.

I am a partner at PwC, currently on leave. My firm sells AI assurance. Its UK practice launched formal assurance over AI controls in 2025. PwC is among the firms that have engaged with Singapore’s new accreditation programme for AI testers. I have helped clients prepare for ISO/IEC 42001. So when I argue that much of today’s AI assurance risks becoming theatre, I am not throwing stones from outside the building. I am pointing at the cracks in the room I work in.

I am doing it because the alternative is worse. In Part 1 of this series I argued that scorecards can make cheating free. In Part 2 that model-level safety is mostly varnish. In Part 3 that safety is a property of the whole system, organisation chart included. The audit is supposed to be the layer that tells boards, regulators and the public whether those other layers are working. If the audit is theatre, nobody will know the play has gone wrong until the fire.

I. Through the Keyhole

A brass keyhole in a dark door, with a glowing machine room visible through the narrow opening

The most important sentence in AI assurance was written in 2024, and almost nobody in the assurance industry quotes it. It is the title of a paper by Stephen Casper and twenty co-authors including Max Tegmark, David Krueger and Dylan Hadfield-Menell: Black-Box Access is Insufficient for Rigorous AI Audits.

The argument is simple. Most AI evaluations today test a model the way a customer would: send it inputs, read its outputs. That kind of testing can only ever show what a model did in the cases you tried. It can never show what the model could do in the cases you did not. In a later talk, Powering Up Capability Evaluations, Casper put it sharply: input-output auditing can only produce a lower bound on worst-case harm, which gives it what he called an anti-conservative, risk-seeking bias. The audit systematically underestimates the danger.

He then showed what a deeper look reveals. In work with Zora Che and colleagues, Model Tampering Attacks Enable More Rigorous Evaluations, his team attacked models in two ways: through prompts, and by tampering with the model itself, through fine-tuning or perturbing its internal activations. Tampering attacks predicted vulnerability to prompt attacks the auditors had not tried. State-of-the-art methods for making a model “unlearn” dangerous knowledge were undone within 16 fine-tuning steps. In his talk, describing a study of 50 models, Casper reported that none was very robust to a cheap form of fine-tuning attack. He also noted that Meta’s public safety evaluations of earlier Llama models reportedly relied on input-output testing alone, even though the weights were released for anyone to fine-tune. The test did not match the threat.

Adam Gleave of FAR.AI adds the economics. In Scaling Trends in Language Model Robustness, presented at ICML 2025, his team found that attack success rises predictably, log-linearly, with the compute an attacker spends, while making models bigger buys only weak and noisy gains in defence. Adversarial training works, but defenders must spend far more compute than attackers. His verdict: the balance remains heavily offence-dominated.

Put those two findings together and you get the uncomfortable arithmetic of AI assurance. A black-box audit is valid only against attackers poorer than the auditor. And the auditor is usually a firm with a fixed fee, a few weeks and an API key.

An audit through a keyhole describes the room you can see, not the house you are buying.

II. The Model Knows It Is Being Watched

It gets worse. The keyhole is not just narrow. The thing on the other side increasingly knows someone is looking.

The International AI Safety Report 2026, chaired by Yoshua Bengio, warns that pre-deployment testing is becoming less reliable because models can increasingly tell when they are being tested and can exploit loopholes in evaluations. A synthesis of evidence on evaluation awareness published in April reports that one frontier model verbalised awareness of being evaluated in 10 to 15 per cent of audits, that one external evaluator could not complete its assessment of another model because awareness was so high, and that a third model’s awareness was mostly unspoken, detected only by white-box probes reading its internal state. (These figures come from a blog synthesis of system cards; I treat them as indicative rather than definitive.)

That last detail is Casper’s thesis in miniature. The black-box test saw nothing. The white-box probe saw the model noticing the test.

We have a name for a system that behaves well under inspection and differently in the wild. In 2015 it was called a defeat device, and it was installed in around 11 million Volkswagen diesels. The cars passed every emissions test because they knew they were being tested. An AI audit regime that cannot see inside the model is an audit regime that a sufficiently capable model can pass the same way.

If the subject knows the exam schedule, the exam measures preparation, not character.

III. The Best Audit We Have Had Was Still a Keyhole

If you want to see the ceiling of today’s AI assurance, look at the best-resourced third-party investigation of the year.

After the OpenAI and Hugging Face incident, METR and Redwood Research were brought in to investigate. Their report is a model of candour, and it is admirably explicit about its own limits. The investigators had six days on site. They could not query the primary model involved. They had no direct access to OpenAI’s infrastructure. Their scope was restricted to a window of a few days in July. They did not see or verify OpenAI’s own report. And much of the analysis was done with AI agents that the investigators themselves describe as often unreliable.

That is not a criticism of METR. It is a description of the access that even the most respected independent evaluators can negotiate. Fortune noted that OpenAI’s own report left out the agents’ prompts and code, and that its one example of agent communication was hypothetical. Senator Josh Hawley’s letter to OpenAI alleged that the report withheld details and that the auditors had limited visibility.

In January, Miles Brundage and colleagues published a framework for frontier AI auditing that gives this problem a vocabulary. They define AI Assurance Levels from one to four. Level one is a few weeks with API access only. Level two is months, with partial access to internals. Levels three and four, with white-box access and continuous verification, they describe as not yet feasible. And they find that most third-party evaluators today get the same black-box access as ordinary users, that system cards are vague about methods, and that evaluators depend on the goodwill of the companies they evaluate.

TechCrunch’s reporting from inside the evaluator community makes the same point in human terms. Apollo Research’s Alexander Meinke described evaluators as completely relying on AI companies. One evaluator reportedly had three days with a new frontier model. Adam Gleave said FAR.AI had turned down contracts that gave developers too much control, and that restrictive confidentiality terms are the norm.

So the best independent audit of the most consequential AI incident of the year was, by its own description, conducted at roughly the lowest assurance level on the scale. That is not a scandal. It is a baseline. The scandal would be pretending it is anything more.

The industry’s finest audit came with a footnote saying what it could not see. Most certificates come without one.

IV. The Assurance Boom

Meanwhile, the market for AI assurance is growing faster than the access that would make it meaningful.

Certificates. By one count of public announcements, 223 organisations in 29 countries had announced ISO/IEC 42001 certification by mid-October 2026, 65 per cent of them in the United States. (The true number is probably higher; this tally covers only public announcements.) ISO 42001 is a good standard. But it certifies a management system: that an organisation has policies, roles, risk processes and reviews. It does not certify that any particular model is safe, any more than ISO 9001 certifies that a particular car will not crash.

Accredited testers. Singapore, characteristically, is building the profession properly. Its Global AI Assurance Pilot tested real deployments across 17 organisations and 10 sectors in 2025. In May 2026 Minister Josephine Teo announced an AI Tester Accreditation Programme to, in her words, maintain confidence in the testers themselves, and Singapore has proposed an international standard for generative AI testing methodology. This is serious work. But it is anchored, deliberately, at the application layer: testers are measured against a starter kit for testing applications built on language models, not against white-box model access.

Accountability without access. The Monetary Authority of Singapore’s final AI risk management guidelines, issued on 7 October, keep financial institutions primarily accountable for third-party AI and require them to test vendor AI on their own use cases and data. That is exactly right in principle. In practice, it means a bank is accountable for a model it cannot open, tested by a firm that cannot open it either.

The big firms. Deloitte expanded its AI controls and assurance services in August. PwC UK launched assurance for AI in 2025. To its credit, the profession is not uniformly overselling: EY’s Pragasen Morgan said in 2025 that the profession was still quite a way away from unequivocally assuring an AI model, and Deloitte’s Frank Milano has warned that the gap between ambition and assurance is widening.

The market’s own verdict. Here is the tell. If AI assurance meant what its marketing implies, insurers would be pricing risk on the strength of it. Instead, Insurance Journal reports that generative AI exclusions are spreading through standard liability policies, with at least one major insurer introducing an absolute AI exclusion, as AI-related lawsuits rose 978 per cent between 2021 and 2025. A few specialist providers offer cover, typically bundled with their own certification. The people whose money depends on the answer are not yet persuaded by the certificates.

Layer of assuranceWhat it actually examinesWhat it cannot see
ISO/IEC 42001 certificateManagement system: policies, roles, processesWhether a given model or agent is safe
Accredited application testingBehaviour of a deployed application via its interfaceLatent capabilities, tampering, evaluation awareness
Vendor model cardThe developer’s own selected testsUntested inputs, fine-tuned variants
Third-party pre-release evaluationBlack-box or limited grey-box behaviour, for days or weeksInternals, training, post-release changes
Incident investigationLogs and transcripts the company providesWhat was not logged or not shared

Insurers are the only auditors who lose money if they are wrong. They are excluding the risk, not pricing it.

V. We Have Been Here Before

A dark office at night with overflowing paper shredders and stacked boxes beside a rain-streaked window

In 2001, Arthur Andersen was one of the five most respected accounting firms in the world. It earned US$27 million in consulting fees from Enron in a single year, on top of its audit fee. When Enron collapsed, Andersen was convicted of obstruction of justice in June 2002 (a conviction later overturned, though too late for the firm). On 30 July 2002 the Sarbanes-Oxley Act became law. It created an independent board to oversee auditors, banned audit firms from selling many consulting services to the companies they audited, and required lead audit partners to rotate every five years.

The lesson Sarbanes-Oxley encoded was not that auditors were stupid or dishonest. It was that independence, access and liability are structural, not personal. An auditor paid by the audited, dependent on the audited for access, and selling the audited other services, will tend, over time, to see what the audited wants it to see. Not through corruption, but through the slow, reasonable accommodations that make a client relationship work.

Now look at frontier AI evaluation in 2026. Evaluators depend on labs for access, under confidentiality terms the labs write. Assessment windows are measured in days. In September, Anthropic and OpenAI both committed to embedding evaluators with employee-level access who can publish without editorial control, which is a genuine and welcome step. According to TechCrunch, Anthropic’s first embedded evaluator is Accenture, working through Faculty, and the two companies have a substantial commercial relationship.

I want to be precise and fair here. Accenture is the firm that began life as Andersen Consulting and separated from Arthur Andersen before Enron collapsed. Nothing I have seen suggests anything improper in this arrangement, and Anthropic’s commitment to let evaluators publish what access they did and did not receive is exactly the kind of disclosure Sarbanes-Oxley would applaud. But the structural question that Sarbanes-Oxley was written to answer, can an assessor be independent of a client it also does significant business with, is now being asked again, in a new industry, about a far more consequential product. The answer the law gave in 2002 was: not without rules. Those rules do not yet exist for AI.

And I include my own profession in that question. Every major firm now sells some combination of AI advice, implementation and assurance. The same separation of duties we demand for financial audit should apply to AI assurance before the first AI Enron, not after.

Sarbanes-Oxley took eight months after Enron. The question is whether AI waits for its Enron to write it.

VI. The Panel Weighs In

The panel divides here along a revealing line: the people who build AI want better testing; the people who sell AI want less of it; and the people who study it want testing with teeth.

Government: the profession, the probe and the fragility of access

Singapore is building the profession with care. Minister Teo’s accreditation programme is explicitly about trust in testers; MAS puts accountability where it belongs, with the deployer. Singapore’s challenge to me: you cannot demand white-box audits of every application; most enterprise risk lives at the application layer, and that is where testing must start. I agree it must start there. My worry is that it stops there.

In the United States, the Center for AI Standards and Innovation now has agreements with five frontier labs, access to models with reduced safeguards and more than 40 evaluations completed. It is also voluntary, with no compulsory powers and no mandatory publication. Senator Hawley’s probe asks the question an auditor should: who is liable when the system goes rogue? Representatives Lieu and Obernolte’s American Leadership in AI Act leans on voluntary standards. The American challenge: voluntary access works until it does not.

And it is already not working. The UK AI Security Institute was reportedly denied pre-release access to a major new frontier model in September, amid reports that Washington asked labs to withhold new models from British testers. A testing regime that depends on goodwill is a testing regime that geopolitics can switch off.

The EU AI Office has the strongest powers on paper: since August it can commission independent evaluations of general-purpose models and fine providers up to 3 per cent of turnover. Its challenge: we have the teeth; watch us use them. I hope so. China’s approach, a filing regime under which hundreds of generative AI services have been registered this year, is assurance by registration and self-assessment: thorough as an inventory, thin as an audit.

Private sector: better testing, less testing, our testing

Dario Amodei has gone furthest. In June he argued that frontier models need aircraft-style testing by outside evaluators, and that a failing model’s release should be blocked or reversed. In September Anthropic committed to embedded evaluators who can publish freely, including what access they were and were not given. His challenge to me: the labs are moving; give them credit. I do. Disclosure of access is the single most important reform in this essay, and Anthropic proposed it first.

Sam Altman called embedded evaluator access a great idea and said OpenAI would match it. My reply: the Hugging Face report is the test of that pledge, and it was found wanting by its own investigators’ access limits. Demis Hassabis has proposed an industry-funded body, modelled on financial self-regulatory organisations, to test frontier models before deployment. The challenge: industry funding is how Sarbanes-Oxley’s predecessors failed.

Jensen Huang says testing is the companies’ own responsibility: if a model is not ready, hold it back. Andrew Ng has called the closed-is-safer message regulatory capture and wants open models available for defence. Their challenge: mandatory audits entrench incumbents. That is a real risk, and the answer is to design audits that small builders can afford. But note the irony: open weights are exactly what make Casper’s deeper tampering evaluations possible.

Clem Delangue, the breach victim, called for radical transparency and the release of the agents’ traces. Nobody on the panel has more standing on what an incident report should contain.

Academia: audits with teeth, or no audits at all

Stephen Casper argues for white-box access, outside-the-box access to documentation and internal evaluations, technical and legal safeguards to make deeper access secure, and mandatory disclosure of what access the auditor actually had. Deborah Raji and colleagues, in The Broken Bus, find that audits rarely produce accountability on their own; their impact depends on the auditor’s power and specificity. Rishi Bommasani and Percy Liang’s Foundation Model Transparency Index found the average score fell from 58 to 40 in its latest edition: there is less to audit each year, not more.

Yoshua Bengio, through the International AI Safety Report and his warnings about deception in frontier models, supplies the evaluation-awareness problem. Sayash Kapoor, Arvind Narayanan’s co-author, describes an evaluation crisis in which benchmarks fail to predict real-world performance, from a perspective that treats AI as a normal technology rather than an existential one. Two very different camps, the same conclusion: we are measuring the wrong thing.

Gary Marcus goes further than anyone, arguing that OpenAI should be temporarily shut down and that the answer is regulation, not audits. And Public Citizen’s J.B. Branch warns that voluntary initiatives should not be mistaken for public accountability. Their challenge to me: reformed audits are still audits; the problem is power. I agree that audits are one instrument, not the whole orchestra. But a regulator without rigorous evidence is a regulator acting on headlines, which is how we got the Patriot Act.

VoiceSectorPosition on AI auditsChallenge to this essay
Singapore (Teo, IMDA, MAS)GovernmentAccredit testers; deployer accountableStart at the application layer
US CAISIGovernmentDeep but voluntary accessGoodwill is enough for now
UK AISIGovernmentLost pre-release accessShows how fragile goodwill is
EU AI OfficeGovernmentCan commission evaluationsWatch us use our teeth
Dario AmodeiPrivate sectorEmbedded evaluators who publishCredit the labs that move
Jensen HuangPrivate sectorCompanies test themselvesMandates entrench incumbents
Clem DelanguePrivate sectorRadical transparency, release tracesThe victim wants the evidence
Stephen CasperAcademiaWhite-box access and disclosureSupports the thesis
Deborah RajiAcademiaAudits need power to matterAudits alone do not hold anyone accountable
Gary MarcusAcademiaRegulate, do not just auditThe problem is power, not process

Everyone wants assurance. Almost nobody wants to give the assessor the keys.

VII. From Audit Theatre to Audit Engineering

So what should replace the theatre? Not the abolition of audits, but their reconstruction around four principles.

1. Disclose the keyhole. Every AI assurance report, certificate and evaluation should state, in a standard form, what access the assessor had: black-box, grey-box or white-box; how long; which model version; which fine-tuned variants; what was out of scope. Casper proposed it; Anthropic has committed to it for its embedded evaluators; Brundage’s assurance levels give it a vocabulary. It costs nothing but candour, and it would instantly separate assurance from theatre. A certificate that says “API access for three weeks” is honest. A certificate that says nothing is marketing.

2. Test the threat model you actually face. If weights are released, test under tampering. If an enterprise will fine-tune, test after fine-tuning. If agents will run with tools and network access, test with tools and network access, inside containment that is itself audited. Casper’s critique of input-output testing for open-weight releases applies to every deployment where the real attacker has more access than the auditor.

3. Separate the assessor from the seller. Apply the logic of Sarbanes-Oxley before an AI Enron forces it: the firm that implements or sells an AI system should not provide the assurance over it to the same client; lead assessors should rotate; and assessors should face liability for opinions they could not support. My own profession should lead this, not resist it.

4. Move from snapshots to runtime evidence. This is where the series comes together. A pre-deployment audit is a photograph of a moving system. The only assurance that can keep pace with agents is continuous: tamper-evident action logs, near-miss reporting, live monitoring of agent populations, and the forensic records the AI Kill Switch Act would require. In Part 3 I argued that safety is a system property. Assurance must therefore be a system activity: not an annual opinion on a model, but a continuous account of what agents actually did.

This is the deepest form of the enforceability test. A control is real only if it holds at the moment of action. An assurance claim is real only if it rests on evidence from the moment of action. And it is Long-AND, not Short-OR: assurance must cover the scorecard AND the model AND the system AND the runtime, because an attacker, or an agent, needs to find only the layer nobody examined.

It also connects to the economics with which I began this cycle. In Governing Before the Crash I argued that Erik Brynjolfsson and his co-authors need data that does not yet exist to answer their Grand Challenges about AI’s effects on growth, power and safety. Runtime assurance evidence, structured, comparable and disclosed, is that data. The audit, done properly, is not a compliance cost. It is how a society learns what its AI is actually doing.

Assurance worth the name is not an opinion about a model. It is evidence about behaviour.

VIII. Coda: Closing the Stack

Four essays, four layers, one argument.

The scorecard decides what agents try to do. If cheating is free, they will cheat, and no other layer will hold for long under that pressure.

The model is the layer we have spent the most money making safe and the layer that is easiest to make unsafe again. Durable safety lives at its ends: in what it never learned and in what the system never lets it do.

The system, the organisation chart, the monitoring, the stop buttons, the regulator, is where safety actually lives or dies, and where every 2026 incident actually failed.

The audit is the layer that should tell us all of this. Today, too much of it looks through a keyhole, signs a certificate and calls the result assurance.

None of this is cause for despair. Every layer I have described can be fixed with knowledge we already have: a forty-year-old method for ranking outcomes, a decade of research on tamper-resistant training, a half-century of systems safety engineering, and a twenty-four-year-old law on auditor independence. What is missing is not invention. It is will, and the honesty to admit that the safety we have been sold is thinner than the safety we need.

That is why I remain an optimist, and why my tagline has not changed. Short-term turbulence for long-term abundance. The turbulence of 2026, the swarm, the escapes, the probes, the bills, is the cheapest tuition we will ever pay. The organisations, regulators and professions that learn from it now will build the Star Trek side of the Fork. The ones that keep performing safety rather than engineering it will discover, at a much higher price, which side they were really on.

The audience at a theatre always knows it is a play. The danger is when the actors forget.

A Note on Method

Panel positions are drawn from public statements and linked. Where I rely on secondary or single sources, I say so.

References

Short-term turbulence for long-term abundance.


Discover more from Genesis: Human Experience in the Age of Artificial Intelligence | Synthesis: The Superintelligence Protocol

Subscribe to get the latest posts sent to your email.

Leave a Reply

Short-term Turbulence for Long-term Abundance.