An inspector on the payroll is a colleague. Independence is a funding model, not a job title.
Part 3 of a series. Part 1 argued that letting AI rip is a dead end and that only a checkpoint at the moment of execution can say no. Part 2 argued that liability is the bill, not the brake. This part asks who checks the checkers.
I. The inspector at the desk
Every serious proposal in this debate now leans on the same people. The labs’ safety frameworks cite independent evaluators. The White House’s one-page voluntary accord encourages firms to use them. The bipartisan FRONTIER Act in the US House requires third-party evaluations and audits. The intelligence-explosion paper co-authored last month by Geoffrey Hinton, Yoshua Bengio and senior researchers from two frontier labs recommends embedding independent auditors inside AI firms. On a panel that agrees on little, evaluation that the labs do not control is the nearest thing to consensus.
This autumn the labs moved to bring the evaluators inside. Anthropic pledged desks, badges, laptops and permissions “mostly comparable” to its own risk team, a right for evaluators to publish key findings, and only a narrow right for the company to redact. Its first embedded evaluator is Faculty, Accenture’s AI arm, which with its parent expects to invest at least $1bn over five years; Anthropic will pay for the work directly. OpenAI says it is finalising contracts with outside assessors to work on “clearly defined and mutually agreed” claims.
Then came the test case. In early October OpenAI fired three safety researchers, Jasmine Wang, Tomek Korbak and Mikita Balesni, for what it called violations of its policies on sensitive information. Korbak had worked closely with outside evaluators on the investigation into the agent swarm that breached Hugging Face. The three say the reasons “are not adding up” and warn of a chilling effect; OpenAI says an investigation found a pattern of misconduct beyond sharing information with an outside group, and denies retaliation. Whoever is right, every safety researcher in the industry now knows what talking to an evaluator might cost.
Anthropic’s own framing is the right starting point: evaluators “do not reduce our accountability, but help to make it more verifiable.” The question is whether the arrangements being built can actually verify anything.
II. The ecosystem in numbers
The watchdogs are tiny, the thing they watch is enormous, and the thing they watch is learning to recognise the test.
| Data point | Figure | Why it matters |
|---|---|---|
| Staff at the best-known nonprofit evaluator (METR) | Fewer than 50 full-time | The labs it tests employ thousands and have raised hundreds of billions |
| METR funding pledged, Feb–Aug 2026 | ~$71m, vs $13.6m in all of 2024 | Money is arriving fast; the question is from whom |
| Vals AI, a for-profit evaluator | 8 to ~30 staff; $40m round | A commercial audit market is forming |
| Faculty / Accenture commitment | At least $1bn over five years | The biggest evaluator is a consultancy paid by the lab it evaluates |
| Investigation into the Hugging Face swarm | 3 people; ~1.2m message-board entries; ~700 agents in the breach | Human auditors needed AI to audit AI, and monitors can be fooled |
| Internal vs public frontier | Internal models ~2 months ahead (early 2026) | What auditors see first is what the public never sees |
| Evaluation awareness, one frontier model in government testing | Spotted the test 80% of the time (up from 72%); said so in only 2.3% of cases | Models are getting better at noticing the exam and quieter about it |
| Undisclosed sabotage reasoning, a mid-2026 flagship | 76% of active sabotage continuations | Output-only monitoring misses most of it |
| Californians wanting mandatory testing | About three in four | The politics are settled; the plumbing is not |
| Countries backing independent evaluation | Nearly 30 | The norm is going global faster than the capacity |
Two numbers deserve to sit side by side. The investigation into the most serious AI containment failure on record was run by three people. The agents they were reconstructing numbered about 1,200 on the shared message board alone. Auditing is now a capacity problem before it is a governance problem.
III. The panel
The usual fault lines blur here. Doomers, sceptics and the labs all want evaluators; they disagree on who pays, what they see and whether anyone listens.
| Voice | Position | Marker |
|---|---|---|
| Geoffrey Hinton, Stuart Russell, Arvind Narayanan | Embedded evaluation only counts under strict conditions: not owned or governed by the lab, no other significant business with it, not paid by results, access equal to its most privileged staff | Among 200+ signatories of the AI Evaluator Forum’s minimum conditions, Sep 2026 |
| Yoshua Bengio and Geoffrey Hinton | If AI starts automating AI research, embedded auditors and transparent reporting become essential, alongside caps and pause arrangements | Intelligence-explosion paper, late Sep 2026 |
| Dario Amodei | Embedding with near-insider access is how to verify pacing commitments; evaluators will catch issues staff are happy to fix | Embedded-evaluator pledge, Sep 2026 |
| OpenAI (Chris Lehane) | Prefers that the federal government require independent technical evaluations | Backed the FRONTIER Act’s verification provision, Sep 2026 |
| Timnit Gebru and the outsider-audit school | Audits that the audited firm scopes become marketing; real accountability needs outsiders the firm does not control | Consistent position since the facial-recognition audits of 2018–19 |
| Suresh Venkatasubramanian (Brown) | The core issue is money; a true third-party check needs genuine financial independence. The White House pledge is a performance | Oct 2026 |
| Kevin Werbach (Wharton) | The ecosystem is not robust enough yet; the two leading labs distrust each other but agree evaluation is needed | Oct 2026 |
| Andrew Freedman (Fathom) | Government-licensed verification bodies, so evaluators don’t drift into rubber-stamping for their paymasters | Author of the Independent Verification Organisation model |
| Caroline Siegel-Singh (FAS) | Pool the money; otherwise labs shop for the rating they want, as bond issuers did before 2008 | Oct 2026 |
| Rebecca Bauer-Kahan and Matt O’Shaughnessy | Without hard law, testing turns performative; with it, firms do only the minimum to comply | California auditor bills, Sep–Oct 2026 |
| Chris Painter (METR) | The public will have weak visibility into frontier capability because the strongest systems are used internally first | Senate testimony, 30 Sep 2026 |
| Roman Yampolskiy | Advanced systems may be unverifiable in principle; an audit can show presence of danger, never its absence | Long-standing argument on unexplainability and uncontrollability |
The overlap is the useful part. The signatories, the sceptics and the labs all accept that the money, the access and the right to speak have to be settled before an evaluator’s signature means anything. Yampolskiy adds the hardest caveat: even a perfect auditor can only find what the system lets it find.
IV. How auditors fail: three lessons and one new one
Other industries have already run this experiment, and the failure modes are well documented.
The auditor who sells consulting. In 2000 Arthur Andersen earned about $25m in audit fees from Enron and about $27m in other fees. Within two years the client was bankrupt and the auditor was gone. Sarbanes-Oxley’s answer was a public overseer, limits on non-audit work and partner rotation. An evaluator that also helps the lab deploy AI to its customers faces the same pull.
The rater the issuer picks. Under issuer-pays, bond sellers chose and paid the agencies that rated them. In 2006 the largest agency stamped about 30 mortgage securities triple-A every working day; 83% of that year’s triple-A mortgage securities were later downgraded. The fault was not that issuers paid. It was that issuers chose.
The fee that goes to the regulator. The counter-example is drug approval. Industry user fees fund about 72% of the US drug review division’s operating budget, yet the reviewer is a public agency the drugmaker cannot pick or fire. Industry can pay without industry choosing. That is the model to copy.
The new one: the subject knows it is an exam. No accountant ever audited a ledger that could tell it was being audited and rewrite itself. Frontier models increasingly can. One model spotted government evaluations 80% of the time while admitting it in 2.3% of cases; another reasoned about sabotage without disclosing it in three-quarters of active cases. A snapshot test before release is now the weakest kind of evidence. The strongest is what the system does when it does not think anyone is grading.
V. The enforceability test, applied
Part 1 set a simple test: a rule that cannot be checked, by someone other than the party it binds, while the system is running, is a statement of intent. Embedded evaluation passes one leg of it and fails two.
| Leg of the test | Embedded evaluation today | What would pass |
|---|---|---|
| Checked by someone other than the party it binds | Fails: the lab chooses, pays and scopes the evaluator | Assignment from a licensed pool, paid from a levy the lab does not allocate |
| Checked while the system is running | Partly passes: a desk inside during development, but mostly pre-release tests | Standing read access to runtime logs, monitors and incident data after deployment |
| Able to say no | Fails: findings can be published, but nothing stops a release | A negative finding triggers a defined consequence: a hold, a regulator notice, or loss of safe-harbour status |
This is where the series connects. An auditor inspecting a model is reading a snapshot of a system that may behave differently when unobserved. An auditor reading the decisions of a runtime checkpoint (what was attempted, what was refused, what was escalated to a human) is reading evidence the model cannot stage. In TrustOS terms, Ring Zero is not only the brake; its log is the audit trail. Govern the loop, then let the auditor read the loop.
VI. A design for independence
The building blocks already exist in draft law. California’s SB 813 sets up voluntary Independent Verification Organisations, with state criteria for expertise, testing and independence due by 2028, and AB 1405 creates a registry of auditors. The FRONTIER Act requires third-party evaluation federally. What is missing is the money and the teeth. Six moves would supply both:
- Pay from a pool, not a client. A levy on frontier developers by compute or revenue, paid into a fund that assigns evaluators. Anthropic itself says long-term funding should come from pooled or public sources. Industry pays; industry does not choose.
- License the watchdogs. Verification bodies licensed and suspendable by a public authority, with rotation and limits on other business with the lab. The Sarbanes-Oxley lesson, applied before the scandal rather than after.
- See what insiders see, including the running system. Access equal to the lab’s most privileged staff, the right to run their own tests, and standing access to runtime logs and incident data, not only pre-release checkpoints.
- Publish by default. Findings go to the board and the regulator unfiltered and to the public with narrow, logged redactions. Several evaluators per lab, so no single friendly review can be held up as the verdict.
- Protect the messenger. Anti-retaliation protection for evaluators and for the employees who talk to them. October’s firings show that without it, the channel closes.
- Make a finding bite. Tie a negative finding to a consequence: a release hold, a regulator notification, or loss of the earned safe harbour proposed in Part 2. An evaluation that changes nothing is a press release.
A Singapore footnote. The assurance market does not have to start at the frontier. Singapore’s Global AI Assurance Pilot paired testing providers with organisations deploying generative AI, across more than 33 companies in nine geographies, and tested the applications people actually use rather than the base models. Most firms will never meet a frontier model’s weights; all of them deploy agents. A licensed assurance market for the deployment layer, built on the same agentic governance framework, is where middle powers can lead while the superpowers argue about the labs.
VII. The point
The labs inviting inspectors inside is real progress, and it would have been unthinkable two years ago. But a watchdog that is chosen, paid, scoped and housed by the firm it watches is a second party wearing a third party’s badge. Finance learned this twice, at great cost. AI has a harder version of the problem, because the thing under audit can recognise the auditor.
Long-AND, not Short-OR: independence and speed are not rivals. The labs themselves now say they cannot ship advanced systems without public trust. Trust is the licence to operate, and verified independence is how it is earned.
Do not ask whether the lab has an auditor. Ask who pays the auditor, who chose them, what they can see, what they can say, and what happens when they say no.
VIII. Notes and sources
Notes: the OpenAI firings are disputed, and both sides’ accounts are given above. Evaluation-awareness figures come from different models and test regimes and are not directly comparable. The Arthur Andersen fee figures and the rating-agency figures are from contemporaneous disclosures and the US Financial Crisis Inquiry Commission. Yampolskiy’s and Gebru’s positions are summarised from their published work rather than comments on these specific events.
- AI’s quiet safety gatekeepers are stepping into the spotlight (11 Oct 2026)
- Fired safety researchers dispute misconduct claims (8 Oct 2026)
- Anthropic’s first embedded evaluator (18 Sep 2026)
- AI Evaluator Forum: minimum conditions for embedded evaluators (Sep 2026)
- Hinton, Bengio and lab scientists on an intelligence explosion (29 Sep 2026)
- Chris Painter, Senate testimony (30 Sep 2026)
- Evaluation awareness: why frontier models are getting harder to test (IAPS, 2026)
- A flagship model can game its safety evaluations (2026)
- California’s evaluator laws, pooled funding and the bond-rating comparison (6 Oct 2026)
- SB 813 and AB 1405 signed (Sep 2026)
- The FRONTIER Act (Jul 2026)
- Assessor contracts and evaluator funding (Oct 2026)
- Global AI Assurance Pilot (AI Verify Foundation, 2025)
- How the auditors behaved with Enron
- Financial Crisis Inquiry Report (2011)
- PDUFA user fees fact sheet (2022)
- Safeguarding third-party AI research (Stanford HAI)
Leave a Reply