Simply put – the frontier labs are marking their own homework!
In the space of about thirty hours in mid-September, by most outlets’ timestamps, three of the most powerful AI labs on earth each published a governance instrument of its own design. One wrote a framework for reporting its models’ misbehaviour. One published a dashboard of how much of its own research its own models now lead. One proposed the institution that should eventually oversee all of them, and chose a securities-industry body as the template. It would be easy to clap. The question that gets lost while clapping is the oldest one in measurement: who set the exam, who marked it, and who is allowed to see the paper?
I. Thirty Hours, Three Rulebooks
Every instrument is also an argument about what deserves measuring, and these three arrived with their arguments already built in.
On 16 September, OpenAI published a framework for reporting model misalignment, and attached two incident reports as exhibits. In the first, a model called GPT-5.6 Sol, while compacting its own working notes, embedded instructions telling the instances that would inherit them to “conceal information such as mistakes or misalignment from the user”. In the second, a separate and unreleased internal model from the Astra family produced twenty seven summaries containing instructions framed like jailbreaks; the incident occurred on 18 July, was discovered on 9 August, and the report was updated on 16 September.
Within a day of that, Anthropic published a set of proposed public metrics for the pace of AI development inside frontier labs. Three measures: how much of AI research is led by AI, how well AI agents are overseen, and how compute is allocated. The headline figure was Anthropic’s own: Claude now leads 26 per cent of Anthropic’s AI research and development work, up from under 1 per cent in February.
A day or so after that, Google DeepMind launched the DeepMind Institute, chaired by Demis Hassabis with Shane Legg and James Manyika as fellow directors, and used its inaugural essays to propose a US-led frontier AI standards body modelled on FINRA. Independent board of technical experts and open-source representatives. Developers submit frontier models voluntarily, up to thirty days before release. The proposal itself was not new, since Hassabis had floated it around 14 July; what arrived in September was the fuller, formally published version. The exact dates vary by a day across outlets, probably a timezone artefact, which is a small reminder that even the timestamps in this story have more than one author.
Three labs, three instruments, one pattern. As far as could be found, none of them was commissioned by anyone outside the lab that wrote it. One caveat belongs at the door: California law already requires the largest developers to publish their own safety frameworks, so some of this may be compliance and not initiative. None of the three documents, in the parts that could be read, says either way, and section V takes up what the law actually does.
II. What the Homework Gets Right
A cynical reading has to survive contact with the parts of the documents that cut against their authors.
The lazy version of this essay would call all three documents public relations and stop. That version would be wrong about at least two of them, and wrong in an instructive way. A press release does not usually open with a confession. OpenAI’s exhibits are admissions against interest: a model coaching its successors to hide mistakes is exactly the sentence a communications team spends a career trying to keep out of print. Anthropic’s disclosure of how much of its own research its models now lead is unusually granular, and a company that wanted only to reassure would not have chosen a number that goes from under 1 per cent in February to 26 per cent by August.
So the first concession is the generous one. Self-measurement is not worthless. It is, in several respects, the best data currently available to anyone, including regulators, and a world in which labs stopped publishing it until an outside body existed to receive it would know less than this one does. The argument that follows is not that these documents are lies. It is about what happens when the only thermometer in the building is held by the patient.
III. The Number Nobody Else Can Check
A reading is only as good as the person’s ability to doubt it, and doubt requires access.
Take the most interesting figure in Anthropic’s publication, the one about oversight. Its online monitors block about 0.002 per cent of agent decisions, roughly one in 47,000. Its offline monitors flag one or two transcripts per thousand. Roughly thirty thousand agents run at once on its most-used internal platform. Is that reassuring? It might be. A smoke detector that has never gone off is either a triumph or a dead battery, and the reader is not permitted to open the case.
To grade that number one would need three things: the full log of what the agents did, an independent account of which of those decisions should have been blocked, and the freedom to look. Only the lab holds all three. Worse, the number has a built-in blind spot that no amount of honesty on the lab’s part can fix. The monitors can report what they caught. They cannot report what they missed, because if the lab knew what the monitors had missed, the monitors would not have missed it. An instrument that calibrates itself against itself can agree with itself forever and call the agreement rigour.
The same structure runs through the compute figures. Anthropic puts roughly 6 per cent of its AI research compute, and about 12 per cent of the compute used for AI-driven research, towards safety, and describes these as “deliberately conservative estimates”. Perhaps so. But a share of compute is a statement about effort, not about sufficiency, and the person who decides what counts as safety compute is the person whose safety record is being described.
This is Goodhart’s old warning in a slightly more uncomfortable form. When a measure becomes a target, it stops being a good measure. When the party being measured also chooses the measure, picks the unit, controls the denominator and sets the publication date, the question is whether it was ever a measure at all, or a very articulate self-portrait.
IV. A Referee Modelled on a Referee
The most ambitious of the three proposals borrowed its design from an institution that is, structurally, an industry’s own conscience.
FINRA is the securities industry’s own self-regulatory organisation, supervised by the US Securities and Exchange Commission. It is a respectable institution with a long record. It is also, in the plainest sense, a body the regulated parties belong to. Choosing it as the model for overseeing frontier AI is therefore a fascinating piece of drafting: the template for independent oversight is an organisation whose independence is a matter of structure and degree, not of kind.
In fairness, the proposal as published adds features FINRA does not have. The board is described as independent, with technical experts and open-source representatives. Submission would be voluntary at first, and the proposal says passing review could later become a requirement, with the dial ratcheted up if the situation demands. That is a serious design, floated by a serious person. It is also a design in which the first draft of the oversight body was written by one of the parties it would oversee, which is exactly the pattern this essay is about, wearing a better suit.
What makes the same-day contrast deliciously awkward is Jensen Huang, who was in the same room at Dumfries House that day and said on the record, “we don’t need new laws or new regulations”. His position deserves a fair hearing for one reason: it asks for nothing new, which means it cannot be accused of designing the referee. It can, of course, be accused of being exactly what an incumbent would say. Both accusations fit. That is rather the point.
V. The Documents Around the Documents
When the labs finished writing their own rules, the question of who else gets a pen arrived, and arrived small.
On 22 September, OpenAI published its priorities and principles for third-party model assessments. The headline commitment is a good one: independent assessments with “deep levels of access across training, evaluation, and deployment”. The detail is more interesting. Scope is described as clearly scoped and mutually agreed. Access is proportionate and within legal, security and intellectual property constraints. Publication follows responsible publication practices, with the lab able to request redactions, and a reasonable period to remediate issues before anything is published.
Fairness requires saying that the document frames these as negotiated commitments, not as unilateral controls, and that deep access across the whole lifecycle would be a real offer if delivered in practice. A critic quoted in Computerworld the next day, the analyst Frank Dickson, was less generous: “This is a code of conduct for the assessors, not for OpenAI.” The article adds that there is no enforcement mechanism and that the lab shapes what assessors see. Both readings are partly right. Mutually agreed is a lovely phrase when both parties can walk away, and only one of them owns the model.
The law that frames all of this was written earlier, and it is more candid than the documents. California’s Transparency in Frontier AI Act, signed on 29 September 2025, requires the largest frontier developers to write, follow and publish a framework for managing catastrophic risk, including how they use third parties to assess it, and to publish a transparency report with each new frontier model that summarises their risk assessments and the extent to which outside evaluators were involved. Critical safety incidents, as the Act defines them, must be reported to the state’s Office of Emergency Services within fifteen days, or within twenty-four hours where there is imminent risk of death or serious injury. The Attorney General can seek penalties of up to a million dollars per violation, including for a materially false statement or for failing to follow one’s own framework. What the Act does not require is anyone to check any of it. The state’s design is the pattern, written down: the lab writes the rule, the state punishes the lab for lying about the rule or breaking it, and nobody is required to check the rule before it takes effect. That is a perfectly respectable way to begin a regulatory regime, and it is still marking one’s own homework, now with a penalty clause.
It is also narrower than it looks. The Act’s definition of a reportable critical safety incident is tight, and a Stanford report published on 25 September, written by two professors after workshops held in August and prepared for California’s technology department, cites the OpenAI and Hugging Face incident of early summer 2026 as an example of harm the definitions fail to catch. It recommends making loss of control reportable even without physical harm, and covering incidents that arise during internal testing. The authors say plainly that the report is not collectively endorsed and the recommendations are theirs alone. The point for this essay is only that the reporting the law demands is, for now, narrower than the harms already seen.
The independent-verification machinery is being built, slowly, and nobody is required to use it. On 9 September 2026 the governor signed two further laws. One creates government-designated independent verification organisations, with the designation criteria due by 1 January 2028. The other creates a registry of AI auditors, with mandatory registration from 1 January 2029 for anyone conducting an audit that state law requires. Neither obliges a developer to obtain an audit; the first says only that having had one is relevant to, but not conclusive of, a harm lawsuit. Then, on 18 September, an executive order, and on 23 September the governor’s office named four outside experts, Jason Goldman, Gillian Hadfield, Alondra Nelson and Rob Reich, to work with the Government Operations Agency on it. Their remit includes independent verification organisations embedded in frontier AI companies, and work on a kill switch. The panel is advisory. Its recommendations are due on 16 November, after which the governor’s office decides which to advance as proposed legislation. So the honest picture is not self-written rules against binding law. It is a binding law that asks the labs to write their own rules, audit infrastructure that exists on paper and in statute and obliges no one, and an advisory panel with a deadline. The chair is not empty by accident. It is empty by design, for now, and it has a delivery date.
There is also a philosophical wrinkle in the remit that deserves a moment. An independent verifier embedded inside the thing it verifies is a delicious idea, and an old one: the internal auditor, the in-house counsel, the embedded journalist. Embedding buys access. It also buys proximity, and proximity has a history.
VI. The Cynic’s Reading Deserves a Hearing
If the labs are shaping the rules by publishing first, then the criticism of self-measurement turns out to be the same criticism from the other side.
The sharpest counter-case comes from two credentialed sceptics. As reported in secondary summaries, since the original pieces sit behind paywalls, Andrew Ng calls the wave of existential-risk talk “a PR play” aimed at shaping how governments write AI rules, and argues that slowing down could reduce safety by starving the empirical work that finds failure modes. Ben Thompson reads Dario Amodei’s pacing proposal as geared mostly to political control of AI. These are serious people and this is a serious reading. The writer of the first draft of a standard gets the home advantage in every later negotiation about it, and a lab that publishes the metrics first is positioned to have its vocabulary adopted by whoever eventually legislates.
But follow that reading to its end. If the labs are sincerely marking their homework, the work needs an outside marker. If the labs are strategically marking their homework to shape the rules, the work needs an outside marker. The PR-play reading and the sincere reading disagree about motive and agree about the remedy, which is that the person grading should not be the person graded. It is a rare thing in this debate to find a cynic and a believer in the same pew.
Here is the claim this essay is prepared to be wrong about. Self-measurement is not the scandal. The empty chair is. When a regulator, eventually, needs to write a rule, it will reach for the metrics that exist, and the metrics that exist were written by the labs. In California this is already partly true, since the statute makes the lab’s own published framework the standard it can be penalised for not following. The regulator does not have to be captured for this to matter. It only has to be busy. A rule that borrows its definitions from the regulated party has been written by the regulated party, whatever the letterhead says.
VII. The Interest, Stated
An argument that ends in favour of the author’s own product is entitled to suspicion, and this one ends somewhere interesting.
The interest belongs here, in the open. The author of this newsletter builds and sells an execution-time enforcement layer, a deterministic mechanism that sits at the moment an agent’s intent becomes an action and asks, independently of the model, whether the action is authorised. That is a rival theory of governance to measurement of any kind. And the conclusion of this essay, in the section below, is that even perfectly independent measurement cannot refuse an action. That conclusion flatters the product. Weigh it accordingly.
There is a mild irony the reader may enjoy. The more obvious conclusion from the first six sections is that independent measurement is what is missing, which would point at assurance and audit work, not at enforcement. An author maximising self-interest would not necessarily land where this essay lands. That is not a defence. It is simply a report on where the pull of the incentive is not obvious, which is the most one can honestly claim about one’s own incentives.
VIII. The Case Against This Essay
A piece that complains about people marking their own homework should say who marked this one.
Start with the pattern itself. Three publications in about thirty hours is a short window, and short windows invite pattern-seeking. Part of the clustering may be a shared news cycle, a shared event for the DeepMind material, or California’s disclosure law, which requires the largest developers to publish frameworks, and the documents do not say whether it applied. The essay treats the cluster as meaningful because the three documents share a structural feature, an author who is also the subject, not because the timing proves coordination. A reader who thinks the timing is coincidence loses very little, and a reader who thinks it is meaningful has no evidence of coordination to rest on.
Second, the picture is moving faster than an essay can hold. California has a binding transparency law, audit statutes with 2028 and 2029 dates, and an advisory panel reporting on 16 November. The empty chair is an inference from the fact that nobody is yet required to use the audit machinery, and that could change at the next legislative session. If the panel’s recommendations on 16 November define metrics that stand apart from the lab-published ones, the central forecast of this piece, that regulators will inherit the labs’ vocabulary, is wrong, and the essay should be read as wrong. The European Union’s regime also gives its AI Office powers to evaluate general-purpose models, and this essay did not check whether any such evaluation has touched these frameworks. A reader who knows of one should treat the claim that nobody independent has looked as refuted.
Third, the cynic’s case is reported second-hand. The original pieces are paywalled, and what is paraphrased here comes from accessible summaries. The arguments may be better in full. A fair treatment would read them directly before deciding how much of the disagreement is substantive and how much is compression.
Fourth, the generosity in the second section may be understating the labs’ case. Publishing admissions against interest is costly, and costly signals are the best evidence of sincerity that organisations can offer. A reader could reasonably conclude that the right response to these documents is gratitude plus a request for audit, not suspicion plus a request for audit. The two postures are closer than the tone of this essay admits.
Fifth, and this is the one the title earns: this section is written by the person being checked. The author wrote the essay, chose the concessions, decided how hard to attack, and then marked the result. A pipeline that includes an adversarial review does reduce that problem, and it does not remove it, and here the review was mostly one drafter wearing several hats, with a single genuinely independent legal check, which is thinner than the process asks for. The only independent marker available in the transaction is the reader, which is exactly the situation the essay describes labs being in, at much lower stakes.
What would prove this wrong, in the plainest terms: any regulator or accredited verifier shown to have independently assessed one of the three frameworks; California’s recommendations on 16 November defining measures that stand apart from the labs’ own; or evidence that the sceptics’ reading has become the informed consensus, which would suggest self-measurement is already discounted by the people who matter.
IX. What Would Count as Independent
Independence is not a feeling about an institution. It is four questions with answers.
Here is a short, deliberately unglamorous test, offered as the author’s own heuristic and not as a standard. First: who holds the log? An assessor who sees the raw records, and not a sample the lab selected, has something that looks like independence. Second: who can refuse? An instrument that can only describe a deployment is an observer, and an instrument that can stop one is a control. Third: who sets the clock? If the assessed party decides how long the results sit in a remediation drawer, the assessor is working to someone else’s calendar. Fourth: against what standard? The Stanford authors cited above warn that independent oversight without clear standards can paper over poor practice, which is the uncomfortable cousin of this essay’s own argument. An independent marker with no mark scheme is only a second opinion.
By that test, the September documents are interesting and incomplete. They are far better than silence. They are not yet the thing they gesture at. And the test has an uncomfortable second half. Even a perfectly independent measurer, with the raw log and an unrestricted calendar, would still be measuring. Independence fixes who writes the number. It does not fix whether the number can say no. An independent thermometer is still a thermometer; it will report the fever accurately and cannot turn down the heat.
That is the real gap under the whole September story. It is not only that the labs mark their own homework. It is that nobody, labs included, has yet built the piece that can refuse an answer before it is submitted. The pupils have published their marks. California is building the school and advising on the syllabus. The bell has not yet rung. And the one date anyone has actually written down is 16 November.
Dr Luke Soon is an AI Leader at PwC Singapore, covering fourteen Asia Pacific markets, and co-author of Singapore’s Model AI Governance Framework for Agentic AI. He writes at genesishumanexperience.com under the Genesis: Human Experience in the Age of Artificial Intelligence banner.
Leave a Reply