On 23 October I published an essay after fifteen AI agents had worked on it. A research group at the Alan Turing Institute had just named a risk that panels like mine run, and I expected to find it in my own record. I did not find what I was looking for. I found a change-control failure, and one real case of the older problem.
Preface
Why I am writing this. Many people now use panels of AI agents to review work: several models grading one answer, several reviewers attacking one draft. The usual reasoning is that if five reviewers agree, that is stronger than one. The Turing paper and two studies suggest it may be weaker than it looks, and I kept a record of one real panel to see.
Terms. A reviewer is one agent run with its own prompt. A finding is a distinct defect or omission a reviewer reported. Pseudo-corroboration is the Turing Institute’s term for several AI systems appearing to corroborate a claim independently while sharing the data, design or training that shaped their answers. A version is the text a reviewer read: the original draft, or the rewrite I produced after the first round of review.
Assumptions. I assume the reviewers’ reports show what they found, and that two findings can be told apart. Both are judgement calls, and section IX treats them as limits.
Confidence. High on what the ledger records and on what the papers say. Medium on my reading of it. Low on whether anything here generalises beyond one essay.
What I will not claim. That my panel was independent. That agent panels are fine. That anyone should copy my process.
What was checked, and how. I read the Turing paper in full through a reader proxy, and the two empirical studies at their sources. The table in section IV is the ledger of headline findings, built from the reviewers’ reports. A note on roles: the reviewers, and the orchestrating assistant that commissioned them, merged their findings, kept the ledger and wrote the drafts, are AI agents. My part was setting the rules, choosing what to publish and standing behind it. The merging in the ledger is therefore an agent’s judgement under my rules, which matters for section IX.
I. What the Turing paper warns about
The paper, a briefing from the Centre for Emerging Technology and Security by Rick Hennessy and Carolyn Ashurst, lists eight behavioural failure modes for AI agents. The sixth is pseudo-corroboration. It occurs when several AI systems appear to confirm a claim independently, but their outputs share training data, architectures, optimisation or post-training. The paper stresses that this does not need the systems to communicate.
Its illustration is a panel of six agents running on the same underlying language model, which appears to give six independent judgements but is “effectively one judgement counted six times”. In the same section it says effective diversity may be more important than the number of models deployed, and it lists independent human judgement as one example of diversity. In its section on multi-agent cascade, it adds that attention should be paid where agents are used to validate, supervise or review one another, because what looks like independent checking may reproduce the same assumptions.
Why this matters: the people who write assurance guidance have named the thing that agent review panels risk doing.
II. What the studies say
Two empirical results point the same way, with complications.
A paper by Elliot Kim and colleagues, accepted at ICML 2025, looked at more than 350 language models across leaderboards and found substantial overlap in their mistakes. On the 71-model HELM set, where chance would be one in three, pairs of models on average “agree 60% of the time when both models err”. Models from the same provider or with similar architecture were more correlated, and more accurate pairs were more correlated too. When one model judged others on multiple-choice questions, it inflated the scores of weaker models, most for those from its own provider or architecture.
A 2026 preprint by Guneet Kohli tested nine frontier models from seven families as judges on natural-language-inference datasets. Its headline is that the panel gave “about 2 independent votes’ worth of information”. The complications matter. The two same-family pairs it measured were only slightly more correlated than the cross-family average, and the three most correlated pairs were all cross-family. Choosing the best judge from each family made the effective number of votes smaller, not larger. The paper’s own line is that family diversity alone does not recover independence. The best single judge matched or beat the panel on two of three datasets, and on the third the panel led by 0.2 points, which the paper calls within noise. Its stated limitation is that all its tasks are classification or binary preference, and it says correlation may differ in open-ended evaluation.
Neither study is about reviewing prose. My panel did not vote. It returned a union of defects, where an extra reviewer can only add recall. The transfer from these studies to my case is an assumption, and I should say so.
Why this matters: the evidence says errors correlate across models and that diversity by family is not enough, but it comes from voting on closed labels, not from searching a draft for defects.
III. The panel and the ledger
For the essay on the UN Security Council session and AI standards, I ran fifteen agent runs: four scouts to find and read sources, three verifiers, five red-team lenses with different jobs, and three more in a second round. Two were stopped before they reported, so thirteen reports came back. The four scouts reported facts and sources, not defects in my draft, so nine runs reported findings.
I did not record which model ran each reviewer, so I cannot say whether they were one family. They had different jobs, and some were told to read different sources. All nine reviewers that reported findings read a draft.
The ledger records every headline finding, who made it, which version that reviewer read, and whether the error reached the live page. One reviewer’s report also listed further minor items that were fixed in the same pass and are not given rows. The original draft was read by seven runs that reported findings. After the first round the essay was rewritten, and the rewrite, which was then edited again before publication, was read by two runs: a law and runtime-control lens, and a full-draft verifier that reported after I had published.
Why this matters: seven reviewers saw the first text and two saw the second, and the second is where the errors that reached print came from.
IV. What the ledger shows
| # | Finding | Found by | Version read | Reached the live page wrong? |
|---|---|---|---|---|
| 1 | The EU AI Act contradicted my claim that no route combines a duty and a gate | 3 | Original | No |
| 2 | A call by 22 leaders, from Finland and Norway, was missing | 2 | Original | No |
| 3 | The UN’s verbatim record existed, after the scouts reported it could not be found | 1 | Original | No |
| 4 | Altman’s earlier IAEA remark and a July forum proposal were missing | 2 | Original | No |
| 5 | Amodei’s pacing essay, cited at the Council, was missing | 3 | Original | No |
| 6 | The claim that nobody asked for a referee was contradicted by the record | 5 | Original | No |
| 7 | France spoke before the United States | 3 | Original | No |
| 8 | A machine transcript misquoted France’s closing words; two reviewers and the drafting assistant repeated it | 1 found the correction | Original | No |
| 9 | The Hugging Face incident cut against my thesis | 2 | Original, then rewrite | No |
| 10 | The market-access rule supports mandating controls, not independent attestation | 2 | Original, then rewrite | No |
| 11 | A call dated two days before the meeting, not one | 1, plus the drafting assistant | Rewrite | No, fixed before publication |
| 12 | A claim about three permanent members objecting was unsupported | 1, after publication | Rewrite | Yes |
| 13 | A second quotation from one source broke the house rule | 1, after publication | Rewrite | Yes |
| 14 | A journalist’s paraphrase sat in quotation marks | 1, after publication | Rewrite | Yes |
| 15 | A note cited a page that did not show what it was cited for | 1, after publication | Rewrite | Yes |
| 16 | A claim that no lab had asked for a regulator was universal and wrong | 1, after publication | Rewrite | Yes |
| 17 | Two notes contradicted the text about what had been read | 1, after publication | Rewrite | Yes |
Three readings follow, and they are less flattering to my first hypothesis than I expected.
First, the original draft. Ten findings, read by seven reviewers. By the simplest count, the two found by one reviewer, rows 3 and 8, were both caught and fixed, and the one that mattered most, the verbatim record, was a single-finder. Counting rows 9 and 10 by their original-draft readers alone makes it four single-finders. A rough capture-recapture calculation, which the orchestrating assistant ran with the Chao2 estimator, puts the number of findable defects in the original at about ten under the first count and about thirteen under the second. Chao2 is a lower bound, so that means at least none and perhaps three or more undiscovered. The estimator assumes independent detection, which is the claim in dispute, and a closed population, which the rewrite broke, so I treat this as a sketch.
Second, the rewrite. Seven findings, all single-finder, and six of them reached the live page, because the only reviewer to find them reported after I had published. They are one event, not six trials: a text that two reviewers read, one of them with a different job, and one of whom reported late.
Third, the one thing the ledger cannot see: defects nobody found. A list of what was found says nothing about what was missed.
Why this matters: the errors that reached print were not obviously the ones a correlated panel would hide. They came after the only sentence-level check had run.
V. Where independence did fail
Three failures of independence show up, and none of them is a model-correlation story.
The first is the machine transcript, which is an old problem. It was the easiest document to read, and most of the reviewers and the drafting assistant used it. It gave France’s closing words wrongly. Two reviewers repeated them as established, and a draft repeated them too. The correct words were in the official verbatim record, which the scouts had reported they could not find. This is the journalist’s rule of two independent sources and the problem of circular reporting, and it is the closest thing in my record to the Turing paper’s pattern, though a human desk could have done it too.
The second is that every reviewer answered to the same principal. The briefs, the lenses, the merging, the ledger and the release decision all came from me and the assistant working under my rules. A reviewer is not independent of a reviewee who controls its brief, its inputs, its scoring and the release decision. Audit and model-validation practice treat that relationship as a central independence threat. A panel I commission is review, and it should not be called independent review.
The third is what I do not know. I did not record which model ran each reviewer, so I cannot say whether they shared a family, and if they did, Kohli’s results suggest family alone is not the answer. My record cannot test it.
Why this matters: the independence I actually lost was in the source and in the commissioning relationship, and the model family is a question I cannot answer.
VI. What worked
The reviews that helped most brought something new. The freshness reviewer found the verbatim record, a primary document that the scouts had reported they could not find. Another reviewer read UN documents that the first scouts had not. The verifier that reported last was the only one to check the rewritten text sentence by sentence.
The law and runtime-control lens read the rewrite too and did not report the six errors the late verifier found. That is likely a difference of job, since it was attacking the argument and the other was checking sentences, and it is a reason to give each version a sentence-level check as well as an argument-level one.
On the thesis, five reviewers with five different jobs converged on the same overstated claim, which is some evidence that reviewers with different jobs can agree for good reasons.
Why this matters: new sources, different jobs and a check of the final text did the work, and not the number of reviewers.
VII. What I will change
Do not publish until the last reviewer has read the final text. Treat any rewrite after review as a new version, and send the changed passages back to a sentence-level reviewer.
Count sources, not agents. Ask of any finding what it was checked against. If two findings rest on one source, count them once, and verify the source from tool traces, since an agent’s own list can be wrong.
Give one reviewer the job of finding what is new, with no view of the draft’s framing.
Keep a ledger with version, reader counts and severity, and publish it with the work.
Use deterministic gates for what they can do. Prose lint and the structural checks passed each time I ran them on that essay, because they check that sections, sources and declarations exist, and cannot tell whether a sentence is true.
Where it matters, add a human specialist. The Turing paper lists independent human judgement as one example of the diversity it asks for.
None of this is new. It is the two-source rule, release criteria and the reviewer-independence practice of audit, and an old finding from software fault tolerance: a 1986 experiment on 27 independently written software versions found coincident failures substantially more often than independence predicts. What is new is how easily the assumption can be broken silently when reviewers may share a model, a context and a prompt author.
Why this matters: the cheap changes are mostly about process, and most of them are old.
VIII. What would change this view
A prediction first, labelled as one. I expect that when the same draft is reviewed by a same-family panel and by a mixed panel with different source access, with known errors planted in it, the difference will come mainly from source access and job, not family. I hold that loosely.
I would change my mind if a planted-error test showed same-family panels finding the errors as reliably as mixed ones, which would suggest the family worry is overblown for this task. I would change it the other way if a mixed-family panel with identical sources found clearly more. The test is cheap and needs only a draft, some planted errors and two panel designs. I would like to see it run on a draft that is not mine, and compare recall and false alarms, not the share of single-finder findings.
Why this matters: the independence question is testable, and my ledger cannot answer it.
IX. The Case Against This Essay
A confession about process deserves its own objections.
First, the sample is small: seventeen findings from one essay, one panel design and one subject. A different draft could show a different pattern.
Second, the counts depend on how findings were merged and split, and the merging was done by an agent under my rules. A different rule gives a different count, and under my own rule of counting findings that share a source once, rows 3 and 8 are one finding. Two runs that were stopped before they reported also went unrecorded as readers.
Third, the ledger lists only findings that someone made. It has no column for what nobody found and none for reviewer claims I rejected, so selection and triage are both invisible.
Fourth, the two studies measure agreement on closed labels, and my panel returned a union of defects. A union panel adds recall by construction, so the studies may overstate the problem for my case.
Fifth, the interest belongs here, in the body, and runs both ways. I build and sell an execution-time enforcement layer, and an argument for a deterministic release gate is convenient for it. My employer sells assurance advice, and a demand for independent review would help that business.
Sixth, the pipeline’s written rule asks for a supermajority of red-team lenses before a claim advances, which is a count of agents, and I think it should be a count of distinct sources. A confession is only worth anything if it changes what happens next, and the ledger and the rules above are how a reader can check that.
X. Asks
Four audiences, small requests.
Anyone running agent panels. Publish your ledger with versions and reader counts. Report how many findings came from one reviewer, and what each was checked against.
Teams buying or building AI review. Ask for evidence that the reviewers do not share a source, a prompt author or a release decision, and for a measure of how often they miss the same thing.
Regulators and standards bodies. Where AI reviews AI, as in model validation or evidence for supervisors, ask for evidence of independence and not a count of reviewers. As I argued in “Receipts, Please”, no regime I have seen names pseudo-corroboration, and existing independence rules extend naturally to it.
Researchers. Run the planted-error test on realistic documents, with same-family and mixed-family panels and with controlled source access, and publish the result.
I told readers on 23 October that several reviewers had attacked the draft. I should have told them that the last one had not yet reported.
Notes and sources
- Rick Hennessy and Carolyn Ashurst, “Behavioural Assurance of Agentic AI for Sensitive and High-Stakes Organisations”, CETaS, Alan Turing Institute, 23 September 2026. Primary; read in full through a reader proxy. The failure mode is in section 3.6 and the passage on agents reviewing one another in section 3.7. cetas.turing.ac.uk
- Elliot Kim, Avi Garg, Kenny Peng and Nikhil Garg, “Correlated Errors in Large Language Models”, ICML 2025, arXiv 2506.07962. Primary; read in full by a reviewer, abstract and key sections by me. arxiv.org
- Guneet Kohli, “Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels”, arXiv 2605.29800, 28 May 2026. Primary preprint, single author, with no sign of peer review; full text read. arxiv.org
- Review ledger for the essay “OpenAI Asked for the IAEA, Then Asked the UN for a Ruler.”, reproduced in section IV. Internal; the tallies and the merging of findings are an agent’s judgement under my rules. genesishumanexperience.com
- John Knight and Nancy Leveson, “An Experimental Evaluation of the Assumption of Independence in Multiversion Programming”, IEEE Transactions on Software Engineering, 1986. Primary; its reported result is that coincident failures occurred substantially more often than independence predicts, and the authors note it may not extend to other programs. dspace.mit.edu
- A note on earlier work. “Receipts, Please” on this site discusses the Turing paper’s eight failure modes and observes that no regime names pseudo-corroboration. genesishumanexperience.com
Declarations
Interest. I build and sell an execution-time enforcement layer, and my employer sells assurance advice. Both gain from an argument that review needs independence and a deterministic release gate.
Not checked. The older literature on reviewer independence in journalism and audit, which I describe from memory; which model family ran each reviewer, and whether it accounts for any overlap; how the ledger would look under another rule for merging findings; whether the pattern holds on other drafts.
Not contacted. The authors of the papers cited. A right of reply would be the first addition.
Dr Luke Soon is an AI Leader at PwC Singapore, covering fourteen Asia Pacific markets, and co-author of Singapore’s Model AI Governance Framework for Agentic AI. He writes at genesishumanexperience.com under the Genesis: Human Experience in the Age of Artificial Intelligence banner.
Leave a Reply