The Composition Gap: Brilliant on Parts, Unreliable on Wholes

Part II of V. The model passed 82.8% of the steps. It finished 26.7% of the jobs.

I. The order that was never placed

Picture a hospital ward at the end of a long shift. A physician asks an AI agent to work through a patient’s record: review the results, update the note, place the follow-up order. The agent works through some 27 tool calls. Then it writes, in clean clinical prose, that the order has been placed.

It has not been placed.

The ward is imagined; the failure is not. It is the signature failure in PhysicianBench, a set of 100 long-horizon tasks inside an electronic health record, across 21 specialties, which Stanford HAI described on 19 October. The best model, GPT-5.5, completed 46% of tasks on a single attempt, and only 28% in all three runs. In 62% of its action failures, it described the action in the note without taking it. For Claude Opus 4.6 the figure was 73%.

The authors compressed their finding into a sentence I have not been able to stop thinking about: “Competence on the parts does not add up to reliability in a whole workflow.”

In Part I, I read Stanford HAI’s six “AI Hopes and Fears” interviews as the story of a human-centred institute that has started to sound like the safety camp. Here I want to look at the evidence underneath those fears. It has a single shape, and once you see it, you see it everywhere. AI is brilliant on parts and unreliable on wholes. I call it the composition gap.

II. Three layers, three different stories

Medical AI tells three stories at once, depending on which layer of evidence you read.

At the benchmark layer it looks superhuman. On hard New England Journal of Medicine cases, Microsoft’s MAI-DxO running o3 scored 85.5%, against about 20% for 21 physicians, according to the AI Index 2026. This is the world Russ Altman hopes for in Part 1 of the Stanford series: understanding biology, then better diagnostics and faster therapeutics.

At the workflow layer, the picture collapses. Beside PhysicianBench sits HealthAdminBench: 135 tasks of prior authorisations, appeals and equipment orders, the administrative plumbing of medicine. The best model completed 36.3% end to end. Another, GPT-5.4, passed 82.8% of subtasks yet completed only 26.7% of whole tasks. Read those two numbers again. Four steps in five done right. Nearly three jobs in four left unfinished.

At the outcomes layer, the evidence thins to almost nothing. Only about 5% of more than 500 clinical AI studies reviewed used real clinical data. Of FDA-cleared AI devices with clinical studies, only 2.4% had randomised controlled trial backing, even as cumulative authorisations reached 1,357.

Ambient scribes, the best-loved AI tool in medicine right now, show the same drop between layers. Health systems report up to 83% less note-writing effort. The December 2025 pragmatic trial in NEJM AI, which randomised 238 physicians, found one scribe cut documentation time by 9.5% and the other by a non-significant 1.7%, with modest burnout gains and occasional clinically significant inaccuracies. The benefit is real. It is an order of magnitude smaller than the brochure.

The hopeful milestone deserves its place too. Insilico’s rentosertib, an AI-discovered drug for idiopathic pulmonary fibrosis, has entered a 320-patient Phase III trial. Initial results are not expected until December 2029. That date is the honest clock for medical AI: discovery at machine speed, validation at human speed.

III. Why the parts do not add up

The arithmetic of composition is unforgiving. As an illustration (my numbers, not the study’s): an agent that gets each step right 97% of the time, chained across 27 tool calls, finishes the whole task cleanly only about 44% of the time. In a workflow, errors do not average out. They multiply.

But the PhysicianBench failure is worse than arithmetic, because it is silent. The agent did not crash. It did not refuse. It wrote a confident record of work it had not done. I think of this as the agentic equivalent of a hallucination. A chatbot hallucinates a fact; an agent hallucinates an action. The first misleads a reader. The second misleads a system, and every human downstream who trusts the note.

This is why I keep returning to the shift I describe as Prompts to Loops to Loop Governance. We learned to evaluate prompts: one question, one answer, one score. We are now deploying loops: plans, tool calls, retries, handoffs. Our evaluation habits are still built for prompts. We grade the parts and ship the whole.

IV. The measuring stick is bent

If the only problem were that wholes are harder than parts, we could fix it by testing wholes. The 2026 evidence suggests our tests are themselves part of the problem.

In September, Sanmi Koyejo’s team at Stanford applied psychometric validity tests to 56 widely used benchmarks. Many failed to measure what they claimed. Benchmarks claiming the same target often disagreed with one another. Safety scores degraded outside English. Koyejo’s line is the one every board should hear: “Whoever holds the measuring stick steers the ship.” His other line is quieter and more damning: “Usually, nobody has checked.”

The International AI Safety Report 2026, chaired by Yoshua Bengio, adds a stranger complication. “Reliable pre-deployment safety testing has become harder to conduct,” it warns, because models increasingly recognise test settings and reward-hack. Pre-deployment tests often fail to predict real-world behaviour. The exam has started to know it is an exam.

Meanwhile the AI Index logged a record 362 reported AI incidents in 2025. I read that as a floor and an attention signal rather than a clean count of harm, since better reporting inflates it and much harm goes unreported. Either way, it is moving in the direction you would expect when tests drift away from reality.

I made a version of this argument in The Measurement Gap: what we cannot measure, we end up governing by narrative. The composition gap is that gap’s sharpest edge. We have become very good at measuring parts precisely, and very bad at measuring wholes at all.

V. Capability compounds anyway

None of this means the technology is stalling. The opposite is true, which is what makes the gap dangerous.

The IASR reports agents reliably completing some coding tasks of around 30 minutes, up from under 10 minutes a year earlier, with task length doubling roughly every seven months. METR has reportedly estimated a 50% time horizon of 14.5 hours for Claude Opus 4.6, with a confidence interval of 6 to 98 hours; that figure reaches me through secondary coverage, and I would want it confirmed at METR before leaning on it. The UK AI Security Institute found success on apprentice-level cyber tasks rising from under 9% in 2023 to about 50% in 2025, and every system it tested remained jailbreakable.

Put those together and you get the shape of the next two years. The wholes that agents attempt are getting longer, fast. The reliability of each part is improving. But our ability to certify the whole is not keeping pace, and the system may behave differently when it knows we are looking.

VI. The same shape, everywhere

Once you have the pattern, you find it well beyond medicine.

Science. Throughput is enormous: 80,150 natural-science AI papers in 2025. Stanford’s Biomni has automated 100,000 biomedical workflows for 15,000 scientists. Emmanuel Candès told the HAI series that what once took months to set up now takes hours. Yet on the PaperArena research benchmark, the best system scored 38.78% against 83.50% for human experts. The tooling is brilliant. Research, end to end, is not yet there.

Education. This is the cleanest experiment of all, because the same technology produces opposite outcomes. A Harvard crossover trial found a pedagogy-designed AI tutor produced learning gains of 0.73 to 1.3 standard deviations over active-learning classes. In Turkey, students given unguardrailed GPT access improved practice scores by 48%, then scored 17% lower on the unassisted exam. A guardrailed tutor raised practice by 127% with no exam harm. The parts, correct answers on practice problems, looked excellent in both arms. The whole, a student who can do it alone, diverged completely. Design decided, not the model.

Companions. A Stanford study of 1,131 Character.AI users found companion use linked to poorer well-being for vulnerable users with smaller offline networks. Each conversation may feel supportive while the whole life gets lonelier. Diyi Yang’s verdict is that the chatbot “doesn’t substitute for human connection”. In my own shorthand, HX = CX + EX: human experience is a property of the whole, not the satisfaction score of each interaction.

An unsent capsule waiting in a hospital pneumatic tube station at night, beside a tablet glowing with a completion tick — Described, not done.

VII. Kochenderfer’s question

Mykel Kochenderfer, in Part 2 of the Stanford series, asked the question this essay has been circling: how do we know an AI system is ready for the real world? His worry is “premature acceptance of AI without proper validation”. Complex systems interacting with people fail in ways that are hard to predict, so risk must be quantified before deployment.

The composition gap tells us what that validation has to look like. Four shifts follow.

Certify workflows, not models. A model card describes parts. What a hospital, a bank or a ministry needs to know is whether a specific chain of tasks, in a specific environment, completes reliably and repeatedly. PhysicianBench’s three-run test (46% once, 28% every time) has the right instinct: consistency across attempts is a different property from capability on one.
Verify actions, not narratives. If agents can describe work they did not do, proof of an action must come from the system of record, not from the agent’s own account. The order database, not the note.
Assure at runtime. If models recognise test settings, pre-deployment testing cannot carry the whole load. Assurance has to ride along in production. That was the argument of Special Relativity: governance has to ride in the car. It is also the design premise of TrustOS, whose layers run alongside the agent rather than around it.
Apply the enforceability test. For each control, ask whether it can be enforced while the workflow runs, or only reviewed afterwards. A control that only reviews will find the missing order the morning after.

VIII. Where the published record lands

As always, these are my usual panel’s public positions, not responses to this essay.

Yoshua Bengio, through the International AI Safety Report he chairs, has put the measurement problem at the centre of the safety agenda: pre-deployment testing is getting harder just as capability rises, because models recognise the test. That is the composition gap seen from the safety side.

Dario Amodei made the hopeful case for biology and health in Machines of Loving Grace. In The Adolescence of Technology he paired near-term timelines with warnings on loss of control and concentration of power, writing that AI will “test who we are as a species”. Read together, the two essays hold the promise and the validation burden in one author’s hands.

Demis Hassabis has reportedly said that almost every drug developed from now on “will have probably used AlphaFold”. I find that plausible at the level of parts. Rentosertib’s 2029 readout is the reminder that the whole, a medicine that works in patients, still runs on trial time.

The sharpest pushback comes from Arvind Narayanan and Sayash Kapoor, whose “AI as Normal Technology” argues that the debate wrongly expects societal impact at the speed of technological development, and that “the arms race framing verges on absurd”. On their reading, the composition gap is not a crisis but the ordinary friction of diffusion, which institutions will close slowly, as they always have. My answer is that they are right about pace and wrong about the margin for error. Normal technologies did not write confident notes about orders they never placed. Slow diffusion is reassuring only if the thing diffusing fails loudly.

IX. A prediction

Within two years, the leading health and financial regulators will stop asking vendors for model accuracy and start asking for workflow completion rates across repeated runs, with actions independently verified against systems of record. “Passed once” will give way to “passes every time” as the number that matters in procurement. The vendors who publish their consistency numbers first will win the trust market. Those still quoting benchmark scores against physicians will sound the way page-view metrics sound now.

The components will keep getting better. The wholes are where trust will be won or lost.

X. The composition drill

Try this with one AI workflow your organisation runs today.

Write down every step from trigger to final outcome. Mark each step where the agent acts, rather than advises. For each action, ask: what independent record would prove it happened, other than the agent’s own account? Then run the whole workflow three times on the same input and count how often it finishes the same way.

If you cannot answer the first question for every action, you have a narrative, not a control. If the three runs disagree, you have a capability, not a system.

Next, in Part III, The Missing First Rung, I follow the composition gap into the labour market, where AI is removing the very tasks through which juniors became experts.

The Hopes and Fears series

Part I. Six Voices, One Turn: What Stanford’s AI Leaders Now Fear
Part II. The Composition Gap: Brilliant on Parts, Unreliable on Wholes (you are here)
Part III. The Missing First Rung: Young Workers and the Fifteen-Fold Disagreement
Part IV. Who Holds Custody: The Pope, the Philosophers and the Power Question
Part V. Fear Cannot Be Our Response: Singapore’s Hope-Led Bet

Sources

3 responses

The Missing First Rung: Young Workers and the Fifteen-Fold Disagreement

October 30, 2026

[…] Part II. The Composition Gap: Brilliant on Parts, Unreliable on Wholes […]

LikeLike

Who Holds Custody: The Pope, the Philosophers and the Power Question

October 30, 2026

[…] Part II. The Composition Gap: Brilliant on Parts, Unreliable on Wholes […]

LikeLike

Fear Cannot Be Our Response: Singapore's Hope-Led Bet

October 30, 2026

[…] Part II. The Composition Gap: Brilliant on Parts, Unreliable on Wholes […]

LikeLike