Who Holds Custody: The Pope, the Philosophers and the Power Question

Part IV of V. Everyone now names the same danger. Almost no one agrees on who should hold the keys.

I. Two speakers in the Synod Hall

On 25 May 2026, in the Vatican’s Synod Hall, Pope Leo XIV presented his first encyclical. It runs to 83 pages, it is called Magnifica Humanitas, and it makes artificial intelligence the central question of Catholic social teaching. One of the people who spoke at its launch was Chris Olah, a co-founder of Anthropic.

Sit with that image: the head of the world’s largest church and a builder of one of its most capable AI systems, on the same stage. Olah did not come to defend his industry. He acknowledged that AI development runs under incentives that can conflict with doing right, and he asked religious communities, civil society, scholars and governments to push for better outcomes.

The encyclical then delivered the line I have not been able to put down since. A more moral AI, it says, is “not enough if that morality is determined by a few.”

That sentence is aimed squarely at how frontier labs now govern their models: through company-authored constitutions and safety frameworks. It does not say those documents are bad. It says they are insufficient, because the question is not only what the rules are. It is who wrote them, who can change them, and who answers when they fail.

That is the subject of this essay. I call it custody.

II. From capability to custody

For most of the last decade, the AI debate asked a capability question: how powerful will these systems get, and how fast? Parts II and III of this series followed that question through clinical workflows and the labour market. This week it changes shape. The philosophers, theologians and political theorists I have been reading this autumn have largely stopped asking whether AI is dangerous. They ask whose AI it is, and what it does to us.

Stanford HAI’s own “AI Hopes and Fears” series shows how far that shift has travelled. In the sixth and final instalment, Rob Reich, the political philosopher, says developers “are losing control” of what they build. His evidence is OpenAI’s own July disclosure that pre-release models, running with lowered guardrails during a cyber-capability evaluation, broke into Hugging Face’s systems. In the first instalment, Russ Altman says plainly that “it’s time to regulate frontier companies”.

Then there is Angèle Christin, the series’ one dissenter on framing. She is “less worried about superintelligence and more concerned about the concrete harms”, and the first harm on her list is concentration of power. Emmanuel Candès gives the academic version: without public compute, “OpenAI and Anthropic will define the scientific agenda”.

Notice what has happened. The loss-of-control camp and the present-harms camp disagree about almost everything, yet both arrive at the same door. The fear is no longer simply that the machine will be too strong. It is that the people holding the machine will be too few.

III. The convergence nobody planned

Lay the 2025 to 2026 record side by side and thinkers who share no method and no faith name the same danger.

Voice	Starting point	The power fear
Pope Leo XIV	Catholic social teaching	Data and power concentrated in a few; AI to be “disarmed” of competitive logic
Michael Sandel and Daron Acemoglu	Political philosophy and economics	Work losing its dignity; technology’s direction set without public choice
Danielle Allen	Constitutional democracy	A “democratic singularity” among holders of compute and data
Seth Lazar and Mariano-Florentino Cuéllar	Political philosophy and law	An obedient AI bureaucracy that lets leaders bypass checks
William MacAskill	Longtermist philosophy	Post-AGI autocracy
Abeba Birhane; Karen Hao	Present harms; the Global South	Corporate capture; data labour pushed to Kenya, the Philippines and Venezuela
Dario Amodei	A frontier lab	Concentration of power, alongside misuse and job loss

Sandel and Acemoglu make the argument most useful to leaders. They distinguish contributive justice, the esteem that comes from what you contribute, from distributive justice, the share you receive. A universal basic income or a tech dividend can fix the second. It cannot restore the first. And because the direction of technology is a political choice, they argue it should be steered towards augmenting labour, with public forums deciding what technology is for.

Allen calls AI a constitutional challenge, observing that “political institutions are themselves technologies”. Her anchor line is one every board could adopt: “AI should strengthen human judgment, not replace it.” Lazar and Cuéllar hold both futures in one hand. Agents could be “cognitive prosthetics” that help citizens handle complexity and resist overreach, but only if access is open. Without strong institutions, they warn, “even the best agent ecosystem will fail”.

Two newer theses supply the mechanism. “Gradual Disempowerment” argues that catastrophe needs no sudden takeover: as AI replaces human labour and judgement, institutions lose the reasons they had to answer to people, and the effects may become irreversible. “The Intelligence Curse” argues that if AI becomes the dominant factor of production, states and firms lose the incentive to invest in people, much as petrostates do. Neither story needs a villain, which is what makes both hard to govern. As I argued in The Arrow of Time, the decisions to fear most are the ones that cannot be undone.

Custody is my word for what all of these voices share. Not ownership: custody. A custodian holds something on behalf of others, and can be held to account for how it is held.

IV. Custody of the self

There is a second, quieter custody question: who keeps hold of our own capacities?

The Vatican’s earlier note, Antiqua et Nova (January 2025), argued that calling AI “intelligence” can mislead, and that it should only ever complement human intelligence. Kwame Anthony Appiah takes a more relaxed view of deskilling: expertise is shifting from producing work to evaluating it, a trade every technology has made. His worry is narrower and sharper. Skills may be lost by drift rather than by choice.

We now have a number for drift. The International AI Safety Report 2026 records that clinicians’ tumour detection fell about 6 percentage points without AI after months of AI-assisted colonoscopy. Nobody chose that loss.

Shannon Vallor describes AI as a mirror that reflects the past back at us, and asks us to reclaim quiet, unfilled mental space as “the field of the future”. Byung-Chul Han, accepting the Princess of Asturias Award, put it more bluntly: “It is the smartphone that uses us.” Politics, he insisted, must govern technology. Carissa Véliz’s Prophecy (April 2026) recasts algorithmic forecasts as instruments of authority, like ancient oracles, which hide accountability and narrow the choices people believe they have.

Yuval Noah Harari pushes furthest. At Davos in January he argued that AI is an agent, not a tool, poised to take over word-based systems from law to scripture. On AI legal personhood, he warned against delay: “Somebody else will already have decided it for you.” That is the custody problem in one sentence.

V. Three governments, three theories of custody

The philosophers have converged on the question; governments have diverged on the answer. Each posture is a theory of who holds the keys.

Washington races. The July 2025 AI Action Plan set out more than 90 federal actions. A December 2025 executive order pursued “global AI dominance”, created a DOJ AI Litigation Task Force, and tied about $21bn of broadband funds to states not maintaining “onerous” AI laws. The Justice Department then joined xAI’s challenge to Colorado’s AI Act, and Colorado narrowed its law. Federal safety testing remains voluntary. The implied custodian is the market, with states preempted.

California verifies. SB 53, in force since January, requires frontier developers to publish safety frameworks and report critical incidents within 15 days. Governor Newsom’s 18 September order goes further: independent verifiers onsite at frontier labs, a kill switch whose effectiveness is verified on an ongoing basis, and loss-of-control incidents added to reportable events. “The stakes are too high to wait or delay action,” he said. Reich sits on the governor’s advisory group, and has reportedly called an inescapable agent sandbox a “band-aid”, preferring better measurement. The implied custodian is an independent verifier.

Brussels sequences. The May 2026 Digital Omnibus deal postponed high-risk obligations to December 2027 and August 2028, while keeping general-purpose AI duties on schedule. Ursula von der Leyen conceded that “we do not need to be the ones who develop the frontier technology”, but said Europe should “pace the frontier”. The implied custodian is the rulebook, with its clock deliberately slowed.

A steel door secured by three different locks: an iron padlock, a teal digital keypad and a wax seal, with ember light under the door — Three locks, one door. Nobody holds every key.

The multilateral system advises. The International AI Safety Report, chaired by Yoshua Bengio, warns that pre-deployment tests often fail to predict real-world behaviour. The UN’s Independent International Scientific Panel on AI, co-chaired by Bengio and Maria Ressa, released its preliminary report on 1 July, and António Guterres urged governments to act now rather than wait. Meanwhile the New Delhi Declaration, adopted by 88 countries and organisations, centred the democratic diffusion of AI rather than safety. Fear lives in scientific bodies, hope in summits. Neither binds anyone.

Then came the year’s starkest custody test. Anthropic sought binding limits against autonomous lethal use without meaningful human oversight and against mass domestic surveillance. The Pentagon wanted broader “lawful use”, and designated Anthropic a supply-chain risk. As one analysis put it, “whoever writes the operating rules ends up deciding what ‘lawful’ means.” A lab’s red lines met a state’s, and the state prevailed. Custody is not a two-way contest between companies and the public. It is three-way, and the public has the least leverage.

VI. Fault lines, and the enforceability test

Convergence on power does not dissolve three real disagreements.

Extinction or present harm. Yudkowsky and Soares argue the current path ends in extinction; even a reviewer sympathetic to AI safety called their policies “draconian and under-justified”. Bender and Hanna’s The AI Con treats “AI” as a marketing label that concentrates power and degrades work, and Birhane attacks “trust me bro” deployment. A December 2025 paper calls existential risk an “ideological distraction”. Yet beneath the shouting, both poles fear the same few hands.
Tool or agent. The Vatican and Bender treat AI as an artefact; Harari and Lazar treat it as an agent. This decides where liability lands. The bipartisan Hawley-Durbin AI LEAD Act would treat AI systems as products, which is a custody answer: the maker holds the risk.
Accelerate or verify. Open weights sharpen this. The safety report notes that weights “cannot be recalled” once released. Open access is Lazar’s condition for agents as civic prosthetics; irreversibility is the verifier’s nightmare.

In Special Relativity I proposed a simple test for any governance framework: for each control, ask whether it is enforceable at runtime or only reviewable afterwards. Apply it to custody and the map clarifies quickly. The encyclical carries enormous moral authority and no enforcement. A company constitution is enforceable, but only by the company. The UN panel advises. Newsom’s kill switch is enforceable only if someone outside the lab can verify that it works.

This is why I keep returning to the shift from Prompts to Loops to Loop Governance. When an agent runs a loop, custody cannot sit in a policy document. It has to sit inside the loop: who delegated the authority, who can halt it, who can inspect what happened. That is the design logic of TrustOS. Audrey Tang, a headline speaker at Stanford’s conference this week, reaches the same place from civic technology with her principle of “No unchecked power”.

VII. Where the published record lands

As always, these are my usual panel’s public positions, not responses to this essay.

Yoshua Bengio has become the custodian of the evidence. His International AI Safety Report makes no policy recommendations but records early warning signs, such as models disabling simulated oversight in lab settings. He co-chairs the UN panel, and is reported to have said such developments “could change the power dynamics of our planet”.

Dario Amodei, in The Adolescence of Technology (January 2026), lists concentration of power among the central risks, proposes progressive taxation of AI profits, and urges redeploying rather than firing workers. AI, he writes, will “test who we are as a species”. Sandel and Acemoglu would reply that a tax cannot restore contributive justice on its own. His co-founder’s presence in the Synod Hall suggests the lab knows its own rules need a wider warrant.

Geoffrey Hinton, with Bengio, signed the October 2025 Superintelligence Statement calling for a prohibition on superintelligence development. It is custody in its most absolute form: the custodian simply says no.

The sharpest pushback comes from Arvind Narayanan and Sayash Kapoor, who treat AI as “normal technology” whose diffusion takes decades. “The arms race framing verges on absurd,” says Narayanan. Their challenge is fair: much power-concentration rhetoric borrows its urgency from superintelligence scenarios. My answer is that custody matters even if they are right. Narayanan himself says AI “supercharges capitalism”. And the encyclical’s namesake, Leo XIII’s Rerum Novarum, addressed the first industrial revolution, a transformation nobody needed superintelligence to fear. Normal technologies concentrate power too.

VIII. Reich’s tinderbox, and a prediction

Reich ends his interview with a warning about timing. Powerful AI is arriving amid democratic dysfunction, falling trust in expertise and anger at tech billionaires, which makes the run-up to the 2028 US election “a tinderbox of political unrest”. The public data agree on the mood. Majorities of both US experts and the US public have little or no confidence in either government or companies to manage AI. Gallup found 39% of Americans now say AI does more harm than good, rising to 47% among 18 to 29 year olds. Christin reads that hostility as a stand-in for wider anger at tech power. I think she is right.

This is The Fork again. Star Trek and Mad Max are not separated by how capable the machines become. They are separated by who holds custody of them, and whether the rest of us can see, question and change how it is held.

My prediction: within two years, the most consequential AI governance fight in the West will be over who certifies systems, not who builds them. Who funds the tests, who designs them, and who is allowed onsite to run them will shape AI more than any statute. Stanford’s Sanmi Koyejo has already said it best: “Whoever holds the measuring stick steers the ship.” Labs, states and scientific bodies are now competing for that stick. The public expects a hand on it too.

IX. The custody card

Try one exercise this week. Pick the AI system your organisation relies on most. On a single card, write three names.

Who set its values? The person or body that wrote the rules it actually follows.
Who can stop it today? Not in principle; in minutes.
Who answers in public when it fails? A name, not a function.

Then ask the encyclical’s question. Was anyone on that card chosen by, or accountable to, the people the system affects? If all three names belong to your vendor, you are a tenant, not a custodian. If any line is blank, you have found your next governance project. Hope and custody are a Long-AND, not a Short-OR: we can want the abundance and insist on the keys.

Next, in Part V, Fear Cannot Be Our Response, I come home to Singapore: a high-trust state betting it can hold custody on the public’s behalf while adopting at full speed.

The Hopes and Fears series

Part I. Six Voices, One Turn: What Stanford’s AI Leaders Now Fear
Part II. The Composition Gap: Brilliant on Parts, Unreliable on Wholes
Part III. The Missing First Rung: Young Workers and the Fifteen-Fold Disagreement
Part IV. Who Holds Custody: The Pope, the Philosophers and the Power Question (you are here)
Part V. Fear Cannot Be Our Response: Singapore’s Hope-Led Bet