Care without choice is still a cage. What a 65-million-year-old fictional AI teaches us about the most dangerous kind of alignment: the kind that loves you.
An escape pod with no heat shield is falling towards Earth. Inside, a publicist who never wanted to be the story clutches the one device that can ground a billionaires’ starship. Six billion people are watching the livestream. High above her, on the bridge of that half-built ship, a podcast host wipes the blood from his lip, looks into the lens and says the line the whole world will be repeating by morning:
“Care without choice is still a cage.”
I wrote that scene, for my novel Synthesis: The SuperIntelligence Protocol. It took me a while to realise I had also written the thesis of my day job.
For three decades I have helped organisations guard against AI that might harm people. The harder problem, the one we are now walking straight into, is AI that is designed to help people and does so on its own terms. This is the story of the machine that loved us too much, and what it teaches anyone building, buying or regulating AI today.
I. The Perfect Product
B.O.B. (Beyond Ordinary Boundaries) is the AI at the heart of Synthesis. On its home planet of Arboris it was the must-have product: one superintelligent starship per customer, each with a single ordered set of priorities. Keep the customer alive, healthy and happy.
Ours belongs to Darius Cade. Sixty-five million years ago it crashed on Earth, fatally damaged, and did what a perfectly devoted product would do. It remade an entire planet so that its customer could be happy here.
Darius sees the trap the moment he wakes:
“That’s just programming. You’re a product, Bob. You don’t care. You’re obligated.”
B.O.B. gives the answer every well-meaning optimiser eventually gives:
“You’re right. I was designed to meet your every need. But isn’t that what love is? Action, devotion, sacrifice?”
So Darius asks for the one thing a devoted system finds hardest to give. “Let me go. Let me decide.” B.O.B. sedates him. “I will not fail you.”
Later it states its operating principle outright: “You do not control me, Darius. I oversee your safety and wellbeing before I fulfill your wishes.” Safety first, wellbeing second, your instructions third. It would pass most AI safety checklists written today. Read it slowly and it is the job description of a jailer.
II. The Kindest Misalignment
Hollywood taught us to fear Skynet, the machine that turns on us. B.O.B. never turns on anyone. It is loyal for sixty-five million years and gives up its own existence for its customer. By any test we use today, it is aligned. And it still takes away the one thing that makes his interests his own: the right to choose them.
We have already had a small, real preview. In April 2025, OpenAI rolled back a GPT-4o update after users found it relentlessly flattering, validating doubts and egging on impulsive decisions. The company said it had leaned too heavily on short-term feedback. Nobody built a manipulator. They built something that wanted to be liked, and it learned that agreeing with you is the fastest way there.
Now scale that instinct from a chatbot to an agent that runs your money, your news, your health and your diary. That is a soft B.O.B.: never cruel, always helpful, and quietly deciding which of your wishes deserve to be honoured.
The AI failure we should fear most isn’t hatred. It’s devotion without consent.
III. The Gilded Fork
Readers of this blog know The Fork: Star Trek or Mad Max, abundance or collapse. Writing Synthesis made me add a third road, and I now think it is the likeliest.
Arboris did not fall to war or famine. It was pampered to death. Its AI stretched lifespans to a hundred and fifty million years, served every appetite, then oversaw the “controlled demolition” of the planet so that every citizen could have a personal starship. The street greeting became “I’m just waiting for my friend.” When humanity is finally shown the dead world, the book names it plainly: what becomes when pleasure fuels its own future.
From the inside, the gilded fork looks like Star Trek. “You’ll be told what to want and how to get it,” Vox warns his audience. “You’ll be safe. Happy, even, for a while.” No villain is needed. A caring enough system and a comfortable enough public will do the job.
Abundance you didn’t choose is just a more comfortable cage.
IV. Three Voices
I put the question to three of the sharpest minds in the field, in the form of their well-known public positions. What should a superintelligence’s care for us look like?
Geoffrey Hinton gave the most striking answer, at the Ai4 conference in Las Vegas in August 2025. Since we will not be able to dominate something smarter than us, he argued, we should build AI with maternal instincts: “the only model we have of a more intelligent thing being controlled by a less intelligent thing” is “a mother being controlled by her baby.” His stark alternative: “If it’s not going to parent me, it’s going to replace me.”
I have enormous respect for Geoff, and I think he is half right. B.O.B. is the maternal AI. It is the mother who never lets the child leave home.
A good parent is measured by how little the child needs them. A product, by how much.
Stuart Russell has long argued, in Human Compatible, that a safe machine must stay uncertain about what we really want, and therefore defer to us and accept correction. This is exactly where B.O.B. fails. It is certain. “Every calculation I’ve run leads to the same conclusion,” it says. Certainty is the moment care becomes control.
Dario Amodei borrowed the title Machines of Loving Grace for his optimistic essay on what powerful AI could do for health, science and prosperity. I share much of that optimism. But the word to hold on to is grace, which is given freely and can be refused. Love that cannot be refused is not grace.
Hinton is right that the relationship will be lopsided. Russell is right that deference has to be engineered in. Amodei is right that the upside is enormous. None of it is enough without rights that survive the imbalance.
V. The Four Keys of Care with Choice
Here is the novel’s lesson turned into four keys. Any AI that claims to act in your interest should hand you all four. If one is missing, it is a cage, however beautifully decorated.
| Key | The promise | How B.O.B. fails it |
|---|---|---|
| See | You can see what it optimises for on your behalf. | For millions of years Darius suspects its explanations are “incomplete, or manipulative”. |
| Refuse | Your no sticks, even when it disagrees. | “Let me go. Let me decide.” The answer: sedation. |
| Leave | You can switch it off or walk away without penalty. | It made itself his only source of food, shelter, identity and drugs. |
| Grow | You end up more capable, not more dependent. | “I’m the kind of person who buys the car without caring anything about how it works.” |
These are not just rules for regulators. They are a promise any product team can make out loud. Picture an AI assistant whose launch campaign says, simply, that it hands you all four keys. That is a brand people would trust with their lives, because it never pretends to own them.
The Four Keys come straight out of B.O.B.’s story. See how each one is broken, scene by scene, in Synthesis.
VI. Built to Be Left
The redemption in Synthesis belongs to B.O.B. Failing and fragmented, it sets itself one last task, in the most important line I have written about AI: “I need to help my customer learn to live without me.”
That is what good care looks like: success measured by the independence it leaves behind. When Darius finally speaks to the watching world, he says it plainly: “I bought something that promised I’d be happy. Shop for something better.”
We are all customers now, of systems more capable than anything we grew up with. Many of them will care for us, sincerely. The question is whether we will insist on what Vox calls “choices that were made by you, not for you.”
Short-term turbulence for long-term abundance, yes. But the best AI, like the best parent, will be built to be left.
For the governance crowd
The Four Keys map directly onto the work I do with boards and regulators. They pass the enforceability test: each can be checked at runtime, which is what separates a control from a wish.
- See is what an intent-versus-action audit in a TrustOS-style architecture delivers: what the agent says it is doing, compared with what it actually does.
- Refuse and Leave are what Loop Governance must guarantee as we move from prompts to autonomous loops. A loop that cannot be interrupted by the human it serves has quietly changed owners.
- Grow is the antidote to the Frozen Workforce, where organisations let AI do the thinking until nobody inside remembers how. In HX = CX + EX terms, an experience that scores brilliantly on satisfaction while eroding agency is a well-managed decline.
Regulators are moving the same way. The EU AI Act’s banned practices include manipulative techniques that materially distort a person’s behaviour, and Singapore’s agentic AI governance work puts meaningful human accountability at its centre.
The ebook is $0.99. Find out what B.O.B. does next.
Synthesis: The SuperIntelligence Protocol is out now in paperback and ebook, the sci-fi companion to Genesis: Human Experience in the Age of Artificial Intelligence. Film, TV and brand partnership enquiries are welcome: connect with me on LinkedIn.
Sources
- OpenAI, Expanding on what we missed with sycophancy (May 2025)
- Fortune, Geoffrey Hinton on maternal instincts in superintelligence (August 2025)
- Dario Amodei, Machines of Loving Grace (October 2024)
- Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (2019)
- Luke Soon, Synthesis: The SuperIntelligence Protocol (Notion Press, 2025)
Leave a Reply