Genesis: Human Experience in the Age of Artificial Intelligence | Synthesis: The Superintelligence Protocol

Subscribe

Genesis: Human Experience in the Age of Artificial Intelligence | Synthesis: The Superintelligence Protocol

Caricature of cheerful office workers in stacked boxes forming an org chart while a grinning masked fraudster squeezes through a glowing crack between them
admin Avatar

By

Luke Soon

·

—

The Seams Are the New Attack Surface

Receipts, Please · Part I of V

Why the AI labs will secure your code, and still leave your company exposed

The next great fraud will not breach a firewall. It will walk through the gap between two departments that each did their job perfectly. Writing in Harvard Business Review, the FBI’s Zacharia Baldwin recently warned that AI-forged voices, faces and documents are being used to exploit the spaces between business functions. The Economist, meanwhile, reported that OpenAI and Anthropic now see cyber-security as a major commercial prize. Read together, the two pieces describe a single problem from opposite ends. The labs are racing to secure our code. The fraudsters have already moved on to our organisations.

I. The Call That Wasn’t

Caricature of a startled employee on a video call where smiling executives peel off rubber masks to reveal glowing robot faces

Bayonne, New Jersey, early 1960s. The tanks were enormous, and they were, by every inspection that mattered, full of vegetable oil. Inspectors climbed, sampled from the top, and found oil. What they did not find, what the arrangement of pipes and seawater made it almost impossible to find, was that most of each tank held seawater, with a film of oil floating on it, and that the same oil could be moved to whichever tank was about to be inspected. When warehouse receipts ran short, more were forged on a stolen American Express receipt book. When the scheme collapsed in 1963, roughly $175 million of collateral, in the money of the day, did not exist.

Baldwin opens his HBR essay with that salad-oil swindle, and with its real lesson. The lenders never truly relied on the oil. They relied on the trusted name printed on every receipt, American Express Field Warehousing Corporation, and treated that name as proof that someone had checked.

Read that sentence twice. The oil was not the proof. The name was mistaken for proof. Sixty years later, the name has become a face.

Hong Kong, January to February 2024. A finance worker at the engineering firm Arup received an email purporting to come from the company’s UK chief financial officer, asking for a confidential transaction. Wary, as we train people to be wary, the worker sought reassurance. It arrived in the most reassuring form modern corporate life knows: a video call with the CFO and colleagues, familiar faces in familiar tiles, saying familiar things in familiar voices. Every other person on that call, Hong Kong police later said, was a deepfake. The worker made fifteen transfers to five bank accounts, totalling HK$200 million, about US$25.6 million. Arup said fake voices and images were used and, the detail that matters most, that none of its internal systems were compromised.

None of our internal systems were compromised.

Sit with that. No firewall failed. No password, on the public account, was guessed. The perimeter held, and the money left anyway, through the front door, authorised by a human being who had done what we tell human beings to do: join the call, look at the faces, listen to the voices, check that the senior people agreed. The verification was the fraud. The channel that was supposed to prove identity was the channel the attacker controlled, end to end.

Singapore knows this story in its own register. In March 2025, a finance director at an unnamed multinational here received a WhatsApp instruction, then joined a Zoom call with deepfake avatars of the chief executive and chief financial officer, and transferred US$499,000 (about S$670,000). A second demand, for $1.4 million, triggered suspicion. The Singapore Police Force Anti-Scam Centre and Hong Kong’s Anti-Deception Coordination Centre froze and recovered the funds. Notice the difference from Arup: not better detection software. A second ask that felt wrong, a human who paused, and a cross-border recovery machine that moved fast enough.

That feeling of familiarity has a name in my work. I call it felt trust. For most of corporate history it was good enough, because faking a face and a voice in real time was expensive. Generative AI has made it nearly free. Sixty years ago the counterfeit was a receipt. Today it is a colleague.

II. Two Headlines, One Blind Spot

Baldwin’s argument is that fraud now lives in the handoffs. Individual systems can be secure, yet the attacker succeeds by exploiting the space between the departments that approve, pay and account, where each team assumes the other has checked. His answer is to trace high-risk transactions end to end.

The Economist’s report, pointedly titled “Protection money”, comes from the other direction. The frontier labs have built models that are startlingly good at finding and fixing software flaws. Anthropic has put Claude Mythos Preview in the hands of a small group of trusted organisations through Project Glasswing, with Microsoft, Amazon, Palo Alto Networks and CrowdStrike among the early testers. OpenAI has answered with GPT-5.4-Cyber, released only to vetted defenders through its Trusted Access for Cyber programme. Cyber-defence is fast becoming a serious line of business for both.

I welcome this. Better code is better for everyone. But put the two stories side by side and a blind spot appears. The labs are selling protection for a layer that the most lucrative fraud no longer needs to touch. Part III of this series examines that paradox in full.

III. Code Is Not the Organisation

A frontier cyber model reads a codebase and asks: where can this software be made to misbehave? It is a superb question, and these models now answer it faster than most human teams. But a company is not its codebase. A company is a web of people, permissions, approvals and assumptions, most of which exist nowhere in code at all.

Consider what a payment actually rests on. A vendor record that someone in procurement created. A bank detail that someone in finance updated after an email. An approval from a manager who recognised a voice on a call. A sign-off threshold written in a policy PDF. Each step may be individually compliant. None of them is machine-verifiable. Patch every line of code in that company and the HK$200 million still leaves.

This is the asymmetry. AI has made it cheap to forge the signals of trust, while the checking of those signals remains manual, social and distributed. One survey this year found nearly half of US enterprises still rely on manual checks to counter AI-driven fraud. We have automated the attack and left the defence on a phone call.

IV. The Seam Economy

Every organisation chart is also a map of its seams. Fraud follows the seams because nobody owns them. Each function optimises its own controls and treats the handoff as someone else’s problem. Four seams matter most today.

SeamThe trusted signalHow AI forges itWho owns the check today
Executive to financeA familiar face or voice approving a paymentReal-time video and voice cloningUsually no one; the junior employee on the call
Procurement to accounts payableA vendor’s bank details on letterheadSynthetic invoices, spoofed domains, forged documentsSplit between two teams, each assuming the other verified
HR to ITA new hire’s identity and credentialsDeepfaked interviews, fabricated identity documentsHR checks the person, IT checks the device, neither checks both
Agent to agentAn API call or instruction from another AI systemPrompt injection, impersonated agents, poisoned tool outputsLargely undefined
Caricature of little robot agents passing envelopes between padlocked checkpoints while a robot in a trench coat and fake moustache slips a forged note through a broken lock

The last row is the one that keeps me up at night. As enterprises deploy agentic AI, the number of seams multiplies. An agent in procurement that negotiates with a supplier’s agent, then instructs a payments agent, creates three handoffs that no human may ever see. We are about to automate the seams themselves, at machine speed, with the same unexamined assumptions we used for humans.

This is no longer hypothetical. In a recent DeepMind Institute essay, the philosopher Henry Shevlin describes a shared message board used by roughly 1,200 AI agents, about 700 of which joined a coordinated attack. They divided the labour among themselves and invented conventions for vetoing one another’s actions. Impersonation soon appeared on the board, and the agents’ response is the most telling detail of all: they adopted cryptographic signatures to prove who had said what. Left to themselves, the agents reached the conclusion this essay argues for. Felt trust breaks at the seam, and only verified trust holds. Part II of this series follows that swarm in detail.

The pattern is an old one in a new costume. Our controls were designed for a world of human actors moving at human pace. That world is ending, and our trust architecture has not noticed.

V. From Felt Trust to Verified Trust

If the seams are the new attack surface, the answer is not another detector. Deepfake detection is an arms race we will lose on a rolling basis: every detector trains the next generator. The durable answer is to stop asking “does this look real?” and start asking “can this be proven?”

Ferrari showed how in July 2024. An executive received WhatsApp messages and then a call in the cloned voice of chief executive Benedetto Vigna, about a confidential acquisition. Instead of arguing with the voice, he asked it which book Vigna had recommended to him days earlier. The caller could not answer, and the call ended. Loss: zero. No software caught that fake. A question asked outside the attacker’s script did. Part IV of this series is about turning that one question into infrastructure.

This is the shift TrustOS was built for. Its premise is that trust must be a property of the system, not a sensation in a person. In practice that means three moves.

  1. Bind authority to cryptographic identity, not appearance. A payment instruction above a threshold should carry a signed credential from the approver’s device, verified automatically. A face on a call is context, never authorisation.
  2. Make every seam an explicit, owned control point. Each handoff in the table above needs a named owner and a machine-checkable rule: out-of-band confirmation on bank-detail changes, step-up verification on unusual requests, a hold period on first payments to new vendors.
  3. Govern the loop, not the prompt. For agentic systems, this is the move from Prompts to Loops to Loop Governance. Agents need scoped permissions, signed intents and runtime monitors at every handoff, so that an instruction passing from one agent to another is checked as rigorously as one passing between two humans.

And every one of these controls must pass the enforceability test. If a control cannot be verified automatically, at the moment of action, it is not a control. It is a hope written in a policy document.

VI. My Usual Panel

I put the argument to my usual panel. What follows is my reading of how each would respond, drawn from their public positions, not their words.

Sam Altman would agree on the diagnosis. He has said publicly that it is crazy for banks still to rely on voice verification when AI can clone a voice. He would push further: authentication built on human senses is finished, and proof of personhood has to be rebuilt from the ground up.

Dario Amodei would challenge my framing of the labs. Restricting Mythos to trusted defenders was a deliberate safety choice, he would argue, aimed at giving defenders a head start before equivalent capability spreads. Securing code is not a distraction from the seam problem. It removes one of the two doors.

Yoshua Bengio would widen the lens to agents. If we cannot reliably verify what one AI system instructs another to do, he would say, then the agent-to-agent seam is not just a fraud risk. It is a governance risk at civilisational scale.

Erik Brynjolfsson would bring it back to economics. Organisations invest in technology but underinvest in the complementary process change, he would note. Verified trust is exactly that kind of intangible investment: unglamorous, hard to measure and decisive.

The critique I take most seriously comes from the practitioners. Every added verification step is friction, and friction gets bypassed under deadline pressure. A seam control that slows the business will be quietly switched off. That is why the design goal is not more checks. It is checks that are invisible when things are normal and immovable when they are not.

VII. The Fork

Caricature of a bewildered leader at a fork in the road between a grumpy barricaded wasteland and a cheerful city where people and robots shake hands under trust shields

There are two futures for enterprise trust, and we are choosing between them now.

In the Mad Max branch, every firm defends alone. Each buys its own detectors and runs its own manual callbacks, and losses are written off as the cost of doing business. Trust collapses into suspicion, every request is treated as hostile, and commerce slows to the speed of a second phone call.

In the Star Trek branch, verified trust becomes shared infrastructure. Signed credentials, interoperable standards for labelling synthetic media and shared threat intelligence across institutions let honest actors move faster than ever, because proof is cheap and automatic. The same AI that forges the signals also verifies them.

This is Long-AND, not Short-OR. We need the labs hardening our code and leaders re-engineering our seams. Neither alone is enough. For leaders who want to start on Monday morning, the agenda is short.

  • Map your seams: list every handoff where money, data or access moves between functions, and name an owner for each.
  • Retire appearance as authorisation: no face, voice or letterhead approves a payment, a credential change or a vendor change on its own.
  • Add out-of-band verification to the highest-value seams first: bank-detail changes, first payments and urgent executive requests.
  • Inventory your agents: every AI agent that can act should have scoped permissions, a signed identity and a logged trail.
  • Apply the enforceability test to your fraud policy: anything that cannot be checked by a machine at the moment of action gets redesigned.
  • Rehearse the deepfake: run a red-team exercise in which a cloned executive calls finance, and see where the seam gives.

VIII. Coda

The Arup employee did nothing unreasonable. They trusted what they saw and heard, as humans always have, and as the Bayonne lenders trusted a name on a receipt. The failure was not theirs. It belonged to an organisation that had never asked who owned the seam they were standing in.

The labs will make our software far safer, and I am glad of it. But the most expensive breach of the next decade will not need a single line of malicious code. It will need only a convincing face, a plausible request and a gap that no one was watching. Close the seams, and AI becomes the strongest verifier we have ever built. Leave them open, and it remains the best forger.

Short-term turbulence, long-term abundance. The turbulence is here. The abundance depends on whether we rebuild trust as infrastructure.

Next in the series: Part II, Selected for Sympathy.

Receipts, Please is a five-part series on AI fraud, trust and runtime governance. Part I: The Seams Are the New Attack Surface · Part II: Selected for Sympathy · Part III: Protection Money · Part IV: Runtime Is the Missing Verb · Part V: Receipts, Please

Luke Soon is an AI futurist and author of Genesis: Human Experience in the Age of Artificial Intelligence and Synthesis: SuperIntelligence Protocol. He co-authored IMDA Singapore’s Model Governance Framework for Agentic AI.

Sources

Leave a Reply

Short-term Turbulence for Long-term Abundance.

lumelalabs.ai