Genesis: Human Experience in the Age of Artificial Intelligence | Synthesis: The Superintelligence Protocol

Subscribe

Genesis: Human Experience in the Age of Artificial Intelligence | Synthesis: The Superintelligence Protocol

admin Avatar

By

Luke Soon

·

—

Leadership in the Age of Agentic and Autonomous Intelligence: Architects of Trust = Governing Loops not People

Architects of Trust?

You can delegate the work. You cannot delegate the answer to “who decided?”

The question I am asked most often in boardrooms is no longer “what can AI do?” It is “what am I for?” Chief executives, permanent secretaries and partners ask it in different words, but the anxiety underneath is the same. If machines can analyse faster, draft better and increasingly act on their own, what is left for the person at the top?

My answer, after more than 25 years working in AI and a good part of the last two helping to write governance frameworks for agentic systems, is that leadership is not shrinking. It is concentrating. Everything that can be delegated will be. What remains is the part that never could be: deciding what matters, interrupting a convincing answer, and owning what happens next. Leadership in the age of AI is the last checkpoint.

That sounds like a comfort. It is actually a warning. The evidence from the past eighteen months suggests most leaders are not yet ready to be that checkpoint, and that AI itself is quietly eroding the very judgment the role depends on.

I. From answers to actions

The shift is no longer theoretical. On a widely used benchmark of agents doing real tasks on a computer, accuracy rose from about 12% to 66% in a single year; the best agent is now within six points of a human. Yet agents still fail roughly one attempt in three. That is the most dangerous zone there is: good enough to hand work to, not good enough to stop checking.

Leaders are moving anyway. In one global survey, 82% of leaders said they expect to add “digital labour” to their workforce within 12 to 18 months. A leading research house forecasts that at least 15% of day-to-day work decisions will be made autonomously by agents by 2028, up from essentially none in 2024. One Gulf government has set itself the target of moving half of its services and operations to agentic AI within two years.

I have described this elsewhere as the move from prompts to loops. A prompt is a question a person asks. A loop is a system that plans, acts, checks and acts again, often many times before a human sees the result. Prompts needed users. Loops need governors. That is the leadership problem in one line.

II. The uncomfortable mirror

The consulting and research houses rarely agree on much. On this they do: the bottleneck is not the technology and not the workforce. It is leadership.

One major consultancy surveyed more than 3,600 employees and nearly 240 senior executives. Only 1% of leaders described their organisation as mature in AI. Leaders estimated that 4% of their staff used generative AI for at least 30% of their daily work; employees themselves reported 13%, three times higher. Executives were more than twice as likely to blame employee readiness as their own. The report’s conclusion was blunt: the biggest barrier to scaling is not employees, who are ready, but leaders, who are not steering fast enough.

The returns tell the same story. A survey of more than 4,400 chief executives this year found that 56% have seen no significant financial benefit from AI so far, and only 12% report gains in both cost and revenue. A university study of hundreds of enterprise deployments found that about 95% of generative AI pilots produced little or no measurable effect on profit and loss. Another consultancy classifies just 5% of companies as built to capture AI’s value at scale, while 60% report minimal gains. A technology company’s survey of 2,000 chief executives found that only a quarter of AI initiatives had delivered the expected return, and 64% admitted that fear of falling behind was driving investment before the value was clear. A research house now predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027.

Read together, these are not technology failures. They are leadership failures that look like technology failures. When AI is treated as a procurement decision, bolted onto old workflows and handed to a central lab, it disappoints. The same university study found that success came where line managers, not just central teams, were empowered to redesign how work is done. The value is in the redesign, and redesign is a leadership act.

III. What the frontier labs see inside their own walls

The most revealing research on AI leadership has come from the companies building the models, because they are running the experiment on themselves first.

One frontier lab studied its own engineers. In a year, the share of their daily work done with its AI rose from about 28% to about 59%, and self-reported productivity gains rose from 20% to 50%. Yet more than half said they could fully delegate only a fifth of their work or less. What they kept for themselves was telling: design, taste and anything hard to verify. They also named what the lab called a paradox of supervision: overseeing the AI requires exactly the skills that relying on it may erode. And senior engineers noticed that junior colleagues were asking them far fewer questions, because the AI had become the first port of call.

Another lab’s survey of enterprise users found workers saving 40 to 60 minutes a day, with the heaviest users sending six times as many messages as the median employee. The gap between frontier workers and everyone else is widening, inside the same organisations.

The lesson for leaders is double-edged. Even in the most AI-fluent organisations on earth, judgment and taste stay human. But the pipeline that produces judgment, the junior who asks the senior, is quietly drying up. If the people building AI are worried about where their next generation of experts will come from, everyone else should be too.

IV. Three ironies every leader should know

In 1983 an engineering psychologist published a short paper on the ironies of automation. Its insight was that the more we automate, the more we depend on human skill at the moment the automation fails, and the less practice humans get in that skill. Four decades on, it reads like a briefing note for today’s boards.

The irony of practice. Leaders are asked to be the final judge of AI output while AI removes the repetitions that build judgment. In one consultancy’s research, half of senior leaders already see de-skilling in their organisations, and more than 60% expect it to become a material threat within three to five years. An early laboratory study found that 83% of people who wrote an essay with an AI model could not accurately quote it minutes later. What you did not think through, you cannot defend.

The irony of persuasion. The best-known field experiment on AI at work, run with hundreds of management consultants, found a jagged frontier. On tasks inside it, AI made people faster and better. On a task outside it, consultants using AI were 19 percentage points less likely to reach the right answer than those working alone, because the output was fluent enough to accept. AI raises speed, not judgment. The more convincing the answer, the more a leader needs the courage to interrupt it and ask whose interests it serves.

The irony of oversight. Watching AI is hard work. People who closely monitor it report 14% more mental effort, 12% more fatigue and 19% more information overload, and tired reviewers approve. This is how a human “in the loop” becomes what researchers call a moral crumple zone: close enough to be blamed, too stretched to control anything. A human in the loop is only a safeguard if that person has the time, the information, the authority and the skill to say no.

V. Five shifts in the leader’s job

From answers to questions. As AI becomes better at solving defined problems, the human premium moves upstream: to defining the problem, framing the question and spotting the crisis before it arrives. The skills employers say are rising fastest in importance tell the same story. In a survey of more than 1,000 employers covering 14 million workers, leadership and social influence rose more than any other skill, and 39% of core skills are expected to change by 2030.

From span of control to span of accountability. Management theory has long held that one person can meaningfully lead only a handful of direct reports. One technology company now proposes a “human-agent ratio” as a measure of productivity. I think it is more important as a measure of governance. There is a limit to how many agents, or agent actions, one accountable human can genuinely supervise. Beyond it, oversight becomes theatre. Organisations should set that span deliberately, and boards should ask to see it.

From approval to checkpoint design. “Define when human review is required” sounds like policy. In practice it is architecture: which actions halt until a named person approves, what that person sees, and what happens if they refuse. In the governance architecture I call TrustOS, this sits at Ring Zero, the layer that can stop an action at the moment of execution rather than investigate it afterwards. Leaders who leave this entirely to engineers have delegated the one thing they cannot.

From teams of people to teams of people and agents. A large field experiment with nearly 800 professionals at a consumer goods company found that individuals working with AI performed as well as two-person teams without it, and that AI broke down the usual divide between technical and commercial thinking. That should change how leaders design teams, run meetings and decide who needs to be in the room. It also raises a harder question: if AI can replace the collaboration, what happens to the relationships, trust and mentoring that collaboration used to create?

From talent pipelines to apprenticeship by design. Entry-level hiring in AI-exposed jobs has already fallen sharply. Juniors ask fewer questions of seniors. This is how the Frozen Workforce forms: not through mass layoffs, but through the quiet disappearance of the first rung of the ladder. Leaders who want experienced people in ten years must deliberately build the apprenticeship that AI is dissolving, including protected time for juniors to learn without it.

VI. The real choice: augment or imitate

Underneath every AI strategy sits a choice that most leaders never make explicitly. They can use AI to imitate people and replace them, or to augment people and extend them.

A leading economist of technology calls the first path a trap: it concentrates value and power with whoever owns the machines, and leaves workers with little bargaining power. Another, a labour economist who has spent decades studying how technology hollowed out middle-skill jobs, argues the opposite is possible: used well, AI can let people with foundational training take on expert work once reserved for doctors, lawyers and engineers, and help rebuild the middle of the labour market. Both insist the outcome is a choice, not a destiny. Both note that today’s incentives, including tax codes that treat labour far more harshly than software, tilt firms towards imitation.

This is where my own frame of HX = CX + EX matters. The human experience of AI is the customer’s experience and the employee’s experience together. Only about half of employees in one survey of nearly 50,000 workers trust their senior executives. A leader who automates without a credible plan for people is spending trust they do not have.

I have described the road ahead as The Fork: one path towards a Star Trek future of abundance, the other towards Mad Max. Technology does not choose the road. Leaders do, one decision at a time. And the choice is not between speed and judgment. It is Long-AND, not Short-OR: the organisations that win will be the ones that keep both.

VII. Seven practices for the last checkpoint

  1. Write down what is non-delegable. A short list, owned by the board, of decisions that must end with a named human: anything irreversible, anything safety-critical, anything that changes someone’s livelihood.
  2. Interrupt the convincing answer. For every major AI-informed decision, someone is tasked with arguing against the model’s recommendation and asking whose interests it leaves out.
  3. Keep your repetitions. Leaders and high-potential staff work some hard problems without AI, deliberately, so the judgment they rely on still exists when they need it.
  4. Set a span of accountability. Cap how many agents or agent actions one person oversees, and staff oversight so that reviewers have the time to say no.
  5. Read the checkpoint log like the P&L. What the agents attempted, what was refused and what was escalated belongs in the monthly management pack, alongside revenue and cost.
  6. Use the tools yourself. A leader who has never run an agent cannot judge one. AI literacy is not a job for the technology team; it is a condition of leadership.
  7. Tell people the plan. Say which work will be automated, which augmented, and what happens to the people affected. Trust is the cheapest control there is, and the easiest to lose.

VIII. Six questions for Monday’s board meeting

  • Which decisions in this organisation can an agent now take without a human, and who signed that off?
  • When did a human last refuse an AI recommendation here, and what happened to them?
  • How many agents does each accountable person oversee, and is that number credible?
  • Are we using AI to augment our people or to imitate them, and did we choose that on purpose?
  • Where will our experts come from in ten years if juniors no longer learn from seniors?
  • If an agent caused serious harm tomorrow, who would answer for it, and would they have had the power to stop it?

IX. The part that cannot be automated

The machines will keep getting better at answers. That is the point of them. What they cannot do is take responsibility for the question, carry the consequences, or decide what kind of organisation, and society, the answers should serve. That part of leadership was never really about information, which is why it is the one part that cannot be automated.

Short-term turbulence, long-term abundance. The turbulence is here. Whether the abundance follows depends less on how intelligent our machines become than on how willing our leaders are to remain the last checkpoint.

A leader in the age of AI will be measured less by the answers they produce than by the answers they are willing to stop.

References

Leave a Reply