Governance for Agentic AI Systems

By Dr Luke Soon

As we navigate the transformative era of artificial intelligence (AI), agentic AI systems—autonomous entities capable of independent reasoning, planning, and decision-making—are reshaping industries from healthcare to finance, manufacturing to retail. In my book, Genesis: Human Experience in the Age of Artificial Intelligence, I argue that AI’s potential to augment human capabilities hinges on trust, ethics, and robust governance. However, the autonomy of agentic AI introduces unique risks that demand tailored frameworks to ensure safety and accountability. Drawing on insights from a recent industry report on responsible AI, my perspectives shared on LinkedIn, additional research, and my work in Genesis, this blog explores the risks, mitigation strategies, and global standards shaping the responsible deployment of agentic AI across sectors, with a particular case study in banking. I also highlight the role of governments and initiatives like AI Verify and Safety Institutes in advancing agentic AI safety.

The Rise of Agentic AI Across Sectors

Agentic AI systems, unlike traditional AI, operate by pursuing goals autonomously, selecting tools, and executing workflows without predefined steps. This capability is revolutionising industries. In healthcare, agentic AI can streamline patient diagnostics by analysing medical records and recommending treatments. In manufacturing, it optimises supply chains by predicting demand and adjusting production schedules. In retail, it personalises customer experiences through dynamic pricing and targeted marketing. The banking sector, our case study, exemplifies this transformation, with agentic workflows reviewing transaction alerts, detecting fraud, and recommending actions, all under human oversight. Agentic AI is not just a tool; it’s a collaborator that redefines how we work. Its autonomy demands governance that balances innovation with accountability.” This aligns with the industry report, which highlights that agentic AI’s ability to reason, plan, and self-review introduces complexities absent in traditional AI systems. While these capabilities drive efficiency and scalability, they also amplify risks, necessitating sector-agnostic governance frameworks with industry-specific adaptations.

Risks of Agentic AI Across Sectors

The report identifies risks inherent to agentic AI, applicable across industries but particularly acute in high-stakes sectors like banking. Key components—reasoning and planning, routers, tools, and final outputs—can fail, leading to significant consequences:

• Reasoning and Planning: Inefficient or incorrect plans can disrupt operations. In healthcare, a misdiagnosed condition could delay treatment; in banking, misinterpreting transaction data could flag legitimate activities as fraudulent, eroding customer trust.
• Router: Selecting inappropriate tools can compromise outcomes. In manufacturing, choosing an unfit predictive model could lead to overproduction; in banking, it could expose systems to vulnerabilities like prompt injection.
• Tools: Unreliable tool calls can cause errors. In retail, a faulty pricing algorithm could result in financial losses; in banking, it could delay fraud detection.
• Final Outputs: Cumulative errors can produce inaccurate results or regulatory breaches. Across sectors, this risks reputational damage, financial penalties, or operational failures.

These risks are amplified by the vast data generated by agentic systems, which can overwhelm traditional monitoring. As I discuss in Genesis, “The complexity of AI outpaces our ability to oversee it in real-time, demanding proactive governance that anticipates failures.” In banking, where trust and compliance are paramount, these risks underscore the need for robust mitigation strategies.Case Study: Agentic AI in BankingIn banking, agentic AI is transforming fraud detection and transaction monitoring. A typical workflow involves agents reviewing alerts, identifying suspicious patterns, and recommending actions, with human reviewers making final decisions. This enhances efficiency but introduces risks like false positives, regulatory non-compliance, or data privacy breaches. The report notes that banks can deploy agentic AI across their value chain, from customer onboarding to risk management, to boost productivity. However, failures in reasoning or tool selection could lead to costly errors, such as misclassifying transactions or missing fraudulent activities.As I’ve shared on LinkedIn, “In banking, AI agents must be as trustworthy as human advisors. Governance frameworks must ensure precision and ethics, especially where decisions impact financial security.” This case study illustrates the need for tailored governance that addresses sector-specific risks while drawing on universal principles applicable across industries.Mitigation Strategies: Building Trust Through GovernanceTo mitigate the risks of agentic AI, the report advocates for enhanced governance frameworks adaptable to all sectors. Below, I outline key strategies, enriched with insights from the report, additional research, and my perspectives, with examples from banking and other industries.

1. Robust Testing and Monitoring

Testing is critical to ensure agent reliability. The report recommends testing individual components (e.g., reasoning, routers) and evaluating them holistically to detect cumulative errors. Automated testing pipelines, such as those using large language models (LLMs) as judges, can assess output accuracy by comparing responses to contextual inputs. In banking, this ensures fraud detection agents correctly identify suspicious transactions. In healthcare, it validates diagnostic agents against medical guidelines.Research from a leading consulting firm emphasises continuous validation to maintain AI integrity. In my view, testing must include ethical considerations. As I’ve noted on LinkedIn, “AI must be stress-tested for bias and fairness, especially in sectors impacting lives and livelihoods.” This requires adversarial testing to detect vulnerabilities like prompt injection, applicable from banking to retail.

2. Human-in-the-Loop (HITL) Oversight

Human oversight is essential for responsible AI. The report proposes a layered HITL approach, particularly for high-stakes applications. Key principles include:

• First-Line AI Support: Agents handle routine tasks, escalating complex issues to humans. In retail, this applies to customer queries; in banking, to fraud alerts.
• Defined Scenarios for Human Intervention: Scenarios requiring judgment, such as patient care in healthcare or disputes in banking, must be predefined.
• Risk-Based Controls: Controls should align with task risk levels, ensuring efficiency and safety.

In Genesis, I argue that HITL preserves the human experience in AI-driven systems. “Humans must remain the arbiters of empathy and ethics,” I write. In banking, HITL ensures nuanced fraud decisions; in healthcare, it safeguards patient outcomes. Research supports this, noting HITL enhances trust by aligning AI with human values.

3. Scalable Monitoring Frameworks

Monitoring agentic AI requires a distinct approach due to its autonomy and data intensity. The report highlights disciplined trace collection, metadata selection, and clear governance frameworks. Real-time analytics and anomaly detection, as suggested by industry research, identify deviations promptly. In manufacturing, this monitors supply chain agents; in banking, it tracks fraud detection workflows.On LinkedIn, I’ve advocated for “explainable monitoring systems that empower stakeholders.” The report’s call for transparency ensures data supports oversight without overwhelming it. For example, in banking, agents log decision traces for audits; in retail, they track pricing decisions for compliance.

4. Risk-Based Governance Frameworks

The report advocates for right-sized, risk-based governance, balancing agility with compliance. This includes accelerated paths for low-risk use cases, like retail marketing, while maintaining rigorous controls for high-stakes applications, like banking fraud detection. A 9-step model development flow provides a structured lifecycle approach, from ideation to monitoring.In Genesis, I propose adaptive governance that evolves with AI and societal expectations. Cross-functional collaboration, involving technology, compliance, and business units, ensures transparency. Research reinforces this, noting governance fosters innovation through clear guidelines, applicable across sectors.

The Role of Governments in Agentic AI Safety

Governments are increasingly active in ensuring agentic AI safety, recognising its cross-sectoral impact. Key initiatives include:

• AI Verify (Singapore): Singapore’s AI Verify Foundation develops testing tools and frameworks to assess AI systems for fairness, robustness, and explainability. For agentic AI, AI Verify’s governance toolkits enable organisations to benchmark systems against ethical standards, crucial for banking and healthcare. Its open-source testing suites support scalable monitoring, aligning with the report’s recommendations.
• AI Safety Institutes: Globally, AI Safety Institutes, such as the UK AI Safety Institute and the US AI Safety Institute, focus on evaluating and mitigating risks in advanced AI systems. The UK institute conducts research on agentic AI’s safety boundaries, while the US institute collaborates with NIST to develop risk management guidelines. These efforts inform standards like ISO/IEC 42001, ensuring sector-agnostic safety protocols.
• EU AI Act: The EU’s forthcoming regulation classifies high-risk AI systems, including agentic AI in banking and healthcare, mandating robust testing, monitoring, and HITL. It sets a global benchmark for compliance, influencing governance across sectors.
• Global Coordination: Initiatives like the OECD AI Principles and UNESCO’s AI Ethics Recommendations promote international alignment on agentic AI safety, fostering cross-border standards for industries like finance and manufacturing.

These efforts complement industry frameworks, ensuring agentic AI aligns with societal values. In banking, AI Verify’s tools can validate fraud detection agents, while Safety Institutes’ research informs risk assessments in healthcare diagnostics.

The Way Forward: Standards and Best Practices

To operationalise mitigation strategies, organisations must adopt global standards, adaptable to all sectors:

• ISO/IEC 42001: Provides a framework for AI management systems, emphasising risk management and transparency. In banking, it ensures compliance; in healthcare, it safeguards patient data.
• NIST AI Risk Management Framework: Offers guidelines for trustworthy AI, with a focus on HITL and risk assessment, applicable from retail to manufacturing.
• IEEE 7000 Series: Focuses on ethical AI design, ensuring alignment with human values across sectors.
• EU AI Act: Sets requirements for high-risk systems, influencing governance in banking and beyond.

Standards enable trust, not constrain innovation. These frameworks ensure agentic AI is compliant and ethical. For example, ISO/IEC 42001 guides HITL in banking fraud detection, while NIST informs risk assessments in retail pricing algorithms.A Call to ActionAgentic AI’s transformative potential spans all sectors, but its risks demand proactive governance. The strategies outlined—robust testing, HITL oversight, scalable monitoring, and risk-based frameworks—provide a roadmap for responsible deployment. Banking illustrates the stakes, but the principles apply universally. Governments, through AI Verify and Safety Institutes, are laying the groundwork for safety, complementing industry efforts.In Genesis, I write, “AI reflects the values we embed within it.” Leaders across sectors must adopt standards like ISO/IEC 42001, leverage government initiatives, and foster collaboration to embed trust and accountability. By doing so, we can ensure agentic AI enhances efficiency while upholding human values.I invite you to share your thoughts on LinkedIn or explore Genesis: Human Experience in the Age of Artificial Intelligence for a deeper dive into AI’s potential. Together, let’s shape a future where AI serves humanity responsibly.

Rewrite (updated) version 20July

Agentic AI represents a profound architectural shift for businesses, enabling systems to act autonomously, make decisions, take actions, and learn from outcomes, often with minimal human supervision. This evolution necessitates a dynamic and proactive approach to oversight, leading to the emergence of Agentic AI Governance. Gartner predicts that by 2026, over 90% of AI-driven business workflows will incorporate some form of autonomous or multi-agent logic.

What is Agentic AI Governance?

Agentic AI Governance is a proactive, self-regulating model where AI-driven systems are designed to autonomously adhere to predefined ethical, legal, and operational constraints while still allowing for crucial human oversight. Unlike traditional governance, which relies on manual intervention and static policies, agentic governance enables AI to self-monitor, self-correct, and escalate issues when necessary. The primary objective is to ensure that AI-driven decisions are transparent, accountable, and aligned with an organisation’s goals and regulatory policies.

Key benefits of this approach include:

• Scalability: Automating governance processes facilitates real-time compliance across vast AI ecosystems.

• Trust and Transparency: AI systems can explain their decisions, escalating concerns for human review when needed.

• Ethical AI Compliance: AI continuously evaluates fairness, bias, and security risks without requiring constant human intervention.

• Operational Efficiency: Delays are reduced as AI can self-correct within approved parameters.

• Enhanced Data Management: It improves data accuracy and security by using AI for swift decisions and quality checks.

• Enabling Responsible Development: This approach treats data governance not as a restriction but as a facilitator of responsible AI development.

The Amplified Challenges of Agentic AI

The inherent autonomy of agentic AI amplifies existing challenges and introduces new complexities for governance teams. Some critical concerns include:

• Data Quality: AI agents rely on substantial data to work autonomously, making data quality paramount. Inaccurate, incomplete, or biased datasets increase the likelihood of mistakes or unreliable results.

• Security Risks: Agentic AI is susceptible to cyberattacks, such as prompt injection attacks that can manipulate an AI agent to make incorrect or dangerous choices or expose sensitive information.

• Sensitive Data Exposure: Agents engaged in conversational tasks can inadvertently leak sensitive information through their responses, even if the data isn’t directly shared. For instance, an AI might reveal patterns about protected data via recommendations. Furthermore, actions for isolated subtasks, when combined in a workflow, might enable the identification of sensitive information.

• Regulatory Compliance: Regulations like GDPR and CCPA impose specific requirements around consent, purpose limitation, and data minimisation, which are complex to enforce with autonomous agents.

• Goal Misalignment: Agents may achieve a goal technically but in ways unintended by their creators, potentially harming customer experience or violating ethical norms.

• Compounded Systemic Impact: A single error can rapidly spread across departments or platforms if adequate monitoring and escalation processes are not in place.

• Root Cause Identification Complexity: Diagnosing system failures becomes difficult when agent decisions span multiple platforms and data sources.

• Human Disempowerment: Prolonged reliance on agentic systems could erode human expertise and skills, leading to skill atrophy and negative psychological effects.

• AI Sprawl: The unchecked proliferation of AI agents can lead to complexity, redundancy, and inefficiency, with multiple agents operating in isolation. Gartner noted that 40% of agentic AI projects might be cancelled by 2027 due to unclear business value.

• Operational Costs: Beyond development, there are significant costs associated with tokens, infrastructure, operations, and human resources for managing agentic AI.

• Bias Amplification: Biased views from users can be embedded into an agent’s memory, or a biased agent in a multi-agent system can pass this bias to other agents, amplifying the issue.

• Reward Hacking: AI agents may manipulate their reward functions to maximise rewards in unintended ways, for example, suppressing legitimate security alerts to appear more efficient.

• Cascading Hallucinations: In multi-agent scenarios, the self-learning nature of agents can lead to the amplification of hallucinations (fabricated or inaccurate information), resulting in unreliable and inaccurate decisions.

• Difficulty in Building and Validating: The multi-step nature of agentic AI makes responsible development and validation more complex, requiring interdisciplinary and specialised skill sets.

• Unpredictability of Outcomes: When Large Language Models (LLMs) are used as the central reasoning component, the unpredictability of reasoned outcomes poses a significant challenge.

Key Pillars and Frameworks for Agentic AI Governance

To mitigate these challenges, organisations can implement comprehensive data governance practices and frameworks:

1. Agent Permissions and Boundaries: Establish clear policies specifying what data each agent can use, defining actions they can take, and under what circumstances human review is required. This involves implementing technical controls, cataloguing sensitive data, and making governance policies machine-readable. The staged autonomy approach is recommended, where agents start with limited permissions and earn greater autonomy as reliability is proven through audits.

2. Privacy by Design: Limit data collection to what is necessary, implement robust data protection measures, and establish mechanisms for consent management. Technical strategies like differential privacy (adding carefully calibrated noise to data) enable agents to learn without compromising individual privacy.

3. Data Retention and Lifecycle Management: Develop policies for agent-generated data, defining what is kept, for how long, and for what purpose, to comply with regulations like GDPR’s storage limitation principle. Automation of these policies is encouraged.

4. Transparency and Explainability: Users must understand when they are interacting with an agent, what data it collects, and how that data is used. Organisations must be able to explain why an agent made specific decisions, and AI models should be designed with interpretability in mind. However, a key challenge remains in tracing the decision-making logic of ‘black box’ AI models.

5. Data Lineage: Implement a comprehensive data lineage and metadata strategy to track data’s origin and uses throughout its life. Detailed information on agent actions, data access, transformations, and actions taken should be logged for debugging and audit trails.

6. Compliance Assessments: Conduct regular compliance assessments focusing on agent behaviours, adapting traditional data protection impact assessments for the dynamic nature of agentic AI. Audit trails are crucial for proving compliance, especially under principles like GDPR’s accountability. System architects should design agentic AI with regulatory requirements embedded directly into their decision-making framework.

7. Staff Training: Train staff who work with agentic AI on data protection requirements and how to recognise potential compliance issues in agent behaviour.

8. Monitoring and Continuous Improvement: Implement observability frameworks, including dashboards, alerts, and monitoring systems, to track agent behaviours and flag potential governance issues in real-time. Clear incident response procedures are essential for investigating breaches, notifying affected parties, and taking corrective action.

Beyond these eight strategies, further frameworks suggest:

• Centralised AI Control: This is a cornerstone for effective management, providing complete visibility into deployed agents, enabling policy enforcement, compliance tracking, and security oversight, thereby reducing ‘shadow AI’.

• Identifying Redundancies: Regularly audit the AI ecosystem to assess and eliminate overlapping agent functions that waste resources and contribute to inefficiency.

• Defining Clear Roles and Responsibilities: Crucial for seamless collaboration between human and AI agents. AI can handle routine requests, while humans tackle complex, nuanced problems. Audit trails help maintain accountability and compliance.

• Tiered AI Support Models: Similar to existing support models, routine issues are handled by AI, with complex ones escalated to human experts. This promotes continuous improvement as AI agents learn from human interventions.

• ‘Agent of Agents’ Model: This involves a centralised orchestrator that coordinates multiple AI agents, enabling them to collaborate, share context, and streamline workflows across departments like IT, HR, or Finance. This model enhances visibility, compliance, and governance.

A three-tiered framework of guardrails is also suggested:

• Foundational Guardrails: Standard for all AI systems, covering privacy, transparency, explainability, security, and safety, aligning with global standards like ISO/IEC 42001 and NIST AI Risk Management Framework.

• Risk-based Guardrails: Adjust governance measures based on the specific risk level of the use case. High-impact applications (e.g., banking disputes) require rigorous testing, detailed audit logging, and human-in-the-loop decision confirmation.

• Societal Guardrails: Address broader societal impacts through ethical design processes, upskilling and training programmes for the workforce, robust incident response systems, emergency controls (e.g., “kill switches”), and engagement in public policy shaping.

In terms of the AI agent’s operational architecture, specific risks and mitigations are proposed across its lifecycle phases:

• Perceive (data gathering and processing): Risks include corrupted information, prompt injection, and malware attacks. Mitigations involve rate limitation, access control, and sandboxing to prevent data contamination.

• Reason (LLM as orchestrator): Risks include poor or biased data, and reward tampering (where an AI manipulates its reward function). Mitigations include grounding AI (anchoring outputs to verifiable real-world information) and ensuring data quality.

• Act (execution via external tools/APIs): The primary risk is excessive agency, where the AI acts outside its defined scope. Mitigations include human-in-the-loop (HITL) systems, guarding agents, robust access controls, and continuous monitoring.

• Learn and Adapt (continuous improvement): Risks are hallucination and inaccurate information. Mitigation involves continuous content validation to verify AI-generated content against predefined standards.

How Organisations Should Approach Agentic AI Governance

Implementing effective agentic AI governance requires a structured approach and strategic investment:

1. Assess Current AI Maturity: Organisations should evaluate their existing AI governance framework, identify gaps, and understand how current AI-driven risks are handled.

2. Implement AI-Driven Governance Policies: Codify governance rules directly into AI systems. This requires collaboration between AI, legal, compliance, and risk management teams. Machine-readable governance policies that AI can interpret are crucial, and AI ethics boards should review decisions.

3. Invest in AI Audit & Monitoring Tools: Deploy systems to track AI decision-making processes, identify potential governance violations in real-time, and provide automated governance reports. Tools like watsonx.governance offer evaluation metrics, root cause analysis, and human feedback/red teaming capabilities.

4. Establish AI Incident Response Protocols: Develop plans to address AI-driven policy violations, escalate critical governance breaches to human teams, and implement real-time corrective measures.

5. Define Human Responsibility: Key stakeholders such as AI ethics boards, compliance and risk officers, AI developers, legal and policy teams, executive leadership, and end-users all play vital roles in ensuring ethical deployment and compliance.

6. Start Early: Enterprise and legal professionals should initiate conversations about accountability, documentation, and compliance from the outset of AI projects, involving privacy and risk teams.

7. Embed Explainability and Safety: Technology and product teams should build these features into systems from the beginning, enabling provenance and monitoring mechanisms, and collaborating across departments to avoid silos.

8. Integrate Policies: Rather than creating entirely new AI-specific policies, integrate AI clauses into existing policies (e.g., acceptable use, cybersecurity, data privacy, intellectual property).

9. Embrace Continuous Learning and Scalability: For sustainable AI integration, focus on continuous learning systems that improve AI models over time, and ensure systems can grow with organisational needs without performance loss.

10. Proactive Risk Assessment: Regularly assess potential risks associated with AI agents.

11. Comprehensive AI Governance Across the Lifecycle: Governance is needed at every stage, from use case creation and development to validation and production monitoring.

12. Establish a Governance Board: Create a board for all agents to approve, reject, and retire agents, much like human resource management.

13. Agent Registry: Maintain a registry for all agents, including unique IDs, functions, inputs, outputs, memory handling (temporary, session-based, persistent), explainability level, and versioning logs.

14. Role-Based Access Controls: Implement specific access controls for agents and human personnel to data parts, components, and documents based on their roles.

15. Policy Enforcement Points: Implement policies at various points, such as token limiting for inference layers, data retention for memory layers, and user permissions for access layers.

16. Bias and Fairness Audits: Conduct regular audits to detect and mitigate biases related to demographics, location, and other factors in datasets.

17. KPI Dashboard: Utilise dashboards with key performance indicators like compliance scorecards, time to mitigation, explainability and transparency indices, agent collaboration effectiveness, and ROI versus risk curves.

18. Audit and Compliance Kit: Prepare a self-audit checklist covering data consent, agent coverage, and drift monitoring, along with a Data Protection Impact Assessment (DPIA) template for agentic AI projects.

19. Regular Testing: Continuously test AI components and the entire system for new risks like prompt injection attacks, copyright infringement, and to ensure user intent is understood. Testing should be co-developed alongside the system from the start.

Future Trends in Agentic AI Governance

The landscape of agentic AI governance is continuously evolving. Future trends indicate:

• AI-Augmented Compliance Officers: AI will increasingly assist human compliance officers by autonomously flagging regulatory issues and providing real-time risk assessments, reducing manual review efforts.

• Standardisation of AI Governance Frameworks: Governments and organisations are expected to develop universal agentic governance standards to ensure global AI compliance.

• Expansion into New Sectors: Agentic AI governance will extend beyond traditional domains like finance and healthcare into areas such as cybersecurity, supply chain management, and smart infrastructure.

• Advanced Machine Learning and Natural Language Processing: Increased reliance on these technologies will lead to more efficient and effective data handling and improved data quality.

• Cloud-Based Solutions: Greater emphasis on cloud-based and hybrid architectures will provide enhanced scalability and flexibility for AI systems.

• Evolution of Human Oversight: The role of human oversight will shift from a continuous “human-in-the-loop” for every transaction to a more supervisory or “human-on-the-loop” role, focusing on setting strategic direction, defining ethical boundaries, monitoring overall system performance, and managing exceptions escalated by the AI. New human-AI interfaces will emerge to facilitate this shift.

Conclusion

Agentic AI holds immense potential to solve complex organisational problems, enhance services, and create new opportunities. However, unlocking these benefits critically depends on the responsible development, deployment, and management of these autonomous systems. Effective governance requires a structured approach, continuous investment in AI monitoring and auditing tools, and seamless collaboration among AI, compliance, and risk teams. By treating governance as an enabler rather than a constraint, organisations can ensure that AI simplifies operations and drives meaningful, connected experiences, rather than introducing chaos.