Governing agentic AI when machines begin to decide at scale

As enterprises move from AI-assisted workflows to autonomous decision-making systems, the focus is shifting sharply from capability to control. Sudhakar Singh, VP and Head of Responsible AI at SAP, explains how organisations must rethink governance, auditability, and human oversight as agentic AI becomes embedded across core business processes.

Drawing on SAP’s experience of deploying agentic intelligence inside enterprise systems, Singh outlines why risk thresholds, reversibility, clean data foundations, and human-in-the-loop models are essential to ensuring that autonomy does not outpace accountability. Excerpts from the conversation.

As AI agents begin making autonomous decisions across finance, HR, and supply chain, how should enterprises rethink accountability and auditability?

Autonomous agents operate within industry-specific and organisation-specific risk thresholds. For instance, different organisations may define varying limits for automatic fund transfers. Any transaction exceeding the configured threshold must trigger human intervention and require explicit approval before the agent proceeds.

For actions approved automatically, enterprises must maintain comprehensive audit logs that capture the complete decision tree. This enables retrospective human review when required. Reversibility is another critical risk management concept. In systems where actions cannot be reversed, such as medical procedures, the risk level is significantly higher, and autonomous agent usage should be restricted or prohibited in such high-risk scenarios.

What new governance frameworks are needed when multiple agents collaborate and create chained decisions without human oversight?

Human oversight remains essential for complex agentic systems and cannot be replaced. Governance frameworks must implement oversight through three mechanisms: human-in-the-loop, human-on-the-loop, and human intervention on demand. Each mechanism addresses different risk levels and operational contexts.

Context drift poses significant challenges in multi-agent interactions. Techniques such as context grounding and periodic realignment help maintain decision accuracy. Implementing traceability checkpoints and agent reasoning review processes further strengthens system reliability.

Industry regulations and jurisdictional compliance requirements mandate comprehensive auditability. Key auditability parameters include action performer identification, whether human or AI, identity attribution, and complete decision flow documentation. These factors are fundamental to establishing accountability and enabling effective audit trails in agentic systems.  

Can agentic systems operate reliably in messy, siloed enterprise data environments? How do you prevent unintended decisions and context leakage?

Agentic systems require clean, comprehensive contextual data to function reliably. Poor data quality directly results in poor outcomes. However, agents can be designed to address data gaps by cross-referencing information from multiple authorised sources.

Data silos present significant challenges for knowledge validation. Agents require access either to a broad-knowledge large language model or direct access to relevant knowledge sources. Organisations face a fundamental trade-off: fully knowledgeable agents with unrestricted data access versus role-specific agents with limited privileges. The former enables better decision-making but increases security risks, while the latter is safer but may lack the breadth of insight needed for optimal decisions.

Preventing context leakage requires repeated context grounding throughout the agent’s operation. SAP implements technical controls including iteration limits and time duration constraints. These controls, combined with mandatory human intervention checkpoints, create hard stops that ensure users retain ultimate control over outcomes, even in worst-case scenarios.

Will AI-first enterprise systems emerge where business logic is evolved by agents? How should leaders prepare for this cultural shift?

Agents analyse existing business data to propose new solutions and data points. When these proposals are accepted and executed, they become part of the business data corpus. This feedback loop enables agents to learn which actions have higher success and approval rates, gradually reducing reliance on repeated human intervention and accelerating automation.

Over time, agents begin developing business logic based on identified patterns. However, this approach has limitations. Decision-making can become commoditised, and organisational uniqueness may erode. Both outcomes impact the workforce: routine tasks become automated, while unique skills risk obsolescence if they are not identified and adapted to the technological shift.

Enabling people to safely adopt AI tools is the way forward. Developer tooling provides a useful analogy. AI-powered coding tools with agent capabilities and Model Context Protocol integration enable rapid prototyping. However, building reliable, maintainable enterprise products still requires developers, domain experts, and business stakeholders. The cultural shift places greater emphasis on critical thinking and analytical skills over rote execution.

What are some real customer use cases where SAP is embedding agentic intelligence into core enterprise workflows? How are clients adapting to AI-driven autonomous decisioning inside their ERP landscape?

SAP has implemented agentic systems across multiple end-to-end business processes. One example is cash collection and cash flow prediction. These agents execute tasks including data collection, historical pattern analysis, industry trend monitoring, geopolitical assessment, and regulatory requirement tracking. The system synthesises these inputs to predict cash flow challenges and their probability.

Agents also generate solution recommendations based on customer-specific business contexts and partner networks. Previously, these analyses required teams of experts and took days or weeks to complete. Agentic systems now provide this intelligence with real-time updates.

Human expertise remains critical for verifying and correcting data patterns and agent outputs. However, turnaround times have reduced significantly. This allows organisations to make faster decisions and simulate multiple solution scenarios before committing to the optimal approach for their specific requirements.