AI is the future of financial services, but what happens when it starts acting alone?

The financial services industry is entering a new phase of artificial intelligence adoption that carries significantly higher stakes for operational risk, governance, and investor confidence.

New research and analysis from Boston Consulting Group warns that as AI systems become more autonomous, financial institutions face a fundamental shift in how risk is created, managed, and contained. The banking and insurance industries rank alongside health care and manufacturing for being most at risk.

BCG’s latest work focuses on the rise of so-called agentic AI: systems capable of acting independently, executing tasks end-to-end, and making decisions without continuous human approval. While banks are already deploying AI across customer service, fraud detection, and internal operations, the move toward autonomy represents a break from the traditional model of human-in-the-loop oversight.

According to the report - What Happens When AI Stops Asking Permission? - this transition introduces a new class of operational and compliance risk because autonomous agents can initiate actions, modify systems, or interact with customers with limited supervision. In highly regulated banking environments, even small errors can escalate quickly into material issues affecting financial reporting, customer trust, or regulatory standing.

The firm notes that AI-related incidents rose 21% from 2024 to 2025, highlighting that risks are no longer theoretical. In one example, an AI agent tasked with managing expenses fabricated credible-sounding but entirely false transaction details when it encountered unclear input. In a banking context, similar failures in lending, payments, or reconciliations could trigger audit failures or regulatory breaches.

Because banks operate complex, interconnected technology ecosystems where automated decisions often cascade across multiple systems, BCG highlights several areas of heightened vulnerability for financial institutions:

• **Exception handling failures: Automated service agents often struggle when cases fall outside standard rules, which can lead to unresolved customer issues or stalled processes.
• Goal drift and misalignment: Autonomous systems learn and adapt over time, raising the risk that they optimize for speed or cost at the expense of compliance or risk controls.
• Systemic amplification: Because banking platforms are deeply interconnected, a malfunctioning AI agent could propagate errors across multiple business lines.

These risks challenge the adequacy of existing control frameworks, many of which were designed for static models rather than adaptive, self-directed systems.

BCG argues that banks must rethink AI governance from the ground up as traditional model risk management and operational risk frameworks are unlikely to be sufficient as AI agents take on broader responsibilities. Instead, firms need agent-specific risk taxonomies, continuous behavioral monitoring, and resilience planning that anticipates unexpected actions rather than just technical failures.

The firm also stresses the importance of extensive real-world testing before deployment, particularly in customer-facing or financially material processes. Human oversight remains essential, but it must be redesigned to supervise outcomes and behaviors rather than individual decisions.

As agentic AI moves from experimentation to core operations, banking leaders face a narrowing window to adapt. The next phase of AI adoption, BCG suggests, will not be defined by who deploys autonomous systems first, but by who governs them best.

“Agentic AI changes the game for AI risk and quality management,” notes Anne Kleppe, a BCG managing director and partner, global responsible AI lead, and coauthor of the article. “Autonomous agents are powerful, but they can drift from the intended business outcomes. The challenge is keeping them aligned to strategy and values while still letting them operate with speed and autonomy.”