The dark side of AI acceleration in financial software development
McKinsey analysed almost 300 publicly traded companies to understand how AI is reshaping software development. It found that the top 20 percent are achieving between 16 and 30 percent improvements in productivity, time to market, and customer experience, along with between 31 and 45 percent gains in software quality. However, while software partners rush to capitalise on these improvements, many are playing fast and loose with their AI guardrails – especially when it comes to highly regulated financial services clients.
“When financial services organisations talk to software partners about AI, they tend to fall into two camps. Some are captivated by the promise of speed, scale and cost efficiency in highly competitive markets. Others are deeply uneasy about risk, regulatory exposure and loss of control over sensitive data. Both instincts are justified,” says Sean Cleworth, CIO at Global Kinetic. “The reality is that the industry is already using AI extensively, often ahead of its governance. The relevant question for banks, insurers and fintechs isn’t whether their partners are using AI, but whether those partners have an appropriate framework for using it responsibly within a regulated environment.”
Cleworth argues that financial institutions should expect visible maturity from their software partners across the people, process and tooling areas.
Building a foundation of accountability
Before any AI licence was issued at Global Kinetic, Cleworth says a formal AI usage policy was implemented and signed by the entire team. The policy sets out which tools may be used for client work, under which circumstances, and what is explicitly prohibited. This includes banning unapproved tools and cautioning staff from pasting sensitive client information, confidential algorithms or personal data into prompts.
The company also built an onboarding programme that everyone, from project managers to senior engineers, must complete.
The aim is to establish a baseline understanding of what the various systems can and can’t do, how prompts may cross borders, what data residency really means in an AI context, and where model providers’ retention and processing practices can create regulatory exposure.
Process matters
Cleworth says different roles inside a financial services project interact with AI in very different ways. For instance, a UX designer using AI capabilities inside Figma faces a different risk profile to a backend developer using Claude to generate code for a payments engine, or a business analyst relying on Microsoft Copilot to summarise product documentation and regulatory texts.
“Good governance means being willing to adapt processes to the client. Some clients are already asking us not to use AI at all on specific codebases. Others are open to it but only after they’ve seen a clear description of when and how it is used, and which steps remain firmly human,” he says. “The reality is that AI is moving much faster than legal and procurement cycles.”
The tooling and data sovereignty conundrum
Cleworth says many enterprise contracts, especially in financial services, already contain provisions about where data may and may not reside, with signed agreements that explicitly prohibit client data from being processed outside certain regions.
However, the reality is that some of the best AI code assistants today are hosted in the US, retain prompts for up to 30 days and, while they may not train on a company’s data, they still persist it on American infrastructure.
“As a client, you should be asking your software partners which AI products they use, on which tiers and in which regions. For instance, if they use Claude, is that the Team tier or an Enterprise deployment with stricter data controls? Where exactly are prompts processed? How long are they retained? Do they ever leave the jurisdiction your contracts require? And what mechanisms exist to stop a well‑meaning developer pasting secrets or private keys into an external model?” he shares.
Cleworth believes clients are also entitled to understand at which points in the lifecycle their partner uses AI on their projects. They should know, for example, whether AI is proposing code that is always reviewed and approved by a human with appropriate seniority, whether it is generating test cases that engineers then validate against regulatory and security standards, and whether it ever influences architectural or design decisions around critical components.
“Mature partners are able to map their ways of working, identify exactly where AI enters the flow and, critically, adjust those touchpoints based on a client’s risk appetite, regulatory obligations and internal policies,” he says.
Transparency the best guardrail – for now
Cleworth says that, over time, the financial services industry will move towards ring‑fenced, sovereign AI environments. These will be controlled safe spaces where models run close to clients’ systems, under their jurisdiction, security model, and policy framework, and can be audited accordingly.
“Today, most organisations operate in a hybrid reality, combining powerful general‑purpose tools with legacy contracts and regulatory regimes that were never written with AI in mind. The reality is technology is evolving too quickly and the regulatory environment is still catching up,” he acknowledges. “For now, what clients should expect is an honest, structured approach. A credible software partner should be able to describe how they govern AI at the level of people, process and tooling for regulated workloads, where the current gaps lie, and how they plan to close them. You want a partner that treats AI not only as a productivity lever, but as a strategic risk and compliance topic.”
