Is AI and cybersecurity an arms race that no one truly understands yet?

“You have to treat AI a bit like an insider threat,” declared Mark West, assistant director of innovation at Roke, at the most recent Foresight Live panel, Attackers, Defenders and Algorithms: AI’s Cybersecurity Double-Edge Sword.

AI is becoming increasingly capable at analysing information across silos, but agents should be subject to the same zero-trust protocols as any employee. This is important, especially because a rogue employee can be removed, while extricating AI from a system presents more complex challenges. And even with narrow use cases for AI, permission creep is a concern, West cautioned.

Yet the genie is out of the bottle. Adversaries are using AI to find exploits and the UK must find a way to deploy its capabilities ethically.

For Clare Johnson, capability lead for cyber and networks at ITSUS Consulting, this is “a huge challenge”. Governance remains patchy, though frameworks are being created by organisations like the Turing Institute. We need to “make sure that what we are using AI for is monitored, checked, critiqued, and that the models that we’re using are ethically and morally robust,” she said.

Awais Rashid, professor of cybersecurity at the University of Bristol, added a crucial caveat: “We have to be mindful that we’re asking for guardrails or ethics,” from automation technology, “where we don’t fully understand how these decisions are arrived at.”

Rashid also cautioned that a fragmented landscape may be created, where some AI tools have built-in ethical mechanisms, while others enforce ethical uses only at the contract level. At best, this fragmentation makes it hard to demonstrate consistent principles when it comes to defence and security. At worst, AI would be technically capable of purposes that conflict with a society’s ethical standards – a real danger in an arms race scenario.

The speed of technological evolution makes this problem acute, and it is “overwhelming and very hard” to fully understand risks and opportunities, West observed, adding that these two “get intertwined very quickly”. And when thinking ahead to future scenarios, “it gets dystopian very fast,” he said. “We are way beyond stochastic parrots. Do I think we have an existential threat? No. But where are we in the middle of those two? I don’t know.”

For Rashid, the explosion of AI adoption echoes an earlier technological shift – that of COBOL, a programming language created more than six decades ago to democratise coding. COBOL code persists beneath countless modern systems, including many banking apps, while the number of people capable of maintaining it have dwindled. LLM-generated code presents the same risk.

“You don’t know what is being produced and you will end up in a situation where nobody knows what is inside the code and how to maintain it,” Rashid warned.

West framed this as determining AI’s role within a team. “Do I move from being someone who writes code to someone who orchestrates and manages agents writing code, which is still a skilled job, because I have to be able to describe what I want in a way that it’s going to be able to deliver that?”

Johnson agreed emphatically. “There is a huge skills deficit that will grow exponentially,” she said. The situation could deteriorate further if AI ever advances to the point where it creates its own programming language. “We’ll have no clue as to what is going on and no knowledge of how to constrain it in any way.”

But there is cause for optimism. While sceptical of some of the claims around Mythos AI, developed by Anthropic to identify exploits, all three agreed that there is a genuine role for AI in securing systems.

“A lot of this stuff should be seen as additive,” noted West, with existing tools remaining essential components of the cybersecurity toolkit.

Rashid reinforced this point, stressing that the same principle applied to physical infrastructure like industrial control systems and IoT devices.

“Good basic security hygiene still works,” he said, also acknowledging, “We do worry about zero days.”

However, a large number of attacks come from mistakes like misconfigurations and poor security practice.

Johnson added that companies should continue following appropriate guidance from the National Cyber Security Centre or ISO-27001.

“Maybe the concern would be if AI gets to a point where it knows how to hide itself from those things. But until that point and until that’s proven, the things that we’re doing day in, day out – which we all should be doing as good practice – still are very valid,” she said.

That AI tools are now discovering exploits that humans previously missed doesn’t indicate that the code was bad, Rashid concluded. The tools have just become more efficient. But this raises a critical question.

“Do we follow the same kind of best practices around responsible disclosure?” And that, for Rashid, boiled down to one fundamental challenge. “What kind of society do we want to be?”