India, July 15: As Indian enterprises and IT services firms race to adopt AI coding tools, new independent research shows the technology's security hasn't kept pace with its ability to get the job done and in some cases, throwing more computers at the problem makes things worse.
AI coding assistants have gotten remarkably good at writing code that works. Whether that code is actually safe to ship is a different story and a new independent study finds the gap between the two is widening, not closing.
The findings come from Capability Without Security: Measuring the Functionality-Security Gap in AI-Generated Code, a study by Ilya Kabanov, independent researcher, The Weather Report Inc.funded by application security firm Checkmarx. Checkmarx says it had no role in designing the study, running the tests, or shaping the conclusions; a separation the firm says was intentional, given how directly the results speak to its own market.
The question the study set out to answer matters to any organisation now leaning on AI to write code which, in India's software and IT services industry, is a fast-growing list. As global technology majors report that a large and rising share of their new code is now AI-generated, Indian enterprises, GCCs, and IT services firms are following the same trajectory, often with less time spent scrutinising what "AI-generated" actually means for security.
How the study was done
Rather than build a new benchmark from scratch, Kabanov re-ran two established ones against today's frontier models: CyberSecEval, a snippet-level test that checks security alone, and SusVibes, a tougher, agentic benchmark that runs full-feature coding tasks inside real open-source repositories, each built around an actual historical vulnerability, a closer proxy for how AI coding tools behave inside a company's real codebase. Because CyberSecEval was also run on GPT-3.5 and GPT-4 back in 2023, the study offers a rare, direct generation-over-generation comparison, rather than a one-time snapshot.
What the study found
Four of today's leading AI coding models Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, and Gemini 3.5 Flash were tested against 200 real-world coding tasks. The newest models solved the coding task correctly 83–95% of the time, a sharp jump from 44–61% just one generation ago. But the share of solutions that were both correct and free of the specific security flaw the task was built around stayed flat at 24–36%.
In practical terms: of the code that worked, roughly two-thirds to three-quarters still carried the exact vulnerability it should have avoided.
Notably, this held even when the vulnerabilities being tested were publicly known well before the models were trained suggesting familiarity with a flaw doesn't stop a model from reproducing it. Coding skill also didn't predict security: in one test, the model that wrote the most correct code ranked last on security.
Knowing versus doing
The research points to a specific pattern behind the failures. It isn't that AI models lack security knowledge; most can correctly describe the right defence when asked, and in several audited cases, a model named the exact defence needed in its own planning notes, then shipped the vulnerable version anyway. The breakdown happens in translating that knowledge into code, not in the knowledge itself.
The study also tested what actually helps with some counterintuitive results. A simple prompt reminding the model to "follow security best practices" added only 1–8 points on real repository tasks. Pushing a model to reason harder about the problem did nothing for security at all, and cut the number of working solutions in half. Two more deliberate interventions worked better: having the model map out threats before writing any code lifted secure-and-working code to 43–49%; adding a dedicated security review afterward pushed that to 47–56%. But even at that ceiling, roughly half of tasks still shipped insecure at four to five times the computing cost of writing the code alone.
"The most striking thing in this research isn't that AI models make security mistakes, it's that they often know better and do it anyway," said Eran Kinsbruner, VP at Checkmarx. "We saw models correctly name the exact defence needed in their own threat model, then ship the vulnerable code regardless. That's not a knowledge gap, it's an execution gap, and it tells you these tools can't be left to secure themselves unchecked."
Why this matters for India
India is both one of the largest producers of software for global enterprises and one of the fastest adopters of AI-assisted development, through its IT services majors, GCCs, and product startups. As AI-written code moves deeper into production systems, including those handling financial transactions, healthcare data, and critical infrastructure, the study's authors argue that security testing needs to keep pace with adoption, not follow years behind it. With India's Digital Personal Data Protection (DPDP) Act tightening expectations around data handling, and sectoral regulators such as the RBI and SEBI raising the bar on cyber resilience, the cost of shipping insecure AI-generated code carries growing regulatory weight, not just technical risk. The study's finding on false-positive noise is a further wrinkle for Indian security teams already stretched thin, where alert fatigue is a well-documented operational problem.