A survey of 500 software engineering leaders and practitioners published this week finds that while nearly all (95%+) believe artificial intelligence tools can reduce burnout well over half (59%) report AI tools at least half the time are creating deployment errors in their code.
As a result, more than two-thirds (67%) said they spend more time debugging AI-generated code, with 68% also noting they are now spending more time resolving AI-related security vulnerabilities. A full 92% of respondents also noted that AI tools are increasing the blast radius of the amount of bad code that needs to be debugged.
Harness Field CTO Nick Durkin said that while AI tools can generate more code faster, most of the tools being used have not been specifically trained using code samples from the production environment where code is destined to run. In the absence of having that context, it should not come as much of a surprise that the code being generated by AI tools doesn’t work in a production environment, he added.
Making matters more challenging, developers then need to spend time debugging code that they were not involved in creating, noted Durkin. That’s generally more difficult than debugging code those developers might otherwise have written themselves, he said.
Many of these issues would become more manageable if organizations had a formal policy concerning which AI tools are employed, said Durkin. The survey finds less than half of developers (48%) are using AI tools that have been approved by their organization. Well over half of respondents (58%) said their organization does not provide clear guidance on which use cases represent the lowest risk for AI adoption.
A full 60% of respondents said their organization lacks any formal processes for assessing AI-generated code vulnerabilities or errors and an equal percentage said the effectiveness of AI coding tools is not being evaluated.
On the plus side, 50% of the engineering leaders surveyed said their organization plans to invest in AI for continuous integration/continuous delivery (CI/CD), with 48% planning to prioritize performance optimization. A total of 42% also plan to use AI to improve security and compliance.
The only way to ensure that organizations are realizing the benefits of AI is to adopt platforms that provide a comprehensive framework for not only creating code using AI but also providing access to AI agents to review that code before it is deployed in a production environment, said Durkin. The issue with using standalone AI coding tools comes down to the simple fact that those tools have no understanding of the production environment where that code is expected to run, he added.
Additionally, many organizations have not yet considered the total impact these tools are having on infrastructure costs or their future sustainability goals, Durkin noted.
Harness’s research notes that if 78% of developers are spending at least 30% of their time on manual, repetitive tasks such as debugging code, that represents $8 million in lost productivity annually for organizations that employ 250 application developers.
There are, as always, concerns to what degree AI might replace developers, with 88+% of respondents still trying to determine the ultimate impact. Many organizations, however, are applying AI to the wrong tasks, said Durkin, Many developers actually enjoy writing code. It’s all the other menial tasks, such as creating tests and debugging code that they would prefer to automate, he said.
Regardless of how anyone may feel about AI, the core technologies are only going to continue to become more capable. The issue now is determining how to best apply it in a way that ultimately leads to more applications being deployed faster.
