Survey: More AI Code Running in Production Environments with Caveats

Survey: More AI Code Running in Production Environments with Caveats

A global survey of 309 software engineering leaders, published today, finds 81% of organizations have to some degree reworked development and release processes to accommodate code generated using artificial intelligence (AI) coding tools. However, only 45% said their organization is running any of that code in production environments.

Conducted by Dimensional Research on behalf of Flux, a provider of an engineering intelligence platform, the survey also finds 35% use AI to write code that they never actually deploy. Flux CTO Aaron Beals said that while AI coding tools provide substantial benefits, they are also a double-edged sword given the additional risks that are often introduced.

Key Risks with AI-Generated Code

For example, the biggest risks organizations are encountering with AI code weekly involve:

• Security issues (49%)
• Dependency changes (48%)
• Impacts on performance (44%)

Adoption of Code Quality Tools

Nearly half (46%) have already purchased code quality analysis tools, while another 39% added automated code review to their workflows. A full 80% of organizations are now spending at least 10% of time on code review, the survey finds.

Code reviews in many organizations have become a significant DevOps bottleneck in the age of AI because concerns about running AI-generated code in production environments continue to persist, noted Beals. Those reviews are still largely being conducted by humans who are being overwhelmed by the amount of code now being generated, he added.

Common Use Cases for AI Automation

In fact, the survey finds that organizations are most likely to rely on AI to automate lower risk tasks such as:

• Documentation (69%)
• Unit testing (66%)
• Simple functions and code review (58% each)

Challenges and Future Outlook

Each DevOps team will need to determine their level of comfort with code generated by AI coding tools, but as advances in the ability of AI models to reason continue to be made, the quality of the code being generated continues to improve.

The issue in the short term is finding a way to review code of uncertain quality that has been generated using earlier versions of AI models. Most of those models were trained using examples of flawed code that results in more vulnerabilities and other security weaknesses finding their way into code that, while well structured, tends to often be overly verbose. That latter issue results in not only a larger attack surface to defend but also higher processing costs.

More challenging still, it’s often difficult for application developers to review code they didn’t write because they lack any of the context about how it was constructed in the first place. Of course, human application developers are far from perfect so many of these issues would be encountered by software engineers anyway. The primary difference is the amount of code that needs to be reviewed continues to exponentially increase.

Eventually, DevOps teams will need to rely more on third-party AI agents that are based on another AI model to review code written by another AI agent. In the meantime, however, it’s clear that when it comes to AI there is no going back. The challenge and the opportunity now is to determine how best to optimize DevSecOps workflows that were designed for what is clearly now a previous era of software engineering.