AI Can Write Tests Faster Than Your Team Can Understand Them

AI coding tools have solved one problem remarkably well: They can produce code extremely quickly. That sounds obviously good. And most of the time, it is. But software development has never really been constrained by how fast we can type. The expensive part comes later. Understanding the code. Reviewing it. Debugging it. Changing it six months later when the person-or model-that wrote it has forgotten why it exists. Test automation is where this becomes especially interesting.

Generating the Test Is the Cheap Part

You can ask an AI coding assistant:

Write Playwright tests for our signup, login, checkout, password reset, dashboard, invoices, settings, and admin pages.

And a few minutes later you might have hundreds or thousands of lines of test code. It feels like incredible leverage. Until the suite starts failing. That’s the argument behind looking at the hidden cost of AI-generated test code. Generation cost has collapsed. Maintenance cost hasn’t. In some cases, AI actually increases it because you now have more code than your team would have written manually.

AI Pull Requests Need Different Review

There’s another subtle problem. Humans tend to judge large AI-generated pull requests differently. When someone on your team writes 80 lines, you probably read them. When an AI assistant generates 1,800 lines? You skim. You look at the filenames. You check whether CI is green. Merge. That’s dangerous for normal application code and potentially worse for test code because a bad test can happily pass for months.

There are good ideas in this guide to testing AI coding assistant pull requests, but the bigger principle is simple: AI-generated tests need validation just like AI-generated product code. “Generated successfully” does not mean “tests the right thing.”

Agents Add Another Failure Mode

Now we’re moving from AI that writes test code to AI that actually decides what actions to take. That introduces a new question: What if the model chooses the wrong tool? An agent might have access to:

• browser interaction
• API calls
• database queries
• file operations
• search
• internal tools

The action itself may work perfectly. It’s just the wrong action. Testing tool-selection failures in AI agents therefore becomes part of testing the product.

And dynamic web applications make this harder. Partial renders, changing state, asynchronous updates, and stale context can cause an agent to confidently interact with yesterday’s version of the page. That combination of tool selection, state drift, and partial renders is probably going to become a much bigger QA category over the next few years.

Self-Healing Can Heal the Wrong Thing

AI test repair is another feature that sounds almost universally good. Selector changed? Fix it automatically. But imagine the application used to have:

Submit Order

And now has:

Submit Quote

The test can no longer find its element. An AI repair system finds the new button and updates the test automatically. Technically, it repaired the selector. Semantically, it may have changed what the test means. That’s why I like the idea of evaluating AI-generated test repairs before trusting automatic fixes. Self-healing should reduce maintenance. It shouldn’t quietly rewrite your specification.

Measure Trust, Not Just Pass Rate

Traditional test dashboards love pass rates. 97.4% passed. Great. But with AI-generated or AI-repaired testing, I think teams need another category of metrics: How trustworthy was the conclusion? That includes things like:

• Was the failure reproducible?
• Was a selector repaired?
• Did the test behavior change after repair?
• Was the failure caused by infrastructure?
• Did AI classify the failure correctly?
• How often do humans override AI conclusions?

That’s why these ideas around measuring AI test failures in CI matter. The next generation of QA dashboards probably won’t just say PASS or FAIL. They’ll tell you why the system believes the result.

AI Should Reduce Abstraction, Not Increase It

This is where I think test automation is heading in an interesting direction. The worst outcome would be: AI generates enormous automation frameworks that humans barely understand, then another AI maintains those frameworks because humans can no longer understand them. That’s technically impressive. It’s also a strange place to end up.

The better outcome is AI generating tests into representations humans can still inspect, understand, and modify. Use AI to remove work. Don’t use it to create a larger black box. Because the real productivity metric isn’t: How much test code did the AI generate? It’s: How much less testing infrastructure does your team have to think about?