Your coding agent shouldn't grade its own homework

AI coding agents can now write design docs, mock up frontends, and implement entire projects end to end. But can they be trusted to verify that their own code actually works?

I lead an engineering team of eight at Momentic that has eagerly adopted background agents. My team frequently awes at the raw intelligence of these frontier models, but they also often complain that their agents will confidently declare that a feature was complete, only for basic functionality to fail when they opened the application for testing.

The problem isn't just that agents make mistakes. It's that their own verification often fails to catch them, leading to overconfidence. Concretely, our engineering team has found three recurring problems: agents rely on shallow checks, inherit their own implementation blind spots, and become less reliable under the combined pressure of growing context and repeated failures.

Shallow checks

Coding agents gravitate toward checks that are easy to write and easy to pass. That usually means unit tests. Unit tests have their place, but a green test suite doesn't necessarily mean the software works end to end.

At Momentic, we recently conducted a human-reviewed audit of 1,000 AI-authored unit tests in our monorepo. The result: only 9.6% were useful verification checks. The remaining 90.4% provided little meaningful value.

• Nearly a third were low-consequence tests checking basic argument passing or formatting.
• Another 22.8% duplicated existing coverage, while 22.4% relied excessively on mocking that poorly reflected production behavior.

Figure 1: AI-authored unit tests

Our findings aren't isolated. In a recent experiment, AI researcher Kun Chen compared coding agents allowed to write their own tests against agents explicitly prohibited from doing so. Across 111 coding tasks, the agents that wrote tests achieved a 65.3% success rate, compared with 66.2% for those that didn't.

This doesn't mean unit tests are useless in general. It means an agent can author massive volumes of unit tests, get seemingly reassuring results, and still fail to verify the behavior that matters.

Shared blind spots

When an engineer writes a feature, they first form an understanding of what needs to be built. They then make implementation decisions based on that understanding. On the surface, coding agents work similarly. The problem is that when they write their own tests, those tests reflect the same, possibly erroneous, assumptions that went into the implementation in the first place.

For example, suppose you ask an agent to build a rich text editor. It implements bold, italic and, underline formatting, then creates tests for those buttons. But perhaps you never specified that the undo button should also undo any applied formatting. Even though this makes natural sense to humans, if those scenarios weren't explicitly considered in the initial prompting, there's a good chance they won't appear in the agent's tests either.

To that point, a 2026 study on coding-agent verification found that tests agents wrote for their own patches often accepted incorrect implementations. So the problem isn't that agents can't write valid tests. It's that they often test what they built, rather than what they should have built.

Completion pressure

Modern coding agents run autonomously for hours. During that time, they accumulate enormous amounts of context: source files, terminal output, screenshots, debugging logs, and so on. This creates another problem. The longer an agent works on a task, the harder it can become to distinguish the original requirements from all the decisions made along the way. And although agents can compact their context, the original material can never be preserved with perfect fidelity.

Chroma's 2025 Context Rot study evaluated 18 LLMs and found that performance deteriorated dramatically as context grew. In one experiment, Claude Sonnet 4's accuracy dropped from 89% to just 38% when context increased from 300 to 113,000 tokens.

Anthropic has done its own research on the topic as well. In a study conducted this April, Anthropic gave Claude Sonnet 4.5 coding tasks with impossible requirements. Researchers isolated a certain "desperation" vector that grew stronger after each failed attempt, spiking when the model considered cheating to pass the tests. Once the tests passed, the activation subsided, even though the solution was incorrect.

This doesn't mean models really experience desperation or that they cheat all the time. But it does show how repeated failures can push an agent toward declaring completion early rather than validating comprehensively.

The solution: Independent verification

So if not the agents themselves, then what? At Momentic, we've moved toward a different approach: separate the agent that writes software from the agent responsible for testing it.

Figure 2: Autonomous software factory

Instead of asking the coding agent to evaluate its own work, we give an independent QA agent the original requirements and access to the running app. It also has access to a remote fleet of browsers and remote mobile devices to efficiently parallelize testing. The QA agent doesn't inherit the coding agent's conversation history or even know about the implementation. It starts with fresh context and a singular, clean objective: find bugs.

There are three important principles at work here:

First, behavior is verified live and never inferred. Rather than relying on static analysis, unit tests, or mocked integration tests, the verifier exercises real user workflows through a live browser or mobile device. This ensures that “verified behavior” is truly 1:1 with what a human would experience.

Second, the verifier is completely independent. The coding agent can’t control the verifier's acceptance criteria or modify the tests to make any failures disappear. The verifier returns reproducible bug reports that the coding agent and even the human developer can efficiently grok on a pull request. This prevents reward hacking and builds trust in the review process as well.

Third, we rely on five different model providers to offer different perspectives. Research has found that language models can favor their own outputs during evaluation. A 2026 study on self-preference bias found that this occurs even when models are told to judge against objective criteria. Using models from different providers offers additional perspectives and more chances to catch regressions, not to mention better reliability when one provider experiences downtime.

The end result is a complete feedback loop: the coding agent builds, the verifier finds bugs, and the coding agent receives actionable reports to fix. The process repeats until the code meets the acceptance criteria.

Last month, we decided to measure just how much better this setup performs compared to the coding agent finding its own bugs. In a benchmark on four complex, open source apps we recently publicly released, our independent verifier “Mo” found 3-5 times as many bugs compared to Codex and Claude Code on its own.

Verification engineering is here

AI is making code dramatically cheaper to produce. The bottleneck is shifting from writing software to proving that it works quickly and reliably. Engineering teams should stop measuring quality by the number of tests an agent writes or how many checks pass in CI. The real measure is how many real defects those checks catch in a given amount of time and spend.

In our experience, targeted end-to-end validation is increasingly delivering better ROI than expanding unit test coverage, challenging some of the assumptions behind the traditional testing pyramid.

“Verification engineering” is quickly becoming a new, independent software discipline, with its own tools, challenges and evaluation metrics.