I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.

Here is a Python function and a test for it. def withdraw(balance, amount): if amount balance: raise ValueError("insufficient funds") return balance - amount def test_withdraw(): assert withdraw(100, 30) == 70 That test gives you 47% line coverage. It gives you a 9.5% mutation kill score. The harness generates 21 small breakages of that module. The suite notices 2 of them. That gap is the whole reason I started this project. Line coverage is the default test-quality signal in most of the industry, and it measures whether a line ran. It does not measure whether anything would have complained if the line were wrong. AI-generated tests are unusually good at producing that shape: high coverage, low detection. Mutation testing measures the real thing. You break the code in small ways and check whether the tests notice. def withdraw(balance, amount): - if amount <= 0: + if amount < 0: raise ValueError("amount must be positive") If no test fails, that is a bug your suite cannot detect. It has been sitting there the whole time. Mutation testing never went mainstream, and I think the reason is simple. It hands you a wall of surviving mutants and no path to fixing any of them. It tells you that you have a problem and then leaves. So the idea was an agent that closes the loop. Find the survivors. Write tests that kill them. Gate each generated test on a hard criterion: keep it only if it passes on clean code and fails on the mutant. Ground truth is a subprocess exit code. No model judges any outcome. I built it over about 30 hours for the micro1 Frontier Engineering Challenge, which had around 7,800 registrants. The tool works, sort of. It is incomplete and I will get to the numbers. But that is not the interesting part of the weekend. The interesting part is that my measuring instrument kept lying to me, and it lied in a consistent direction. The finding that broke my own premise Before I ran a single agent call, I ran the harness across 12 widely-used, well-maintained Python libraries: cachetools, validators, natsort, dictdiffer, toolz, voluptuous, python-slugify, python-dotenv, shortuuid, boltons, aiofiles, tenacity. 455 mutants generated. 133 survived the existing test suites. Then I checked something I had assumed I would not need to check. Of those 133 survivors, how many sit on a line the tests actually execute? 53. The rest were never run at all. Not weakly tested. Not vacuously tested. Untested. I suspected my test commands were scoped too narrowly, so I widened them per target, between 6 and 40 times more test code. If the "executes but does not assert" category was real and I was just missing it, that number should climb. It went from 54 to 53. Down. And where widening changed anything, it converted unreachable mutations directly into kills. It did not move them into the middle category. It skipped it. So in mature, human-written Python, the vacuous test failure mode is rare. Where these suites fail, they fail by not running the code at all. The story I had absorbed about tests that execute everything and assert nothing is a story about AI-generated tests. It is not a story about human ones. I had to reframe the project before I had built the main part of it. That was the first sign that what I was actually building was a measuring instrument, and that I had not been treating it like one. Eight bugs in the instrument Every one of these would have produced a confident, wrong, publishable number. Editable installs made mutations invisible. pip install -e on src-layout packages resolves imports back to the original checkout. My mutations were written to a temp copy, so they never executed. Three targets silently scored 0.000. That would have read as "the agent fails on src-layout packages," which is a finding. It is just not a true one.Concurrency corrupted one target. Running mutants in parallel gave me three different survivor sets across four runs, on the one target doing real async I/O. I had already drafted a result of "0.27 to 0.77" off that. It was noise. Note which way it pointed: spurious failures get counted as kills, and kills are the number every arm is trying to increase. A file picker chose the wrong test file. On the hardest target. Which means the model would have been shown irrelevant context in exactly the place where context mattered most. A classifier was about to run on the wrong unit. It classified batches, not individual tests. One strong test in a batch of 69 would have marked all 69 as strong. A reconstruction step dropped shared imports. This manufactured test failures that were not real failures. An extractor only scanned top-level functions. So a perfectly valid unittest.TestCase response got discarded as "no test found." Worse, the agent's retry loop then received a harness error instead of real pytest output. That quietly disabled the exact mechanism I was trying to measure.self.assertEqual(...) was classified as "no assertion." This one would have manufactured precisely the finding I was hypothesising. It would have handed me my own conclusion.A pre-registered metric was not computable on dunder-dispatched code like call and__or__ . It read as a real near-zero rate rather than as undefined. The two things they had in common They all pointed the same way. Every single one of those eight would have made my result look better, cleaner, or more publishable. Not one of them would have made the agent look worse than it was. I do not think that is a coincidence, and I do not think it is a conspiracy either. It is attention. When a number disappoints you, you go looking for the reason. When a number pleases you, you write it up. So the measurement bugs that survive all the way to publication are disproportionately the ones that helped you. That is a selection effect operating on your own debugging, and you cannot fix it by being careful. Careful people are exactly as motivated to stop investigating when the number looks good. None of them was found by reading code. Every one was caught by running a check whose outcome I had predicted in advance, and getting the wrong answer. The clearest case was bug 5. My prediction was: remove this one known-bad test and the suite goes green. It did not go green. That contradiction is the only reason I found the dropped-imports bug before I trusted the numbers it was feeding me. I would not have found it by rereading the function. I had already read the function. The results, with the caveats attached Three arms, same model, same token ceiling. | Arm | What it does | Mutants killed | |---|---|---| | A | One prompt: "write as many tests as warranted." The brief's specified baseline. | - | | B | One test per call, same call count as the agent. No mutation hint, no gate, no retry. | 1 | | C | The agent: mutation diff in context, execution gate, one retry with real pytest output fed back. | 9 | On the 15 mutants the agent covered, B killed 1 and C killed 9. Keep rate was 60%, so the gate was genuinely filtering rather than rubber-stamping. Nine retries fired and three succeeded. Now the parts that matter just as much. The agent ran on 2 of 10 targets. The API budget ran out mid-run. I did not swap in a substitute model to finish the sweep, because then the comparison would not be a comparison. Those two targets are the ones where the baseline performed worst. That is not a random sample, and I have no way to argue it is representative. All 9 kills were on the two cheapest mutation types. There was zero cross-function transfer. Seven of the nine kept tests kill exactly the one mutation they were written for, and nothing else. That is a real limitation, not a rounding error. One result that inverted my hypothesis I had pre-registered a prediction that gate-passing tests would mostly be vacuous. Bare existence checks. Tests with no assertion that "kill" a mutant by crashing rather than by detecting anything. That is not what happened. The none category was empty. Eight of the nine kills were real assertion failures. And of the six discarded drafts, zero failed on clean code. All six were valid, passing tests that simply did not detect the bug. So the gate was not catching broken tests. It was catching working tests that miss. That is a more interesting failure mode than the one I predicted, and I only know it because the prediction was written down first and was wrong in a specific way. The determinism check that quarantined my own target Three hours before the deadline I did a clean-clone reproduction run to verify the reproducibility claim. The determinism check runs each target three times serially and requires the survivor sets to be byte-identical. Eleven of twelve reproduced exactly. The twelfth varied. I reported it in the README instead of fixing it. A check that has never caught anything is indistinguishable from a check that cannot catch anything, and the first thing mine ever caught was one of my own targets. Removing that from the record would have made the project look better and the instrument look worse. I missed the submission by 11 minutes. That is annoying in a way I do not want to dress up. But the reproduction run is what found the twelfth target, and running it is the reason I trust the other eleven. What I would tell you to take from this Before you measure an agent, write down what your instrument would look like if it were lying to you. Then build the check that catches exactly that. And write down which direction each possible lie would push your result. That second list is the important one, because it tells you which checks you will be least motivated to run. The repo is private while I finish the write-up. If you build evaluations for agents and you have hit this, I would like to compare notes, particularly on catching measurement bias before it reaches a number you have already started believing. Top comments (0)