Your eval suite passes. I built the tool that checks whether it checks anything.

Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red? I didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: inject a known defect into the system under test, run the eval suite, and report which checks stayed green. A surviving mutation is a hole in the eval. Not an argument - a hole, with a name and a reproduction. It's called evalmut, and it's now public: - pip install evalmut - the CLI runs against a plain Python suite file - 18 mutation operators, every one provenance-gated: an operator only exists if a real, documented defect exists that it reproduces (mined from production failures and issue trackers - never authored to pad a coverage number) - Deterministic end to end. No LLM judges anything. Red/green is reproducible. - The catalog of what it caught - including in suites I trusted - ships in the repo as FINDINGS.md The part I'd defend in a design review: the tool went through eight rounds of adversarial cold-critique before I trusted it, because a mutation tester that's wrong is worse than none - it hands out false confidence about false confidence. Early rounds found real false positives in the tool itself. By round six, tool-fault false positives on a well-formed suite reached zero and stayed there. An empty suite exits nonzero on purpose: a mutation tester must never report "no holes" on a suite that checked nothing. I work with AI agents openly - Claude Code wrote much of this under an adversarial loop I operate; the discipline above is how I keep either of us from grading our own homework. Repo: github.com/egnaro9/evalmut (MIT, tagged v0.1-paper - there's a short paper in /paper if you want the method written up properly) If you run an eval suite you trust: point this at it before you trust it more. I'd genuinely like to hear what survives. Top comments (0)