Four products passed their tests. Then I tested them.

Every one of these had a green suite. 147 tests on the portfolio, 77 on the budget proxy, 53 on the conformance checker, 78 on the retrieval system. All passing. Each had a benchmark, a published result, and a live deployment. Then I wrote scenarios that hit the live URLs instead of the code, and one of them was overrunning its central guarantee by 303% in production. This is what that exercise found, including the four times my own harness accused a product of a bug it did not have. The rule A scenario had to be something a person would plausibly do, described in their words. "A runaway agent loop burns through its budget overnight" is a scenario. "Assert reserve() returns False" is a unit test. And nothing in the scenario suite may import product source code. It gets what a stranger gets: a URL over the public internet. That second rule is the one that mattered. What it found A spend ceiling that did not hold. 25 concurrent workers against a budget with room for two. Twenty were admitted. Overrun: 303%. The cause was 25 serverless instances, each holding its own in-memory ledger, each enforcing the $0.05 ceiling perfectly against its own slice of traffic. There were 25 ceilings instead of one. The uncomfortable part is where the answer already was. The store's own docstring read "correct for one replica and useless for two - which is precisely measured failure #1". The deployment doc said not to run it behind a load balancer. The baseline benchmark had measured the single-replica ledger as the first of four ways budget enforcement fails open. Three layers said the right thing and the product was still wrong, because none of them ran against the deployment. The first version of the test passed Worth dwelling on. My scenario asked the service how much it had spent: ✓ 12 workers fired concurrently - 4 allowed, 8 refused ✓ spend after the burst $0.008400 - ceiling $0.05 Both reads landed on an instance that had served one request. The suite was green while the product overran by 233%. A test that asks the system under test whether it is behaving cannot detect a system that is wrong about itself. Spend is now derived from what a provider would bill - admitted calls times observed cost per call - which does not depend on the service being right. The ledger's own figure is recorded beside it, and the disagreement between the two is the finding. Twelve workers also was not enough: it fanned across fewer instances and passed on some runs. Twenty-five made it deterministic. A test that fails intermittently gets rerun until it passes. Four times the harness was wrong The honest ratio is one real defect to four false accusations, and every false one was mine. It claimed a portfolio route was broken. It assumed /work , /about and /writing existed, got a 404, and reported the site as broken. There is no /work index. Routes now come from sitemap.xml . A scenario that invents the interface measures its author's assumptions. It claimed the citation mechanism was undeployed. It scraped work pages for citation links and found zero. Citations are not static links - the site ships a build-time allowlist of pre-verified passages, and the chatbot may only cite from that set. So a model cannot invent a citation even when it invents a claim. A stronger design than I had assumed, and verifying the allowlist verifies every citation the site can ever produce. It claimed 2 of 34 citations were unresolved. The passages contained apostrophes rendered as ' , and my tag-strip did not decode HTML entities. The text was on the page and would highlight correctly in a browser. It claimed a side channel in a project about side channels. It asserted that "no token" and "unknown token" must return byte-identical responses. But an attacker already knows whether they sent a credential, so telling them carries no information. The property that matters is that two rejected credentials are indistinguishable - verified against five plausible guessed principal names, all returning the same body. Each correction is documented in the code rather than quietly amended, because a test that flags correct behaviour costs more than no test: the next real finding gets ignored alongside it. What passing scenarios are worth Eighteen now pass. A few are worth more than the rest. One tests that a documented weakness is still real. The retrieval project publishes a limitation: padded results share no query terms, so the recipient can spot the filler and recover what was withheld. The scenario passes when that reproduces against production. A published weakness that turned out to be overstated would be its own kind of dishonesty. One tests that the baseline still leaks. The comparison only means something if the unfixed path is genuinely exploitable. Counts across nine identities come back as 0, 2 and 4, and the deficit tracks permission - the identity with least access infers the most, the CEO infers nothing. If the baseline did not leak, the enforced result would be measuring nothing. One tests that the good work stays findable. Three projects were added to the portfolio with live URLs and reproducible benchmarks. The failure mode that would quietly undo that is not a 404 - it is the pages existing while being absent from the sitemap, the chat corpus and the MCP tool list. The site would keep answering, from the generically described employer work, and nobody would notice the verifiable material had dropped out. What I would tell someone starting this Write the scenario against the URL, not the module. The import is what makes a test blind to deployment. Compute the thing you care about independently of the system that is supposed to be computing it. If you are checking a spend ceiling, calculate the bill yourself. Turn the concurrency up until the failure is deterministic. An intermittent failure is a failure you will explain away. And expect your harness to be wrong more often than the product. Mine was, four to one. That ratio is not a reason to skip it - one of the five was a 303% overrun on the exact guarantee the product existed to provide. Scenario suite, findings and reproduction: the scenarios/ directory. Every number here traces to a benchmark or a scenario result; a launch post that states an unmeasured figure fails its own build gate. Top comments (0)