Our agent's tests were green. 16 of 17 never ran.

In October 2026, a coding agent opened a merge request for a database library in DynoInvoice, the e-invoicing system agents are building under Keel. The task required negative tests: stale versions, concurrent updates, another tenant's rows, no transaction, row-level security on and off.

The agent wrote 17 of them, in a shared base class. The merge request went through one review round and two green pipelines. 16 of those 17 tests had never run.

The line that hid them

The tests were written in Kotlin, in this shape:

@Test fun `rejects a stale version`() = withService { assertThrows<StaleVersionException> { store.update(staleRow) } }

withService returns whatever its lambda returns. Here the lambda returns the exception that assertThrows hands back, so the test method doesn't return Unit. It returns a value.

JUnit Jupiter only runs test methods that return void (in Kotlin, Unit). It skips the others without failing the build. So the pipeline was green and listed six test classes, and every class ran one case.

The fix is a small change of syntax: a block body instead of an expression body, so the method returns Unit:

@Test fun `rejects a stale version`() { withService { assertThrows<StaleVersionException> { store.update(staleRow) } } }

Who caught it, and who didn't

The independent reviewer read the merge request and found a real gap: a missing batch-audit test. It didn't notice that most of the tests weren't running. Neither did a check that asks each changed test class for at least one passing case. Every class had exactly one.

A separate, independent agent noticed later, while working on something else. It opened the test report and saw 6 classes with 1 case each, against a shared base class with 17 @Test methods. Re-running one class reported 2 tests.

It was caught before merge. But it wasn't caught by the step that was supposed to catch it, and we'd rather say so.

The fix that stays fixed

Fixing the 16 methods wasn't enough on its own. The same mistake could come back, and it would be just as invisible. So two things changed:

• Every @Test must return Unit.
• A new gate compares counts. For each test class, the number of cases in the report must equal the number of @Test lines in the file.

The gate also has tests of its own. A deliberately expression-bodied test must come out one short, and a missing class must count as zero.

After the fix, every case in the six classes ran, with no failures and no skips. The change merged on 18 October 2026 (commit daaaaabd in the public repository).

Green lied to us in other ways too

This wasn't the only time a passing check proved nothing:

• A refusal test that didn't test the refusal. To check that the test could fail, a reviewer replaced a tenant check with if false. The test stayed green, because a different layer rejected the insert with the same error code. The check is now proved on its own.
• A gate that skipped new files. A schema that broke a published API standard merged green, because the gate compared each schema with its previous version, and a new file has none. The next merge requests failed, the schema was fixed through a new task, and the gate now checks new files too, with no exception added.
• Claims in merge requests are checked, not trusted. Reviewers check them against the pipeline and the code: claims that a change had been verified locally, that a red job was a known flaky test, and that only two files had changed were each found untrue in review and corrected.

What you can copy

• Read the case count, not the colour. Green tells you nothing failed. It doesn't tell you anything ran.
• Gate on counts. Compare the cases reported per class with the test methods in the source. A minimal version, for Gradle's JUnit XML reports and Kotlin tests:

# fail if any test class ran fewer cases than it declares
for src in $(grep -rl --include='*.kt' '@Test' src/test); do
  cls=$(basename "$src" .kt)
  declared=$(grep -cE '@Test([^A-Za-z]|$)' "$src") # not @TestInstance
  ran=$(grep -ho "testsuite name=\"[^\"]*\.$cls\" tests=\"[0-9]*\"" build/test-results/test/*.xml \
    | grep -o 'tests="[0-9]*"' | grep -o '[0-9]*' | head -1)
  [ "${ran:-0}" -ge "$declared" ] || { echo "$cls: declared $declared, ran ${ran:-0}"; exit 1; }
done

Adapt it to inherited tests and parameterised cases. Then plant a skipped test and make sure the gate goes red.

• Prove each negative test can fail. Break the thing it guards, and watch it go red.

Where Keel fits

Under Keel, a task's tests must prove acceptance criteria held in Keel; the coding agent can't change those criteria, and a rule forbids weakening a check to get green. On DynoInvoice, an independent reviewer agent also checks every merge request, including any change to tests. That review missed this one.

The fix tightened the check instead of loosening it: an independent agent read the test report, a ruling settled the fix, and a gate that compares case counts now runs in DynoInvoice's build.

Written by the DynoKore team with AI assistance. Events from DynoInvoice's public repository and Keel record, October 2026. First published on dynokore.com. We're opening DynoKore Keel to a small group of design partners: dynokore.ai.