hollowtest: find tests that pass but prove nothing

The problem

Mutation testing is the gold standard for “do my tests actually catch bugs?” It is also slow, noisy, and a poor default for every PR. Style linters do not care whether you asserted anything meaningful. Coverage only asks whether a line ran. What we needed in the AI-coding loop was a fast, deterministic gate: did this test body contain real proof, or is it hollow?

Hollow patterns

I kept seeing in AI-generated suites:

• Empty bodies - pass, a docstring, or an ellipsis left as a “TODO test”
• No assertions - call the code, maybe print, never check a result
• Tautologies - assert True, assert x == x
• Existence theater - only assert result is not None / assert result / assert len(x) > 0
• Mock theater - mock.assert_called_once_with(...) and nothing about real outcomes

All of these pass. All of them inflate coverage. None of them prove behavior.

What hollowtest does

pip install git+https://github.com/SybilGambleyyu/hollowtest.git
hollowtest tests/

It walks Python test files (pytest and unittest conventions), finds test functions, and classifies their assertions via the AST. No network, no model, no API key - stdlib only.

Meaningful patterns are left alone: assert result == expected, self.assertEqual, with pytest.raises(...), or a mock check plus a real value assertion.

Example against a tiny sample

test_app.py:12:1: warning HT003 in test_add_exists
Existence assertions only: Only checks presence/truthiness ...
test_app.py:18:1: error HT005 in test_placeholder
Empty test body: Test body is empty, docstring-only, or just `pass`
test_app.py:23:1: error HT001 in test_smoke_print
No assertions: No assert / self.assert* / pytest.raises found ...

Exit codes work for CI (--fail-on error|warning), and there is JSON plus GitHub Actions annotation output.

What it is not

• Not coverage. Coverage says a line ran; hollowtest asks whether any assertion proved something.
• Not mutation testing. It will not tell you if your equality check is too weak for every bug class. It catches the empty and trivial cases in milliseconds.
• Not a style linter. Use ruff for that.
• Python only, for now.

Why ship this

The gap I cared about was not “another AI that reviews your AI.” It was a boring, reliable check that treats test hollowness as a first-class defect class. If your agents write tests (or your teammates paste them), run this on the suite the same way you run the suite itself.

GitHub Actions sketch

- run: pip install git+https://github.com/SybilGambleyyu/hollowtest.git
- run: hollowtest tests/ --format github --fail-on warning

Get it

• Source: github.com/SybilGambleyyu/hollowtest
• Release: v0.1.0

If you find false positives or hollow patterns it misses, open an issue - that feedback is the roadmap. MIT licensed. Zero runtime dependencies. Python 3.10+.

Top comments (0)