A hundred and eighty six tests passed on every laptop and CI never collected one of them

In July a customer was given two discounts on one order that our rules say can never be combined. Fixing the code took ten minutes.

What took the afternoon was finding that the test for exactly that case already existed, had been written in February, and was failing.

How the tests went missing

Our pricing package had been rebuilt that spring by a contractor team, who named their test files after the modules they covered: discount_tests.py, basket_tests.py, tax_tests.py.

Our pipeline runs pytest from the repository root and lets it discover the tests. pytest only collects files whose names match its python_files setting, which by default is test_*.py and *_test.py. A plural tests matches neither.

The files sat in the tests directory, imported cleanly, and were never opened.

Why it looked green

The authors had not been careless. They ran their tests the way most of us run the tests we are working on, by passing the file path to pytest, and pytest collects any file named on the command line whatever it is called. On their laptops all hundred and eighty six passed.

Later changes broke some of them, and by then nobody was running them by path any more. In CI the job reported a green run of a little over two thousand tests, roughly the number it had reported before the rewrite, because the old pricing tests had been deleted in the same pull request.

Nothing failed, so nothing told us. A test count that stays flat looks exactly like health.

What we changed

We renamed the files and fixed the eleven tests that had rotted, two of which were real bugs we had already shipped.

The pipeline now runs pytest with --collect-only first, compares the result with a plain search of the repository for test functions, and fails if any file that defines one was not collected.

The python_files pattern is written out in pyproject.toml, so the rule sits where people look instead of in a default nobody reads.

The collected count is posted on every pull request next to the count on main, and a drop of more than ten needs a reviewer to say why. And when a pull request adds a test file, the reviewer checks for it in the CI log, not on the author's machine.

A test runner only runs what it can find. We had checked that our tests passed and never that anything was looking at them.

• Sergey Shinder