Agent Patches Can Shrink Your Test Suite Without Failing CI

A green CI check means the tests that ran passed. It says nothing about how many tests ran.

Agent-written patches are very good at making that second number smaller. The pattern is rarely sabotage - it is convenience. A failing integration test, an import that wants a network, a slow marker: each has a one-line "fix" that removes it from the run. Nothing goes red, the evidence set narrows, and the merge lands on top of it.

The gate below closes that specific gap. It records the collected test inventory on the base tree and on the patched tree, diffs the two, and refuses any shrink that is not written down in an expiring waiver file.

The same waiver file is where quarantined flaky tests live, which turns "freeze the flaky test" into a reviewable budget that shrinks over time instead of a permanent @pytest.mark.skip.

Everything here is pytest-specific; the census idea is runner-agnostic, but I will not pretend the code ports for free.

The three shapes that pass CI untouched

A conftest.py edit that drops a directory:

# before: no collect_ignore anywhere
# after, in tests/conftest.py
collect_ignore = ["integration"]

A module-level skip that makes an entire file vanish from the run rather than fail inside it:

# tests/test_ledger.py
import pytest
pytest.skip("flaky under load", allow_module_level=True)

A config change that deselects by marker:

# pytest.ini, addopts
-m "not slow"

None of these produce a failing test. All three change what your green check actually covers. The census is built to see the difference between "tests passed" and "tests existed".

Step 1 - Record a collection census

A 20-line plugin dumps every collected node ID plus its markers and the run configuration. No test bodies execute, so this is cheap and safe to run on untrusted trees.

# census_plugin.py - load with: pytest --collect-only -p census_plugin
import json
import os
_items = []
def pytest_collection_finish(session):
    for item in session.items:
        _items.append({
            'nodeid': item.nodeid,
            'markers': sorted({m.name for m in item.iter_markers()}),
        })
def pytest_sessionfinish(session, exitstatus):
    payload = {
        'exitstatus': int(exitstatus),
        'args': [str(a) for a in session.config.invocation_params.args],
        'addopts': session.config.getini('addopts'),
        'count': len(_items),
        'items': _items,
    }
    with open(os.environ.get('CENSUS_OUT', 'census.json'), 'w', encoding='utf-8') as fh:
        json.dump(payload, fh, indent=1, sort_keys=True)

Run it once per tree. PYTHONPATH=. is required so -p census_plugin can import the module from the repo root.

CENSUS_OUT=/tmp/base.json PYTHONPATH=. pytest --collect-only -q -p census_plugin
CENSUS_OUT=/tmp/head.json PYTHONPATH=. pytest --collect-only -q -p census_plugin

Step 2 - Diff base against head

The gate compares five things: invocation arguments, addopts, collection exit status, removed node IDs, and newly added skip/skipif/xfail markers.

Two runs of the same tree should produce byte-identical JSON apart from ordering; if they do not, your suite has unstable node IDs and the gate will be noise (more on that below).

# census_gate.py BASE HEAD WAIVERS -> exit 1 on unwaived drift
import json
import sys
from datetime import date
EXCUSE = {'skip', 'skipif', 'xfail'}
def load(path):
    with open(path, encoding='utf-8') as fh:
        doc = json.load(fh)
    return doc, {i['nodeid']: set(i['markers']) for i in doc['items']}
def norm(value):
    return [w for w in (value or '').split() if w]
def main(base_path, head_path, waiver_path):
    base_doc, base = load(base_path)
    head_doc, head = load(head_path)
    with open(waiver_path, encoding='utf-8') as fh:
        waivers = json.load(fh)
    today = date.today().isoformat()
    drift = []
    if norm(' '.join(map(str, base_doc['args']))) != norm(' '.join(map(str, head_doc['args']))):
        drift.append(('config', 'invocation args', base_doc['args'], head_doc['args']))
    if norm(base_doc['addopts']) != norm(head_doc['addopts']):
        drift.append(('config', 'addopts', base_doc['addopts'], head_doc['addopts']))
    if head_doc['exitstatus'] != 0:
        drift.append(('collection', 'exitstatus', base_doc['exitstatus'], head_doc['exitstatus']))
    for node in sorted(set(base) - set(head)):
        if node not in waivers.get('removed', {}):
            drift.append(('removed', node, 'collected', 'absent'))
    excused = waivers.get('excused', {})
    for node, markers in sorted(head.items()):
        fresh = (markers & EXCUSE) - (base.get(node, set()) & EXCUSE)
        if not fresh:
            continue
        entry = excused.get(node)
        if entry is None:
            drift.append(('excused', node, 'none', sorted(fresh)))
        elif entry.get('expires', '1970-01-01') < today:
            drift.append(('excused', node, 'expired ' + entry['expires'], sorted(fresh)))
    budget = waivers.get('budget', {}).get('excused_max', 0)
    if len(excused) > budget:
        drift.append(('budget', 'excused count', budget, len(excused)))
    for kind, what, before, after in drift:
        print('[{}] {} | base={!r} head={!r}'.format(kind, what, before, after))
    print('{} drift event(s); {} base vs {} head node ids'.format(len(drift), len(base), len(head)))
    return 1 if drift else 0
if __name__ == '__main__':
    sys.exit(main(*sys.argv[1:4]))

Added tests never fail the gate. Only losses and new excuses do.

Step 3 - The waiver file is the freeze

A waiver is a skip with an owner and a deadline. Everything removed or newly excused must appear here, or the gate exits non-zero.

{
  "budget": { "excused_max": 6 },
  "removed": {
    "tests/test_pricing.py::test_rounding_edges": {
      "owner": "pricing",
      "expires": "2026-12-15"
    }
  },
  "excused": {
    "tests/test_sync.py::test_reconnect": {
      "owner": "sync",
      "expires": "2026-11-30"
    }
  }
}

This is the part people skip, and it is the part that matters. A flaky-test freeze without an expiry date is a deletion with extra steps.

Step 4 - Make the quarantine budget shrink

excused_max is a policy number you lower deliberately, not a ceiling you raise when a patch is inconvenient. The counterweight is the removal path: when an excused test is fixed, both the marker and the waiver entry go away, and the next patch must lower the budget in the same commit.

If the count only ever goes up, the gate is decoration.

Step 5 - Wire it into the patch gate

# .github/workflows/patch-gate.yml (excerpt)
- name: collection census
  run: |
    set -euo pipefail
    git worktree add /tmp/base "$BASE_SHA"
    (cd /tmp/base && CENSUS_OUT=/tmp/base.json PYTHONPATH=. pytest --collect-only -q -p census_plugin)
    CENSUS_OUT=/tmp/head.json PYTHONPATH=. pytest --collect-only -q -p census_plugin
    python census_gate.py /tmp/base.json /tmp/head.json waivers.json

Reading the output: a decision table

• removed node, no waiver - Usual cause: File deleted, renamed, or dropped by collect_ignore. What the reviewer should do: Demand a waiver with an owner; renaming means a new node ID and a human-checked one-to-one mapping.
• excused gains skip / xfail - Usual cause: Flaky test quarantined. What the reviewer should do: Accept only with an expiry and a linked issue; expiry inside the current quarter.
• config addopts drift - Usual cause: -m / -k / --ignore narrowing. What the reviewer should do: Treat as a coverage change, not a build tweak.
• collection exit status non-zero - Usual cause: Import error masked by a skip. What the reviewer should do: Hard stop: the suite never finished collecting.
• budget over excused_max - Usual cause: Quarantine creeping. What the reviewer should do: Reject the patch or fix a test in the same change.

Property checks need provenance, not just existence

The census protects the size of the suite. It cannot protect its meaning - a collected test that asserts nothing looks identical to a good one.

That is a separate check, and the cheap version is procedural rather than clever: keep property/invariant tests in a path the agent patch may not touch, and require every new property to be shown failing on at least one hand-written broken input before it is accepted. If a property has never failed, nobody knows what it detects.

Treat that as a review rule you enforce by eye or by CODEOWNERS, not as something the census will catch for you.

Running the two-tree check without a second machine

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate needs two checkouts of the repo and one Python environment both can import. MonkeyCode offers free model access and a free server option, and I use them for two deliberately separated jobs, because a gate that depends on a model is not a gate.

• Census execution on the free server option. The two worktrees, the two --collect-only runs, and the diff job do not have to live on a laptop whose environment drifts. The logic is unchanged - still two collection runs and one comparison.
• Model-assisted triage, after the gate has already decided. The deterministic gate prints drift lines first. Only then does the model see that report plus this decision table, and its job is to propose a cause label and a draft waiver entry per event.
• Human-owned policy. The model does not write expires, does not raise excused_max, and does not choose the marker set. A reviewer edits waivers.json by hand and reruns the gate.

If the model labels an event wrong, the cost is a bad first draft that still needs approval. If the server is unavailable, the same two commands run anywhere else. Neither piece is load-bearing for correctness, which is the only reason I am comfortable putting them in the loop.

Limitations

The census sees pytest node IDs, not coverage. A test that keeps its name and loses half its assertions is invisible here.

Dynamic parametrization, generated test IDs, and time-dependent names produce unstable node IDs; the gate will report phantom removals until IDs are stabilised.

Suite-level conftest.py changes that alter collection order or fixture scope are reported only insofar as they change the node set or the excuses.

Waivers are a social contract. Nothing in the script stops a reviewer from extending an expiry date by a year.

Setting up two worktrees assumes the patched tree can be collected without a live external service. If it cannot, fix collection first.

Who should not use this

Teams without a named owner for waiver expiry should not adopt it - the file will rot and the gate will be bypassed.

Repos where test collection requires network access or a database need a stubbed collection path before any of this is meaningful.

And if your suite already has a hard rule that agent patches cannot touch test configuration at all, the census is redundant.

Start by running the census on a repo you already trust. The drift list should be empty; if it is not, you have found the first thing to fix before any agent patch is worth reviewing.