Your Characterization Tests Passed. The Side Effects Changed Order.

Return values are the easy part of a refactor. Side-effect order is where messy repos break quietly.

A green suite can still merge a file that now gets written twice. It can still ship a temp file that is deleted before its reader runs. Characterization tests pin outputs. They rarely pin sequence.

This post builds a small ledger that records ordered side effects. You capture it before the refactor. You diff it after the smallest safe change.

The failure a return-value test cannot see

Here is a real shape. A nightly exporter writes a CSV, renames it into place, then shells out to a mailer. Your test asserts the returned path and the row count. Both stay correct while the internal order flips.

# Before: write -> replace -> notify
df.to_csv(tmp)
os.replace(tmp, final)
subprocess.run(["mailer", final], check=True)
# After: notify -> write -> replace (still returns the same path)

The test passes. The mailer reads a stale file. Nobody sees it until a customer asks where the report went.

Step 1 - Name one scenario, not the whole repo

One entry point, fixed inputs, no network. Keep it boring.

# scenarios/nightly_export.py
from reports.exporter import build_nightly

def run():
    build_nightly(dest="/tmp/export-check")

Do not try to ledger the whole package on day one. A scenario that touches ten modules gives you a diff you cannot read. Start where the change will land.

Step 2 - List candidates with grep, not with a hunch

grep -rnE "open\(|\.write\(|os\.replace|shutil\.(move|copy)|subprocess\.|requests\.(post|put)" reports/ \
  | grep -v tests/

You want the boundaries the scenario actually reaches. Grep gives you candidates, not proof. Confirm each hit by running the scenario once and printing the files it touched.

Step 3 - Capture an ordered ledger

Sixty lines. Patch a few boundaries, record every call in arrival order, normalize volatile strings.

# effect_ledger.py - ordered, normalized side-effect capture
import builtins
import hashlib
import json
import os
import re
import subprocess
from contextlib import contextmanager
from unittest import mock

LEDGER = []
_TMP = re.compile(r"/tmp/[A-Za-z0-9_.\-]+")
_TS = re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}")

def _norm(value):
    text = str(value)
    return _TS.sub("<TS>", _TMP.sub("<TMP>", text))

def _record(kind, name, payload=None):
    LEDGER.append({
        "seq": len(LEDGER),
        "kind": kind,
        "name": _norm(name),
        "payload": None if payload is None else _norm(payload),
    })

class _WriteTracer:
    """Records a content hash when the handle closes."""
    def __init__(self, handle, path):
        self._h, self._path = handle, path
        self._sha = hashlib.sha256()
        self._closed = False

    def write(self, data):
        self._sha.update(data.encode() if isinstance(data, str) else data)
        return self._h.write(data)

    def close(self):
        if not self._closed:
            self._closed = True
            _record("write.close", self._path, self._sha.hexdigest()[:16])
        return self._h.close()

    def __enter__(self):
        return self

    def __exit__(self, *exc):
        self.close()
        return False

    def __getattr__(self, name):
        return getattr(self._h, name)

@contextmanager
def ledgered():
    LEDGER.clear()
    real_open, real_replace, real_run = builtins.open, os.replace, subprocess.run

    def open_hook(file, mode="r", *args, **kwargs):
        handle = real_open(file, mode, *args, **kwargs)
        if set(mode) & set("wax+"):
            _record("write.open", file, mode)
            return _WriteTracer(handle, file)
        return handle

    def replace_hook(src, dst, *args, **kwargs):
        _record("os.replace", dst, f"from={src}")
        return real_replace(src, dst, *args, **kwargs)

    def run_hook(*args, **kwargs):
        cmd = args[0] if args else kwargs.get("args")
        _record("subprocess.run", cmd)
        return real_run(*args, **kwargs)

    with mock.patch("builtins.open", open_hook), \
         mock.patch("os.replace", replace_hook), \
         mock.patch("subprocess.run", run_hook):
        yield LEDGER

def dump(path):
    payload = json.dumps(LEDGER, indent=2)
    with builtins.open(path, "w", encoding="utf-8") as fh:
        fh.write(payload)

Note the tracer. A path name tells you a file was opened. A hash tells you it held what you expected.

Step 4 - Prove the baseline is stable before you trust it

Run the same scenario three times under different hash seeds.

for seed in 0 1 2; do PYTHONHASHSEED=$seed python capture_ledger.py nightly_export "ledgers/base-$seed.json"; done
python diff_ledger.py ledgers/base-0.json ledgers/base-1.json
python diff_ledger.py ledgers/base-0.json ledgers/base-2.json

If those diffs are non-empty, stop. Your scenario is nondeterministic and the gate is worthless. Sort directory listings. Freeze the clock. Pin the temp prefix. Only then record a baseline.

Step 5 - Make the smallest safe change

Extract one block into a helper. Rename one variable. Change no behavior.

python capture_ledger.py nightly_export ledgers/after.json
python diff_ledger.py ledgers/base-0.json ledgers/after.json

Exit code 1 means the effect sequence moved. Exit code 0 means this scenario saw the same effects in the same order. That is a narrow claim. Keep it narrow and it stays trustworthy.

Step 6 - Read the diff as a decision, not an alarm

Commit before.json next to the harness. A baseline you cannot diff later is just a log file.

What this ledger will not catch

• Threads and asyncio. Interleaving order is not what you recorded.
• C extensions and direct syscalls below open and subprocess.
• Effects on boundaries you did not patch, such as a database driver or an SDK client.
• Paths your single scenario never executes. Coverage is still your job.

It also cannot tell you whether the new order is correct. Only whether it changed.

Who should skip this

If you already have an integration suite that exercises the real boundary, use it. This harness exists for repos with no harness at all.

Skip it for a rewrite. A ledger gate protects mechanical moves, not redesigned flows. Skip it for concurrency-heavy code, where the sequence is genuinely not fixed.

Where a free model and a free server fit

Two places, both modest. First, boundary discovery. Paste a module's imports into a free model and ask for candidate side-effect boundaries. Verify every candidate with grep, because models miss dynamic dispatch and mixins.

Second, isolation. Run the seed loop on a disposable Linux box instead of your laptop. The free server option covers a capture run for a small repo; it is not a reason to move your whole build there.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat free model access and the free server option as the operator's claims. I did not benchmark throughput, quotas, or hardware here, and none of the numbers above came from them.

Start with one scenario

One entry point, one boundary list, one baseline, one extraction. Then diff. If you try this, draft the scenario and the boundary list first. The free tier is enough for both, and the diff will tell you the truth.