DEV Community

The Locked Store That Unlocked Itself

From the agentinbox v0.1.0 release, part of agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses. This story is finding #1 of the v0.1.0 field test report.

AI disclosure: this article was drafted and edited with AI assistance. The field test, the defect, the fix, and every number are real and linked below.

I shipped a tool whose only job is to refuse when something is wrong. Then a field test found a path where it quietly fixed the wrong thing and carried on.

The scenario sat in the plan like a formality. Lock the store's permissions with chmod 000, then check that the product refuses to operate. Every security person writes this kind of case and half of them never expect it to fire.

My fault matrix said a permission-locked store must fail closed. A case named FT-SC-24-LIVE existed to prove it, on a real box, with no fixtures in between. The tool looked at the locked store, unlocked it, and printed no pending asks.

The line that undid the lockout

The driver opened its database like this:

if not self.path.exists():
    descriptor = os.open(self.path, os.O_CREAT | os.O_RDWR, 0o600)
    os.close(descriptor)
else:
    _chmod(self.path, 0o600)

If the database does not exist, create it private. Fine. If it does exist, chmod it to 0600. Also fine, if you wrote that line thinking about a bad umask or a file someone copied around.

Now put the attack next to it. chmod 000 inbox.db means nobody can read or write that file, the owner included. The next open should die loudly. sqlite3.OperationalError. Anything.

The else branch ran first instead. chmod 0600 on the locked file, run by the owner's own process, handed the permissions back. The lockout lasted one function call. The connection succeeded, the inbox rendered, the CLI exited 0.

Then doctor, my own shipped posture check, looked at the file and reported mode 600. Healthy. Green. Nothing to see here.

Why 930 green tests never noticed

The unit suite was thorough. Store conformance, crash recovery, migrations, permission checks. Nine hundred and thirty tests green. One of them was literally named test_store_does_not_loosen_permissions_on_open.

The gap is structural, and this is the part I keep thinking about. A unit test builds its own store. It makes a temp directory, initializes a fresh database, and asserts against the thing it just made. That fixture can never simulate a permission change that happens outside the process, after creation, by somebody else. Not because it is hard. Because the shape of the test never asks.

FT-SC-24-LIVE differed in one way. It ran on a real filesystem, and the step script did the attacking:

chmod 000 "$HOME/inbox.db"
agentinbox inbox; rc=$?

The assertion wanted rc=4, the reserved fail-closed exit code. The field returned rc=0. That was the whole discovery. No clever tooling, no fuzzing. A fault case the plan told us to run, run on the day it was scheduled, recorded as a fail against my own product.

Your tests cannot find what your fixtures pre-forgive. The unit tests were not wrong about what they tested. They were wrong about the world. They assumed a store is only ever touched by the process that owns it.

The fix, and what made it stick

The fix turned out to be small. Permissions change at exactly one moment in a store's life: creation. Everything after that is either a read of the current state or a refusal.

if not self.path.exists():
    # a new db is created private
    descriptor = os.open(self.path, os.O_CREAT | os.O_RDWR, 0o600)
    os.close(descriptor)
else:
    # never chmod a file we cannot read
    mode = stat.S_IMODE(self.path.stat().st_mode)
    if mode & 0o444:
        # readable but permissive: tighten only
        _chmod(self.path, 0o600)

A file the owner cannot read now falls through to the connection attempt. sqlite3.OperationalError, wrapped as StoreError, mapped by the CLI to exit 4. Fail closed, mode untouched, doctor tells the truth.

The patch was not the part that mattered. The discipline after the patch did:

  • Write the regression test first, against the broken build. tests/store/test_locked_store.py chmods a real database to 000, asserts load_store raises StoreError, and asserts the mode stays 000 afterward. I watched it fail, then applied the fix, then watched it pass.
  • Add a second test asserting a fresh store is still created 0600, so the fix cannot overcorrect into refusing to create private files at all.
  • Re-run FT-SC-24-LIVE on the fixed build, in the field. The locked store exits 4 there now, not just under pytest. Without step 3, I would have had a claim. The re-run is what made it evidence.

What I now believe about fault cases

I used to file fault cases under paranoia, the padding at the end of a test plan. This sweep moved them up my list, for three reasons and one that stings.

The one product defect came from the fault case. Fifty-three field units ran. Exactly one product defect surfaced, and a fault case found it, not a unit test. The other five failures were harness problems and a deliberate decision to run no human study.

The fault has to happen outside your process. A fault your fixture creates is a fault your fixture controls. The chmod came from the test runner shell, on a filesystem nobody in the test just initialized.

Watch what your diagnostics say after recovery. The bug was not the recovery. Recovery is fine. The bug was that recovery made the diagnostic lie. When your doctor reports healthy after your fail path fires, the doctor is the vulnerability.

The uncomfortable one: the field case cost about a day of harness work to run, and the unit suite would never have caught this. Test only what your fixtures can build, and there is a class of bug you will ship with confidence and later learn about from a stranger.

The receipt

Everything above is checkable:

  • The defect, the repro, and the fix live in the field test report, defect catalogue row 5 and finding 1 in the field notes.
  • The field evidence sits under results/s16-live/cases/FT-SC-24-LIVE/, verdict files included.
  • The regression test is tests/store/test_locked_store.py.
  • The released build is agentsec-agentinbox==0.1.0 on PyPI.

Has a test ever told you your own fail path was healthy? What did you do about it?

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.