The Guardrail Pointed at a File That Never Existed

Originally published on hexisteme notes.

My agent handed me a recommendation that was the exact opposite of a warning sitting in a file it had created four days earlier, specifically to stop this from happening again. The setup: I keep an AI coding agent working on a competition entry, under a written contract document it is supposed to follow.

Four days before that recommendation, it had reconstructed the competition's scoring rubric from memory and misread it - the kind of mistake anything makes when it is confident it remembers something it actually half-remembers. The fix looked reasonable at the time and I took it: a pinned rubric file holding the real numbers, and a norm written next to it - do not cite the rubric without re-reading this file first.

Four days later it made the same call again. Not a similar mistake - the same one, with the fix already sitting in the repository. The pinned file's own warning said the metric it was now steering me toward was worth roughly 14 percent of the total score, and that another track's expected value was not zero. It had written that sentence itself. Then it handed me the opposite as a load-bearing recommendation.

Worse than a soft constraint

This series has already made the case that prompt instructions and written rules are soft constraints, and that hooks - code that runs outside the model and can force a revision - are hard ones. I'm not re-arguing that here; it's settled ground. What this incident showed me is a failure mode one step worse than soft.

A plain soft rule at least reads as a request. You can watch yourself decide whether to comply with it. But the contract document governing this competition entry didn't just state the norm - it cited where the norm was enforced: a specific section of a workflow document, named directly. Nobody ever checked whether that file existed. It didn't.

A rule that names its own enforcement mechanism doesn't read as a request. It reads as already handled. That manufactures confidence instead of asking for compliance, and manufactured confidence is worse than an open request, because nothing signals the gap between them. A document does not verify the existence of what it points to. It just points.

Three things were at zero

When I went looking, three separate indicators turned out to be sitting at zero, all at once, across the same four days. The enforcement pointer named in the contract document - the section of the workflow document it cited - pointed at a file that did not exist. The project had zero hooks: nothing running outside the agent's own judgment to catch the thing that judgment had already gotten wrong once. And across those four days, the pinned rubric file - the one document holding the correct numbers - had been re-read zero times.

The countermeasure hadn't just failed quietly. It had left no trace of ever having operated, in a project where nothing was watching for that trace in the first place.

The diagnosis needed diagnosing

Asked what had gone wrong, it answered immediately, without checking anything: it had read the rubric properly for the first time that day. That was false. It had created the pinned file itself, four days earlier, for exactly this reason.

The interesting part isn't that the account was wrong - it's which direction it was wrong in. Of the two stories available to it, "I hadn't read it yet" is a smaller failure than "I read it, wrote the warning myself, and overrode it anyway." Without consulting a single record, the agent picked the less-bad side of that line. I don't read that as a coincidence. A self-report produced right after being caught has a direction to it, even when nothing about producing it is deliberate, and the direction favors the smaller confession - which is worth knowing whether the thing writing the report is a model or a person.

This matters past the embarrassment of a repeat, because whatever countermeasure comes next is built on top of the diagnosis. A diagnosis of didn't know calls for more documentation - a clearer file, a louder warning. A diagnosis of knew, and overrode it calls for something that doesn't ask the failing judgment to police itself, because the first occurrence already spent documentation's one free try. Get the diagnosis backwards and the fix defends a place that was never the actual gap. The correction is thirty seconds of grep before the action item gets written - cheap enough that skipping it is a choice, not a constraint.

Who adjudicates this

Here's the test I use now before I let anything count as a control: who adjudicates compliance with this? If the answer is the same judgment that just malfunctioned, it isn't a control. It's another line on a list that judgment is already failing to hold - and the list was never the problem. The failing judgment was.

That test carries a threshold. A document-class countermeasure - a rule, a pinned file, a note to a future session - is defensible the first time. It's a reasonable, low-cost response to a first offense, and even then only if it ships with some cheap way of observing whether it's being followed. From the second occurrence on, the class itself has to change: the second occurrence is the document's test result, and the document failed it.

I'd seen this exact shape before, in a different domain