Your Retry Budget Is Not a Safety Net

*Second in a series on The Factory. Previously: The Factory That Merged 37 Tasks. The harness is at github.com/frozer/factory. The public description of my task harness ends on a claim: a packet that's wrong about the world fails identically on every retry. That sentence cost me four dead tasks and nine commits spent repairing task definitions instead of writing code. It reads like something you'd arrive at by thinking. I arrived at it by watching the same failure scroll past three times in a row. What three attempts is actually good for max_attempts = 3 felt like obvious hygiene. Models are stochastic. Sometimes a run goes sideways for no reason you can name - a bad turn, a truncated response, a tool call that gets refused. Retry it and it works. That's real, and a retry budget handles it well. The strength is exactly the constraint. A retry budget assumes the next attempt will differ from the last one. It buys you a second sample from a distribution. But a retry doesn't hand the model a fresh situation. It hands it the same packet back. Same file, same claims, same instructions. If the packet says a file lives at a path where no file lives, attempt three fails precisely where attempt one did, and the only thing three attempts bought was three times the bill. Failure without variance isn't flakiness. It's a specification defect wearing a reliability costume. Nobody had opened the files Here's what that looked like in practice. B03 was a loader for a national census dataset. Three attempts, all burned, all against a file shape that existed nowhere: wrong directory, wrong filenames, and a Data / Valor JSON envelope that appears nowhere in the actual data tree. Every attempt produced a parser for a document that doesn't exist. The packet was the defect, not the model. Nobody had opened the actual files before cutting it. Rewritten from the real JSON, the truth was a flat metadata /data envelope, four files - one of which shouldn't be loaded at all - and a long-format pivot with genuine missing rows. Nothing about that is inferable from a design document. It's only knowable by opening the file. The same check applied to two sibling packets turned up a schema mismatch, a missed pivot, and a real design collision: two source files sharing five of six category labels verbatim, with no column to tell them apart. Loading both, as the packet instructed, would have silently overwritten data. Also, the packet's confident claim about Spanish decimal commas was simply false for that data - zero commas in any total column, across all three files. A11 is the cleaner case, because it removes the model from the argument entirely. Its invariant required two fields - a source SRID and an attribution string - routed through a typed config. Those fields did not exist on the type. And the file that defines the type was outside A11's editable set. So no diff the implementer was permitted to produce could satisfy that invariant. Not a hard task. An impossible one, three times, at full price. C09 was the expensive kind. Its steps read build → validate → write , which is a sensible order and reads perfectly. But the build call persisted internally before returning - so build had already written. All three attempts validated after the write, and dutifully logged the failure that the invariant existed to prevent. Nothing in the worktree tells a model any of this. It fails, gets handed the same packet, and fails the same way. The rule A retry budget is for variance, not for error. Before you raise max_attempts , read the three failures side by side and ask one question: did they differ? If attempt three failed differently from attempt one, retries are doing their job. If all three failed identically, you are not looking at an unreliable model. You are looking at an instruction that cannot be followed, and you are paying full price to have that confirmed repeatedly. The retry loop is the last place to fix this and the first place everyone looks. What it forced The fix wasn't a better implementer prompt. It was admitting that the cheapest place to catch a defective packet is before any model has cost anything. So the planning step got its own two agents. One cuts packets from the spec. A second one - with no ability to edit anything, because a run that can edit what it's judging will eventually repair a defect quietly instead of reporting it - assesses each packet and rules on every checkable claim in it: held , violated , or unverifiable , each with a path:line or a command's output. The queue now refuses any packet without a passing assessment pinned to that packet's exact hash. The cutter works from a ground-truth ladder: files on disk first, then the code the task will call, then what earlier packets produce, then the toolchain - and the spec ranks below all of them, as evidence of intent rather than of fact. That ordering is the whole lesson. A spec tells you what someone wanted. Only the file tells you what's there. The part I didn't expect I backtested the assessor against those nine repair commits as a labelled set. All five known-bad packets came back recut , each naming the same defect the human fix had found. The two I'd already repaired came back ready . Three packets that had shipped came back recut - for true reasons. And assessing A11 surfaced a live bug in merged code: a loader calling a snapshot_id property on a directory path, where the property reads a single file's bytes. That code had passed the linter, passed the tests, passed review, and merged. No test could have caught it. The dataset isn't in this checkout, so the offline suite never reaches the call. A green test suite told me that code was fine. A model reading the packet next to the code found it in one pass. Top comments (0)