Your Agent Loop Is Lying About Being Done: Writing Verifiable Stop Conditions

TL;DR - A loop is only as trustworthy as its stop condition. "Done" has to be checkable by a different model - tests pass, lint clean, schema validates - never the worker's own claim. Below: why self-grading fails, the four rungs of verification, a checker-agent template you can copy, and the three caps that stop a loop running forever. Part 3 of the Loop Engineering series on ShipWithAI. Read the full article → Part 2 gave you the parts list. Part 1 promised this installment would be entirely about designing stop conditions, "because they are the hardest part to get right." Here it is. The thesis, in one line: A loop that grades its own work will tell you it is done before it is. Why self-grading fails A stop condition is a claim about reality. If the agent that wrote the code is the only thing asserting the claim, you don't have a verification - you have a self-report. The author's framing: a maker grading its own output is structurally optimistic, so it declares victory early. And this has a name - Anthropic's own writeup on building effective agents lists the failure mode explicitly: premature victory and fake-done features. The four rungs The rungs are an escalation of who checks: | Rung | Mechanism | What it adds over the rung below | |---|---|---| | 1 | Bare while ! npm test; do claude -p ...; done + iteration cap | A machine-checkable exit code as the gate | | 2 | Stop hook (exit 2 or {decision:"block"} ) | Blocks completion until the check passes | | 3 | /goal - a separate model grades each turn | A grader that is not the maker | | 4 | Scheduled goal runs (Routines) | The condition is re-checked unattended | Rung 1 checks with an exit code, rung 3 checks with a second model, and the gap between them is exactly the maker's optimism. Worth knowing the mechanics of rung 2 vs 3: a Stop hook blocks completion with exit 2 until a check passes, and Claude Code auto-overrides after 8 consecutive blocks. /goal (v2.1.139+) instead has a separate model - Haiku - grade the stop condition each turn. Rewriting a wish into a condition "Run until the feature is done" is not a stop condition, it is a wish. A stop condition is a command with an exit code. | Bad condition | Why it fails | Verifiable rewrite | |---|---|---| | "the feature is done" | Not falsifiable, no check | "all tests in test/auth pass AND lint clean" | | "code looks good" | Not verifiable by a machine | "passes: true for every story in prd.json " | | "the bug is fixed" | No boundary | "the new regression test exits 0 and no other test breaks" | The three-attribute test for any condition you write: - Verifiable - something can confirm it (a command, a count, a second model) - Falsifiable - it can fail, with a clear failing signal - Cheap - you can evaluate it every single turn That last one is the one people underrate - "cheap" here means literally you can run it every turn, not that it's nice to have. The checker agent Copy this. The whole design is in what it's forbidden from doing: --- name: stop-condition-checker description: Grades whether the loop's stop condition holds. Not the maker. model: haiku --- You verify, you do not fix. Run the project's check command and report only: - PASS if bash tests/run.sh exits 0 AND bash scripts/lint.sh exits 0 - FAIL otherwise, with the first failing line Never edit code. Never report PASS on the maker's say-so; run the command. The checker's value comes entirely from not being the maker. Different instructions, a cheaper model, and a clean context each turn are what make its PASS mean something. The article's actual prescription: a checker with different instructions, optionally a different model, and ideally a clean context each turn. The different model is an option, not the requirement - the requirement is not being the maker. Splitting maker from checker is called "the single highest-leverage move in loop design." One real trade-off worth knowing: the official ralph-wiggum plugin runs the checker in the same session as the maker, and the community has flagged that this deviates from fresh-context Ralph, where each lap starts clean. The article doesn't pick a side - same-session is cheaper; fresh-context is harder to fool. Pick deliberately. The hands-on run Read the disclaimer first, because it's the most useful part. The run did not use rung 3. /goal was the natural fit, but it's an interactive in-session grader and the author wanted a captured, reproducible run - so he used the rung-1 bare-loop equivalent: a fresh-context claude -p as the maker each turn, and the bash test exit code as the checker. An honest maker/checker split, graded by a machine-checkable artifact rather than by Haiku. He also notes token and dollar cost weren't instrumented, so he won't quote a number he doesn't have. The bug, in ShipWithAI's content-agent repo: scripts/check-draft-seo.sh compares every keyword word against only the title's first three words - so any keyword longer than three words gets flagged, even when it leads the title verbatim. The failing test: ⚠ keyword_placement: keyword "how to build an agent loop": not in title first 3 words FAIL: multi-word keyword that leads the title was wrongly flagged. exit=1 The stop condition - note that it guards against collateral damage, not just the target bug: bash tests/test-check-draft-seo-multiword-keyword.sh exits 0 AND bash scripts/test-rubric.sh still exits 0 The fix, one line of real logic: all_title_words = re.findall(r"[A-Za-z...]+", title.lower()) kw_words = re.findall(r"[A-Za-z...]+", kw_lower) title_words = all_title_words[:max(3, len(kw_words))] Result: one fresh-context maker turn. Pass on turn 1, cap of 8 never reached. He also ran a control draft to prove the fix isn't just a check that now always passes - worth copying as a habit. The loop stopped because a command exited 0, not because the agent felt finished. That is the only kind of "done" you can leave a loop alone with. Cap it three ways The stop condition says when to stop on success. The cap says when to stop anyway. A loop you trust has both. - Max iteration count - claude -p --max-turns N , plus the Stop hook's 8-block auto-override - No-progress detection - continuous-claude's --stall-threshold - Budget ceiling - --max-cost and--max-duration For lived precedent: ShipWithAI's own content-agent review loop caps at 3. The hands-on run above capped at 8. Pick per job, not by rule. Before you try this - A working harness and a minimal loop you can already run - Claude Code v2.1.139+ for /goal - A repo with a test command that exits non-zero on failure Try it this week Take the loop you built after Part 2 and ask one question: who asserts that it's done? If the answer is "the same agent that did the work," you haven't reached rung 1 yet. Write the condition as a shell command. Add a second agent whose only job is to run it. Then pick your three caps. This is a condensed summary. The full article walks each rung with working config, the complete hands-on trace, and the FAQ on /goal vs Stop hooks: 👉 Stop Conditions: Making "Done" Mean Something - Part 3, ShipWithAI Earlier: Part 1 - Why You Should Stop Prompting · Part 2 - Anatomy of a Loop. Part 4 - Memory Outside the Context Window - ships next. Top comments (0)