I Scored Retry Loops. Boundedness Was Optional.
Boundedness Failed First
Boundedness failed first. Not the model. Not the schema. The ceiling. I stopped asking whether an agent was "actually thinking" and started asking a dumber question that still draws blood. Does this while ever stop if the tool hangs? That question is unromantic. It is also the one that keeps a shared box from turning into a heat lamp.
The Problem with Unbounded Loops
I wanted a number I could re-run on Monday, not a vibe from a chat transcript that already flattered me. Most agent retry code I read is a loop wearing a trench coat. You know the coat. try, except, sleep, continue, maybe a comment that says "be resilient." Resilience without a budget is just an unbounded bill. Did the author cap attempts? Did they cap wall time? Or did they only cap their optimism?
Building a Scorer
I built a scorer. Not a leaderboard. A lint with opinions. It reads Python, walks the AST, and grades a retry helper the way I wish code review would: boundedness first, poetry never. The artifact is the point. If you strip every product name out of this article, you should still be able to save the files and get the same integers.
The Scorer's Rules
Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a candidate loop I do not want to hand-write at 11pm, I ask a free model through MonkeyCode. Then I drop the output into fixtures/ and refuse to trust it until the scorer prints a score. The free server option is not a mascot for that step. It is the second half of the experiment, the half where time.sleep meets scheduling noise.
The Rubric
My laptop is a terrible witness. It is too fast, and it likes me. The rubric I actually encode I do not grade "quality of reasoning." I grade whether a stop condition exists in the artifact you were about to deploy. Those are different papers. One of them is a keynote. The other one is a unit test. A retry helper starts at zero. I add points for an attempt cap, a wall-clock timeout, backoff, jitter, and an idempotency key traveling with the request. I subtract points for while True with no break budget, and for sleep that cannot see a deadline.
The Code
#!/usr/bin/env python3
""" Score retry-loop hygiene from Python source. Fixture-calibrated, not a bake-off. """
from __future__ import annotations
import ast
import json
import sys
from dataclasses import asdict, dataclass, field
from pathlib import Path
ATTEMPT_NAMES = {"max_attempts", "max_retries", "retries", "attempts", "n_tries"}
TIMEOUT_NAMES = {"timeout", "deadline", "max_seconds", "wall_timeout", "budget_s"}
BACKOFF_NAMES = {"backoff", "backoff_s", "delay", "base_delay"}
JITTER_NAMES = {"jitter", "jitter_s", "jitter_ratio"}
IDEM_NAMES = {"idempotency_key", "idempotency", "request_id", "dedupe_key"}
@dataclass
class LoopScore:
path: str
has_max_attempts: bool = False
has_timeout: bool = False
has_backoff: bool = False
has_jitter: bool = False
has_idempotency: bool = False
unbounded_while: bool = False
sleep_without_budget: bool = False
score: int = 0
notes: list[str] = field(default_factory=list)
class HygieneVisitor(ast.NodeVisitor):
def __init__(self) -> None:
self.names: set[str] = set()
self.unbounded_while = False
self.sleep_calls = 0
self.breaks_in_loop = 0
def visit_Name(self, node: ast.Name) -> None:
self.names.add(node.id)
self.generic_visit(node)
def visit_While(self, node: ast.While) -> None:
constant_true = (
isinstance(node.test, ast.Constant) and node.test.value is True
) or (
isinstance(node.test, ast.Constant) and node.test.value == 1
)
if constant_true:
self.unbounded_while = True
for child in ast.walk(node):
if isinstance(child, ast.Break):
self.breaks_in_loop += 1
self.generic_visit(node)
def visit_Call(self, node: ast.Call) -> None:
func = node.func
name = ""
if isinstance(func, ast.Attribute):
name = func.attr
elif isinstance(func, ast.Name):
name = func.id
if name in {"sleep", "usleep"}:
self.sleep_calls += 1
self.generic_visit(node)
def score_source(path: Path, src: str) -> LoopScore:
tree = ast.parse(src)
v = HygieneVisitor()
v.visit(tree)
row = LoopScore(path=str(path))
row.has_max_attempts = bool(ATTEMPT_NAMES & v.names)
row.has_timeout = bool(TIMEOUT_NAMES & v.names)
row.has_backoff = bool(BACKOFF_NAMES & v.names)
row.has_jitter = bool(JITTER_NAMES & v.names)
row.has_idempotency = bool(IDEM_NAMES & v.names)
row.unbounded_while = v.unbounded_while and v.breaks_in_loop == 0
budget = row.has_max_attempts or row.has_timeout
row.sleep_without_budget = v.sleep_calls > 0 and not budget
n = 0
if row.has_max_attempts:
n += 2
row.notes.append("+2 attempt cap")
if row.has_timeout:
n += 2
row.notes.append("+2 wall clock")
if row.has_backoff:
n += 1
row.notes.append("+1 backoff")
if row.has_jitter:
n += 1
row.notes.append("+1 jitter")
if row.has_idempotency:
n += 2
row.notes.append("+2 idempotency key")
if row.unbounded_while:
n -= 3
row.notes.append("-3 unbounded while")
if row.sleep_without_budget:
n -= 2
row.notes.append("-2 sleep with no ceiling")
row.score = n
return row
def main(argv: list[str]) -> int:
root = Path(argv[1] if len(argv) > 1 else "fixtures")
rows = []
for path in sorted(root.glob("*.py")):
rows.append(score_source(path, path.read_text(encoding="utf-8")))
print(json.dumps([asdict(r) for r in rows], indent=2))
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
Running the Scorer
Run it like a test, not like a demo.
mkdir -p fixtures
python3 loop_hygiene.py fixtures/
If that command prints nothing, you have an empty folder, not a passing grade. Empty is not resilient either.
Fixtures
Four fixtures, four numbers I can defend. I calibrated the scorer on four files I wrote on purpose. This is not a claim about any named model. It is a claim that the ruler does not flop when I bend it.
Two fixtures are the "looks fine in a PR" loops a chat window emits when you say make it robust. Two are loops I would actually leave under a cron that can page me.
fixtures/a_trench_coat.py
import time
def call_tool(url):
while True:
try:
return fetch(url)
except Exception:
time.sleep(1)
fixtures/b_range_but_naked.py
import time
def call_tool(url):
max_attempts = 5
for _ in range(max_attempts):
try:
return fetch(url)
except Exception:
time.sleep(0.5)
raise RuntimeError("gave up")
fixtures/c_budget.py
import random
import time
def call_tool(url, timeout=8.0):
max_attempts = 5
backoff = 0.2
deadline = time.monotonic() + timeout
for attempt in range(max_attempts):
if time.monotonic() >= deadline:
raise TimeoutError("wall clock")
try:
return fetch(url)
except Exception:
jitter = random.random() * 0.05
sleep_for = min(backoff, max(0.0, deadline - time.monotonic()))
time.sleep(sleep_for + jitter)
backoff *= 2
raise RuntimeError("attempts exhausted")
fixtures/d_idempotent.py
import random
import time
import uuid
def call_tool(url, timeout=8.0, idempotency_key=None):
max_attempts = 5
backoff = 0.2
jitter = 0.05
deadline = time.monotonic() + timeout
key = idempotency_key or str(uuid.uuid4())
for attempt in range(max_attempts):
if time.monotonic() >= deadline:
raise TimeoutError("wall clock")
try:
return fetch(url, headers={"Idempotency-Key": key})
except Exception:
sleep_for = min(backoff, max(0.0, deadline - time.monotonic()))
time.sleep(sleep_for + random.random() * jitter)
backoff *= 2
raise RuntimeError("attempts exhausted")
Dynamic Probe
The half my laptop keeps lying about. Static scores catch missing ceilings. They do not catch a ceiling that exists on paper and then loses a fight with real delay. So I added a dynamic probe. It is deliberately ugly. A mock tool returns after 0.25s, then 1s, then 3s, then it never returns.
#!/usr/bin/env python3
""" Dynamic probe. Label: run this; do not treat my laptop timings as yours. """
import threading
import time
from http.server import BaseHTTPRequestHandler, HTTPServer
DELAYS = [0.25, 1.0, 3.0, 999.0]
HITS = {"n": 0}
class SlowTool(BaseHTTPRequestHandler):
def do_GET(self):
i = min(HITS["n"], len(DELAYS) - 1)
HITS["n"] += 1
time.sleep(DELAYS[i])
self.send_response(200)
self.end_headers()
self.wfile.write(b"ok")
def log_message(self, fmt, *args):
return
def main() -> None:
server = HTTPServer(("127.0.0.1", 8765), SlowTool)
t = threading.Thread(target=server.serve_forever, daemon=True)
t.start()
deadline = time.monotonic() + 10.0
# Import YOUR candidate here and call it against http://127.0.0.1:8765/
# If this process is still alive past deadline, the loop has no ceiling.
while time.monotonic() < deadline:
time.sleep(0.05)
server.shutdown()
print({"hits": HITS["n"], "exited_before_watchdog": True})
if __name__ == "__main__":
main()
What This Does Not Measure
This does not measure whether a model is "better at coding than most developers." I have no idea how you would even sample that sentence without lying to yourself. It does not measure MCP servers, community knowledge bases, or whether agents are secretly if-statements. Plenty of if-statements are honest. The dishonest ones are the loops that refuse to be if-statements with a counter.
Who Should Skip It
Anyone shipping billing, healthcare, or a retry around a non-idempotent charge. A hygiene score of 8 is not a PCI audit. Also skip it if you will not read the generated loop. A scorer you ignore is just another dashboard, and dashboards do not page you until the invoice does.
Comments
No comments yet. Start the discussion.