Debugging a failed agent run costs more than the run itself

Originally published on Loop & Retry - field notes on building LLM agents that survive production.

Here is the intuition to kill: a failed run is cheap because you can just run it again. A deterministic system earns you that assumption - the bug is sitting there, reproducible on demand, and debugging is a bounded search through code you control. An agent is not that system. When a run fails, the run that failed is gone. What you re-run is a different run that happens to share a prompt, and it may well succeed.

The token cost of the failure was never the expensive part. The expensive part is that you have to pay, over and over, to make the failure happen again in front of you. This post is about that second bill - the cost of reproduction - and why it dominates the cost of the actual fix. It’s the cost twin of the measuring-failure-in-production post: that one was about noticing silent failures; this one is about what it costs to understand one once you’ve noticed it. Together with the $200 postmortem, this is why the cost of a failure often has nothing to do with the failure itself and everything to do with what you do (or don’t) to recover from it.

The failed run is a crime scene with no recording

A traditional bug report comes with a stack trace, an input, and a promise: feed the input back in and you’ll see the trace again. That promise is what makes debugging tractable. You bisect, you add a log line, you re-run, you narrow. Every re-run is free information because every re-run is the same run.

An agent breaks that chain in two places. First, the model call is stochastic - same prompt, different sampled tokens, different tool calls, different path. Second, the world moved: the API the agent called returns different data now, the row it read got updated, the rate limiter is in a different state. So the failed trajectory isn’t a function of inputs you still have. It was a function of inputs plus two sources