Runaway agent loop: incident runbook
RB-01 ยท v0.1 ยท MAGSpace Runbooks ยท by Muhammad Arshad ยท CC BY 4.0
When to use this
- Token spend for one run, customer or agent is far above its normal rate.
- The same tool is called again and again with near-identical arguments.
- The agent keeps producing output but makes no progress toward its goal.
Public example: a 3-person B2B team reported that a customer's agent retried a failing tool 31,000 times between 1:12am and 6:50am. It spent $4,700 in tokens on a $400/month plan. They had a global rate limit and total-spend alerts, but nothing per customer. (r/SaaS, October 2026)
Source: https://www.reddit.com/r/SaaS/comments/1xh0cq5/an_agent_got_stuck_in_a_loop_and_burned_4700_of_a/
Severity
- SEV-1: spend is still rising, or the loop touched customer data or side-effecting tools.
- SEV-2: contained, but the cost is unbilled or someone has to absorb it.
- SEV-3: caught by existing caps; no customer impact.
1. Contain (first 15 minutes)
- Stop the run: kill switch, pause the queue, or disable that tenant's agent key.
- Freeze, don't delete: keep traces, prompts, tool responses and billing exports.
- Add a temporary hard cap per tenant in front of the model call.
- Write down every containment action with the time and who did it. Mark each one TEMPORARY.
2. Reconstruct: 11 questions
Label every answer VERIFIED (from logs or billing), PROPOSED (a hypothesis) or UNKNOWN. Unknowns stay unknown until evidence arrives.
| # | Step | Question |
|---|---|---|
| 1 | Deviation event | What exactly happened, and between which timestamps? |
| 2 | Root driver | What made the agent keep going? (error text, retry policy, missing budget) |
| 3 | Discovery | How was it found, by whom, and how long after it started? |
| 4 | Observation | What did the traces and the bill show? |
| 5 | Escalation | Who was told, and how fast? |
| 6 | Tacit knowledge | What did the on-call person know that isn't written down anywhere? |
| 7 | Alternatives and trade-offs | Which other fixes were considered, and why were they rejected? |
| 8 | Authorization | Who approved the containment and the fix? |
| 9 | Loss and impact | Tokens, money, customer trust, side effects. |
| 10 | Containment | What stopped it? Is that change temporary or permanent? |
| 11 | Prevention | Which control would have stopped it at step 3 instead of step 31,000? |
3. Usual causes
- Retries on errors that can never succeed (validation, auth, not found).
- Tool errors returned as plain text that the model reads as "try a slightly different prompt".
- No per-run budget for steps, tokens or wall-clock time.
- Spend checked only in total, not per tenant or per run.
- Loop detection that only catches exact repeats, while the agent varies its prompt slightly.
- A tool timeout that triggers a re-call while the first call is still running, duplicating side effects.
4. Permanent controls
- Per-run budget: max steps, max tokens and max wall-clock time, checked before every model call.
- Per-tenant budget: a soft alert at 1x plan and a hard stop at 2x (the fix the team in the example adopted).
- Retry policy by error class: retry only 429s, 5xx and timeouts, with backoff, at most 3 times.
- No-progress detector: the same tool with the same normalised arguments 3 times stops the run and escalates.
- Tool errors returned as structured observations with
retryable: false. - Idempotency keys on every side-effecting tool.
class Stop(Exception): ...
class Escalate(Exception): ...
def guard(run, tool, args, err=None):
if run.tokens > run.budget.tokens or run.steps > run.budget.steps:
raise Stop("run budget exceeded")
key = (tool, normalize(args))
run.seen[key] = run.seen.get(key, 0) + 1
if run.seen[key] >= 3:
raise Escalate(f"no progress: {tool} called 3x with the same arguments")
if err is not None and not err.retryable:
raise Escalate(f"{tool} failed with a non-retryable error: {err.code}")
5. Label the actions: automate, never automate, escalate
Score each action on four dimensions: safety and compliance, deterministic heuristic, reversibility, authorization. A fail on safety or reversibility rules out automation.
| Action | Label | Why |
|---|---|---|
| Stop a run that exceeds its budget | Automate | Deterministic and reversible |
| Retry a timeout or 5xx up to 3 times | Automate | Deterministic, bounded, reversible |
| Raise a tenant's budget | Escalate | Needs a person to authorise it |
| Credit or refund the customer | Escalate | Money and customer relationship |
| Delete data or rotate production keys to "fix" the loop | Never automate | Fails reversibility |
6. A workaround is not a fix
The emergency cap you added at 3am does not become permanent agent behaviour until it has been reviewed, tested and authorised. Track it like any other change.
7. Close-out checklist
- [ ] Spend stopped and confirmed in the provider's billing
- [ ] Traces and prompts preserved
- [ ] 11 questions answered and labelled VERIFIED / PROPOSED / UNKNOWN
- [ ] Per-run and per-tenant budgets in the code path, with tests
- [ ] Retry policy by error class
- [ ] No-progress detector, with a test that replays the incident
- [ ] Temporary workarounds reviewed and either removed or approved
- [ ] Customer told, by a person
Need a second pair of eyes? This runbook is published by MAGSpace. I'm Muhammad Arshad, its founder and former EVP Engineering at BITLOGIX. If an agent loop or bill spike hit you this month, I run a 48-hour AI bill / agent audit for $500. I reconstruct the incident from your own logs, rank the causes and hand you the fixes. Your data and code stay with you.
Book it by LinkedIn DM: https://www.linkedin.com/in/marshadofficial/
Comments
No comments yet. Start the discussion.