Every dashboard was green while my agent made things up. Here is how I debugged it.

A user asked our support agent how to reset two-factor auth, and it confidently walked them through steps that do not exist in our product. Made up, start to finish, but well-written and plausible.

I went to check what broke, and every dashboard was green. The request returned a 200. Latency was normal. The error rate had not moved. As far as our monitoring was concerned, nothing had happened at all.

That is the thing about agent bugs. The worst ones do not throw. They return a clean, confident, wrong answer, and your normal tools call that a success. Here is how I actually track these down now, without a single line of code in this post.

Why your normal monitoring cannot see this

A regular monitoring stack treats the whole agent as one HTTP call. It sees the request go in, the response come out, and a 200 in between. It has no idea that inside that one call the agent did a retrieval, made two model calls, and picked a tool.

So when the retrieval comes back empty and the model invents an answer, your dashboard still sees one successful request. The failure is real and completely invisible, because the layer that failed is a layer your monitoring never looked inside. To debug an agent you have to see inside the call, not just around it.

Step one: see the whole run as a tree

The first thing that changed everything was viewing each run as a tree of steps instead of a single event. Every step the agent took becomes its own row, nested under the step that called it:

• the agent run at the top,
• the retrieval underneath it,
• the model calls,
• each tool call,
• any guardrail or check.

Each row shows how long it took, whether it succeeded, and what went in and came out. Once you can see the run this way, "where did it go wrong" stops being a guess. You are reading the actual sequence of what happened instead of staring at the final answer trying to reverse-engineer it.

Step two: follow the failure up the chain

Here is the part that cracked my two-factor bug open. When a step fails deep in the run, you almost never see the error where it happened. You see a bad answer at the very top. So the move is to find the step that actually failed first and follow it upward.

In my case the chain read like this:

• The retrieval ran and came back empty. No documentation matched the question.
• The next step ran anyway, building a prompt with an empty context.
• The model, handed no real information, filled the gap by inventing an answer.
• The top of the run returned a clean 200, because technically nothing crashed.

Seen as a flat result, this looks like a random hallucination. Seen as a chain, the cause is obvious: an empty retrieval that nothing downstream checked for. The bug was never the model. It was the missing guard between an empty retrieval and a model call.

Step three: diff the bad run against a good one

Most agent regressions come from something changing: a prompt edit, a tool that now returns a slightly different shape, a retriever pulling different chunks after a reindex.

So the fastest way I have found to catch a regression is to line the failing run up next to a recent successful run for the same task and compare them step by step. The retrieved chunks, the prompt, the model response, the tool inputs and outputs, side by side.

Almost every time, one row is clearly different from the good run, and that row is your cause. It turns "something changed somewhere" into "this exact thing changed."

Step four: the fix is usually not code

The bug I described did not get fixed in the model or with some clever retry. It got fixed with two small changes:

• Make the step short-circuit when the retrieval is empty. If there is no documentation, the agent should say it could not find anything, not push an empty context into the model and hope.
• Add a check on that path that flags when an answer is not grounded in retrieved content, and let it fail the build in CI so the same hole cannot reopen quietly.

That is the pattern for most agent bugs I have hit. The failure feels like a model problem and the fix is almost always a missing guardrail or a prompt change. Very rarely is it the deep code fix your instinct reaches for first.

What I do differently now, in general

A handful of habits that turned agent debugging from an afternoon into a few minutes:

• Trace every tool and step, not just the entry point. A run that only records the top-level call hides exactly the failures you most need to see.
• Keep every failed run, and only sample the healthy ones. The failures are rare and precious. Do not let cost-saving sampling throw them away.
• Tag each run with the release, the feature flag, and the user journey. Diffing and grouping only work if that context is on the run.
• Score every run for quality, not just latency. A groundedness or instruction-following check is what catches the confident-but-wrong answers that a 200 hides.
• Skim the failures daily on anything high-traffic. The pattern shows up long before any user complains, if you are looking.
• Share the fix with whoever owns the prompts. Since most fixes are prompt or guard changes, the person who can actually apply them often is not you.

The lesson I keep relearning is that a green dashboard means the plumbing held, not that the agent did the right thing. The whole game is being able to look inside a single run and read what actually happened, step by step.

If you want a step-by-step version of this with the exact span setup and a worked example, this walkthrough is a good one. If you have chased a silent agent failure like this, I would love to hear what the root cause turned out to be. Mine is almost always something upstream returning empty and nothing checking for it.