I built an agent that asks your app what happened to it on SigNoz

I built an agent that asks your app what happened to it on SigNoz

Built for the Agents of SigNoz hackathon, July 2026.

When production breaks at 2am, the evidence already exists. Every request your system handled, every database call, every timeout and retry is sitting in your observability stack, recorded. Reading it is the problem. You need the query builder, a mental map of the architecture, and about forty minutes that the person on call doesn't have.

So I built Why Did It Break. It's an investigation agent on top of SigNoz. You ask a question the way you'd ask a colleague ("why is checkout slow?") and the agent runs live queries against SigNoz, follows leads like an engineer would, and comes back with the root cause, the evidence, a suggested fix, and how confident it is. Every claim links to the exact trace in SigNoz that proves it.

• Live demo: https://whatnow.up.railway.app
• Code: https://github.com/fozagtx/whatNow

The one rule: no mocks

Before writing any code I set one constraint. Nothing gets faked. No seeded demo data, no canned answers. If the agent can't support a claim from live telemetry, it has to say "I don't know" and drop its confidence. This one rule shaped the whole project, because every feature had to survive contact with real data instead of a happy-path script.

The demo system is HotROD, a multi-service ride-dispatch app that actually runs, under continuous real HTTP load. Its slowness is real (a MySQL SELECT that takes over a second under contention) and its failures are real (intermittent Redis timeouts in the driver lookup path).

Architecture

your question       SigNoz alert (webhook)
       |                     |
       v                     v
+-----------------------------------------------+
|                   RCA agent                    |
|   loops, choosing tools per step:             |
|     get_service_stats    per-service p99/err  |
|     compare_windows       now vs. before      |
|     get_slow_spans        slowest raw spans   |
|     get_error_spans       recent failures     |
|     search_spans          free-form drilldown |
|     get_logs              error text by trace |
+-----------------------------------------------+
                |
          every call = live POST /api/v5/query_range
                v
{ root_cause, evidence[] -> trace links, fix, confidence }

There's no hardcoded sequence of queries. The model gets six tools and picks its own path. Usually it starts with per-service stats, drills into the suspect's slowest spans, pulls one specific trace by id to confirm the parent-child timing, and stops when it has enough.

The UI prints the full tool-call trail under every answer so you can see which queries it chose. That trail is the trust mechanism… the agent has to show its work.

How SigNoz is used

SigNoz is the data plane, the trigger, and the proof. Install went through Foundry, the one-config-one-command path. casting.yaml and the generated casting.yaml.lock are in the repo. For the hosted demo I rebuilt the whole stack (ClickHouse, ClickHouse Keeper, Postgres, the otel-collector, the SigNoz server) on Railway by translating Foundry's generated compose manifests into Railway services with private networking. That pack is committed under deploy/railway/.

The v5 query_range API is the agent's entire toolset. All six tools hit POST /api/v5/query_range with a service-account key on the viewer role:

• raw builder queries over traces: slowest spans ordered by duration_nano, recent has_error = true spans with their status_message, and drill-downs with filter expressions like trace_id = '...' or service.name = 'x' AND duration_nano > 1000000000
• scalar aggregations: count() and p99(duration_nano) grouped by service.name, which give the agent its first read on who's slow and who's failing
• the same aggregations over an offset window, which powers "what changed in the last half hour?"
• the logs signal filtered by trace_id, so once the agent has a suspect it can read the actual error text

And then alerts. Asking questions is the reactive half. The proactive half is a trace-based alert rule (error count over a threshold on the Redis driver-lookup path) wired to a webhook notification channel that points at my app. When SigNoz fires, the agent investigates on its own and the result lands in an incident feed before anyone opens a dashboard.

This fired for real during the hackathon: live traffic pushed the error rate over the threshold, SigNoz's evaluator fired, the webhook delivered. The agent measured an 18.5% error rate against the alert's 10% threshold. It checked the alert's own claim against the raw spans.

Every evidence item deep-links to the trace view in SigNoz. The agent saying "the MySQL SELECT takes 1.19s inside the 1.6s dispatch" is one thing. Clicking through to the flame graph and seeing exactly that span hierarchy is what makes someone trust it.

What broke while I built it

The agent ran away. Early on, with SigNoz unreachable, my prompt said "adapt when tools fail." The model adapted, fifty calls in a row, burning real API credit on an investigation that could never succeed. I fixed it in layers: a hard request budget per investigation, a stop rule after two identical failures, and a preflight health check that answers instantly (and spends nothing) when the backend is down.

Empty log queries sent it fishing. When get_logs returned nothing, the agent retried it nine times with wider and wider windows, up to a 100,000-minute lookback. Prompt-level advice didn't stop it. What worked was making the tool itself return "this system ships no logs, do not call this again."

HotROD exported OTLP over HTTP to port 4317, which is the gRPC port. Every span vanished into "transport connection broken" errors while ClickHouse sat at zero rows. Fifteen minutes of staring at collector logs taught me more about OTLP than any tutorial: 4317 is gRPC, 4318 is HTTP, and the error message tells you which one you got wrong.

On deadline day I ran out of paid API credits entirely. I filtered the OpenRouter catalog for free models with genuine tool-calling support and switched the agent to a free 550B model. It investigates correctly but likes to cite trace ids in prose instead of the structured evidence array, so there's a backfill that pulls any trace id it mentions into a clickable evidence card.

Where it stands

The whole thing runs in the cloud now. SigNoz across five services, the demo app under continuous load, and the agent. Ask it "why are ride dispatches slow?" and it answers correctly in a handful of self-chosen queries: the dispatch call is dominated by a slow MySQL SELECT behind the customer service, plus Redis driver-lookup errors adding retries. Each claim has a trace link you can click and check.

The data was always there. Now anyone can ask it questions.