ur rag can return 200 and still be cooked - how we built Goose on SigNoz

The failure mode nobody puts on the dashboard

Normal API: wrong = 5xx or timeout. RAG chatbot: wrong = success arc. Retriever returns something, LLM writes a smooth answer, handler returns 200, span status: OK (fr) but asked category ≠ queried category. User asks a custom fact, pipeline silently pulls billing docs. Or tool output is empty/invalid and the model just freestyles from “general knowledge”. On HackRx that silent miss is the diff between “cool demo” and “i would never ship this”. We needed signals for is this answer actually grounded / routed right - not just is the process alive.

What Goose even is

Short version:

• Demo RAG emits OTel quality junk every turn
• SigNoz eats traces/metrics/logs and shows the cliff
• Alert when quality.score tanks
• Agent hits SigNoz MCP, pulls evidence, writes Markdown RCA
• Optional auto-remediate (POST /admin/fix-retriever for our injected break)

Track is AI & Agent Observability.

Demo flow we used: run ~100 custom-fact questions healthy (answers deliberately weird - “we live on Mars” - so you can tell it’s grounded in our Chroma store not Wikipedia). Then break retrieval. Same questions. Still 200s. Scores go brrr into the floor. Answers flip. SigNoz shows the cliff. Agent investigates + can fix.

How we actually used SigNoz

1. OTLP everything into one place

RAG app ships OTLP into SigNoz. Each chat turn ain’t just a request span - we slap quality attrs + metrics on it:

• quality.score - the main cliff metric
• quality.entity_match - asked category match retrieved?
• tool.output_valid - was retriever/tool output even usable
• gen_ai.* + zone attrs (asked / queried)
• Structured quality logs for the same mess

Silent fails are cross-signal by nature. Trace can look fine (OK) while the metrics panel shows quality.score falling like 1.0 → 0.2. In SigNoz we open the dashboard cliff then jump a span that still says OK but tool.output_valid=false. That combo is the whole thesis.

Drop your dashboard screenshot here - quality.score healthy then cliff.

2. Dashboard for the cliff, not vanity charts

We didn’t want another “cpu go up” board. Ours is basically:

• Latest / avg quality.score
• entity_match / misroute vibes
• tool.output_valid rate
• Traffic still healthy while quality dies (that’s the joke)

If volume stays up and quality dies, you know it’s not “the box is down”, it’s “the answers got dumb”.

Screenshot: healthy avg ~1.0 vs broken ~0.2.

3. Alert = answers got dumb, not cpu spicy

Alert: quality.score below 0.6. Channel = webhook into our agent on :8020/webhook/alert. This is the bit that made it feel real. SigNoz isn’t just a museum of charts. It’s the pager for quality. Same muscle memory as a classic outage page, except the outage is semantic.

Screenshot: alert rule / fired alert.

4. MCP agent as the on-call that reads telemetry

Alert fires → Goose doesn’t invent lore. It talks to SigNoz MCP, grabs recent quality evidence, dumps a Markdown RCA (what tanked, misroute hints like asked vs queried zones, what to do). For the hackathon remediator we call POST /admin/fix-retriever when the report / alert value smells like our injected misroute. IRL you’d swap that for reindex / model rollback / feature flag whatever - pattern stays: SigNoz evidence → agent decides → action.

Also being honest: auto-fix does not magically fix “we changed embeddings and everything drifted”. Our remediator is for the routing break we demo. Don’t overclaim that part, judges can smell it.

Screenshot: agent report Markdown.

E2E we actually ran (not vibes)

Healthy traffic → break → webhook → fix. Autoheal off on purpose so the agent had to be the one that healed it.

• Healthy (~5 min, 25 q) - avg score 1.0, retriever not broken
• Broken (5 q) - scores 0.2, tool.output_valid=false, answers drift to earth/general knowledge etc
• Agent - webhook → MCP investigate → POST /admin/fix-retriever OK
• Verify - Mars answer back, score 1.0, retriever healthy

This is literally the loop we wished we had during HackRx crunch. Not only “did the batch score drop in a csv” but “did prod quality just cliff and can something investigate without me clicking around for 40 mins”.

Instrumentation (the part you can copy)

Every turn we roughly do this then export OTLP:

span.set_attribute("quality.score", quality_score)
span.set_attribute("quality.entity_match", 1.0 if asked == queried else 0.0)
span.set_attribute("tool.output_valid", tool_output_valid)
span.set_attribute("quality.asked_zone", asked)
span.set_attribute("quality.queried_zone", queried)
quality_score_gauge.set(quality_score)

Point OTEL_EXPORTER_OTLP_ENDPOINT at SigNoz OTLP.

Gotcha we ate so you don’t have to

If you start the RAG before SigNoz is up, OTel can basically no-op / disable and then your dashboards look haunted (empty) even though the app “works”. Restart RAG after SigNoz is healthy. We wasted time on this. Not proud.

Another dumb one: localhost vs 127.0.0.1 on Mac - IPv6 being spicy - our e2e died with connection refused until we pinned 127.0.0.1. Tiny, but it’s the kind of detail you only get from actually running the loop at 3am.

What worked / what didn’t

Worked

• Quality metrics next to traces in one UI (SigNoz) = way less context switch brain death
• Alerting on quality.score feels correct. CPU alerts would’ve slept through this incident
• MCP for RCA is actually useful when the evidence is already in SigNoz, not in a random spreadsheet

Didn’t / don’t lie about it

• In-app autoheal can steal the demo if you forget to turn it off (we did, then fixed)
• One remediator ≠ every root cause. Embedding change ≠ misroute break
• Starting services in the wrong order will gaslight you into thinking instrumentation is broken

What I’d tell HackRx-me

Instrument semantic quality on day 1. Batch evals are cute. Production still needs a cliff on a graph.

Goose exists cuz top 30 at HackRx still left a hole - strong RAG, blind w