I Gave My AI Agent Pipeline a Nervous System with SigNoz

Why I'm writing this before the hackathon even starts

I'm building an AI-powered platform to help patients navigate rare and undiagnosed disease workups - routing them toward the right specialists and India's government-designated Centres of Excellence instead of years of misdiagnosis. The architecture has been frozen for a while now: a LangGraph-based multi-agent system with twelve components - an Orchestrator, a Symptom Intake Agent, an HPO Extraction Agent, a Clinical Triage Engine, a RAG Retrieval Engine, a Differential Reasoning Agent, a CoE Navigation Agent, and a few more, all passing state to each other, all calling out to LLM APIs, a vector store, and Postgres.

Here's the uncomfortable truth I ran into while finishing the documentation: I could describe every agent's responsibility in detail, in a Word document, with data flow diagrams. What I couldn't tell you was how long the HPO Extraction Agent actually takes when the RAG Retrieval Engine is slow, or whether a failed differential reasoning call fails loudly or just silently returns an empty list that the next agent quietly accepts.

That gap - between "I designed this system" and "I can see this system while it runs" - is exactly what the WeMakeDevs × SigNoz warm-up challenge is built around. So before the actual hackathon sprint (Track 01, AI & Agent Observability, is the obvious fit for my project), I self-hosted SigNoz and wired up an instrumented slice of my pipeline to see what "flying blind," a phrase the hackathon page uses almost accusingly, actually feels like - and what the alternative feels like.

Scope note: this post covers a representative two-service slice of the full pipeline (an API gateway that mimics the Orchestrator, plus one downstream agent), not the entire twelve-component system. The goal here is to genuinely understand SigNoz's traces/metrics/logs/dashboards/alerts before the real build, not to pretend I shipped the whole platform in a weekend.

The problem: my architecture was frozen, my visibility wasn't

Multi-agent systems have a specific failure mode that single-service apps don't: the bug is rarely in the code you're looking at. If the Differential Reasoning Agent returns a bad answer, is it because:

• the RAG Retrieval Engine handed it thin context,
• the HPO Extraction Agent mis-parsed a symptom,
• the LLM call itself timed out and something upstream swallowed the exception,
• or the Orchestrator passed stale state from a previous turn?

With print() statements and scattered log files, answering that means grepping through however many terminals you have open, lining up timestamps by eye, and hoping the clocks agree. That's not a debugging process, it's archaeology. And it gets exponentially worse the moment you add a second agent, a database, and an external API - which describes almost every serious AI system being built for this hackathon.

Logs, metrics, traces - and why you actually need all three

It's tempting to think of these as three flavors of the same thing. They're not; they answer different questions, and observability tools only get powerful when they can correlate across all three.

• Logs are discrete, timestamped events with (usually) unstructured or semi-structured text - "started processing symptom set for session abc123". Great for the what happened, terrible for the how does this relate to everything else that happened.
• Metrics are aggregated numbers over time - request rate, p99 latency, error percentage. Great for spotting that something's wrong (a graph spikes at 3:14pm), useless on their own for explaining why.
• Traces are the connective tissue: a single request's journey across every service, function, and network hop it touches, represented as a tree of timed spans. A trace is the only one of the three that can answer "which specific step, in which specific request, took 4 seconds and why."

For a multi-agent pipeline specifically, traces matter more than they do in a typical CRUD app, because the "request" isn't one HTTP call - it's a chain of LLM calls, tool invocations, and retrieval hops, and the only way to see that chain as one thing is distributed tracing with correctly propagated context.

Why SigNoz, specifically

There are plenty of observability tools. What made SigNoz the right fit for this specific problem:

• It's OpenTelemetry-native, not "OTel-compatible" as an afterthought. Since OpenTelemetry is already the closest thing to a standard for instrumenting LLM and agent workflows, I'm not locking my instrumentation code to a proprietary SDK I'd have to rip out later.
• It unifies traces, metrics, and logs in one place, with correlation between them - click a span in a trace, jump straight to the logs emitted during that span.
• It's self-hostable and open source, which matters a lot for a healthcare-adjacent project where I want to control where telemetry data lives, not ship patient-adjacent request metadata to a third-party SaaS by default.
• It's the sponsor's own platform for the AI-agent tracks in this hackathon, so the muscle memory I build now (Query Builder, alert rules, dashboard panels) carries directly into the real submission.

Setting up SigNoz: what the docs don't warn you about

SigNoz's self-host install has changed recently - if you've seen an older tutorial reference a docker-compose up command straight out of the SigNoz repo's deploy/ folder, that path has been deprecated in a recent release. The currently supported installer is Foundry, a declarative CLI that generates and manages the Compose (or Kubernetes) files for you.

The actual flow is refreshingly small:

# 1. Install the Foundry CLI
curl -fsSL https://signoz.io/foundry.sh | bash

# 2. Declare what you want: Docker Compose, single machine
cat > casting.yaml << 'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

# 3. Deploy
foundryctl cast -f casting.yaml

cast isn't one opaque command - it chains three stages (gauge validates your Docker setup, forge renders the actual Compose files into a pours/ directory, cast brings the stack up), and you can run each stage separately if you want to inspect the generated Compose file before trusting it with docker compose up. That separation is a nice bit of design: I could see exactly what containers were about to start (ClickHouse for storage, a Postgres-backed metastore, a ClickHouse Keeper node, the OTel collector, and the SigNoz UI/API server itself) before committing.

Prerequisite check: Docker Engine 20.10+, the Compose v2 plugin, and at least 4GB of memory allocated to Docker. If containers keep restarting, that memory ceiling is the first thing to raise - SigNoz's docs call this out explicitly, and ClickHouse in particular is memory-hungry on a cold start.

Once docker ps shows everything healthy, the UI comes up at http://localhost:8080. Ports 4317 (OTLP gRPC) and 4318 (OTLP HTTP) are what your application will actually talk to. Although the deployment itself was straightforward, the most useful confirmation came from seeing the services come up cleanly and the SigNoz UI become reachable locally. That moment confirmed the observability stack was ready to receive telemetry from my application.

Instrumenting a real agent, not a toy Flask app

Rather than spinning up a "hello world" Flask app just to say I instrumented something, I stood up a stripped-down FastAPI service that mirrors the shape of my real Symptom Intake Agent: it receives a symptom payload, calls out to a downstream "extraction" service (standing in for the HPO Extraction Agent), and returns a structured response. Two services, one synchronous hop between them - small enough to reason about, real enough to show actual cross-service tracing instead of a single-span demo.

SigNoz's Python guide recommends opentelemetry-distro for zero-code auto-instrumentation, which detects your installed libraries and wraps them automatically:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install

That second command is doing more than it looks like - it scans your installed packages (FastAPI, requests, whatever database driver you're using) and installs the matching OpenTelemetry instrumentation library for each one. Run it after your app's dependencies are installed, or it has nothing to detect.

Environment variables point the SDK at self-hosted SigNoz instead of the cloud ingestion endpoint:

export OTEL_RESOURCE_ATTRIBUTES="service.name=symptom-intake-agent,service.version=$(git rev-parse --short HEAD)"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_METRICS_EXPORTER="none"  # traces first, metrics deliberately off for now

Callout - why disable metrics here: opentelemetry-distro enables metrics by default, and the auto-instrumentation for HTTP clients like requests and httpx will start emitting a duration histogram for every outgoing call the moment metrics are on. For a chatty agent pipeline making several LLM and retrieval calls per request, that's a lot of unplanned data volume before you've even decided you want it. Turning it off at first keeps the traces experiment clean; flipping it to otlp later is a one-line change.

Then the run command changes from uvicorn main:app to:

opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000

The gotcha that would have cost me an hour if I hadn't read the docs first: opentelemetry-instrument does not support Uvicorn's --workers flag, and --reload outright breaks instrumentation for any framework, not just Uvicorn. The reason is unglamorous but important - the OpenTelemetry SDK's batch span processor runs on a background thread, and Python's fork model doesn't safely carry that thread into forked worker processes unless the server explicitly re-initializes it after fork. Uvicorn's worker mode doesn't do that; Gunicorn (loading the app fresh in each forked worker) does.

So for multi-worker production instrumentation, the documented pattern is Gunicorn with Uvicorn workers:

opentelemetry-instrument gunicorn -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:8000

For a single-worker dev instance, plain opentelemetry-instrument uvicorn without --reload is fine, and that's what I used for this warm-up.

One lesson I took away during setup is that instrumentation deserves the same attention as application code. A small configuration mistake can prevent telemetry from appearing, so validating the OpenTelemetry configuration early saves significant debugging time later.

Following one request through the pipeline

To make the tracing story concrete, I sent a single synthetic symptom payload through the two-service slice - a POST /intake call carrying a small set of HPO-style symptom terms - and watched it end to end instead of looking at aggregate graphs first.

Conceptually, the request's journey looks like this:

Client
  │
  ▼
symptom-intake-agent (FastAPI, span: POST /intake)
  │  validates payload
  │  calls downstream extraction service (span: HTTP POST /extract)
  ▼
extraction-service (FastAPI, span: POST /extract)
  │  normalizes symptom terms
  │  returns structured HPO-like terms
  ▲
symptom-intake-agent (assembles response)
  │
  ▼
Client

Because both services are auto-instrumented with OpenTelemetry and both export to the same SigNoz collector, the HTTP call between them isn't two disconnected traces - it's one trace with a parent span in symptom-intake-agent and a child span in extraction-service, stitched together by W3C trace-context propagation happening automatically inside the instrumented requests/httpx client and the receiving FastAPI middleware. That stitching is the entire point of distributed tracing, and it's also the thing that's genuinely hard to bolt on after the fact with hand-rolled logging.

Traces: the flamegraph that explained my own architecture back to me

Opening the Traces tab in SigNoz and drilling into that request shows a flamegraph / Gantt-style view: the parent span for POST /intake as a wide bar at the top, and the nested POST /extract call as a shorter bar underneath it, positioned exactly where in time it started and ended relative to the parent.

This is a small example, but it points at something I wouldn't have appreciated without seeing it laid out visually: in a chain of agents, latency is additive and attribution is not. If the whole request takes 900ms and the extraction call alone accounts for 700 of them, that's not a fact you can eyeball from logs with two separate timestamps in two separate files - you'd have to manually subtract wall-clock times and hope your service clocks are synced. In the trace view it's just... the size of the bar.

For the real twelve-agent pipeline, this is the difference between "the whole request is slow" (unhelpful) and "the RAG Retrieval Engine is 60% of total latency on differential-diagnosis-heavy queries" (actionable, and exactly the kind of finding that should shape which agent gets optimized first).

Watching the request appear as a complete parent–child trace reinforced the value of distributed tracing. Rather than viewing isolated events, I could understand the entire request lifecycle as one connected flow.