Evidence First, Answer Second: Building an Observable Industrial AI Agent with SigNoz

Evidence First, Answer Second: Building an Observable Industrial AI Agent with SigNoz

Most AI systems are designed to give an answer. That is useful in a chatbot. On a factory floor, it can be dangerous. While building Industrial IoT Anomaly Control, I kept coming back to one question: What should an AI agent do when it detects a real problem but does not have enough evidence to explain the cause safely?

My answer was simple: it should stop, show what it knows, and send the case to a human. This project is a real-time monitoring system for a simulated water-treatment plant. It streams live sensor data from six industrial assets, detects unusual behaviour, searches a knowledge base for similar incidents, and then chooses between two paths:

• recommend a safe action when the evidence is strong;
• escalate for human review when the evidence is weak.

SigNoz is what makes this decision process visible. Instead of seeing only the final AI response, I can inspect the full path from the incoming sensor reading to anomaly detection, knowledge retrieval, policy checks, agent explanation, and recovery.

The problem with traditional alarms

Factories already collect large amounts of telemetry such as vibration, temperature, humidity, sequence numbers, and timestamps. The problem is not missing data. The problem is turning that data into a useful decision. A traditional threshold alarm may say: Vibration is above the configured limit. That still leaves the operator with several questions:

• Is the machine actually failing?
• Is the sensor or gateway sending bad data?
• Has this pattern happened before?
• Is there enough evidence to recommend maintenance?
• Why did the AI reach this conclusion?

Repeated threshold alerts can also create alarm fatigue. If one fault produces dozens of alerts, operators may start treating them as noise. I wanted the system to create one investigation instead of another flood of alarms.

What I built

The demo represents six assets in a water-treatment plant. A TCP simulator can run in healthy mode or a seeded faulty mode that creates repeatable equipment and data-quality problems. The system detects:

• sudden vibration spikes;
• gradual vibration drift;
• missing or intermittent readings;
• duplicate events;
• sequence gaps;
• stale or reversed timestamps.

One design decision became very important: bad telemetry is not the same as a broken machine. Duplicate events and sequence gaps become data-quality incidents and can never turn into mechanical recommendations. Related abnormal readings are grouped into one investigation, avoiding a new alert for every reading.

How the agent uses knowledge

When the system detects an equipment-condition incident, it searches a Chroma knowledge base containing structured water-treatment scenarios. The search is filtered before similarity matching. Equipment type, sensor type, and incident category are used to prevent unrelated incidents from becoming evidence. For example, a vibration problem on a centrifugal pump should not be explained using a humidity issue from a different asset.

The retrieval step returns the strongest matches, similarity scores, verification status, and the gap between the first and second result. This is where my project differs from a basic RAG application. A traditional RAG flow often looks like this:

Question → Retrieve documents → Generate an answer

My flow is closer to:

Anomaly → Collect evidence → Retrieve incidents → Check confidence → Recommend or escalate → Explain the approved result

The agent is not allowed to treat every retrieved document as enough evidence.

The LLM does not own the safety decision

I did not ask the model to generate a confidence number. A language model can return “95% confident,” but that number may not represent a tested probability. For an industrial use case, that is not a safe decision rule. Instead, the application calculates confidence from measurable signals:

• anomaly strength and persistence;
• agreement between detectors;
• current data quality;
• similarity of the best retrieved incident;
• distance between the first and second matches;
• whether the precedent was verified;
• configured recommendation and escalation thresholds.

The main safety rule is:

assert not ( decision == "RECOMMEND" and confidence < configured_threshold )

If the evidence passes policy, the system recommends a bounded inspection action. If the match is weak, unclear, unverified, or missing, the decision becomes ESCALATE. The LLM only explains the result. It cannot override policy or invent evidence. If the provider is unavailable, a deterministic fallback answers while sensor processing continues. That separation was one of my biggest lessons from the project: The application should own safety. The model should help communicate it.

Making the full decision visible with SigNoz

Adding observability meant moving beyond “the API is up” and “the model returned 200.” I wanted to observe the quality of the agent’s decision. The application emits OpenTelemetry traces, metrics, and structured incident logs through OTLP HTTP, keeping monitoring separate from business logic.

A normal reading creates a root span named: sensor.process

An anomalous investigation can include:

sensor.process
├── detectors.evaluate
├── incident.evaluate
│   └── policy.evaluate
├── knowledge.enrich_incident
│   ├── knowledge.retrieve
│   └── policy.re_evaluate
├── agent.explain
└── agent.monitor_recovery

The first policy decision is created immediately. Retrieval and explanation run in the background, because a slow embedding or LLM call must never pause the live stream. The trace connects details such as: equipment and sensor identity; incident ID and category; detector names and anomaly severity; retrieved precedent ID; final policy decision; whether the model or fallback created the explanation; recovery progress. Raw measurements, prompts, keys, and repair notes are deliberately excluded from telemetry.

The dashboard that changed how I understood the agent

I created a SigNoz dashboard called Water Treatment Agent - Trust & Recovery. The most useful panels are:

• telemetry throughput by equipment;
• detector activity;
• recommendation and escalation counts;
• knowledge retrieval outcomes;
• agent assessment latency;
• automatic incident resolutions;
• incident recovery time.

One real issue I hit was using the wrong metric reduction for automatic resolutions. A rate on a total-count panel produced an empty or confusing result. Using an increase across the selected range and displaying the sum fixed it. I learned this by watching real telemetry, not by only reading documentation.

I also grouped agent latency by both the attempted mode and final mode. That lets me distinguish:

• Mistral attempted and Mistral completed;
• Mistral attempted but deterministic fallback completed;
• deterministic mode used directly.

A fallback is therefore a visible operational signal, not a hidden log line.

Following an incident from detection to recovery

The operator dashboard stores the W3C trace ID with each incident. From the active incident, the operator can click Open trace in SigNoz and jump directly to the full investigation. This connects the product directly to observability:

1. Operator sees incident
2. Opens exact trace
3. Checks detector evidence
4. Checks retrieved knowledge
5. Checks policy decision
6. Checks model or fallback latency
7. Watches recovery

The agent does not stop after generating an explanation. It keeps watching the asset. An equipment incident is automatically resolved only after five consecutive healthy readings. The recovery span records the current healthy count and whether the incident was safely closed. Operators can record the real finding and fix. It becomes clearly labelled local knowledge without replacing verified evidence.

What I learned

The biggest lesson was that agent observability is not only token counts and model latency. Those metrics do not answer the most important question: Why did the system believe it had enough evidence to recommend an action? Meaningful observability required connecting:

• raw event handling;
• detector output;
• incident state;
• retrieval quality;
• policy rules;
• model behaviour;
• fallback behaviour;
• recovery.

Abstention should also be visible. Escalation is not a failed response; it is often the safer outcome. SigNoz shows how often the agent abstains and whether retrieval or data quality caused it. I also kept the live path independent from slow AI work. The safe detector-only result reaches the operator first, then retrieval and explanation enrich it.

Conclusion

Industrial IoT Anomaly Control is not designed to always sound confident. It is designed to build an evidence chain, follow clear rules, and stop when that evidence is not strong enough. SigNoz turns that evidence chain into something an operator or engineer can inspect-from the first unusual reading to the final decision and recovery.

The principle behind the project is simple: Evidence first. Answer second.

Project links

• GitHub: https://github.com/dawgdevv/AI-Powered-Industrial-Anomaly-Detection
• Demo video: https://www.youtube.com/watch?v=uuN28q6kNXw

References

• SigNoz self-hosted installation: https://signoz.io/docs/install/docker/
• SigNoz OpenTelemetry Python instrumentation: https://signoz.io/docs/instrumentation/opentelemetry-python/
• OpenTelemetry documentation: https://opentelemetry.io/docs/