I Thought Building Agent Observability Was a Detector Problem. I Was Wrong.

When I started building agent-exec-trace , I thought I knew where the difficulty would be. OSS agent-exec-trace is an observability layer for AI agents. It instruments agent runs using OpenTelemetry-style traces, then analyzes those traces for loops, retry storms, cost spikes, hallucination patterns, and other bad runtime behavior. The goal is simple: when an agent run feels wrong, I want something better than logs and intuition to explain why. I thought the hard part would be the detector logic. Define the anomalies. Pick the thresholds. Wire the traces. Run the analytics. Ship the repo. That was the clean version of the story. The real version was uglier. On my first large pass over 100,000 real agent traces from Hugging Face, my empty_response detector fired on 100% of them. Every trace. At the same time, 28 of my 35 rule-based detectors never fired at all. That is the moment the project stopped being "build some detectors" and turned into a learning experience. The detectors were not wrong. The traces were the wrong shape. And once I saw that, a bunch of other assumptions broke with it. I had a very similar feeling when I shipped my agent eval harness. Different tool. Same pattern. Real systems are very good at ruining tidy design stories. What I Was Actually Trying To Build The problem I cared about felt pretty obvious. Traditional observability can tell me if a service is up, slow, or failing. It cannot tell me why an agent called the same tool eight times, changed its mind three times, burned tokens on a dead-end path, or quietly drifted into behavior I would never want repeated in production. That gap matters more than people admit. Most teams can tell you the run felt wrong. Very few can point to the exact step where it started going wrong. So I built agent-exec-trace : an OpenTelemetry-style observability layer for agent behavior. The idea is simple enough to explain. Instrument the run. Capture the behavioral path. Emit spans for planning, tool calls, retrieval, memory, approvals, and cost. Then run detectors over those traces and surface the bad runs in a UI someone can actually use. The repo is here: github.com/deghosal-2026/agent-exec-trace It ships as three PyPI packages, has 794 Python tests and 34 Playwright end-to-end tests, and the SDK is deliberately small enough that someone can wrap a real agent without redesigning their system. from agent_exec_trace import AgentTracer, trace_agent, tool_span AgentTracer.setup(otlp_endpoint="http://localhost:4317") @trace_agent(agent_name="my-agent", agent_version="1.0.0") async def handle(query: str) -> str: with tool_span("search", tool_args={"q": query}): return await search(query) That part was not the lesson. The lesson was what happened after I had something real enough to test. The First Thing I Learned: Real Traces Fight Back I did plan field testing. Just not early enough. That distinction matters, because the mistake was not forgetting it. The mistake was scheduling it late enough that the rest of the system had already started to feel done. The WBS was solid. The PRD was thorough. The detector catalog was documented. The milestones were real. The quality gates were strict: ruff clean, mypy strict clean, tests green, coverage above 90%. All of that was good discipline. None of it protected me from the fact that mock agents are polite and real traces are not. That is what "too late" looked like in practice. By the time field testing showed up, parts of the detector catalog, the assumptions about trace shape, and even some milestone gates already felt validated. Field testing did not arrive early enough to shape the design. It arrived late enough to expose where the design had been validated against the wrong reality. The Hugging Face corpus was the first slap. My detector expected response content in one place. The corpus stored it in many different shapes. So empty_response was not telling me "all these agents are broken." It was telling me "your assumptions about trace shape are too narrow." That one bug forced four normalization passes. Different response keys. Different tool-name conventions. Different operation names. Timestamp parsing issues. Parent-child inconsistencies. Bad validation data does not just create noise. It teaches you the wrong lessons with confidence. By the end of that cleanup, the headline got more honest and less comforting. The original compatibility number looked decent. The real one was 42.4%. That was the moment I stopped thinking about observability as a detector problem and started thinking about it as a data-shape problem. Because a detector that is perfectly correct in theory is still useless if the data it depends on almost never exists in the wild. The Second Thing I Learned: Synthetic Data Can Save You And Mislead You Once the real-trace corpus showed me its limits, I went in the other direction. I built a synthetic trace generator. One million traces. Ten fake agents. Fourteen tools. Deliberate behavior modes: loops, retries, timeouts, inactivity gaps, intervention waits, token explosions, memory bursts. That solved one problem immediately. Structural compatibility jumped to 99.2%. Now the detectors had something they could actually see. Twenty of the 35 rule-based detectors fired. That sounds like a win. And it was. But synthetic data lies in a different way. My hallucination detector fired on 98% of synthetic traces. Not because I had accidentally built the greatest detector in history, but because the synthetic outputs and the synthetic tool evidence had a fake relationship that made the detector's job too easy. The same thing happened with cost. My cost_spike detector had a real threshold. My synthetic generator produced mostly cents-level costs. So the detector almost never fired there either. Synthetic traces can prove a detector runs. They cannot prove it matters. That gave me a second lesson I needed badly: Synthetic traces are great for proving a detector can run. They are not enough to prove a detector is calibrated for reality. So now I think of validation in layers. Unit tests tell me the logic works. Synthetic traces tell me the detector can see the fields it needs. Real traces tell me whether any of this matters outside my own sandbox. That sequencing sounds obvious when written down. It did not feel obvious when I was in the middle of building it. The Third Thing I Learned: Green Gates Can Lie This one annoyed me more than any detector bug. I had marked the OTLP export milestone done. The gate was green. The docs looked fine. The demo looked fine. And the end-to-end export path had never actually been verified. Two bugs had cancelled each other out. The OTel collector's gRPC port was not exposed in Docker Compose. And the SDK path I was using configured local tracing instead of OTLP export. So I had a completed milestone for a feature that had not really worked. That was a worse lesson than a failing test. A failing test is honest. A green gate created by two cancelling bugs is dishonest in a way that looks disciplined. That changed how I think about milestones. I trust a green checkmark a lot less now. If the gate cannot prove that a real agent emits, Jaeger receives, analytics ingests, and the API serves the result, then the gate is incomplete. I do not care how many sub-checks passed before it. That is not an observability lesson only. That is just software engineering. The Fourth Thing I Learned: Structure Without Content Is Not Enough I started fairly conservative on privacy. Metadata-only felt responsible. No raw tool arguments. No full tool responses. No memory values by default. Reasonable instinct. But it taught me a more uncomfortable tradeoff. An observability SDK that only captures structure can become blind in exactly the places where you most want judgment. My hallucination detector is the clearest example. If the detector cannot see what the tool returned, it cannot meaningfully judge whether the agent's claim matches the evidence. Once I allowed truncated content instead of metadata-only, the hallucination false-positive rate dropped sharply. That did not make the privacy question go away. It just made the tradeoff explicit. "Safer by default" and "useful by default" are not always the same choice. Structure without enough evidence is just a cleaner way to stay blind. I do not think enough tooling says that out loud. The Fifth Thing I Learned: Shipping OSS Is Not The Same As Finishing The Idea The repo is public. The packages are published. The test suite is real. The product works. And I still would not call the problem solved. That matters to me. I think too many OSS launch posts flatten everything into "here is what shipped" and skip "here is what I still do not trust." Here is what I still do not trust fully: - the 28 detectors that have not yet fired on real corpora in a meaningful way - the LLM detectors on production workloads - span-tree materialization in the API, which is still weaker than I want - any claim that one threshold set will generalize cleanly across workloads - any green gate that has not been proven end-to-end That does not mean the project should not ship. It means the honest version of shipping is: this tool is useful now, and I understand its edges much better than I did when I started. That is a good outcome. It is also a very different outcome from the clean version I had in my head. What I Would Do Differently Next Time I would move field testing much earlier. Not because I failed to think of it, but because planning it and planning it early are not the same thing. Late field testing turns into audit. Early field testing shapes the design. By the time mine ran, it was strong enough to expose bad assumptions, but too late to stop me from building confidence on top of them. I would treat the trace corpus as a design artifact, not just test input. I would insist on an end-to-end smoke path much earlier. And I would stop pretending that detector logic i