What You Cannot See Will Break Your LLM App: A Practitioner Guide to Production Observability

TL;DR - Key Takeaways - Traditional APM can show a healthy system even when an LLM is producing hallucinations, malformed outputs, unsafe responses or truncations. - The four major production problem classes are quality drift, prompt failures, cost anomalies and latency degradation. - Structured logging should capture model, workflow, token usage, latency, time to first token, finish reason, cost and quality signals. - Finish reasons are especially valuable because truncated responses can still appear as successful API calls. - Rule-based checks on every response combined with model-graded evaluation on a sample can catch quality regressions without doubling inference costs. Traditional application observability was built around a simple mental model: Your code runs, metrics come out and when something breaks, the logs tell you why. Large language models (LLMs) break that model in ways that are not obvious until you have shipped one and watched it misbehave in production. An LLM-powered application can be up, serving requests, returning HTTP 200 responses and still be failing catastrophically - producing hallucinated content, silently truncating outputs, drifting toward unsafe responses or degrading in quality because the model provider quietly updated the underlying checkpoint. Standard infrastructure monitoring tells you nothing about any of this. Over the past two years, I have built and operated a production LLM application that processes tens of thousands of requests daily. The observability stack I run today is substantially different from what I started with, and most of the changes came from incidents I could not have anticipated without experience. This article shares the architecture and tooling that actually work - not the theoretical monitoring stack, but the one running right now. Why LLM Observability is Different Conventional APM tools track latency, error rates and throughput. These are necessary but not sufficient for LLM systems. The failure modes that matter most are semantic, not structural. A conventional API returns a well-typed response or throws an exception. An LLM returns a string. That string might be exactly what you asked for, a plausible-sounding but incorrect answer, an output in the wrong format that breaks a downstream parser, a response that violates content policies or a truncated completion because the context window was exceeded silently. None of these show up as an error in standard monitoring. Four distinct problem classes in LLM production require dedicated observability signals, and each needs a different instrumentation approach. The Four Problem Classes Problem Class 1: Quality Drift The output quality of your application degrades over time, usually for one of three reasons: The provider updated the underlying model, your prompt encountered input distributions it was not tested against or a downstream change altered the context your LLM receives. Quality drift is invisible without a baseline to compare against and a measurement approach that is not purely human-reviewed. Catching this requires automated evaluation: A set of representative inputs with expected outputs or rubrics, run on a schedule against production traffic samples, with scores tracked over time. The signal is not a single evaluation run but the trend line. A sudden drop in average score on the evaluation set is an early indicator of a problem before users report it. Problem Class 2: Prompt Failures Prompts are code. They can fail due to inputs that the prompt was not designed to handle: Unexpected languages, edge-case formatting, adversarial inputs or simply unusually long inputs that cause important context to be dropped. Prompt failures often look like partial successes: The model returns something, but it is wrong in a way that requires domain knowledge to recognize. Catching these requires logging full prompt-response pairs (with privacy handling for PII), tagging the output with structured metadata about the result type and sampling enough traffic for human review to catch failure patterns. Pattern-based prompt failures often affect a narrow slice of inputs and will not show up in aggregate metrics. Problem Class 3: Cost Anomalies LLM API costs are token-based and can spike unexpectedly. A bug that causes your application to include a large context document in every request, a prompt template that grew too large or a change that triggers multi-turn conversation where single-turn was expected can multiply your token consumption by 10x overnight. By the time the billing statement arrives, the damage is done. Cost observability requires token-level tracking per request type, not just total spend. You need to know the average token count for each workflow, see that number in real time and alert on anomalies before they accumulate into a large bill. Problem Class 4: Latency Degradation LLM API latency is variable in ways that server-side APIs are not. Time to first token and total generation time depend on server load at the provider, prompt length, output length and model family. Latency can degrade without any change on your side and without the provider posting a status update. Monitoring p50 latency is insufficient - LLM latency distributions are fat-tailed, and p95 and p99 are where user experience breaks down. The Minimal Instrumentation Stack You do not need a commercial observability platform to get meaningful LLM monitoring. The following instrumentation can be built into any application in a day, and it covers 80% of the failure modes that matter in practice. Step 1: Log Every LLM Call as a Structured Event { “request_id”: “abc123”, “timestamp”: “2026-06-19T14:00:00Z”, “model”: “gpt-4o”, “workflow”: “document_summarization”, “prompt_tokens”: 1842, “completion_tokens”: 312, “latency_ms”: 2240, “ttfb_ms”: 480, “finish_reason”: “stop”, “cost_usd”: 0.00318, “output_quality_score”: null } This structured log record is the foundation of everything else. It gives you the data to compute per-workflow cost trends, latency distributions and finish reason breakdowns (how often does your model hit the token limit instead of reaching a natural stop?). Finish reason analysis alone will surface truncation issues that users notice but that look fine in error rate dashboards. Step 2: Track Finish Reasons Explicitly The finish_reason field is one of the most underused signals in LLM monitoring. A high rate of ‘length’ finish reasons means your model is being cut off before completing its output. This is almost always a problem - it means users are receiving partial results - but it registers as a successful API call in every standard monitoring system. Alert when the length finish reason rate for any workflow exceeds 5% of requests. Investigate when it exceeds 2%. In most cases, the fix is adjusting max_tokens, reformulating the prompt to produce more concise output or implementing chunked generation. Step 3: Per-Workflow Token Budget Alerting BUDGET_ALERTS = { “document_summarization”: {“prompt_tokens_p95”: 3000, “total_tokens_p95”: 3500}, “slide_generation”: {“prompt_tokens_p95”: 5000, “total_tokens_p95”: 5800}, “qa_response”: {“prompt_tokens_p95”: 800, “total_tokens_p95”: 1200}, } Define expected token ranges per workflow based on your baseline measurements. Alert when p95 token count for a workflow exceeds the budget by more than 20%. This catches context bloat early and surfaces regressions from prompt changes that inflate token usage. Automated Quality Evaluation Logs and metrics tell you about the mechanics of your LLM calls. Automated evaluation tells you whether the outputs are actually good. Setting this up is the highest-leverage observability investment you can make for a production LLM application. The practical approach for most production systems is not model-graded evaluation running on every request - that doubles your inference cost. It is a two-tier system: Lightweight rule-based checks on every request, and deeper model-graded evaluation on a sample. Tier 1: Rule-Based Checks on Every Response - Format Validation: Does the output match the expected structure (valid JSON, correct number of sections, presence of required fields)? This catches a surprisingly large proportion of prompt failures at zero additional LLM cost. - Length Sanity: Is the output within the expected character or word range for this workflow? Outputs that are dramatically shorter than expected indicate truncation or refusal; outputs dramatically longer indicate prompt leakage or runaway generation. - Content Safety Signals: Apply a lightweight classifier for harmful content categories. This is especially important if your application accepts arbitrary user input as part of the prompt. Tier 2: Model-Graded Evaluation on a 5% Sample Sample 5% of production traffic (or more if volume allows) and run a secondary evaluation prompt against the original request and output. The evaluation prompt asks a smaller, cheaper model to score the output on dimensions relevant to your application: Accuracy, completeness, format adherence, tone. Store the scores alongside the original log record. The power of this approach is the trend line. A stable average score that suddenly drops is a high-confidence signal that something changed - a model update, a prompt regression or a shift in input distribution. Without the trend line, you are flying blind until users complain. Distributed Tracing for Multi-Step Pipelines Various production LLM applications are not single-model calls - they are pipelines: Retrieve context, summarize, generate, validate and sometimes retry. Standard metrics do not show you where time is spent in the pipeline or where quality degrades. Applying distributed tracing to LLM pipelines - using OpenTelemetry (OTel) spans with LLM-specific attributes - gives you visibility into the full execution path. The key convention is to create a span for each LLM call with attributes that match the structured log format above. A tr