Beyond Logs: Why Observability's Next Era Is Comprehension

The Pattern That Needs to Die: Mining Metrics Out of Logs

For years, the default pipeline looked like this: emit a log line for every event, ship it to a log aggregator, then write a query that counts, sums, or averages fields inside those log lines to approximate a metric. Log-based metrics dashboards, logs_to_metrics processors, regex-parsed latency histograms scraped out of access logs - all variations on the same idea.

It works, in the sense that it produces a number. But it's the wrong tool doing the job of the right one, for a few concrete reasons:

• Cost. A log line is bytes of unstructured text, indexed and stored per event. A metric is a pre-aggregated number. Deriving a metric from logs means you pay full log-ingestion cost for data whose only purpose is to become four numbers (count, sum, min, max) at query time - over and over, on every query.

• Latency. Metrics-from-logs are only as fresh as your log pipeline's indexing latency. A real metrics pipeline aggregates at write time, so a spike shows up in seconds, not after a batch job or index refresh.

• Fragility. Regex-parsing a log line to extract status_code breaks the moment someone changes the log format. A metric emitted directly by the application, with a typed field, doesn't have that failure mode.

• It hides intent. If you actually want a metric, emit a metric. Making it derivable from logs means the actual measurement - the thing you're paying to compute and alert on - lives implicitly inside a text format designed for narrative, not aggregation.

If you're extracting a metric from a log line today, that's a signal the underlying event should be instrumented as a metric (or an exemplar-linked span) directly, at the source. We'd go as far as saying: nobody should be building new log-to-metrics pipelines in 2026. If you're doing it, it's almost always because metrics weren't instrumented where the event actually happens - fix that, don't work around it downstream.

Metrics: Aggregate, Cheap, and Only as Good as Your Cardinality Discipline

Metrics answer "how much" and "how often," aggregated over time, cheaply, at any scale. A counter, a gauge, a histogram - pre-aggregated numeric time series, indexed by a small, fixed set of label dimensions. That's what makes them fast to query and cheap to store: the shape of a metric is bounded, no matter how much traffic flows through it.

That boundedness is also the entire contract. The moment a label's possible values grow unbounded - user IDs, request IDs, raw URLs, IP addresses, session tokens - you've broken the assumption the whole system is built on. Every unique combination of label values creates a new time series. A metric with method, route, and status_code labels might produce a few hundred series. Add user_id as a label and you've just told your metrics backend to create a new time series per user, forever. That's not a slow leak - it's an explosion, and it's the single most common way teams take down (or bankrupt) their metrics backend.

When to Reach for Metrics

• Request rate, error rate, duration (the RED signals) per service, route, and status class
• Resource utilization - CPU, memory, queue depth, connection pool saturation
• Business counters - signups, jobs processed, cache hit ratio
• Anything you want to alert on or put on a dashboard as a trend over time

The Cardinality Test

Before adding a label to a metric, ask: "How many distinct values can this take, across all time?" If the answer is bounded and small (HTTP methods, a fixed list of status classes, a fixed list of service names), it's a metric label. If the answer is "one per user," "one per request," or "unbounded and growing," it does not belong on a metric - full stop.

This is exactly where traces come in. When you find yourself wanting per-user, per-request, or per-tenant granularity - "show me the latency for this specific customer's this specific request" - that's not a cardinality problem to solve with more labels. That's a signal you've outgrown metrics for that dimension and need a trace.

A span can carry a user_id, a request_id, a full SQL statement, an order ID - arbitrary high-cardinality, high-dimensionality attributes - because a trace is stored and queried per-event, not pre-aggregated into a time series. You don't pay a cardinality tax on a span attribute the way you do on a metric label.

A concrete example: you're running an API gateway and want to know if a specific enterprise customer's traffic is degrading independently of the fleet average. Don't add customer_id to your http_request_duration_seconds histogram - that's thousands of new series the instant you onboard your thousandth customer. Instead, keep customer_id off the metric, and rely on traces (filtered or sampled by customer_id as a span attribute) to answer that question when it comes up. The metric tells you the fleet is healthy or not; the trace tells you why for one specific case.

Traces: Causality and the Request's Actual Path

A trace answers "what happened, in what order, across which services, and where did the time go" for one specific unit of work. It's the signal built for cardinality - every span can carry rich, unique, per-request context, because you're not aggregating it into a running total, you're storing (or sampling) the event itself.

When to Reach for Traces

• Any request that crosses a service boundary - you need to see the whole path, not one service's local view
• Latency breakdowns - which downstream call, which DB query, which lock actually cost the time
• Debugging one specific failing request, not the aggregate failure rate
• Anything that needs high-cardinality context attached to a single event: a user ID, a tenant ID, a specific query, a specific input payload

And the inverse smell, mirroring the metrics section: if you find yourself emitting a log line on every single request - "request started", "request completed", one line per call, thousands of times a minute, all with the same shape - that's not really a log. It's a metric wearing a log's clothing. A per-request log line whose only real content is "this happened, it took N ms, it returned status X" should almost always be a counter and a histogram, not free text shipped to a log index. Keep the log for the exceptional case - the one request that failed, that took an outlier amount of time, that hit an edge case worth narrating - and let the trace carry the structured, per-request detail instead of a flood of near-identical log lines.

Logs: The Narrative Signal, Used Sparingly and on Purpose

Logs are still valuable - they're just not the default anymore. Logs are for the things that don't fit a number and don't fit a span: a stack trace, an unexpected state transition, a one-off narrative of "here's exactly what the code was doing when this broke." They're the signal you want when something you didn't anticipate happens, and you need arbitrary, unstructured detail to understand it after the fact.

When to Reach for Logs

• Errors and exceptions, with full context, at the point of failure
• Startup/shutdown and configuration state - things that happen once, not per-request
• Debug-level detail during active investigation of a specific issue
• Audit trails where the literal sequence of events, in text, is the deliverable

What Logs Are Not For

Counting things (use a metric), tracking a request's path across services (use a trace), or serving as the default instrumentation for every code path "just in case." If a log line fires on every request with the same three fields every time, you already know what it should have been.

Profiling: The Fourth Signal Nobody Instruments Until It's Too Late

Profiling - continuous CPU, memory, and allocation profiling - answers a question none of the other three signals can: "which line of code is actually burning the resources." A trace tells you a span took 400ms. A metric tells you p99 CPU-bound latency crept up over the last week. Neither tells you which function inside that span is the one eating CPU.

Profiling closes that gap, and it does it continuously and cheaply enough (with modern eBPF-based profilers) that "we'll add profiling later" is no longer a good excuse. Treated as a first-class signal alongside metrics, traces, and logs - not a special tool you reach for once a quarter - profiling turns "this service is slow" into "this function, in this code path, under this load" without a single manual pprof session.

OpenTelemetry Semantic Conventions: The Reason Unification Is Possible at All

None of the above works as "one system" unless every signal agrees on what things are called. This is the part that's easy to skip past, and it's the part that actually makes signal correlation possible instead of aspirational.

Before semantic conventions, every team invented its own vocabulary. One service logs status, another logs http_status, a third logs statusCode, a fourth logs resp_code - all describing the exact same field. Multiply that across logs, metrics labels, and span attributes, and you get a system where a single logical concept ("the HTTP response status code") has a dozen different names depending on which signal, which team, and which service you're looking at. You cannot correlate what you cannot name consistently.

OpenTelemetry's semantic conventions fix this by defining a standard, versioned vocabulary of attribute names and their expected types, scoped by domain:

• HTTP: http.request.method, http.response.status_code, http.route, url.path
• Database: db.system, db.namespace, db.query.text, db.operation.name
• Messaging: messaging.system, messaging.destination.name, messaging.operation
• RPC: rpc.system, rpc.service, rpc.method
• Kubernetes / cloud infra: k8s.pod.name, k8s.namespace.name, cloud.provider, cloud.region

Crucially, these attribute names are signal-agnostic. http.response.status_code means the same thing whether it shows up as a span attribute on a trace, a label on a metric, or a structured field in a log record. That's what makes cross-signal queries possible in the first place: "show me the trace for the request behind this metric spike" or "show me the logs correlated with this specific span" only works because both sides used the same key for the same concept.

A Few Concrete Mechanisms Worth Calling Out

Resource attributes. Every signal - every log record, every metric data point, every span, every profile - is tagged with the same Resource: service.name, service.version, deployment.environment, k8s.pod.name. That's what lets you pivot from "this service's error-rate metric spiked" to "show me every signal from this exact service instance during that window" without re-deriving identity per signal.

Trace context propagation. trace_id and span_id aren't just trace concepts - OpenTelemetry's logging instrumentation injects them into log records automatically. A log line emitted mid-request carries the trace_id of the request it happened in, for free. That single shared ID is what turns "a pile of logs" and "a pile of traces" into one correlated timeline.

Exemplars. A metric is aggregated by design - you lose the individual event. Exemplars solve this by attaching a sampled trace_id to a specific bucket in a histogram. So when your http.server.request.duration histogram shows a spike in the 900ms-1s bucket, the exemplar hands you a real trace ID from a request that actually landed in that bucket - a direct jump from "the aggregate looks bad" to "here's one concrete example of why."

One collection pipeline. The OpenTelemetry Collector processes logs, metrics, traces, and (increasingly) profiles through the same pipeline, with the same resource-detection processors, the same batching, the same export targets. You configure identity and enrichment once, not four times per signal.

This is the actual argument for "use all the signals together, not logs by default": semantic conventions plus shared resource context plus trace-context propagation are what make it mechanically possible to move between signals without translation. Without them, you have four separate data silos that happen to describe the same system. With them, you have one telemetry graph with four views into it.

The Next Era: Telemetry That Humans and AI Can Both Understand

Semantic conventions solved the machine-to-machine problem - one signal's span can be correlated with another signal's log because they agree on vocabulary. But there's a second-order effect that matters more every quarter: a consistent, typed, well-named telemetry graph isn't just queryable by a human with a dashboard. It's legible to a model.

An LLM reading status: "INFO" on a field meant to hold an HTTP status code has no idea what happened. An LLM reading http.response.status_code: 503 on a span, correlated via trace_id to the exact log line and the exact metric bucket it came from, can reconstruct the incident the same way a senior engineer would - because the data is structured, named consistently, and linked by shared identifiers.