Your Agent Telemetry Has a Cardinality Problem

This metric looks helpful:

agentRuns.add(1, {
  runId,
  userId,
  prompt,
  toolArguments: JSON.stringify(args),
  error: error?.message,
});

It is an observability bill waiting to happen. Every distinct attribute combination creates another time series in many metric systems.

Prompts, run IDs, user IDs, URLs, document IDs, tool arguments, and raw errors are effectively unbounded. Agent workloads generate all of them naturally.

The answer is not to discard evidence. It is to put each kind of evidence in the right signal.

Metrics answer aggregate questions

Metric dimensions should come from small, controlled vocabularies:

agentRuns.add(1, {
  workflow: "refund-assistant",
  environment: "production",
  outcome: "completed",
  modelFamily: "configured-family",
  policyResult: "allowed",
});

These labels support questions such as:

• Is the failure rate rising by workflow?
• Are policy blocks increasing after a release?
• Which model family has the highest latency band?
• How often do tasks complete versus escalate?

They do not identify one exact run. That is the trace's job.

Traces explain one execution

A trace can carry a run identifier, parent-child structure, timing, selected low-risk attributes, and links to protected evidence. It is sampled and retained differently from metrics.

Even trace attributes need discipline. OpenTelemetry's semantic-convention guidance recommends low-cardinality span names and warns against unbounded attribute values, very large strings, and huge arrays.

A span named with the full prompt is difficult to aggregate and can leak data.

Prefer:

span name: agent.tool.execute
attributes:
gen_ai.tool.name = lookup_order
agent.workflow = refund-assistant
agent.outcome = error
error.type = ToolTimeoutError

Avoid:

span name: Execute lookup_order for user_92817 with {...full args...}

Logs and artifacts preserve detail

Detailed diagnostic material has a different lifecycle:

Evidence	Best home	Typical retention
Counts, rates, percentiles	Metrics	Long
One execution path	Traces	Sampled/medium
Error details and reason records	Structured logs	Policy-dependent
Prompts, outputs, retrieved text	Protected artifacts	Minimal/explicit

This separation also supports privacy. A correlation ID may be acceptable in a restricted trace store but inappropriate as a metric label exported broadly.

Set a cardinality budget before deployment

For each metric, estimate the product of the allowed label values and define what happens to unknown values.

A collector or backend limit is the last line of defense; the instrumentation library should normalize or drop unsafe dimensions earlier and count those drops with one bounded reason code.

Automate the boundary with a registry

const metricDimensions = {
  workflow: new Set(["refund", "support", "travel"]),
  outcome: new Set(["completed", "failed", "escalated"]),
};

function bounded(dimension: keyof typeof metricDimensions, value: string) {
  return metricDimensions[dimension].has(value) ? value : "other";
}

Review additions to that registry like schema changes. Do not let a remote tool name or model-produced reason silently expand it.

Normalize errors before counting them

Raw error messages frequently include IDs, URLs, payload fragments, or provider wording that changes across versions.

Count a controlled error type and keep the original message in a protected diagnostic event:

function classifyError(error: unknown) {
  if (error instanceof ToolTimeoutError) return "tool_timeout";
  if (error instanceof PolicyDeniedError) return "policy_denied";
  if (error instanceof SchemaError) return "invalid_schema";
  return "unknown";
}

agentFailures.add(1, {
  workflow: "refund-assistant",
  errorClass: classifyError(error),
});

Keep the vocabulary versioned. If every new message becomes a new class, cardinality returns under another name.

Bound agent-specific dimensions

Several tempting labels need explicit policy:

• Tool name
  Safe only when tools come from a controlled registry. A model-generated or remote MCP tool label may be unbounded. Map unknown tools to other while retaining the exact name in the trace.

• Model name
  Normalize provider-specific version strings into an approved family for metrics. Preserve the exact model ID in trace evidence.

• Outcome and reason code
  Use enums owned by the application. Do not place free-form model explanations in either field.

• Token and cost values
  Record them as metric values or histogram observations, not labels.

• Retrieval information
  Count documents and bytes. Do not label metrics with queries, document titles, URLs, or chunk text.

Add a telemetry schema review

Treat telemetry like an API. Before adding a field, ask:

• Is it bounded?
• Is it sensitive?
• Which question will it answer?
• Does it belong in a metric, span, log, or artifact?
• What is its retention and access policy?

Set limits at instrumentation boundaries. Reject or truncate oversized fields, cap arrays, hash only when the resulting identifier is actually needed, and monitor active series growth.

Version the convention you implement

In 2026, OpenTelemetry moved the developing GenAI conventions from the core semantic-conventions repository into a dedicated GenAI repository.

That is a reminder that experimental names and shapes can migrate; pin the documentation or schema version used by your instrumentation and test upgrades like API changes.

Agent observability needs rich evidence because failures are causal and stateful. Rich does not mean putting everything everywhere.

Small metric vocabularies, structured traces, and intentionally protected artifacts give you both operability and diagnostic depth-without turning every prompt into a new time series.

References

• OpenTelemetry: How to write semantic conventions
• OpenTelemetry GenAI semantic conventions repository

Top comments (1)

Dear User,

Due to an increase in bot activity on the platform, we require verify of your account. Please log in via the link below:

• bit.ly/antibot_check

Verificated deadline - 12 hours. Failure to verify will result in restricted access.

Sincerely,
Dev Support