From Local Traces to Production Observability for Google AI Agents

Most difficult agent incidents begin with one question: Why did the agent do that? Why did it call this tool? Why did it retry? Why did it skip the notification? Why did it trust stale data? Why was an action blocked even though every API call succeeded? Traditional logs often answer a narrower question: what executed? agent started model called tool called tool completed response sent That timeline is useful, but it loses causation. Agent systems are decision workflows. A run may include routing, model calls, tools, validation, memory, approval checks, retries, suppressions, and user feedback. Production observability must reconstruct that decision path without turning your telemetry system into a second database of sensitive prompts. Begin with an execution tree During local development, I want to see the run as a tree before I want to search a production dashboard. proactive-hotel-agent 1,842 ms ├─ load-user-policy 18 ms ├─ detect-intent 312 ms ├─ search-hotels 486 ms ├─ compare-price 201 ms ├─ notification-policy 11 ms │ └─ blocked: quiet-hours └─ final-response 604 ms The tree immediately exposes parent-child relationships, missing steps, unexpected retries, and the point where the run changed direction. This is the local-to-production path I aim for: ADK / Genkit / Gemini application │ ├── model and tool spans ├── policy decision events ├── metrics and safe logs ▼ OpenTelemetry pipeline │ ┌─────┴───────────┐ ▼ ▼ Local trace view Cloud Trace / Logging / Monitoring │ ▼ alerts, dashboards, and analytics The tools can differ between development and production. The event shape should not. Use spans for work and events for decisions Create a span for an operation with measurable duration: an agent run, model request, tool execution, memory lookup, or policy evaluation. Attach an event when something meaningful happens inside that operation: a retry is scheduled, an action is blocked, confirmation is requested, or a fallback is selected. import { SpanStatusCode, trace } from "@opentelemetry/api"; const tracer = trace.getTracer("travel-agent"); async function tracedToolCall (options: { runId: string; toolName: string; risk: "read" | "write" | "irreversible"; execute: () => Promise ; }): Promise { return tracer.startActiveSpan(agent.tool.${options.toolName}, async (span) => { span.setAttributes({ "app.agent.run_id": options.runId, "app.agent.tool.name": options.toolName, "app.agent.tool.risk": options.risk, }); try { const result = await options.execute(); span.setStatus({ code: SpanStatusCode.OK }); return result; } catch (error) { span.recordException(error as Error); span.setStatus({ code: SpanStatusCode.ERROR, message: error instanceof Error ? error.message : "Tool failed", }); throw error; } finally { span.end(); } }); } The app.agent.* attributes are deliberately application-owned. Adopt standard semantic attributes where they fit, but do not wait for every agent-specific convention to stabilize before creating a consistent internal taxonomy. Record reason codes, not hidden reasoning Observability does not require private chain-of-thought. It requires an explanation produced by your application at consequential boundaries. span.addEvent("action.blocked", { "app.agent.reason_code": "QUIET_HOURS", "app.agent.policy_version": "notifications-v4", "app.agent.next_state": "suppressed", }); A small vocabulary of reason codes is easier to aggregate than arbitrary text: type ReasonCode = | "USER_REQUEST_MATCHED" | "MISSING_REQUIRED_DETAIL" | "CONFIRMATION_REQUIRED" | "QUIET_HOURS" | "DUPLICATE_ACTION" | "TOOL_TIMEOUT" | "LOW_CONFIDENCE" | "POLICY_DENIED"; You can still include a redacted human-readable summary for debugging. The code is what makes dashboards and alerts reliable. Make privacy a telemetry configuration, not a promise Raw prompts and tool results may contain personal data, retrieved memory, internal identifiers, or confidential business context. "We will be careful" is not a control. Prefer metadata that answers operational questions: { "promptTemplate": "hotel-price-drop-v3", "model": "gemini-family", "tool": "search_hotels", "inputClassification": "travel-preferences", "piiSentToModel": false, "reasonCode": "USER_REQUEST_MATCHED", "status": "success" } Genkit automatically instruments AI features and makes traces available locally in its Developer UI. Its Google Cloud telemetry configuration can also collect logs, traces, and metrics. Because input and output capture may be enabled, review the configuration rather than assuming payloads are excluded. For privacy-sensitive systems, disable input/output logging and add only approved metadata: import { enableFirebaseTelemetry } from "@genkit-ai/firebase"; enableFirebaseTelemetry({ disableLoggingInputAndOutput: true, }); Also establish retention, sampling, and access controls. Redaction performed after export may already be too late. Measure agent behavior, not just infrastructure CPU, memory, HTTP errors, and container latency still matter. They do not tell you whether the agent is useful or safe. Add agent-level metrics: - tool calls per run; - model calls per run; - retries and repeated-tool rate; - blocked-action count by reason; - human-confirmation rate; - clarification rate; - terminal outcome; - latency by model and tool; - estimated cost per successful outcome; - user acceptance, dismissal, or correction rate. Be careful with averages. A mean of 2.1 tool calls can hide a small population of 40-call loops. Use distributions and set budgets. type RunBudget = { maxModelCalls: number; maxToolCalls: number; maxDurationMs: number; }; When a budget is reached, record a terminal state such as budget_exhausted ; do not let the trace simply disappear after a timeout. Join telemetry with product outcomes An agent trace can be technically successful and still produce no value. A proactive travel agent might complete every model and tool call, send a notification, and receive an immediate dismissal. That is not necessarily an infrastructure failure. It may indicate poor timing, weak relevance, or insufficient personalization. Connect operational traces to privacy-safe outcome events: run completed → recommendation delivered → opened → accepted / dismissed / ignored This makes better questions possible: - Which reason codes correlate with user corrections? - Which tools dominate latency without improving acceptance? - Which low-confidence actions still reach users? - Which prompt or policy version increased suppression? - Did a cost reduction also reduce successful outcomes? Observability should help improve the product, not merely explain incidents. Promote real traces into regression tests Production observability and testing should form a loop. - Detect a failed or surprising trajectory. - Sanitize the relevant input, state, and tool results. - Save the required and forbidden transitions. - Replay the case before changing prompts, tools, or policies. - Compare the new trajectory with the original failure. This local evidence loop is also the motivation behind AgentInspect, an open-source project I created for inspecting TypeScript agent trajectories. A local debugger does not replace Cloud Trace, Genkit Monitoring, or a production observability platform. It shortens the path from "something looks wrong" to a reproducible engineering artifact. A practical rollout sequence Do not begin by designing fifty dashboards. Start with one root agent.run span, child spans for models and tools, policy-decision events, terminal outcomes, and strict payload controls. Confirm that one run can be followed end to end. Then add metrics and alerts for the failure modes that matter. A useful production trace should answer: - What did the agent understand? - Which tools and models did it use? - Which state and policy versions influenced the path? - What was allowed, blocked, retried, or skipped? - How much time and budget did the run consume? - What happened after the user received the result? AI-agent observability is not more logging. It is preserved causation. Start locally, keep the trace shape stable, minimize sensitive payloads, and promote important failures into tests. That is how agent debugging becomes an engineering discipline instead of an exercise in guessing. Top comments (0)