Instrumenting a Go API with Prometheus

RED as the baseline, not the ceiling

Request rate, error ratio, and latency:

• http_requests_total{method, path, status} for volume and error breakdown
• http_request_duration_seconds as a histogram, buckets tuned to the actual latency distribution
• http_requests_in_flight as a gauge for concurrency pressure
• http_response_size_bytes to catch payload bloat before it manifests as latency

The label that matters most here is path. Using the resolved URL instead of the route template turns every /users/42, /users/43 ... into its own series - cardinality explodes on nothing but normal traffic, and unmatched routes (scanner noise, bad paths) make it worse. Route templates as labels, unmatched paths collapsed into a single bucket, and the metrics endpoint itself excluded from the middleware so scrape traffic doesn't pollute its own numbers.

Connection pool saturation

Database exhaustion rarely announces itself early through query timing alone. Wiring NewDBStatsCollector against the connection pool surfaces open/in-use/idle counts plus wait count and wait duration - the earliest indicator that a pool is under pressure, well before it shows up as user-facing latency.

That instrumentation only means something paired with actual limits: max open connections, idle connection caps, connection lifetime. Without bounds, a spike exhausts the database directly. With them, the same spike shows up as queuing in the metrics - visible, diagnosable, alertable.

Security events belong in the metrics pipeline

Structured logs answer "what happened." They don't answer "is this happening right now, at what rate." Auth failures, failed logins by reason, account lockouts, rate-limit triggers, and forbidden-access attempts got a parallel path into Prometheus - security_events_total{event, reason}, logins_total{result} - incremented alongside the existing log calls. That's the difference between discovering a brute-force pattern by grepping logs after the fact, and alerting on the rate the moment it starts climbing.

Reliability and runtime signals

• panics_recovered_total from the recovery middleware - stability issues that would otherwise blend into a pile of 500s
• cache_operations_total{cache, result} on hit/miss paths - actual numbers about cache effectiveness
• app_build_info{service, version, env} - correlating any metric back to the exact build and environment it came from
• Default Go collectors, registered once: goroutines, memstats, GC pauses, CPU, resident memory - no custom code, and usually the first sign of a leak long before it's user-facing

The principle underneath it

Every metric is a standing commitment - a series that has to be maintained - and every label is a multiplier on its cost. The goal was never maximum coverage. It was a deliberately small set of signals that answer "is this healthy, and if not, why" without opening a log file first. The mechanics of Prometheus are the easy part. Choosing what deserves to be measured, and building the guardrails that keep it cheap as traffic grows, is the actual work.