Delivery Exception Detection - Node.js Metrics API Queries Feeding Lambda Webhooks

Short answer: for a small Node.js notification service, poll one delivery-failure metric from a separate scheduled function, evaluate a versioned rule, and send a deduplicated webhook; move to a full incident-management system only when rotations, escalations, and acknowledgement state become requirements. That answer is deliberately narrower than “install an observability platform.” A logistics team needs to know when delivery notifications stop reaching recipients, but rollback safety changes the shape of the solution: the alert evaluator must not share a deployment fate with the service it watches, and rolling back either component must not replay an old incident or silently reinterpret stored state. The least complex useful design is therefore two small programs with one explicit contract between them. This isn't a claim that polling wins everywhere. It wins only while the alert surface is small, a few minutes of detection latency is acceptable, and the team already has a trustworthy metrics query endpoint plus a webhook destination. The delivery-failure ledger comes before the monitor Start with the symptom the operator can act on. “Process is running” is a weak signal for a notification service; “delivery attempts are failing” is closer to the logistics outcome. The Google SRE monitoring guidance separates latency, traffic, errors, and saturation. Here, errors are the primary signal, traffic supplies the denominator, and latency can be a second rule if delayed delivery is operationally different from failed delivery. Define the query contract before the poller. For each closed time window it should provide an event time, an attempted-delivery count, and a failed-delivery count. Keep retries straight: if one shipment notification is attempted three times, decide whether the metric represents three transport attempts or one final delivery outcome. Either is defensible. Mixing them is not. A rising attempt-level error rate can reveal provider trouble early, while final-outcome failures map more directly to shipments that need intervention; the dashboard label and runbook must say which one the alert uses. Missing data needs its own state. Zero failures, zero attempts, and no query result are three different observations. Treating all three as zero produces a pleasant graph and a dangerous monitor. A closed five-minute window with at least 20 attempts and at least three final failures is a reasonable example policy for illustrating the state machine, not a universal threshold and not a benchmark. I'm not sure what threshold fits a particular delivery network without its normal traffic distribution, retry policy, and acceptable detection delay; a week of representative metric history would resolve that. Silence lies. Keep cardinality bounded. Region and notification channel may be useful alert dimensions because they identify an owner or a containment action. Shipment ID and recipient ID belong in traces or logs, not metric labels. The poller should receive aggregate facts, then attach a link or query recipe that lets an operator inspect individual failures under the access controls already used for customer data. How does a Node.js poller turn a metrics API query endpoint into alerts? It shouldn't. The Node.js app should publish the metric, while a scheduled observer polls it from another failure domain. A Lambda-style function is one way to host that observer, but the important property is independent scheduling and deployment, not the product category. If the application event loop stalls, its self-check cannot be the only mechanism expected to report the stall. The observer can stay boring: fetch a normalized internal query result, reject incomplete windows, evaluate the versioned rule, derive a stable incident key, and post a webhook only on a state transition. The Python below assumes an adapter has normalized the metrics provider's response into window_end , attempted , and failed ; that tiny adapter is where provider-specific query syntax belongs. Both URLs come from configuration, so the monitor does not pretend that every metrics service shares a route layout. import hashlib import json import os from datetime import datetime, timezone from urllib.request import Request, urlopen RULE_VERSION = "delivery-final-failure-v1" MIN_ATTEMPTS = 20 MIN_FAILURES = 3 def read_json(url: str) -> dict: request = Request(url, headers={"Accept": "application/json"}) with urlopen(request, timeout=10) as response: return json.load(response) def evaluate(sample: dict) -> dict: attempted = int(sample["attempted"]) failed = int(sample["failed"]) window_end = datetime.fromisoformat(sample["window_end"]) now = datetime.now(timezone.utc) if window_end.tzinfo is None or window_end > now: raise ValueError("window_end must be an aware, closed-window timestamp") firing = attempted >= MIN_ATTEMPTS and failed >= MIN_FAILURES incident_source = f"delivery-final-failure:{window_end.isoformat()}" incident_key = hashlib.sha256(incident_source.encode()).hexdigest()[:20] return { "incident_key": incident_key, "rule_version": RULE_VERSION, "state": "firing" if firing else "ok", "window_end": window_end.isoformat(), "attempted": attempted, "failed": failed, } def post_webhook(url: str, payload: dict) -> None: body = json.dumps(payload, separators=(",", ":")).encode() request = Request( url, data=body, method="POST", headers={"Content-Type": "application/json"}, ) with urlopen(request, timeout=10) as response: response.read() def handler(event, context): sample = read_json(os.environ["METRICS_QUERY_URL"]) decision = evaluate(sample) post_webhook(os.environ["ALERT_WEBHOOK_URL"], decision) return decision This sample intentionally stops short of claiming production-grade deduplication. A hash makes the identity deterministic, but exactly-once delivery doesn't appear because a function computed a key. Consider one five-minute window ending at 10:35: the observer posts its firing decision, the webhook receiver commits it, and the connection closes before the observer receives the acknowledgement. The scheduled runtime invokes the observer again. Without a unique constraint or conditional state write, the same logistics failure becomes two operator messages even though every component behaved within an ordinary retry contract. The receiver must therefore enforce uniqueness on incident_key , or the observer must perform a conditional write to durable state before sending; the second choice introduces another transition to model, because a crash after that write but before the webhook would otherwise suppress the alert. A small outbox record with pending , sent , and an immutable payload makes that transition inspectable. If neither side can enforce idempotency, accept at-least-once notifications explicitly and design the destination around duplicates. The ambiguous outcome is unavoidable - replay must be harmless. There is another sharp edge: the example posts every evaluated result to keep the contract visible. In a deployed monitor, persist the last confirmed state and send only ok → firing and firing → ok transitions. Record query failures separately from delivery failures, because “the observer cannot read metrics” is not evidence that customer notifications recovered. No magic here. Stored state decides whether rollback is safe A rollback changes code, but the damage usually enters through state. Suppose evaluator version 2 renames failed to terminal_failures , writes the new shape, and is then rolled back. Version 1 may read the record incorrectly or treat it as absent, producing a duplicate firing transition. The safe design uses additive state evolution: retain old fields during the compatibility window, give every record a schema version, and make the previous release read the new record before version 2 is allowed to send alerts. Rule versions and incident identities serve different purposes. Include rule_version in the payload so an operator can reconstruct why the decision fired, but keep it out of the incident key when two evaluator releases represent the same operational condition. Otherwise a rollback creates a new identity for the same five-minute failure window. If a rule change truly represents a different condition, such as moving from final delivery failures to provider-attempt failures, give it a distinct rule name and run it in shadow mode first. The deployment gate should exercise four fixtures: a quiet closed window, a firing window, missing data, and a replay of an already-recorded incident. Then deploy the observer without notification authority, compare its decisions with the active version, enable sends, and retain the prior artifact plus its readable state schema. A kill switch should disable outbound webhooks without disabling metric evaluation; that preserves evidence while containing alert noise. Rollback safety also means the notification application's release cannot redefine the metric without coordination. During a field rename, publish old and new series long enough for both observer versions to query them. This costs temporary duplication, yet it is easier to reason about than a synchronized “flag day” across an application, a metrics backend, a scheduled function, durable incident state, and a chat or ticket webhook. Failure domains matter more than feature lists The choice is less about feature count than ownership. A small team with one actionable condition can own a poller. A team promising round-the-clock response across several services needs acknowledgement, escalation, scheduling, audit history, and tested delivery paths; rebuilding those capabilities around a webhook would turn a small monitor into an incident-management project. | Shape | Best fit | Rollback advantage | The catch | |---|---|---|---| | Scheduled query plus webhook | A few low-urgency rules with an existing metrics endpoint | Evaluator releases can be canaried independently | The team owns state, deduplication, retries, and webhook d