Your Serverless Cron Job Failed Silently at 3AM: Making Event-Driven Jobs Reliable

Moving scheduled tasks off a cron box and onto serverless functions solves the "one server, one crontab, one single point of failure" problem. It also introduces a quieter one: a job that used to fail loudly in a log you'd eventually read now fails silently, retries in ways you didn't design, and sometimes runs twice. The reliability work doesn't disappear when you delete the crontab - it moves into your function code and your queue configuration, and most teams discover this the first time a nightly billing job double-charges someone. This is the follow-up to migrating scheduled tasks to serverless: not how to move the jobs, but how to run them without getting paged. The three things that actually matter are delivery semantics (at-least-once, almost always), idempotency (so at-least-once is safe), and observability (so you find out a job didn't run, not just when it errors). Why does the same job sometimes run twice? Because nearly every serverless event source is at-least-once, not exactly-once. A scheduled EventBridge rule, an SQS message, a Pub/Sub push, an Azure Queue trigger - all of them can deliver the same event more than once. This isn't a bug you can configure away; it's a property of distributed message delivery. A network blip between the broker and your function acknowledgment means the broker didn't hear "I got it," so it redelivers. The consequence: any job that mutates state - sending email, charging a card, incrementing a counter, writing a row - must be safe to run twice with the same input. On a single cron box this rarely bit you because there was one process and one clock. In a serverless fabric with retries built in, duplicate execution is the default, and the code has to assume it. The takeaway: treat every scheduled function as if it will be invoked at least twice with identical input, because eventually it will be. How do you make a job safe to run twice? Idempotency. The practical pattern is an idempotency key plus a durable record of what you've already processed. Derive a stable key from the event (not a random UUID generated inside the handler - that changes on every retry), then check-and-set before doing the real work. import hashlib def idempotency_key(event: dict) -> str: # Stable across retries: same event -> same key. raw = f"{event['job']}:{event['target_date']}:{event['user_id']}" return hashlib.sha256(raw.encode()).hexdigest() def handle(event, db): key = idempotency_key(event) # Atomic insert; fails if the key already exists. inserted = db.try_insert_processed(key) if not inserted: return {"status": "duplicate_skipped", "key": key} do_the_actual_work(event) db.mark_completed(key) return {"status": "done", "key": key} Two details make or break this. First, the insert has to be atomic - a INSERT ... ON CONFLICT DO NOTHING in Postgres, a conditional write in DynamoDB with attribute_not_exists . A read-then-write leaves a race window where two concurrent retries both see "not processed." Second, decide what happens if the process dies after try_insert_processed but before mark_completed . Storing an explicit state (pending / completed ) rather than mere existence lets a later retry detect a half-finished job and either resume or alert, instead of silently skipping it as a "duplicate." The takeaway: an idempotency key only helps if the check-and-set is atomic and the record captures completion state, not just "seen." What happens to a job that keeps failing? This is where a dead-letter queue (DLQ) earns its keep. Without one, a poison message - malformed input, a downstream dependency that's down, a bug that throws on one specific record - gets retried until the source gives up, then vanishes. You lose the job and the evidence. The pattern is: bounded retries with backoff, then route the failure to a DLQ instead of dropping it. For an SQS-triggered Lambda, that's a redrive policy on the source queue: { "RedrivePolicy": { "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:jobs-dlq", "maxReceiveCount": 5 } } After five failed receives, the message lands in jobs-dlq where it sits durably until a human or a redrive job looks at it. The equivalents elsewhere: Google Cloud Pub/Sub has a deadLetterPolicy with maxDeliveryAttempts ; Azure Service Bus has built-in dead-lettering with a max delivery count; a plain Postgres-backed queue needs you to add a failure_count column and a WHERE failure_count 0 on a DLQ should page someone, because it means a job is failing in a way retries won't fix. The takeaway: a dead-letter queue converts silent data loss into a visible, alertable backlog - but only if something is actually watching its depth. How do you even know a scheduled job didn't run? This is the failure mode serverless makes worse, and it's the one teams under-invest in. Error alerting tells you when a function ran and threw. It tells you nothing when the function never fired - a disabled EventBridge rule, a bad IAM permission, an account-level throttle, a deploy that removed the trigger. The nightly report just... doesn't show up, and you find out when a stakeholder asks where it is. The fix is a heartbeat or dead-man's-switch pattern: the job records a successful completion, and a separate check alerts on the absence of that record within the expected window. def handle(event, db, clock): do_the_actual_work(event) db.record_heartbeat(job="nightly_report", at=clock.now()) # A separate, independently-scheduled watcher: def check_liveness(db, clock): last = db.last_heartbeat("nightly_report") if last is None or (clock.now() - last).total_seconds() > 26 * 3600: alert("nightly_report has not completed in over 26 hours") The watcher runs on its own schedule and, ideally, its own infrastructure - if both the job and its monitor depend on the same broken EventBridge rule, neither fires and you're blind. Hosted dead-man's-switch services (the "ping this URL every day or we alert you" kind) exist precisely because the monitor should not share a failure domain with the thing it monitors. The takeaway: alert on the absence of success, not just the presence of errors - a job that never starts produces no error to catch. Which reliability mechanism do I actually need? Not every job needs the full apparatus. Match the mechanism to the job's blast radius. | Job characteristic | Idempotency | Dead-letter queue | Heartbeat / liveness | |---|---|---|---| | Read-only (e.g. cache warmer) | Optional | Nice to have | If business-critical | | Mutates state (billing, emails) | Required | Required | Required | | Fan-out over many records | Required (per record) | Required | On the orchestrator | | Idempotent by nature (full recompute) | Built in | Recommended | If deadline-sensitive | | One-shot, non-critical | Skip | Skip | Skip | The mistake I see most is applying all three to a harmless cache-refresh job while a money-touching job has none of them. Reliability effort should follow consequences. The takeaway: a cache warmer and a payment run deserve different reliability budgets - spend where a duplicate or a miss actually costs something. What does this cost you in complexity? Honestly: a fair amount, and it's worth naming the trade-off. A crontab line is one line. The reliable serverless equivalent is a function, an idempotency table, a DLQ, an alarm on the DLQ, a heartbeat table, and a separate liveness monitor. That's real operational surface, and for a genuinely trivial job on a single stable machine, plain cron is still a defensible choice - don't cargo-cult serverless onto a job that a cron line already handles fine. The reason to pay the complexity cost is when you have many jobs, need them to survive an instance dying, or want per-job scaling and isolation. At that point the crontab-on-one-box model fails in ways that page you at 3AM regardless, and you're just choosing which complexity to own. The takeaway: serverless reliability is more moving parts than cron - adopt it for scale and fault-tolerance, not because scheduled functions sound modern. Bottom line If you've moved scheduled tasks to serverless, assume at-least-once delivery and make every state-changing job idempotent with an atomic check-and-set - this is non-negotiable, not an optimization. Add a dead-letter queue with a monitored depth for anything whose failure you can't afford to lose silently, and add a heartbeat-plus-liveness check for any job where "it never ran" is a real incident. Skip all three only for read-only, non-critical jobs where a duplicate or a miss costs nothing. The goal isn't maximal machinery - it's that no job fails silently at 3AM without something, somewhere, noticing. Top comments (0)