DEV Community

Recovering Logistics User Reminders - 4 Wrong Cron Expression and Timezone Safeguards

TL;DR For logistics reminders, make a short, fixed-rate scheduler discover due rows in UTC, claim each reminder with a durable delivery key, and let a retry worker deliver the outbound webhook. A cron expression should trigger discovery; it should never be the only record that a reminder was owed. This design survives a wrong timezone, a missed run, pause and resume, and a few seconds of jitter without turning recovery into duplicate delivery.

I have been paged for both sides of this failure: a job that never fired and a retry that delivered twice. The invariant from those incidents is blunt: schedule time decides eligibility; durable state decides whether work may run. If the scheduler owns both decisions, its clock mistake becomes a customer-visible data mistake.

How should user reminders handle cron timezone, missed runs, pause, and resume?

Start by separating a user's civil-time intent from the machine's execution time. Store the named timezone, such as Europe/Paris or America/Chicago, alongside the reminder rule. Compute the next occurrence with a timezone database, then persist the resulting UTC instant. Don't store a fixed numeric offset as the timezone: offsets change, while the named zone expresses the user's rule.

This distinction matters at daylight-saving transitions. A local wall-clock time may be skipped or repeated. Your product must choose a policy for each case before code chooses one accidentally: move a nonexistent time forward, skip it, or reject it at creation; for an ambiguous time, select the earlier occurrence, the later occurrence, or both.

I'm not sure there is one correct product policy for every reminder. The decision depends on whether "09:00 local" or "once per elapsed interval" is the real promise. What matters operationally is that the policy is explicit, tested, and recorded with the computed UTC deadline.

Treat cron as a polling cadence, not the reminder definition

For example, a process can scan every few seconds for records where due_at <= now and status = pending. The scan window should overlap the previous tick, because exact-once timing is imaginary once processes restart, networks pause, or clocks move slightly. Overlap is safe only when claiming a reminder is atomic and keyed by a stable identifier.

Pause is state, too. While an account is paused, leave its reminder rows durable and exclude them from claims. On resume, choose one of three documented policies: discard deadlines that passed, deliver only the latest one, or replay every still-valid reminder in order. A scheduler that offers "no backfill" is suitable only for the first policy. It cannot infer the other two from cron history, so the application must query overdue rows and apply the retention rule itself.

Keep this rule in the runbook: resume changes eligibility; it does not erase delivery history.

The delivery ledger is the real duplicate boundary

Consider a shipment-status reminder due for a recipient in Europe while the carrier webhook endpoint runs in the US. A worker claims the reminder, sends the webhook, and loses the response before recording success. The row still looks eligible when the worker restarts, so a second worker can claim it after the lease expires. Retrying is correct from the sender's view, but the carrier may already have advanced the shipment workflow or notified a driver. If the second request uses a new identifier, the receiver has no stable fact with which to recognize the same logical occurrence.

This is the incident sequence to reproduce in a test: commit the claim, accept the request at the receiver, discard the response, restart the sender, and inspect the next request's delivery key. Scheduling accuracy cannot close that gap; a durable identity shared across attempts can.

Duplicates hurt. Give each logical delivery a deterministic key, such as shipment_id + reminder_kind + scheduled_occurrence. Insert that key into a delivery ledger under a unique constraint in the same transaction that changes application state or creates an outbox record. A relay then publishes committed outbox rows. The transactional outbox pattern exists for this boundary: the database update and the message record commit together, without requiring a distributed transaction. Its catch is important - a relay can publish more than once, so the consumer still needs to be idempotent.

For outbound HTTP, send the delivery key in an idempotency header or in the signed payload contract when the receiver supports it. When it doesn't, your ledger still prevents concurrent workers from initiating the same attempt, but it cannot prove the remote side did not accept a request whose response was lost. In that case, favor a receiver-provided lookup or reconciliation endpoint. Blind retry is a business decision, not a transport trick.

The preventative path can stay small

The interfaces below assume the repository implements an atomic claim, an attempt record, and a comparison-and-set completion. They deliberately keep calendar calculation outside the worker.

package delivery

import (
  "context"
  "errors"
  "time"
)

type Reminder struct {
  ID string
  DeliveryKey string
  DueAt time.Time
  Payload []byte
}

type Repository interface {
  ClaimDue(ctx context.Context, now time.Time, limit int) ([]Reminder, error)
  BeginAttempt(ctx context.Context, deliveryKey string) (bool, error)
  MarkDelivered(ctx context.Context, deliveryKey string, at time.Time) error
  ScheduleRetry(ctx context.Context, deliveryKey string, next time.Time) error
}

type Sender interface {
  Send(ctx context.Context, deliveryKey string, payload []byte) error
}

func RunOnce(ctx context.Context, repo Repository, sender Sender, now time.Time) error {
  due, err := repo.ClaimDue(ctx, now.UTC(), 100)
  if err != nil {
    return err
  }
  for _, reminder := range due {
    started, err := repo.BeginAttempt(ctx, reminder.DeliveryKey)
    if err != nil {
      return err
    }
    if !started {
      continue
    }
    if err := sender.Send(ctx, reminder.DeliveryKey, reminder.Payload); err != nil {
      next := now.UTC().Add(retryDelay(reminder.DeliveryKey))
      if scheduleErr := repo.ScheduleRetry(ctx, reminder.DeliveryKey, next); scheduleErr != nil {
        return errors.Join(err, scheduleErr)
      }
      continue
    }
    if err := repo.MarkDelivered(ctx, reminder.DeliveryKey, now.UTC()); err != nil {
      return err
    }
  }
  return nil
}

func retryDelay(key string) time.Duration {
  // Production code derives bounded jitter from the stable delivery key.
  return 30 * time.Second
}

That fixed 30s return is the policy boundary, not a recommended retry sequence. In production, calculate a capped delay from the attempt count and add bounded jitter. Deriving jitter from the stable key makes tests repeatable and avoids every worker choosing a new random schedule after a restart. Retry only failures classified as transient by the webhook contract; permanent rejection belongs in a terminal state with enough context for an operator to act.

A FIFO queue can preserve ordering within its documented grouping and deduplication model, which is useful when shipment events must reach one destination sequentially. It does not replace the application ledger. Queue deduplication has service-specific scope and timing, while the business key may need to remain unique for the full lifetime of a shipment occurrence.

Four checks before the next scheduler change

First, test calendar semantics with named zones. Include a normal day, a skipped local time, an ambiguous local time, and a user who changes zones after creating a reminder. Assert the persisted UTC deadline and the next civil-time occurrence separately. These are different outputs.

Second, test recovery by stopping the scheduler across several scan intervals. On restart, verify that overdue rows are discovered according to the backfill retention policy. Then pause an account, allow deadlines to pass, resume it, and verify the chosen discard, latest-only, or replay behavior. No guessing during an incident.

Third, inject the dangerous delivery boundary: let the receiver accept a request while the sender observes no usable response. The next attempt must reuse the same delivery key. Verify that a cooperating receiver returns the prior result or suppresses the repeated side effect. A transport timeout is not evidence of non-delivery; duplicate-delivery pages make that distinction hard to forget.

Fourth, measure outcomes instead of cron ticks. Alert on the age of the oldest eligible reminder, claim-to-send latency, retry attempts by classification, terminal failures, and duplicate-key conflicts. Scheduler heartbeat is supporting evidence. A healthy heartbeat beside a growing overdue queue is still a missed-delivery incident. Seconds of jitter should appear as a latency distribution, not as a binary claim that every reminder ran at an exact second.

The trade-off is additional durable state and reconciliation work. A simple in-process timer is a better fit when reminders are disposable, the process lifetime is the intended retention boundary, and duplicate effects are harmless. Stick with a queue's native delayed-delivery feature when its maximum delay, ordering rules, retention, and redelivery semantics match the entire requirement. The ledger-and-scanner design is not suitable when the backing store cannot perform atomic claims or enforce a unique delivery key; fix that boundary or choose infrastructure that can before adding more retry workers.

One last deployment check: roll out schema and readers before writers begin depending on new delivery states. During rollback, old workers must ignore or safely preserve states they do not understand. A retry system becomes much harder to recover when two versions disagree about whether a row is eligible.

References

Further reading

The two primary references above cover the queue and transaction boundaries used here:

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.