Logistics Renewal Deadlines: When Should Failed Webhook Jobs Use a Delayed Queue or Cron?
Short answer: use a delayed queue when each failed webhook job needs its own retry time and audit trail; use cron to run a bounded redrive or reconciliation sweep when a little extra latency is acceptable and the database already owns retry state.
A public HTTPS endpoint is for receiving the webhook or a cron callback, not for exposing a private retry worker.
For a logistics system, the decision is easiest to make from the business deadline. A renewal reminder may be useful before a carrier contract closes, but it is worse to send the same reminder twice than to send it a few minutes late. Latency matters, yet the retry mechanism must not turn one delivery failure into two business effects.
I treat that as a ledger problem: every attempt gets a durable identity, every state transition leaves an audit record, and the receiver remains idempotent. The implementation can be small. The correctness argument cannot.
What does latency cost when failed webhook jobs wait for a delayed queue or cron?
A delayed queue keeps eligibility with the individual message. When delivery fails, the worker can record the attempt, calculate the next eligible time, and defer that job while other work proceeds. The retry clock belongs to the failed delivery rather than to a periodic scan of every outstanding delivery.
Cron is a control-plane mechanism. It wakes up on a schedule, calls a configured public HTTPS endpoint, or starts a process that selects due rows. That makes it useful for a bounded redrive, a dead-letter review, or a reconciliation pass. It does not, by itself, provide an idempotency ledger, a claim protocol, or evidence that a particular attempt owns a particular effect.
The latency-versus-cost choice follows from that distinction. Choose the delayed queue when retry timing is part of the delivery contract, when failures arrive unevenly, or when operators need to inspect one job without reopening the entire backlog. Choose cron when the retry volume is modest, a scan is already part of the database design, and the deadline tolerates the next scheduled run.
The catch is operational ownership. A queue makes the application own duplicate consumption and dead-letter handling. A cron redrive makes the application own leases, row claiming, pagination, clock boundaries, and the cost of scanning. Neither option makes exactly-once delivery happen by declaration.
Delivery evidence and retention rules
Webhook delivery is usually at least once in practice:
- a worker can time out after the receiver commits, then retry;
- a process can restart after sending but before recording success;
- two workers can observe the same due item if the claim expires.
A retry count does not solve those cases because it counts attempts, not effects. The record should make the distinction inspectable long after the worker has gone away: business event, destination, idempotency key, attempt number, claim owner, decision time, outcome class, and the next eligible time.
That is a deliberately boring schema, but it gives reconciliation a stable answer when an operator asks whether a reminder was sent, merely attempted, or made durable at the receiver. Do not fold these facts into an unstructured log line; logs are useful for diagnosis, while the delivery ledger is the evidence used to decide what may be redriven.
The durable key should identify the business event and the intended recipient. Store it with the attempt record, enforce uniqueness at the effect boundary where possible, and make the receiver return success for a safely repeated request. Keep the payload separate from the audit history when retention or privacy rules require different treatment.
The transactional outbox pattern is a useful starting point: write the business change and its notification intent in one transaction, then let a relay publish the job. Publication can still be repeated, so the outbox is not a substitute for idempotency. It closes a different gap: the application does not commit a renewal change while silently losing the instruction to notify.
Here is the shape of a worker. It deliberately acknowledges only after the delivery result and attempt record are durable; the repository and sender are generic so the retry contract stays visible.
type Delivery struct {
ID string
IdempotencyKey string
Payload []byte
Attempt int
}
type Repository interface {
RecordAttempt(id string, attempt int, outcome string) error
MarkDelivered(id string) error
}
type Sender interface {
Send(idempotencyKey string, payload []byte) error
}
func Handle(d Delivery, repo Repository, sender Sender) error {
if err := sender.Send(d.IdempotencyKey, d.Payload); err != nil {
if recordErr := repo.RecordAttempt(d.ID, d.Attempt, "retryable failure"); recordErr != nil {
return recordErr
}
return err
}
return repo.MarkDelivered(d.ID)
}
That code does not claim a universal retry schedule. The schedule belongs in policy: transient transport failures may be retried, while an invalid signature, an invalid payload, or a revoked destination may need review. Your mileage may vary because the retention, compliance, and privacy limits around a logistics message are organization-specific.
Cron implementation: table claims and leases
The database version needs a real state machine. A row should move from pending to claimed with an expiry, from claimed to delivered after a durable success, or back to due after a retryable failure. A claim must be conditional on the current version or lease, so two workers cannot both believe they own the same delivery.
The scan also needs stable pagination. If updating a row changes the ordering column while the scan is in progress, offset pagination can skip or repeat records. Keyset pagination over a stable tuple, such as (next_attempt_at, id), makes the traversal easier to reason about, but it still does not replace the claim condition.
Cron adds a shared latency floor: a job that becomes due just after the sweep may wait for the next sweep. That can be the cheapest and easiest arrangement for low-volume reminders, especially when the service already has a durable retry table and operators already know how to inspect it. It becomes a poor fit when a large backlog causes every run to scan the same population, or when a renewal deadline requires per-job timing.
Do not make a cron callback the delivery worker merely because it has a public URL. The callback can enqueue a bounded redrive request; a private pull worker can then process messages without accepting arbitrary internet traffic. A paused scheduler may not reconstruct every missed trigger, so the database state must remain authoritative and the next run must select due work explicitly.
Evaluation: public HTTPS endpoint latency and operating burden
βCheapestβ and βeasiestβ are architecture questions, not stable properties of an endpoint. The request itself is only one cost. Scan volume, queue operations, storage for attempt evidence, on-call diagnosis, and the time spent proving that duplicate reminders are harmless all belong in the comparison.
| Choice | Good fit | Main trade-off |
|---|---|---|
| Delayed queue with pull workers | Uneven failures, per-job backoff, and low retry latency | Requires consumer idempotency, durable attempt records, and dead-letter operations |
| Cron plus a retry table | Modest volume and deadlines that tolerate a sweep interval | Requires leases, conditional claims, stable pagination, and repeated scans |
| Direct cron-to-HTTPS delivery | A small control-plane task with no private worker to protect | Couples scheduler execution to endpoint availability and makes batch failure harder to isolate |
The public HTTPS endpoint should authenticate the caller, validate the payload before creating work, and expose enough request identifiers for correlation without logging sensitive shipment or customer data. Return a response only after the receiver has made a durable decision; an early success response tells the sender to stop retrying before the business effect is safe.
There is no honest universal winner. Stick with cron when its existing retry table is already auditable and the deadline allows schedule latency. Use a delayed queue when each job needs an independent clock or when backlog isolation is worth the additional consumer operations. In either case, test the timeout-after-commit case before judging the system by its happy-path latency.
A rollout should prove reconciliation before increasing throughput
Start with one reminder class and a small bounded redrive. Emit an event for every attempt, including the stable delivery key, selected route, outcome class, and next eligible time. Alert on age of the oldest due job, as well as process health; a worker that is alive but cannot make progress is still a failed scheduler.
Stop guessing. Run duplicate tests deliberately: deliver the same key twice, expire a claim during a send, restart after the receiver commits, and replay a dead-letter item. Then reconcile the delivery ledger with the business record. The expected result is one business effect and a complete explanation of the repeated attempts.
Keep the boundary clear during migration. A cron task may enqueue work, and a queue worker may redrive a failed item, but only one component should decide whether the business notification is delivered. This prevents a temporary dual path from creating two independent retry clocks.
The practical rule is short: pay for per-job timing when the deadline or audit requirement needs it; accept a cron sweep when its latency and scan cost are explicit, measured, and acceptable. A public HTTPS endpoint is an interface boundary. It is not the reliability model.
Comments
No comments yet. Start the discussion.