Backend Cron Alternatives: A Guide to Scheduled Scripts with Reliable HTTP Delivery
Short answer: For scheduled scripts that make HTTP calls, choose backend cron alternatives that enqueue durable work; keep retries and duplicate handling outside the timer. The deciding constraint is delivery guarantees, not a bare cron trigger or a low headline price.
A scheduled media cleanup job should enqueue webhook work and make each delivery idempotent; it should not keep retrying inside the cron process. A timer tells you when to start. It does not make an HTTP request durable, ordered, or safe to repeat.
I build CLIs and SDKs for developers, so I measure the first useful call and count every piece of glue. A five-line scheduler that loses a delivery at the first network timeout is not small. It is unfinished.
What does a schedule actually guarantee?
A scheduler can provide a trigger window. The outbound request needs a separate contract. For a media cleanup run, that contract should answer four questions: what event is being delivered, how many attempts are allowed, how is a duplicate recognized, and where does a permanently failing item go?
The simplest useful model is at-least-once delivery with an idempotency key. The sender records an event before sending it. A worker claims the event, sends the request, and marks success only after an acceptable response. If the process dies after the receiver commits but before the sender records success, the next attempt repeats the request. That gap is normal. The receiver must make the repeated operation harmless.
Exactly-once delivery sounds cleaner. Across two systems connected by HTTP, it is usually a naming problem unless both sides share a transaction boundary. I would rather state the duplicate behavior plainly than hide it behind a scheduler label.
type Delivery = {
id : string ;
kind : "media.cleanup.completed" ;
payload : { libraryId : string ; removedCount : number } ;
attempts : number ;
nextAttemptAt : string ;
status : "pending" | "delivered" | "dead" ;
} ;
The id is the deduplication key, not a timestamp and not a hash of fields that may change. Keep it stable through retries. The receiver can store it with the side effect, or reject a previously seen key before doing the side effect again.
The smallest implementation that survives a timeout
The scheduled script should create work, not perform all the work. One test can check enqueueing, another delivery behavior, and a third the dead-letter path. This is less glue than putting retries, logging, and scheduling into one process.
type Queue = { send : ( delivery : Delivery ) => Promise < void > } ;
export async function scheduleCleanup ( queue : Queue , id : string , libraryId : string , removedCount : number , ) : Promise < void > {
await queue . send ( {
id ,
kind : "media.cleanup.completed" ,
payload : { libraryId , removedCount } ,
attempts : 0 ,
nextAttemptAt : new Date ( ) . toISOString ( ) ,
status : "pending" ,
} ) ;
}
The worker needs bounded retries, a deadline, and a durable final state. It should classify failures instead of treating every non-2xx response as the same kind of problem. A malformed payload is not repaired by waiting. A timeout may be.
type DeliveryStore = {
markDelivered : ( id : string ) => Promise < void > ;
reschedule : ( id : string , at : Date , attempts : number ) => Promise < void > ;
moveToDead : ( id : string , reason : string ) => Promise < void > ;
} ;
export async function recordOutcome ( delivery : Delivery , store : DeliveryStore , status : number , ) : Promise < void > {
if ( status >= 200 && status < 300 ) {
await store . markDelivered ( delivery . id ) ;
return ;
}
if ( status >= 400 && status < 500 ) {
await store . moveToDead ( delivery . id , `non-retryable status ${status}` ) ;
return ;
}
const attempts = delivery . attempts + 1 ;
if ( attempts >= 5 ) {
await store . moveToDead ( delivery . id , `retry limit reached at ${status}` ) ;
return ;
}
const delayMs = Math . min ( 60_000 , 2 ** attempts * 1_000 ) ;
await store . reschedule ( delivery . id , new Date ( Date . now ( ) + delayMs ) , attempts ) ;
}
These numbers are policy examples, not universal defaults. The important properties are bounded work and persisted state. A retry count that exists only in process memory resets when the worker restarts. That is an expensive illusion of control.
Should backend cron alternatives run scheduled scripts over HTTP?
A backend cron alternative is often compared by its bill or whether it can issue an HTTP call. Those are quick filters. Delivery semantics decide the architecture.
A Kubernetes CronJob is good at starting a scheduled workload and expressing concurrency policy. It does not, by itself, turn a request into a durable delivery record. The application still needs persistence, retry classification, and an operational home for failures. That can be right when the team already operates the cluster. It is a poor fit if the only requirement is one durable outbound event.
Google Cloud Tasks puts more of the dispatch concern behind a queue-shaped interface. Its value is the separation between task creation and HTTP execution. Receiver idempotency, payload versioning, authentication, and terminal failure remain application choices.
Amazon SQS with a dead-letter queue gives operators a place to inspect messages that exhausted their retry policy. That differs from silently dropping an event and from retrying forever. The dead-letter record should retain the event ID, endpoint identity, attempt history, and stop reason.
Zapier, n8n, and GitHub Actions occupy different layers. Zapier is oriented around configured app-to-app workflows. n8n can be self-hosted and exposes workflow steps. GitHub Actions is tied to repository workflows and suits a scheduled script that belongs to a delivery pipeline. Those distinctions matter more than a checkbox saying βHTTP.β None removes duplicate handling at the receiver.
I would run one failure test before choosing: accept the request at the receiver, kill the worker before its success write, restart it, and inspect the result. If the system cannot explain the duplicate, the integration is not ready for media cleanup.
The failure tests I would run before shipping
- Failure: Connection timeout / Sender state: pending / Required behavior: retry with bounded backoff
- Failure: Receiver returns a validation error / Sender state: pending / Required behavior: move to dead-letter state and alert
- Failure: Receiver commits, sender crashes / Sender state: pending / Required behavior: retry the same idempotency key
- Failure: Worker restarts during backoff / Sender state: pending / Required behavior: recover time from storage
- Failure: Retry limit is exhausted / Sender state: pending / Required behavior: retain payload and reason
Then test the schedule itself. A missed run should be detectable. A long cleanup should not overlap another cleanup unless concurrency is intentional. Clock drift and daylight-saving transitions deserve tests when a schedule uses local time.
The cleanup data may be boring. The clock is not.
Log a correlation ID, delivery ID, attempt number, response status, elapsed time, and final state. Avoid putting the full media payload into logs by default. Metrics should expose pending age, retry count, dead-letter count, and delivery latency. An uptime alert tells me little; an old pending event tells me the user-facing problem.
I benchmark the path that matters: scheduled trigger to durable enqueue, queue-to-request latency, and recovery after a forced crash. I do not benchmark one clean 200 response and call the system reliable.
What I would change at scale
At larger volume, I would separate scheduling, dispatch, and receipt confirmation. I would add payload versioning before the first breaking schema change, rotate endpoint credentials without changing event identity, and cap concurrency per receiver so a backlog does not become a traffic burst.
I would make replay an explicit operation with an audit trail. βRetryβ means the system is still following policy. βReplayβ means an operator chose to send a terminal event again. Mixing those verbs makes incident review harder.
The trade-off is operational weight. A durable queue, a store, metrics, and a dead-letter workflow require more moving parts than a scheduled HTTP request. For a disposable internal script, that may be needless machinery. For a media cleanup that removes stale files, updates an index, and then informs another service, a crash can leave those systems disagreeing about what happened. The next scheduled run may repeat the event, and an operator may replay it again while investigating.
That is why I would spend the extra moving parts on the event boundary when the webhook causes deletion, indexing, billing, or publication; losing the boundary between βstartedβ and βdeliveredβ is the bigger cost.
Duplicates happen. The decision rule is straightforward: choose the lightest scheduler that can create durable work, then put delivery guarantees in the queue and receiver contract. A scheduler is a clock. A reliable webhook pipeline is a state machine.
References
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- https://cloud.google.com/tasks/docs/dual-overview
- https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
- https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule
- https://zapier.com/apps
- https://docs.n8n.io/
Comments
No comments yet. Start the discussion.