Designing a Distributed Job Scheduler That Runs Jobs Exactly Once

The Core Problem

A scheduler has two responsibilities that are easy to confuse: deciding when a job should run, and actually running it. The first is a time problem. The second is an execution problem. Bundling them into one process is what makes single-node cron simple and also what makes it fragile. Once you distribute, you separate them.

The two hard guarantees are:

• A job that is due must eventually run (durability)
• A job should not run twice when you did not ask it to (correctness)

Most naive designs quietly violate one of these the first time a worker crashes.

Key Design Decisions

Store jobs durably with a next-run timestamp. Every job is a row: its schedule (cron expression or interval), its next fire time, its payload, and a status. The scheduler's job is to find rows where next_run is in the past and hand them off. If the store is durable, a scheduler crash loses nothing because the source of truth is the database, not memory.

Poll a time-indexed store instead of holding timers in memory. Holding a million in-memory timers means a restart forgets everything. Instead, index jobs by next_run and periodically query for the due ones. The polling interval trades latency against load: poll every second and jobs fire within a second of their target, at the cost of a query per second.

Separate the dispatcher from the workers with a queue. When a job is due, the dispatcher pushes it onto a queue and workers pull from it. This lets you scale execution independently and absorb bursts. It also means the dispatcher stays lightweight and only cares about timing.

Solve exactly-once with idempotency, not wishful thinking. True exactly-once delivery is impossible in a distributed system, so you aim for at-least-once delivery plus idempotent execution. Give each scheduled run a unique run id. Before a worker executes, it claims the run with a conditional write (compare-and-set the status from pending to running). If the claim fails, another worker already has it. If a worker dies mid-run, a visibility timeout or lease expiry lets the run be reclaimed.

Avoid thundering herds at popular times. Everyone schedules jobs at the top of the hour. If ten thousand jobs all fire at once, downstream systems get hammered. Add jitter to spread them, or shard the time buckets.

Recurring Jobs and Drift

For a repeating job, after a run completes you compute the next fire time from the schedule and update the row. Compute it from the intended time, not from now, or you accumulate drift. If a run is late, decide the policy: skip the missed occurrence or run it immediately. Both are valid, but pick one on purpose.

How the Real Systems Do It

Systems like Quartz, Airflow, and cloud schedulers share this shape: a durable store of scheduled work, a dispatch layer that polls for due items, a queue, and a worker pool that leases and executes with idempotency. Leader election (often via a lock in the database or a coordination service) keeps multiple scheduler instances from dispatching the same job. The database or log is always the source of truth, and workers are treated as disposable. When a worker vanishes, its lease expires and the work flows to another one.

The mental model that carries you through the interview: timing and execution are different problems, durability comes from the store not the process, and exactly-once is really at-least-once plus idempotent claims.

I wrote the full breakdown, with the lease protocol and the schema, here: https://www.systemdesign.academy/interview/design-distributed-job-scheduler