Two Workers Both Held the 'Only One' Lock. The Clock Was Never Synced.

The Problem

We had a nightly billing-reconciliation job that was only supposed to run once. To enforce that across three worker instances, we did the standard thing: grab a Redis lock with SET lock:reconcile <owner> NX PX 30000 before starting, release it when done, and let the TTL clean up after a crash.

One night it ran twice, at the same time, on two different workers. Same customer batch, processed twice, twice the reconciliation entries written. Nobody deployed anything that day. The lock code hadn't changed in months.

Why It Happens

The lock logic was correct. The assumption underneath it wasn't. PX 30000 means "expire in 30 seconds," and Redis measures that 30 seconds using its own server clock.

Worker A acquires the lock at what its own local clock reads as 10:00:00.000 and plans to finish in 12 seconds, well inside the 30-second budget. But Worker A's local clock - the one it used to decide "I'm still safe, no need to renew" - was running about 9 seconds behind the Redis host's clock, thanks to an NTP daemon that had silently stopped syncing after a container restart weeks earlier.

Redis's clock said the key expired at 30 seconds from its view of "now," which came 9 seconds sooner in wall-clock terms than Worker A expected. The lock expired mid-job. Worker B's health check tried to acquire it, found it free, and started the same reconciliation run. Worker A, still working under its own (wrong) assumption that it had 18 seconds of runway left, kept going too. Two processes, one "exclusive" resource, both convinced they were the only one holding it.

This is the sharp edge of any lease-based lock (Redis, Zookeeper ephemeral nodes, DynamoDB conditional writes with TTL, etc.): the lock's validity is defined by the lock server's clock, but the thing deciding "am I still safe to keep working" is the client's clock. If those two clocks disagree by more than your safety margin, the lock can expire out from under you without either side doing anything wrong locally.

Clock drift like this is rarely dramatic. It's usually single-digit seconds, invisible in normal operation, and only bites when a job's runtime gets close to the lock's TTL - which is exactly the case that matters for a lock.

What to Do About It

Three changes, in order of how much they actually helped:

Auto-renew the lock instead of trusting a fixed TTL

A background thread extends the lease every few seconds (PEXPIRE with a check-and-set on ownership) as long as the job is alive. If the process dies, renewal stops and the lock expires naturally. This removes the "did I estimate my own runtime correctly" problem entirely - you stop trying to predict duration up front.

def renew_lock(redis, key, owner, ttl_ms=30000, interval_s=5):
    while not job_done.is_set():
        # only renew if we still own it
        redis.eval(RENEW_IF_OWNER_SCRIPT, 1, key, owner, ttl_ms)
        time.sleep(interval_s)

Fence the actual write, not just the lock acquisition

Give every lock acquisition a monotonically increasing fencing token. When writing the reconciliation results, include the token, and have the write path reject any token lower than the highest one already seen. Even if two workers both believe they hold the lock briefly, only the one with the newer token's writes land.

Fix the actual clock drift

We found the NTP daemon silently failing on two of three workers - chronyc tracking showed one host 9s off and another 4s off, both marked as "not synchronized" for weeks with no alert on it. We added a monitoring check on NTP sync status itself, not just on the job outcomes.

Notice the order: fencing tokens fix the correctness problem even if clocks are never perfectly synced. Clock monitoring fixes the root cause but won't help you retroactively. You want both - the fence is your safety net, the clock fix is why you rarely need it.

Key Takeaways

• A lease-based lock's TTL is measured by the lock server's clock, not the client's - if they drift apart, "I still have time" can be wrong without any bug in your lock code.
• Prefer renewing a lock in the background over estimating job duration up front and hoping it fits inside the TTL.
• Fencing tokens on the actual write are what make a lock's mutual exclusion durable even when the lock itself briefly fails - don't rely on acquisition alone.
• Monitor NTP sync status as its own signal. A host silently drifting for weeks produces zero symptoms until a job's timing happens to land on the wrong side of it.