Clock Synchronization in Distributed Databases

Part 1: The Physical Clock Problem, Quartz, NTP, and Why Time Lies This is Part 1 of a 5-part series on clock synchronization in distributed databases, covering CockroachDB and Aurora DSQL. - Part 1: The Physical Clock Problem (you are here) - Part 2: Atomic Clocks, The Source of Truth - Part 3: The Hybrid Logical Clock, CockroachDB's Solution - Part 4: Uncertainty Intervals, MVCC, and Transaction Restarts - Part 5: Aurora DSQL, When You Can Assume Atomic Clocks Why Time Matters in Distributed Databases Time seems simple. It isn't, especially in a distributed database. When your database spans multiple nodes across regions, every transaction needs a timestamp. That timestamp drives two critical things: - MVCC versioning: distinguishing before and after versions of a row - Serializable isolation: determining whether two concurrent transactions conflict The challenge is that no two clocks across different machines ever read exactly the same time. They drift. They jump. They disagree. If your database trusts them blindly, you get subtle, hard-to-reproduce consistency violations. This series walks through how CockroachDB and Aurora DSQL tackle this problem, starting from the physics of quartz crystals, through NTP and atomic clocks, and ultimately to bounded uncertainty intervals and transaction restarts. The Physical Clock, What It Actually Is Every server has a quartz crystal oscillator on its motherboard. When electricity passes through the crystal, it vibrates at a predictable frequency. The operating system counts those vibrations to track time. Quartz oscillator vibrates -> OS counts vibrations -> System clock value (what software reads as "current time") Two terms you will see used often refer to the same underlying reality at different levels of abstraction: | Term | What it means | |---|---| | Quartz oscillator | The physical silicon chip that vibrates | | System clock / Physical clock / Wall clock | The OS time value built by counting those vibrations | They are not two separate things. The oscillator produces the ticks. The system clock is the counter that reads those ticks. The Drift Problem Quartz crystals are not perfect. Manufacturing tolerances, temperature changes, CPU load, aging, and power fluctuations all cause the vibration rate to shift slightly over time. Left uncorrected, every server clock will slowly wander away from true time. Drift rate: roughly 1 second per 11 to 12 days Over 30 days without correction: Server A clock: 12:00:02.700 True time: 12:00:00.000 Server C clock: 11:59:57.400 Difference between A and C: over 5 seconds In a single-node system, clock drift rarely causes problems. Everything uses the same clock, so events are still ordered correctly relative to each other. In a distributed system it is a different story. Two nodes writing the same key at the "same time" may disagree by seconds about what that means. A read on Node A may silently miss a write on Node B that happened moments earlier, simply because Node A's clock is behind. NTP, How the World Corrects Drift Network Time Protocol (NTP) is the standard answer to clock drift. Before going into how it works, it is worth being clear about what NTP actually is: NTP is not a clock. It is a synchronization protocol. It does not improve the quality of your quartz oscillator. It periodically asks an external server what time it is, then adjusts your local system clock toward that answer. How NTP Measures the Offset NTP uses a 4-timestamp exchange to estimate how far your clock is from the server's clock: Step 1: Your server sends a request, recording its local time T1 Step 2: NTP server receives it, records its time T2 Step 3: NTP server sends a reply, recording its time T3 Step 4: Your server receives the reply, records its local time T4 Round-trip delay = (T4 - T1) - (T3 - T2) Clock offset = ((T2 - T1) + (T3 - T4)) / 2 Your clock is then adjusted toward the correct time based on the calculated offset. Slewing vs. Stepping, How the Correction Is Applied There are two ways NTP can apply a correction. Slewing is a gradual adjustment: Target correction: +50ms Slew rate: roughly 0.5ms per second Time to complete: about 100 seconds The system clock is nudged slowly and continuously. No abrupt jump. Time always moves forward. Stepping is an abrupt jump: Target correction: +50ms Applied instantly in a single adjustment. The system clock jumps forward (or backward) immediately. For databases, slewing is strongly preferred. An abrupt step that moves the clock backward, even by a few milliseconds, can cause a new write to receive a lower timestamp than an older write on the same node. One more important clarification before moving on: NTP usually does not physically change the quartz crystal’s natural oscillation rate. Instead, it disciplines the software/system clock by adjusting its effective rate and offset on top of that hardware clock source. The NTP Stratum Hierarchy NTP organizes time sources into a hierarchy called stratums. The lower the stratum number, the closer the source is to a physical atomic clock. Stratum 0 Physical atomic clocks, GPS receivers (not directly on the network) | Stratum 1 Servers directly connected to Stratum 0 | Stratum 2 Servers synced from Stratum 1 | Stratum 3 Servers synced from Stratum 2 | Your Server Typically Stratum 3 or 4 Each hop adds network jitter and error Typical accuracy by level: | Source | Typical Accuracy | |---|---| | Stratum 0 (atomic clock / GPS) | Nanoseconds | | Stratum 1 (directly connected) | 1 to 10 microseconds | | Stratum 2 (LAN) | ~1 millisecond | | Stratum 3 to 4 (internet) | 1 to 50 milliseconds | Your server is generally 3 to 4 hops away from an atomic clock. Each hop adds error that cannot be fully recovered downstream. Why NTP Has a Hard Accuracy Ceiling Even with a perfect atomic clock at Stratum 0, NTP cannot give you nanosecond accuracy at your server. Several factors create a floor on NTP error that no amount of tuning can eliminate. Network Jitter Each NTP measurement is a network round-trip. The time that round-trip takes varies unpredictably: Request 1: 5ms round-trip Request 2: 23ms round-trip (router was busy) Request 3: 8ms round-trip NTP averages and filters these, but cannot eliminate the variance. The resulting offset estimate carries residual error from jitter. Path Asymmetry NTP assumes the message takes the same time in each direction. In practice it rarely does: A to B: 5ms B to A: 15ms NTP computes: round-trip = 20ms, assumes one-way = 10ms Actual one-way A to B = 5ms Systematic offset error = 5ms, undetectable and uncorrectable This asymmetry error is silently absorbed into the offset measurement. It is one reason why CockroachDB's default max_offset is a conservative 500ms rather than something close to typical NTP accuracy. Software Timestamping NTP timestamps are added in software, after the OS has processed the packet. OS scheduling delays, interrupt handling, and context switches all introduce noise, typically microseconds to low milliseconds on a busy server. Virtualization In a virtual machine, the guest OS clock depends on the hypervisor. If the VM is paused for migration, or the host is under CPU pressure, the guest clock can drift significantly and NTP corrections can lag behind. VMware and KVM documentation both note that VM clock discipline requires special configuration, making this a real operational concern for any database running in virtual environments. Leap Seconds Occasionally a leap second is inserted into UTC to keep it aligned with Earth's rotation. This is a 1-second discontinuity that looks like a catastrophic clock jump to software that does not handle it carefully. Leap second event: 23:59:59 UTC 23:59:60 UTC (the extra second) 00:00:00 UTC Systems that assume 60 seconds per minute can crash, produce duplicate timestamps, or show impossible log entries. Google and AWS handle this with leap smearing, distributing the extra second gradually over a 24-hour window so the clock advances continuously without any abrupt change. This approach is strongly recommended for CockroachDB deployments. There is one important caveat though: all nodes in a cluster must use the same smearing implementation. Mixing smeared and non-smeared NTP sources introduces a 1-second offset between nodes, which will trigger the cluster's self-protection mechanisms. Where We Stand After Part 1 The problem: Quartz oscillators drift. Every server clock wanders from true time. In a distributed database, nodes disagreeing on time leads directly to consistency violations. NTP's role: NTP periodically corrects the system clock toward an external reference. It corrects the software reading, not the oscillator frequency itself. Slewing (gradual) is safe for databases. Stepping (abrupt) is not. NTP's limits: Network jitter, path asymmetry, software timestamping, virtualization, and leap seconds all create a hard floor on NTP accuracy. Typical accuracy: 1 to 50ms over the internet, ~1ms on a good LAN. The gap: For most applications, millisecond-level accuracy is fine. For a distributed database where transaction ordering depends on timestamps across nodes, that level of uncertainty requires careful engineering to handle correctly. In Part 2, we go to the source of truth: atomic clocks. What they are, how they work, why they are so accurate, the different types that exist, and how AWS built atomic clock infrastructure into every region so that Aurora DSQL can operate with microsecond-level certainty instead of millisecond-level guesses. References Kulkarni, S., Demirbas, M., et al. Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases (2014). http://www.cse.buffalo.edu/tech-reports/2014-04.pdfCockroachDB Blog: Living Without Atomic Clocks: Where CockroachDB and Spanner Diverge. https://www.cockroachlabs.com/blog/living-without-atomic-clocks/CockroachDB Blog: Clock Management in CockroachDB: Good Timekeeping is Key. https://www.cockroachla