Tailscale's 6-Month Hunt for a 16-Year-Old SQLite Bug: A Debugging Playbook for Java Teams

"Our uptime was pretty shaky." That is how Alex Chan, an engineer at Tailscale, opens the postmortem of the strangest database incident his team has ever chased (the writeup, 1,141 points and 217 comments on Hacker News at the time of writing). Over six months, a single bug corrupted shard databases across their control plane 19 separate times. No pattern. No obvious cause. No way to reproduce it on demand. Just "impossible" corruption, recurring, until months of forensics revealed that the fault had been sitting inside SQLite itself for at least 16 years. I am a Senior Software Engineer II at BS23 in Dhaka, and I have spent the last six years building Spring Boot services and, more recently, running my own AI agent infrastructure. Database corruption is the bug class I dread most, because it does not announce itself with a stack trace. It shows up as a checksum error in a backup pipeline, or a transaction that quietly did not persist, or an index that disagrees with the table it points at. Full disclosure: everything in this article comes from Tailscale's published postmortem and the Hacker News thread; I have not touched their codebase. What I can stand behind is the debugging methodology, because it is the same discipline I use when a Spring Boot service starts misbehaving in ways the docs say cannot happen. This postmortem is one of the best debugging case studies I have read this year, and almost all of it transfers directly to Java teams running Postgres, MySQL, or even SQLite inside a Spring Boot app. Here is what happened, how they cracked it, and the checklist I now use when a database says "impossible." The incident: 19 corruptions in six months Tailscale's control plane, the service that coordinates every tailnet, is split into shards. Each shard has an SQLite database holding the configuration data for the tailnets on it, and a single Go process exclusively accesses that database. That single-writer design is exactly how SQLite is meant to be used, which makes what happened next so unsettling. Their backup pipeline takes a complete snapshot of the database every few minutes and uploads the whole file to an S3 bucket. It ran without incident from early 2023 until August last year, when a data pipeline reading those S3 backups reported an error. The team ran SQLite's PRAGMA integrity_check against the backup and confirmed it: the database was corrupted. - The first instinct was to treat it as a one-off. SQLite corruption is possible but rare, and not something you expect in normal operation. They repaired the affected database and investigated the cause. They found nothing. - Then it happened again. And again. In total, 19 separate corruption incidents over six months before the underlying bug was resolved. - Each incident meant real downtime. When corruption was detected, the control plane process on that shard had to stop while the database was repaired or restored. Early incidents took over an hour of downtime per shard. Tailnets on that shard lost the admin console and the API during recovery, and devices coming online could not learn about the network until the control plane was back. The data itself was never catastrophic: these databases hold tailnet metadata, not private keys or traffic. But a company whose entire product is reliable connectivity cannot afford repeated status-page events, and every incident eroded trust even when most shards were unaffected. Why it resisted every standard technique Here is the part every developer who has ever debugged a heisenbug will recognize. The bug did not want to be found. - There were no recent changes to blame. Nobody had touched the low-level SQLite interaction code for years. A fine-toothed review found nothing. - There were no common factors. Corruption was not tied to a single shard, customer, tailnet feature, time of day, or load level. - It could not be reproduced synthetically. With no reliable trigger, the only option was to deploy passive forensic telemetry into the live environment and wait for the next corruption to happen. To make it worse, the corruption had no schedule. Incidents could be hours apart or weeks apart. There was a six-week period between October and December with zero incidents, and then they returned "as an unwelcome Christmas present." The team made two moves that I think are the real story of this postmortem. First, they bought a professional support contract from the SQLite developers (their support page), which gave them direct access to the people who wrote the database. Second, they systematically mapped out theories, ruled them out with data, and refused to guess. The candidate theories included broken POSIX locks on close() , mismanaging memory owned by SQLite, and using SQLite from multiple threads while thread safety was disabled. Every incident produced more data, and every theory was eliminated one by one. The clue: a transaction that vanished While the root cause was still unknown, the team had a live platform to keep running. They automated hard-stops on corruption, deployed a backup monitor that continuously ran PRAGMA integrity_check over backups, and improved runbooks. That cut recovery time to under an hour. Then they built something clever: a transaction logging pipeline. The idea was simple. Stream every SQL statement that modifies the database to a separate log file. Because SQLite is a single-writer database with serializable transactions, the transaction history is completely linear and deterministic. Replaying those transactions against the last known-good backup should restore the database to its most recent state, safely bypassing the corrupted pages. This is a technique I want to call out for Java teams, because it only works in single-writer systems. It would not hold in a multi-writer Postgres or MySQL deployment. The pipeline worked. And then it produced the clue that cracked the case. In two incidents, the transaction logs failed to replay cleanly. Data written and committed by one transaction was inexplicably invisible to later transactions. A write had vanished into thin air without raising any error. That should be impossible. In a serializable, single-writer database, a committed write cannot disappear. The fact that it did pointed directly at the checkpointing layer, the only component with enough concurrency to hide a lost write. The bug: a 16-year-old race in the WAL To understand the fix, you need the two-file model SQLite uses with Write-Ahead Logging. The database file is a series of pages. When you update data, new pages are not written directly to the database file. They go into the WAL file first, for performance and concurrency. At some point, a checkpoint copies those pages back into the main database file. In most deployments SQLite decides when to checkpoint on its own. Tailscale, however, takes manual control of the checkpoint process so it can run fast, consistent backups. That non-standard choice matters, because it is what exposed them to the bug. During corruption incidents, their metrics showed SQLite reporting that it copied more pages from the WAL file than actually existed there. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong. The SQLite developers had been building a new debugging tool for exactly this layer: a wrapper around the virtual filesystem that writes additional tracing logs about changes to the database. It is called the tmstmpvfs shim, and the source lives in the SQLite public repository. Tailscale deployed it into production and waited. They did not have to wait long. The logs exposed a rare data race between a checkpoint and a write transaction. If a write occurs at a specific moment during a checkpoint, the checkpoint gets confused: it thinks some pages have been copied from the WAL into the main database file, but they have not. Those pages are never written, and the data is permanently lost. Worse, other pages that reference the missing ones, such as an index, do get written. The database file becomes structurally corrupt, which is exactly what PRAGMA integrity_check was detecting all along. The SQLite developers named it the WAL-Reset bug, and they estimate it had been in SQLite for at least 16 years. It survived that long because it was rare enough that they had to add code to deliberately trigger it in their test environment. The fix adds a check to the checkpointing function that detects when the WAL has been reset by another thread. The false alarm that nearly derailed the fix The fix shipped as SQLite 3.52.0, and Tailscale rolled it out carefully: a few canary shards first, then the rest of the control plane. Then their backup monitor promptly turned red, reporting corruption in 13 different databases. This is my favorite part of the whole story, because it is a lesson in how even a correct fix can be misread. The 13 databases had not suffered real corruption. SQLite 3.52.0 also contained an optimization that subtly changed rounding behavior for text-to-floating-point conversions, and Tailscale stored high-precision timestamps as text, converting them to floating point in a VIRTUAL generated column. Stale expression indexes, where an index on a computed value no longer matches after the computation changes, get reported as corruption by PRAGMA integrity_check . The canary shards simply did not have any timestamps that triggered the changed rounding, so the phased rollout missed it. - The SQLite developers withdrew 3.52.0 entirely and published 3.51.3 containing only the WAL-Reset fix. - Tailscale fixed its side by storing timestamps as integer seconds instead of text, since text-to-integer conversions are unambiguous. - SQLite added an automated self-healing index feature in 3.53.0 to prevent the stale expression index problem permanently. Then came the most disciplined part of the investigation. An absence of corruption is not proof of a fix, because the team had already lived throug