Why CockroachDB refused writes to a healthy 155 KiB row

A worksheet in prod stopped saving. The pod was healthy. 404 MiB of a 2 GiB limit, 655m of 1500m, no restarts. I didn't believe that, so I went and looked at the database too. Three active queries cluster-wide, 12% CPU, all three nodes live. Idle. Nothing was exhausted, nothing had crashed, and the service still couldn't write. The software It's a collaborative editor. Teachers build worksheets, whiteboards and lesson plans, and several people can have the same document open at once. Every document is a CRDT, built on Loro. The browser holds a replica and applies edits to it locally, then pushes them over a WebSocket to a sync server. The server keeps its own copy of each open document in memory, merges whatever arrives into it, and writes the result to CockroachDB v25.x. That last step is the one that matters here. Persisting a document means exporting the entire Loro doc as a snapshot and writing it into a single BYTEA column, on a single row. Not an append-only log of updates, which is the usual way to store a CRDT. The whole document, on every save. The document that stopped saving was 155 KiB. Its range was 1 GiB. A wild goose chase to find the root cause The red herring: the same service had an unrelated CPU problem running that day, readiness probes flapping, the node pegged, hundreds of timeout errors in the logs. I went through all of it. Every bit real, none of it connected to this. Two separate problems on one service on the same day, and the louder one wasn't the one refusing writes. A second false trail: I noticed payload sizes varied a lot from one document to the next and read that as clients sending incremental deltas, which would make the write volume real edits. That was wrong. Varying payload size doesn't imply a delta. A CRDT snapshot of a changing document is a different size every time, the same way two zip files of slightly different inputs come out different sizes. The check that settles it is dividing the payload by the stored snapshot: | document type | writes per resource | payload ÷ snapshot | |---|---|---| | worksheet | 995 | 0.81 | | lesson plan | 57 | 0.94 | | whiteboard | 5.9 | 0.87 | | text document | 18 | 0.82 | Near 1.0 means the client sent as many bytes as the entire stored document. Every document type was doing it. Worksheets weren't doing anything different in kind. They were doing it 169 times more often than whiteboards, and that was the whole difference between a wasteful system and a broken one. How to wedge a CockroachDB range The error was in the logs the whole time, buried at a much lower volume than the noise. split failed while applying backpressure to Put [/Table/111/60/"..."/0] on range r725: could not find valid split key Four things had to be true at once for that, each one reasonable on its own. - CockroachDB is MVCC (Multiversion Concurrency Control), so a write never overwrites anything. Every write stores a new copy of the row under the same key at a new timestamp, and the previous copies stay exactly where they are. The key in the storage engine isn't the row; it's the row plus a timestamp. That's what lets a transaction read a consistent view of the database without locking the rows it reads. A transaction reading at timestamp T sees the newest committed version at or below T of every key it touches. CockroachDB runs SERIALIZABLE by default and there is more machinery than that behind it, since reads leave marks in the timestamp cache that push later writers, and a read that meets an unresolved intent below its own timestamp has to wait on it. But keeping every committed version around is what the rest is built on top of. It's also what AS OF SYSTEM TIME , follower reads and incremental backups are built on. All three are reads at an older timestamp, and they only work if the data as of that timestamp is still on disk. So old versions can't be dropped at write time. Something has to guarantee they're still there for anyone reading in the past. They get collected later by the MVCC GC queue, once they're older than gc.ttlseconds , which was four hours here. Which means the storage a row occupies isn't its size. It's its size multiplied by how many times you wrote it in the last four hours. The whole row is one key. CockroachDB stores a row as one key per column family, and this table never defined any beyond the default, so every column sits in the same one. One document, one key, however large the snapshot gets. A split has to cut between two keys. Ranges are kept under range_max_bytes by splitting, and a split picks a key and cuts the keyspace there: everything below goes to one range, everything above to the other. If every byte in a range belongs to one key and the copies differ only by timestamp, there's nowhere to put the boundary. They can't be separated anyway, because the range is what serves reads of that key at any timestamp, so all of them have to live together.The client was pushing every 2.2 seconds, whether or not anything had changed. Here's what the range actually looked like: keys 1 versions 6,766 val_bytes 1024.02 MiB live 0.151 MiB One key. Nearly seven thousand copies of it. A gigabyte of stored versions against 155 KiB of actual row. 0.47 writes per second against a 14,400 second GC window predicts 6,768 versions. There were 6,766. The range was holding exactly one GC window of writes, which is where this stopped being a mystery and became arithmetic. I liked that part a lot. Nothing was queued or deferred to get there, which is worth being explicit about. Every one of those writes applied immediately: proposed, replicated, committed, visible to the next read. The range grew because that is what a range does when you write to it. Splitting is not part of the write path. Splitting happens on the split queue. Each store walks its replicas on a timer, reads their size straight off the MVCC stats it already maintains, and queues anything over range_max_bytes . That's deliberately asynchronous, because a split isn't a local operation. It's a distributed transaction that carves the keyspace in two, writes a new range descriptor, and updates the meta ranges that tell the rest of the cluster where keys live. You don't want that on the hot path of a Put . So there's always a gap between "this range is too big" and "this range has been split", and under normal load, the queue closes it in seconds. Backpressure is what stops a range from outrunning the queue when it doesn't. At twice range_max_bytes , the KV layer stops letting writes into a range with a split pending: range_max_bytes 536,870,912 (512 MiB) backpressure at 1,073,741,824 r725 1,073,844,534 It doesn't reject them outright. It holds the batch, waiting for the range to come back under the threshold, and the write fails only when the request runs out of time. That distinction is why the failure surfaced to us as persist timeouts rather than as a clean error, and it's the whole design assumption: the split you're waiting on is going to happen. 100 KiB over the line. And the split was never going to happen. I sampled the version count twice, 25 seconds apart, to be sure writes were genuinely frozen rather than merely slow. 6,766 both times. While wedged, the range produced about 390 log lines every 15 minutes, continuously, because failed persists retried with no backoff. Which leaves GC as the only thing that could end it, and GC runs on a queue too, with the same asynchronous, scored shape as the split queue. Each replica tracks a statistic called gc_bytes_age , the volume of collectable garbage multiplied by how long it's been collectable, and the queue prioritises by that rather than by raw size. When it gets to a range it computes a threshold of now - gc.ttlseconds , drops every version older than that, and advances the range's own GC threshold so that later reads below it are refused rather than served wrong. Two things follow from that shape. GC can never reach anything inside the TTL window, so during a wedge a range can only shed what has already aged past it. And because the queue is scored and periodic rather than continuous, recovery begins when the queue reaches the range, not when the first version becomes collectable. The second of those is the part I can't fully account for. Getting back under the line needed almost nothing, since the range was sitting 100 KiB over a 1024 MiB threshold, and yet writes stayed refused for 75 to 90 minutes every time. Aging alone doesn't explain a gap that size, so what dominates it has to be when the GC queue got round to the range. I never pinned that down more precisely, and the incident was resolved before it mattered enough to. The cycle itself is legible enough without it. Roughly two hours of rewriting to rebuild a gigabyte, then the wedge, then GC clears it and it starts over. Five times across two days, always the same row. And gc.ttlseconds is a floor on retention rather than a target. Retained bytes are write rate times version size times that window, and nothing in the system pushes back on the product. Raft never failed in any of this. No quorum loss, no elections, nothing. But it sits underneath every part of it, and it's the reason the size limit exists at all. A range isn't a storage bucket. It's a Raft group: three replicas by default, one of them holding the lease. Every write to that row was a Raft proposal, which the leaseholder proposed, a quorum accepted, and each replica then applied to its own copy. So those 6,766 versions weren't 6,766 disk writes. They were 6,766 rounds of distributed consensus, each shipping a full 155 KiB snapshot across the network, and the gigabyte existed three times over, once per replica. Size matters to Raft in two more places. A replica that falls far enough behind can't be caught up from the log, because the leader has already truncated the entries it would need, so it gets sent a Raft snapshot instead: the entire range, over the network. Same story when a node is decommissioned and it