One NUL byte made 14,994 signed records verify as tampered

I build Chron, a local audit log for AI coding sessions. Every message is hashed into a chain, so if a row is edited after the fact, verification fails and says which row. Last week I added a path that imports existing Claude Code transcripts. I ran it against my own machine, 33 transcripts, about 70,000 events - and then, instead of spot‑checking one session, verified all of them. The biggest one failed:

Verifying 3847fff0 - 14,994 messages Hash chain ✗ Row 8842: content_hash mismatch - row tampered

Nothing had tampered with anything. The rows were minutes old and nothing but my own import had ever written them. For a tool whose entire job is telling you whether a record was altered, a false positive is worse than a missed detection. A missed detection is a gap. A false positive teaches people to ignore the alarm.

What I assumed, and what it was

My first guess was ordering. Imported transcripts keep the client's line order rather than timestamp order, because 11,591 lines in my corpus carry a timestamp earlier than a line before them. Get that wrong and the chain links up in the wrong sequence. That guess was wrong, and the error message said so.

A prev_hash mismatch means the rows are in the wrong order. A content_hash mismatch means this row's stored hash doesn't match this row's stored fields. Ordering was fine. The row itself disagreed with itself. Which shouldn't be possible.

The code hashes a string and inserts the same string, in the same function, a few lines apart. So I looked at what was actually in the row:

SELECT length ( content ) AS chars , length ( cast ( content AS blob )) AS bytes , instr ( content , char ( 0 )) AS nul_at FROM messages WHERE id = '2d9746d3…';

charsbytesnul_at
298530299
298 characters. 530 bytes. And a NUL at position 299 - past the end of a string that is only 298 long.

The mechanism SQLite stored every one of those 530 bytes. It is perfectly happy to hold a NUL inside a TEXT value. What it will not do is pretend the NUL isn't there when something asks for the value as a string. length() stops at the NUL. So does the driver, handing the value back to JavaScript.

wrote: 530 bytes
read: 298 characters, 308 bytes

So the write hashed 530 bytes of content and the verification hashed the 298 characters that came back. Two different inputs, two different digests, one “row tampered”.

The content was tool output. Some program in some session printed a NUL byte, the way programs sometimes do, and it rode all the way into an evidence store. One byte, in one row, out of 73,526 - enough to invalidate a 14,994‑event chain, because every row after it inherits the break.

The fix that doesn't work

The tempting fix is to strip the NUL inside the hash function. It fails, and it's worth seeing why:

• Write: hash sanitize(content) , storecontent - still 530 bytes, still with the NUL.
• Verify: read back the truncated 298 characters, hash sanitize(truncated) . sanitize doesn't help, because the two sides aren't hashing the same content in the first place. The stored value is the problem. You have to change what goes into the database, not just what goes into the digest.

The rule I ended up writing on the wall: hash what you can read back. Not what you were handed - what the storage layer will give you again later.

Find out whether your stack does this

Rather than fix the one byte I'd found and move on, I went looking for what else wouldn't survive the trip. This is short enough to run against your own database:

import { createClient } from '@libsql/client';
import { createHash } from 'crypto';

const db = createClient({ url: 'file::memory:' });
await db.execute('CREATE TABLE t (id TEXT PRIMARY KEY, body TEXT)');

const sha = s => createHash('sha256').update(s).digest('hex').slice(0, 12);

const cases = {
  'embedded NUL': `before${String.fromCharCode(0)}after`,
  'lone high surrogate': `before${String.fromCharCode(0xd800)}after`,
  'lone low surrogate': `before${String.fromCharCode(0xdc00)}after`,
  'emoji (valid pair)': 'before\u{1f600}after',
  'CRLF and tab': 'before\r\n\tafter',
  'ESC and DEL': `before${String.fromCharCode(0x1b)}${String.fromCharCode(0x7f)}after`,
};

for (const [name, wrote] of Object.entries(cases)) {
  await db.execute({ sql: 'INSERT INTO t VALUES (?, ?)', args: [name, wrote] });
  const read = (await db.execute({ sql: 'SELECT body FROM t WHERE id = ?', args: [name] })).rows[0].body;
  console.log(
    `${read === wrote ? 'ok ' : 'BROKEN'} ${name.padEnd(20)} ` +
    `len ${wrote.length} -> ${read.length} sha ${sha(wrote)} -> ${sha(read)}`
  );
}

On @libsql/client 0.17.3 and Node 23:

BROKEN embedded NUL len 12 -> 6 sha 92e7bd379d66 -> 6db7d803e74f
BROKEN lone high surrogate len 12 -> 12 sha a95d623c5792 -> a95d623c5792
BROKEN lone low surrogate len 12 -> 12 sha a95d623c5792 -> a95d623c5792
ok emoji (valid pair) len 13 -> 13 sha cd9b426a0837 -> cd9b426a0837
ok CRLF and tab len 14 -> 14 sha 51daa3413452 -> 51daa3413452
ok ESC and DEL len 13 -> 13 sha 9cc6db066164 -> 9cc6db066164

Everything else I threw at it survives. Control characters are fine. CRLF is fine. Emoji are fine, as long as the surrogate pair is intact.

The near‑miss in that output

Look at the two surrogate rows again. read === wrote is false - the string that came back is not the one that went in. But the hashes match. I nearly wrote this up as “NUL and lone surrogates both break the chain.” They don't, and the reason is more interesting than if they did.

A lone surrogate is not encodable as UTF‑8. The driver substitutes U+FFFD on the way through. And Node's UTF‑8 encoder performs the identical substitution when you hash the string:

Buffer.from('a\uD800b', 'utf8') -> 61 ef bf bd 62
Buffer.from('a�b', 'utf8') -> 61 ef bf bd 62

Both sides mangle it the same way, so the digests agree and verification passes. The integrity check survives on a coincidence, that two unrelated components happen to implement the same replacement rule. Nothing documents that. Nothing enforces it.

Hash bytes instead of a string, move to a runtime whose encoder throws instead of substituting, and the coincidence stops holding. A load‑bearing accident is still an accident.

So I normalise both, for different reasons. The NUL, because it breaks verification today. The surrogate, because storing content the caller never sent is bad on its own terms, and because I would rather not depend on two encoders agreeing by luck. Both become U+FFFD rather than being deleted. That's the standard marker for a character that couldn't be represented, and it keeps the fact that something was there.

Proving the test would have caught it

Writing a test after fixing a bug tells you nothing until you've watched it fail. I reverted the normalisation and ran the new tests against the old behaviour:

11 of 21 failed, including verification through every write path and a twelve‑row chain.

Put the fix back, all 21 pass. That's the part I'd skip if I were in a hurry, and it's the part that tells you the test is real.

What I'd take from this

Hash what you can read back. Any scheme that hashes content and stores it separately has a serialisation boundary in the middle, and the boundary is allowed to change your data. Content‑addressed stores, signed records, dedup keys, ETags - same shape, same exposure.

A silent substitution is worse than an error. Every layer here behaved reasonably on its own. SQLite stored the bytes. The driver returned a C string. Node's encoder replaced what it couldn't encode. Nothing logged a warning, and the defect surfaced as an accusation against innocent data.

Verify all of it, once. This was one row in 73,526. Any sampling strategy misses it. The only reason I found it before a user did is that I got suspicious and checked everything instead of the one session I'd picked.

And be careful which failures you claim. I was one draft away from publishing a confident, wrong explanation of the surrogate case. The probe was right there; I just hadn't read its output closely enough.

I build Chron, a local audit log for AI coding sessions. Every message is hashed into a chain, so if a row is edited after the fact, verification fails and says which row.