DEV Community 11h ago

What happens after a write? Reworking Squirix's WAL in preview.6

In my first Squirix article, I wrote about why Squirix keeps a strict boundary between the client and the server. Applications use a typed client over gRPC; the server owns cache state, routing, persistence, recovery, and operational endpoints. That boundary makes ownership clear. It does not answer the uncomfortable question underneath it: what exactly happens between accepting a write and telling the client it succeeded?

Preview.6 goes one level deeper into that path. What if the process dies after writing to disk but before sending the response? What survives a torn journal tail? How can a client retry without applying the same mutation twice? Those questions shaped the release: a pipelined binary write-ahead log, tighter snapshot and journal recovery, and idempotent mutation outcomes that survive a restart.

Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during 0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

Follow one write

The basic ordering rule is simple: Append the mutation to the journal before applying it in memory, and do not return success before the required durability boundary. For a normal durable mutation, the path looks like this:

validate and admit the mutation
append a binary WAL record
cross the configured durability boundary
apply the mutation in memory
return success

The order is deliberate. If the server changed memory first and appended to the journal afterward, a crash in between could make an acknowledged write disappear after restart.

Idempotent RPCs take one deliberate variation. Squirix appends the mutation, applies it in memory to produce the response, appends that response as an idempotency outcome, then waits for a single durability boundary covering both records. The response is not returned before that wait completes. The mutation and its retry result share one durability commit, without pretending the network response is atomic with the disk write.

Journal-first ordering avoids acknowledged-but-unrecoverable writes, but it exposes a different case. Suppose the journal becomes durable and the process dies before the response reaches the client. Recovery may replay the mutation even though the client observed only a timeout or broken connection. That outcome is not an ordinary failure. It is commit unknown: the operation may have committed, but the client did not receive a definitive answer.

This is different from a rejection before the journal. If the memory-admission gate rejects a growing entry, Squirix has not appended anything and has not changed cache state. That rejection is definitive. Once durable bytes may exist, the answer has to be more careful.

Moving file I/O off request threads

Preview.6 replaces the previous journal path with a pipelined binary WAL backend. The goal was not simply to make file writes faster. It was to give journal work one owner and make backpressure visible.

Request threads do not independently open files, encode records, and call fsync. They submit append work to a bounded ring. A dedicated background thread named squirix-journal-io owns the journal event loop:

request threads
    |||
    vvv
+-------------------+
| bounded WAL ring  |
+-------------------+
    |
    v
+------------------------+
| single journal thread  |
+------------------------+
    |
    v
binary WAL segments

The journal thread drains accepted work, encodes and batches frames, writes segment bytes, services durability deadlines, performs flushes, and coordinates segment rolls. The queue is bounded because storage can always fall behind incoming traffic. An unbounded queue would turn that mismatch into unbounded memory growth; the ring turns it into visible backpressure instead.

The single writer also gives segment state one clear owner. Offsets, batches, durability state, and roll decisions do not need to be coordinated among arbitrary request threads.

Why the journal is binary

The new WAL writes binary frames rather than passing persistence records through JSON text. That gives the storage path predictable record sizes, explicit field validation, CRC32C checksums, and span-based encoding for the hot parts of the format. It also makes versioning a property of the storage format rather than an accidental consequence of a general-purpose serializer.

This is partly a performance choice, but the main benefit is control. Recovery needs to distinguish a complete record from arbitrary bytes at the end of a file. A framed binary format can say exactly how long a record is and whether its checksum is valid.

The event loop also coalesces encoded records into write batches. Writing bytes and declaring those bytes durable are separate steps, which is where group commit enters the design.

Sharing an fsync with group commit

Flushing every mutation separately gives straightforward semantics, but it also makes each request pay the full durability cost. With group commit enabled, several appends can share one flush:

append A ─┐
append B ─┼── write batch ── fsync ── complete A, B, and C
append C ─┘

Each waiter is completed only after a durability flush covers the relevant appends. The journal thread flushes the current write batch, performs the storage flush, and then completes the captured batch of waiters. If the flush fails, every waiter in that batch fails; no caller is told that partial durability was enough.

The group is bounded by configuration-a maximum wait and a maximum batch size-trading a small, controlled delay for fewer durability flushes. Workloads that prefer the strictest per-write latency can disable group commit.

The important contract is not merely that an fsync happened somewhere. It is that a completed waiter is covered by that durability flush.

The response can still get lost

Even with correct WAL ordering, the server cannot make the network response atomic with the disk write. The failure window is easiest to see as a short sequence:

append operation A
flush operation A
crash before the response reaches the client
recover operation A from the WAL
client retries operation A

Without a stable request identity, step 5 could apply the same logical mutation twice. The risk is greatest for mutations whose effect is not naturally idempotent.

Squirix mutation RPCs therefore carry an opaque operation_id. The server associates the ID with a request fingerprint and the serialized response. A retry with the same ID and fingerprint can return the original result. Reusing the ID for a different request is rejected.

In preview.6, that identity survives restart. Recovery rebuilds successful durable outcomes from operation metadata in the journal and from retained idempotency records in snapshots. The operation ID is also preserved when a receiving node forwards a mutation to the key owner.

client
    |
    | operation_id = 4f...
    v
receiving node
    |
    | same operation_id
    v
owner node
    +--> durable mutation record
    +--> durable outcome metadata

This is not a distributed transaction, and I will not present it as one. It gives a supported mutation a stable identity across forwarding, timeouts, retries, and restarts-exactly the situations where commit unknown becomes operationally important.

Recovery starts by distrusting the files

A snapshot is not valid merely because a file with the right name exists. Squirix reads and validates the complete snapshot into a temporary load result before applying its entries. The startup gate remains closed until snapshot restore and journal replay have both completed.

The current recovery flow is:

Read the manifest.
Locate the referenced snapshot.
Decode cache entries and retained idempotency records into a temporary load result.
Validate the complete snapshot.
Apply the validated snapshot state.
Replay journal records after the snapshot watermark.
Open the startup gate.

manifest
    |
    v
snapshot ── validate all frames ── apply recovered state
    |
    v
last applied sequence N
    |
    v
replay WAL records where sequence > N

The watermark matters. A snapshot represents state through a particular journal sequence. Replaying records at or before that sequence would apply mutations twice; skipping later records would lose committed state. Preview.6 freezes the metadata that describes the snapshot cut so that the entries and their sequence boundary refer to the same logical point.

Snapshot publication follows the usual temporary-file pattern: write and flush a temporary file, publish the completed snapshot, and only then update the manifest reference. A crash during the temporary write must not make an incomplete snapshot authoritative.

If the referenced snapshot is missing, unreadable, truncated, or fails checksum validation, Squirix discards all snapshot-derived state. It falls back to journal-only recovery only when the required journal history is still available. Otherwise recovery fails instead of guessing across a gap.

Preview.6 also fixes the valid fallback path so replay starts from the earliest available segment when there is no snapshot watermark.

A torn tail does not erase the valid prefix

A process can stop halfway through the final WAL frame. Recovery must not interpret those bytes as a complete mutation, but it should not discard every earlier record either.

[valid frame][valid frame][valid frame][partial bytes...]
                                          ^ recovery stops here

Journal replay proceeds through complete, validated records in segment order. During startup, Squirix scans the active segment to the last valid frame boundary and truncates an incomplete tail before reopening it for writes.

This is intentionally conservative: never invent a record from partial bytes; never replay beyond an invalid boundary; preserve the valid committed prefix. The same principle applies to snapshots: do not start applying entries until the complete file has been decoded and validated. A clean journal fallback is easier to reason about than trusting a partly decoded snapshot.

Compaction must preserve retry semantics

Compaction cannot keep only the latest key/value state. Imagine that operation A committed, its result was retained for deduplication, and old journal segments were then compacted away. If the snapshot contained cache values but not the idempotency record, a delayed retry of A could lose its original identity and execute again.

Snapshots therefore include retained idempotency outcomes alongside cache entries. After compaction and restart, Squirix needs to recover both:

cache state + retry identity
durable outcomes

Durability is not only the ability to reconstruct values. It also means reconstructing enough protocol state to handle uncertain retries safely.

What preview.6 changes

The durability work in 0.1.0-preview.6 includes:

a pipelined binary WAL
a dedicated single-writer journal event loop
bounded append backpressure
configurable group commit
journal-only recovery from the earliest available segment when the journal topology is valid
a stable snapshot cut and replay watermark
durable operation outcomes across restart
operation-ID propagation between nodes
retained idempotency records in snapshots
compaction cancellation during shutdown
lower allocation pressure on WAL hot paths

Some changes improve throughput. Most exist because of awkward timing: a crash after a journal flush, a lost response, a truncated final frame, compaction followed by an old retry, or shutdown while persistence work is active. That is where durability engineering lives. The happy path is necessary, but it is rarely the part that keeps me thinking after the code is written.

What preview.6 does not promise

This work makes the durability model easier to reason about, but Squirix is still early software. The on-disk format may change during 0.x. Durability is per node; preview.6 does not add replication or automatic failover. Operations spanning multiple owners are not globally atomic transactions. Serializer compatibility and restart behavior still need to be validated for the mutation paths a workload depends on.

A WAL is not a substitute for replication, backups, or recovery testing. Preview.6 is a stronger durability foundation, not a production-readiness declaration.

Closing thought

Appending bytes is the easy part of a write-ahead log. Deciding what those bytes mean after a crash is the real work. The difficult part is naming every boundary precisely: accepted; appended; flushed; applied in memory; visible to the client; recoverable after restart; safe to retry.

Preview.6 makes those boundaries more explicit. That gives future work-replication, failover, and more advanced cluster behavior-a durability model it can build on instead of work around.

If you work on storage engines or distributed .NET systems, I would be glad to hear where you think the remaining sharp edges are. The project is open source, and feedback on the WAL, recovery semantics, and retry model is especially welcome.

Links

Read on DEV Community ↗ ← Back to News

What happens after a write? Reworking Squirix's WAL in preview.6