DEV Community 1h ago

When the disk fills up: pressure and tenant isolation in Squirix preview.7

In my preview.6 article, I wrote about what happens after a write is accepted: binary WAL frames, group commit, crash recovery, and durable retries. That work made the durability boundaries easier to name. It does not answer the next uncomfortable question: what happens when the node should stop accepting more durable work? A write-ahead log that never refuses appends will eventually fill the disk. A shared concurrency gate that never distinguishes callers will let one noisy tenant starve everyone else. From the outside, both failures can look like “the cache is down,” even while the process is still alive.

Those questions shaped 0.1.0-preview.7: journal disk quotas that reject durable writes with a stable error, and per-principal backpressure keyed by JWT subject (or connection id when there is no subject). Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during 0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

The failure mode we wanted to avoid

Imagine a node that keeps appending journal segments until the volume is full. Depending on the OS and I/O path, the next write might throw, hang, or leave a torn tail. Operators then see process crashes, opaque I/O exceptions, and readiness flaps that take the node out of load balancers - exactly when health and metrics are most useful.

The contract we want instead is boring and explicit:

journal on-disk size approaches configured cap
|
v durable append that would exceed the cap is rejected
|
v client gets JOURNAL_DISK_QUOTA (HTTP 429 / gRPC ResourceExhausted)
|
v /health/ready stays healthy
  /health/ready/details exposes journalDisk pressure

Rejection is definitive for that attempt. The process stays up. Readiness stays scrapeable so operators can reclaim space or raise the limit without first fighting a dead host. That is different from the “commit unknown” case in preview.6. There, durable bytes may already exist and the client only saw a timeout. Here, the server refused the append before it crossed the durability boundary. Clients should not invent success; they should back off until operators reclaim headroom or raise the cap.

A hard cap with soft observability

Pipelined journal settings already expose JournalMaxTotalBytesMb - a hard on-disk size for journal segments, not snapshots or manifests. Preview.7 maps that cap into controlled write rejection. The soft high-water for readiness details is fixed at 80% of the hard limit. Soft high-water is observability only: writes that still fit under the hard cap continue. Hard limit is the observed critical state when usage reaches the configured maximum.

0% -------- 80% (high) -------- 100% (critical / hard cap)
|           |                   |
| writes OK | writes OK if fit  | durable oversize append rejected
|           | details show high | JOURNAL_DISK_QUOTA

A few details matter in practice:

The quota covers journal segments only.
Rejected durable writes do not crash the process.
/health/ready remains healthy so the node can still expose pressure state.
/health/ready/details includes journalDisk with state, maxBytes, usedBytes, highWaterBytes, and writeRejectionActive.

The operator path is intentionally unexciting: confirm pressure, trigger or wait for snapshot plus compaction/retention, raise JournalMaxTotalBytesMb only after confirming disk capacity, and treat JOURNAL_DISK_QUOTA like other capacity rejections.

Why readiness stays healthy under quota

It is tempting to mark a node not-ready when the journal is full. That can remove it from traffic, but it also removes the cheapest way to inspect why durable writes stopped. Preview.7 keeps /health/ready healthy and puts pressure into details:

load balancer / orchestrator
|
| GET /health/ready --> 200 (still ready)
|
| GET /health/ready/details
v
journalDisk.state = critical
writeRejectionActive = true

Clients already receive 429 / ResourceExhausted. That is the write-path signal. Readiness is the operator-path signal. Mixing them turns every capacity event into a membership event.

Backpressure is not memory pressure

Squirix already separates several admission policies. Preview.7 does not merge them; it makes one of them fairer under multi-tenant load.

Transport limits still protect auth, payload size, and deadlines.
Memory pressure still admits work based on estimated cache size.
Journal disk quota still protects on-disk WAL growth.
Runtime backpressure protects concurrent cache operations, queues, slowdown, and optional rate limits.

Runtime backpressure sits after validation and before memory admission. A rejection there happens before logical cache operations enter memory admission or clustered/local paths, so rejected work does not append journal records, mutate local memory, update memory accounting, or record idempotency outcomes. That is the same “fail closed before durability” idea as memory admission - applied to concurrency and rate, not to estimated bytes.

Isolating callers by principal

Global in-flight and queue limits protect the node. They do not stop one authenticated client from consuming the whole budget. Preview.7 resolves a backpressure client id for each cache operation and applies per-client concurrency and rate limits against that id:

request
|
+--> JWT sub / NameIdentifier? -- yes --> jwt:{subject}
|
+--> HttpContext present? ------- yes --> conn:{connectionId}
|
+---------------------------------------> runtime

Authenticated external callers with a subject land in jwt:{subject}.
Anonymous or subject-less HTTP callers land in conn:{connectionId}.
In-process callers without an HttpContext share one runtime bucket.

v0.1 external auth is JWT-only. Inter-node cluster forwarding uses mTLS on the internal listener and typically lands in the conn: or runtime bucket rather than a shared external JWT subject. That is deliberate: external tenant isolation should not treat owner-forwarded RPCs as one giant client. Anonymous loopback clients on distinct TCP connections also do not share one JWT principal bucket. Without a principal, connection identity is the best stable key the host has.

What preview.7 changes

The pressure work in 0.1.0-preview.7 includes:

Mapping the journal total-size hard cap to controlled durable write rejection (JOURNAL_DISK_QUOTA).
Exposing soft high-water (80%) on readiness details without blocking writes that still fit.
Keeping readiness healthy under quota so pressure remains observable.
Keying per-client backpressure off JWT subject, then connection id, then the shared runtime bucket.
Tightening path validation and symlink helpers.
Clearing analyzer/Sonar debt on client and server hot paths.

Some of that is housekeeping. Most of it exists because awkward timing is not only about crashes. A full volume, a noisy neighbor, or a readiness flap at the wrong moment can erase the operational story as thoroughly as a torn WAL frame.

What preview.7 does not promise

This work makes overload and disk exhaustion easier to reason about, but Squirix is still early software. Quotas are per node and cover journal segments on that node; there is no cluster-wide disk accountant. In-process callers share runtime, and internal mTLS paths are not the same as external JWT tenants. Quota buys a clear error and time to act; it is not a substitute for capacity planning, compaction, backups, or recovery testing. Preview.7 is stronger operator-facing pressure control on top of the preview.6 durability foundation, not a production-readiness declaration.

Closing thought

Durability work asks: what survives a crash? Pressure work asks: what happens when survival is no longer free? A journal that always accepts appends will eventually lose the ability to flush cleanly. A concurrency gate that treats every caller as interchangeable will eventually amplify a noisy neighbor into a whole-node outage. The difficult part is naming the refusal precisely: rejected before durability; rejected for this principal only; still ready enough to inspect; safe for clients to treat as definitive for that attempt.

Preview.7 makes those refusals more explicit. That gives operators a capacity policy instead of unexplained I/O chaos, and it gives future multi-tenant work a clearer place to hang fairness without rewriting the WAL story. If you operate multi-tenant .NET services or care about cache overload semantics, I would be glad to hear where this model still surprises you. Feedback on quota versus readiness, JWT versus connection isolation, and client reactions to JOURNAL_DISK_QUOTA is especially welcome.

Links

dotnet add package squirix --version 0.1.0-preview.7
dotnet add package squirix.server --version 0.1.0-preview.7

Read on DEV Community ↗ ← Back to News

When the disk fills up: pressure and tenant isolation in Squirix preview.7