Fill a WAL Volume on Purpose Before ENOSPC Turns Your Health Check Green

A service reports healthy because its process is alive, while WAL writes fail with ENOSPC. Log rotation also fails, retries amplify load, and deleting one file does not prove durable writes resumed.

Bounded Local Fault Injection

mkdir -p ./disk-fixture
dd if=/dev/zero of=./disk-fixture/filler bs=1M count=900

Run only inside a disposable volume with a declared size. Drive a fixed write workload and record free bytes, successful writes, failed writes by error class, WAL/checkpoint age, retry count, and health state. Never run this against a workstation root or shared production volume.

Thresholds and Actions

Readiness Probe Design

A useful readiness probe performs a bounded durable-write check or observes a recent successful checkpoint; a process-only liveness probe should remain separate.

inject -> ENOSPC -> readiness false -> bounded cleanup -> integrity/checkpoint -> test write -> readiness true

Test log rotation failure, deletion of an open file that does not release space, and a restart before checkpoint. Cleanup must remove the fixture, confirm file descriptors release blocks, run integrity checks, and prove a new write survives restart.

This local container drill does not model cloud-volume burst credits, remote filesystems, or every database WAL.

Which signal should remove your instance from service first: free bytes, checkpoint age, or the first failed durable write?