Free Workers Don't Ship Free Disk: Cap Artifact Retention First

At 03:12 the alert fired on queue age. Not on errors, not on latency. The batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts. Nobody had set a retention policy.

Three days of runs had filled the worker's disk. Writes started failing. The consumer requeued failed jobs. Tokens were spent again on work already paid for once. The evidence said storage. The dashboard said AI capacity. Which action do you take first?

Separate what you saw from what you inferred

Observed: OSError: [Errno 28] No space left on device, then a growing retry count.

Observed: p95 queue age crossed 15 minutes while model latency stayed flat.

Inferred: the model got slower. That inference was wrong in this incident, and it usually is. Disk pressure looks like throughput pressure from the outside. Retries hide the real cause inside the token bill.

Treat the retention budget as part of the workload spec

A free worker gives you compute. It does not give you unbounded storage. Write the budget down before you enqueue anything:

required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes

reserve_bytes = total_disk_bytes * reserve_ratio

Declared workload for the reference run below:

• queue_depth = 4000
• avg_artifact_bytes = 2_000_000 (JSON result, stdout log, preview)
• safety_factor = 1.3 (variance between jobs)
• total_disk_bytes = 20 GB, reserve_ratio = 0.2

That is 10.4 GB of artifacts against a 4 GB reserve. Tight, not impossible. Without the reserve, it fails quietly at the end of the batch.

Pre-flight admission check

Reject the batch before it starts, not after job 3,400.

# admit.py - refuse a batch the disk cannot finish
import shutil
def admit(store, queue_depth, avg_bytes, reserve_ratio=0.20, safety=1.3):
    total, used, free = shutil.disk_usage(store)
    need = int(queue_depth * avg_bytes * safety)
    reserve = int(total * reserve_ratio)
    verdict = (free - need) >= reserve
    return verdict, {
        "free_bytes": free,
        "need_bytes": need,
        "reserve_bytes": reserve,
        "headroom_bytes": free - need - reserve,
    }

if __name__ == "__main__":
    ok, report = admit("/srv/artifacts", 4000, 2_000_000)
    print(report)
    raise SystemExit(0 if ok else 75) # 75 = EX_TEMPFAIL

Exit code 75 tells your scheduler this is retryable-later, not a bug. Do not enqueue on False.

Check the same number from the shell before you trust it:

df -B1 --output=source,avail,pcent /srv/artifacts
find /srv/artifacts -type f -newermt '-24 hours' -printf '%s\n' \
  | awk '{s+=$1; n++} END {printf "files=%d avg_bytes=%d\n", n, s/n}'

A pruner you can run in dry-run first

Default is dry-run. Deletion requires --apply. Pinned artifacts survive every pass.

#!/usr/bin/env python3
# retain.py - cap artifact store before the worker disk fills
import argparse, os, shutil
from pathlib import Path

def pinned(store):
    p = Path(store) / ".pinned"
    if not p.exists():
        return set()
    return {line.strip() for line in p.read_text().splitlines() if line.strip()}

def candidates(store):
    keep = pinned(store)
    out = []
    for entry in Path(store).iterdir():
        if not entry.is_dir() or entry.name in keep:
            continue
        size = sum(f.stat().st_size for f in entry.rglob("*") if f.is_file())
        out.append((entry.stat().st_mtime, entry, size))
    return sorted(out) # oldest first

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--store", required=True)
    ap.add_argument("--max-bytes", type=int, required=True)
    ap.add_argument("--keep-newest", type=int, default=50)
    ap.add_argument("--apply", action="store_true")
    a = ap.parse_args()
    items = candidates(a.store)
    protected = items[-a.keep_newest:]
    used = sum(size for _, _, size in items)
    freed = 0
    for _, path, size in items:
        if used - freed <= a.max_bytes:
            break
        if (path.stat().st_mtime, path, size) in protected:
            continue
        print(f"{'rm' if a.apply else 'would rm'} {path} {size}")
        if a.apply:
            shutil.rmtree(path, ignore_errors=True)
        freed += size
    print(f"used={used} freed={freed} target={a.max_bytes} apply={a.apply}")

if __name__ == "__main__":
    main()

Run it on a timer, not in your head:

python3 retain.py --store /srv/artifacts --max-bytes 12000000000 --keep-newest 50 # inspect, then arm it
python3 retain.py --store /srv/artifacts --max-bytes 12000000000 --apply

Failure drill: fill the disk on purpose

Do this on a scratch worker before production finds it for you.

dd if=/dev/zero of=/srv/artifacts/fill bs=1M count=3000 status=progress

Run admit.py. Confirm it returns exit 75 once headroom goes negative. Submit a 200-job batch with admission disabled. Confirm the first write error appears. Confirm the consumer requeues, and watch the token counter climb without a new artifact. Delete fill, run retain.py --apply, confirm the store drops under the cap. Re-enable admission and confirm the batch starts.

Step 4 is the one to instrument. Emit these fields per job:

• job_id, attempt, artifact_bytes, write_ok
• disk_avail_bytes at start and end
• tokens_in, tokens_out, tokens_wasted_on_retry

If tokens_wasted_on_retry is non-zero, storage is charging you twice.

What the ledger shows

Arithmetic on the declared inputs above, not measurements from your cluster:

• Artifacts at 2 MB per 1,000 jobs: 2 GB, grows linearly with retention window
• Retry tokens after write failure: 1x generation cost, artifacts lost, model output not cached
• Pruner CPU seconds: rmtree on local disk
• Object storage offload per GB-month: check egress before assuming cheap

Expected output shape from admit.py on a healthy worker:

{'free_bytes': 14800000000, 'need_bytes': 10400000000, 'reserve_bytes': 4000000000, 'headroom_bytes': 400000000}

A 400 KB headroom is a warning, not a pass. Raise safety_factor or split the batch.

When the free worker is the wrong bet

Stop and re-plan if any of these hold:

• Artifacts must be retained beyond a week for audit or replay.
• Average artifact size exceeds 5% of total disk per batch.
• A single worker's throughput cannot drain the queue inside your deadline.
• Artifacts are inputs to downstream training or billing.

In those cases, write artifacts to durable object storage and keep only recent pointers on the worker. Pay for storage, not for retries.

Where the free tier fits

I use MonkeyCode's free model access and free server option for batch experimentation. Both are operator-supplied availability claims, not measured guarantees here. Treat the disk as a budget you declare yourself. The workflow above does not depend on them. Run it on any worker with a local disk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

This is a single-node pattern. It does not give you replication, cross-zone durability, or shared storage across workers.

Skip it if you need multi-writer concurrency on the same store. Skip it if your compliance team requires immutable retention. Skip it if your artifacts are larger than your reserve can absorb.

Do not use the pruner as your only backup. It deletes. That is the point. If the arithmetic above says the disk cannot finish the batch, then free capacity is the wrong bet for that batch.

Cleanup and rollback

Before arming the timer, record the current state:

du -sb /srv/artifacts > /srv/artifacts.size.before
find /srv/artifacts -maxdepth 1 -type d | wc -l > /srv/artifacts.count.before

Rollback is two commands:

systemctl disable --now artifact-retain.timer
rm -f /srv/artifacts/fill

Then restore what you deleted from your offload bucket. If you have no offload bucket, you did not have a rollback. Start with a dry-run on one day of artifacts. Compare the file list against what you expected to keep. Only then arm --apply.

If you want a cheap place to rehearse this drill, the free server option is a reasonable scratch target. Keep the pin list short and the blast radius smaller than your resume.