DEV Community

NonoDB in practice #2: background jobs with leases and retries, no Redis

The second-most-common reason I reach for Redis is a job queue. Not a high-throughput event bus - just "send this email", "sync this customer", "generate this PDF". A few hundred jobs a day on a single VM.

And yet the default advice is to add BullMQ or Sidekiq, which means adding Redis, which means another process to monitor, another persistence model to reason about, and another backup job nobody wrote.

There is a middle path that fits the same small deployment as the app: keep the jobs in the same database as the data, and make the queue semantics explicit with a single atomic operation.

In this post we build a durable job queue with:

  • atomic claim (no two workers run the same job),
  • fenced leases (a slow worker cannot finish a job after losing it),
  • exponential backoff and a maximum attempt count,
  • idempotent enqueue,
  • at-least-once delivery, and handlers that can live with that.

The database is NonoDB, a single Rust daemon over LMDB. The same pattern works against any document store with an atomic find-and-update, but NonoDB's find_and_modify makes the claim a one-liner.

The job document

Everything is a document in the _jobs collection. There is no hidden queue state.

{
  "_id" : "01J8Z6Q3K4",
  "type" : "send_email",
  "status" : "pending",
  "payload" : { "template" : "invite", "to" : "n**@example.com" },
  "attempts" : 0,
  "max_attempts" : 5,
  "due_at" : 1762800000000,
  "created_at" : 1762799940000,
  "updated_at" : 1762799940000
}

status is one of pending, running, done, failed. due_at is a Unix timestamp in milliseconds; a job is claimable when status == "pending" and due_at <= now. A claim adds lease_id, lease_until, worker_id, and started_at.

The two operations that matter are: Claim - select the oldest due job and move it to running with a new lease, in one transaction. Complete or fail - write the result only if the lease still matches.

The claim

In NonoDB, find_and_modify selects one document, applies the update, and returns the image in the same writer transaction. There is no read-modify-write gap between "is it pending?" and "mark it running".

import uuid
from datetime import datetime, timedelta, timezone

def claim_job(jobs, worker_id: str, lease_seconds: int = 60):
    now = datetime.now(timezone.utc)
    lease_until = now + timedelta(seconds=lease_seconds)
    lease_id = uuid.uuid4().hex
    job = jobs.find_and_modify(
        {
            "status": "pending",
            "due_at": {"$lte": int(now.timestamp() * 1000)},
        },
        {
            "$set": {
                "status": "running",
                "lease_id": lease_id,
                "lease_until": int(lease_until.timestamp() * 1000),
                "worker_id": worker_id,
                "started_at": int(now.timestamp() * 1000),
                "updated_at": int(now.timestamp() * 1000),
            },
            "$inc": {"attempts": 1},
        },
        sort=[("due_at", 1), ("_id", 1)],
        new=True,
    )
    return job

sort=[("due_at", 1), ("_id", 1)] gives FIFO-ish behavior and a deterministic tie-break. With the compound index below, the claim is an index range scan, not a collection scan:

jobs.create_index(fields=["status", "due_at"])
jobs.create_index(fields=["status", "lease_until"])

If claim_job returns None, there is nothing due. Sleep, then poll again. That is the whole scheduler.

The important detail is that this remains correct with N workers. The writer is serialized; the first transaction flips the job to running, and the next worker's filter no longer matches it. No SELECT ... FOR UPDATE SKIP LOCKED, no Lua script, no distributed lock service.

Complete with a fence, not a hope

A lease without fencing is a bug waiting for a slow handler. Suppose worker A claims a job with a 60-second lease, then pauses for 90 seconds - GC, a slow HTTP call, a container freeze. Worker B recovers the expired lease and starts the same job. Now A wakes up and writes "done". The job runs twice and the bookkeeping lies about who finished it.

That is why every claim writes a fresh lease_id and every completion filters on it:

def complete_job(jobs, job: dict, result: dict | None = None) -> bool:
    now = int(datetime.now(timezone.utc).timestamp() * 1000)
    res = jobs.update_result(
        {"_id": job["_id"], "status": "running", "lease_id": job["lease_id"]},
        {
            "$set": {
                "status": "done",
                "result": result or {},
                "finished_at": now,
                "updated_at": now,
            },
            "$unset": {"lease_id": "", "lease_until": ""},
        },
    )
    return res["matched"] == 1

If worker B recovered the lease, it overwrote lease_id. Worker A's update matches zero documents and returns False. A can log "lost lease, discard result" and move on. The unique fence per claim is what makes the lease safe, not the timestamp.

Use find_and_modify with remove=True instead of an update when the job document should disappear on success.

Fail with backoff

A failed handler should not spin. Return the job to pending with a growing due_at, up to max_attempts:

def fail_job(jobs, job: dict, error: str) -> str:
    now = datetime.now(timezone.utc)
    attempt = job["attempts"]
    max_attempts = job.get("max_attempts", 5)
    if attempt >= max_attempts:
        jobs.update(
            {"_id": job["_id"], "lease_id": job["lease_id"]},
            {
                "$set": {
                    "status": "failed",
                    "last_error": error[:2000],
                    "finished_at": int(now.timestamp() * 1000),
                },
                "$unset": {"lease_id": "", "lease_until": ""},
            },
        )
        return "failed"
    delay = min(300, 2 ** attempt)  # 2s, 4s, 8s ... capped at 5 min
    retry_at = now + timedelta(seconds=delay)
    jobs.update(
        {"_id": job["_id"], "lease_id": job["lease_id"]},
        {
            "$set": {
                "status": "pending",
                "due_at": int(retry_at.timestamp() * 1000),
                "last_error": error[:2000],
                "updated_at": int(now.timestamp() * 1000),
            },
            "$unset": {"lease_id": "", "lease_until": ""},
        },
    )
    return "retrying"

Again, the filter includes lease_id. A worker that lost its lease cannot mark someone else's job as failed.

Recover expired leases

Crashed workers leave jobs stuck in running. A periodic sweep moves them back to pending:

def recover_expired(jobs) -> int:
    now = int(datetime.now(timezone.utc).timestamp() * 1000)
    return jobs.update(
        {"status": "running", "lease_until": {"$lt": now}},
        {
            "$set": {"status": "pending", "updated_at": now},
            "$unset": {"lease_id": "", "lease_until": "", "worker_id": ""},
        },
        multi=True,
    )

update(multi=True) is one transaction, so the sweep cannot interleave with a claim that is already in flight. Run it at worker startup and every 30 seconds while running.

Does the recovered job double-run? It may, if the original worker is still alive. That is exactly why handlers must be idempotent. The fence protects the queue state; it cannot un-send an email. At-least-once is the honest contract.

Enqueue idempotently, and atomically with your data

The producer side is a plain insert, but production code usually needs one of two guarantees.

Deduplicate by business key. Use a deterministic _id and treat a duplicate as success:

from nonodb import NonoDBError

def enqueue(jobs, job_id: str, job_type: str, payload: dict):
    try:
        jobs.insert(
            {
                "_id": job_id,
                "type": job_type,
                "status": "pending",
                "payload": payload,
                "attempts": 0,
                "max_attempts": 5,
                "due_at": 0,
                "created_at": 0,
                "updated_at": 0,
            }
        )
    except NonoDBError as exc:
        # A duplicate _id is an already-enqueued job, not an error.
        if exc.code != "duplicate_key":
            raise

A better job_id than a random UUID is f"{job_type}:{entity_id}:{version}". Then "enqueue twice because the request retried" becomes a no-op.

Write the job with the business change. If an order insert must always produce a sync job, a crash between the two writes would silently lose work. NonoDB's write_batch commits both operations or neither:

db.collection("orders").write_batch(
    [
        {
            "op": "insert",
            "coll": "orders",
            "doc": {"_id": "o1", "status": "new", "total": 1200},
        },
        {
            "op": "insert",
            "coll": "_jobs",
            "doc": {
                "_id": "sync_order:o1",
                "type": "sync_order",
                "status": "pending",
                "payload": {"order_id": "o1"},
                "attempts": 0,
                "max_attempts": 5,
                "due_at": 0,
                "created_at": 0,
                "updated_at": 0,
            },
        },
    ]
)

That is a transactional outbox without a separate outbox table and without a CDC pipeline.

Run the worker

The repository ships a small Python helper that implements exactly this contract. A handler module is just functions registered by name:

# jobs_handlers.py
from nonodb_worker import Job

def send_email(job: Job):
    payload = job.payload
    # send it, with an idempotency key derived from job.id
    return {"sent": True}

def register(runner) -> None:
    runner.register("send_email", send_email)
PYTHONPATH=client:workers/python:examples \
NONODB_SOCK=/run/nonodb/nonodb.sock \
python3 -m nonodb_worker \
--module jobs_handlers \
--collection _jobs \
--recover-expired \
--lease-seconds 60 --once

--once claims and runs at most one job, which is handy for a systemd timer if the jobs are infrequent. A long-running worker with a one-second poll is the low-latency option. The queue semantics are the same either way.

What you get, and what you give up

You get:

  • one data directory and one daemon,
  • a queue you can inspect with find, the CLI, or the GUI,
  • atomic claim and fenced completion,
  • retries and backoff,
  • idempotent enqueue and transactional outbox,
  • no Redis persistence tuning.

You give up:

  • Throughput. One writer thread serializes claims. That is fine for typical background workloads, not for a firehose.
  • Exactly-once. Nobody honest offers exactly-once side effects. Assume a handler can run twice and make it idempotent.
  • Priority queues with strict fairness. Add a priority field, sort by priority then due_at, and accept that low-priority jobs can starve. For strict fairness you need more state than one collection.
  • Cron-style recurrence. Store a recurring schedule as a job type that enqueues its next run; keep the schedule document separate from the jobs.

When a dedicated queue is the right call

If you already run Redis for caching, adding BullMQ is cheap. If you need hundreds of thousands of jobs per second, per-job priorities with fair scheduling, or cross-region workers, use a system built for that.

But if the real requirement is "a few durable background tasks on the same box as the app", a collection with two indexes and one atomic operation is a lot less infrastructure than a message broker.

The next article takes the same daemon and adds an audit trail and one-click undo, again without a second database.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.