You Probably Don't Need Redis: Put the Job Queue in Your SQLite File

The Hidden Bug in a Separate Queue

Picture the normal flow. A user places an order, so you do two things: write the order to your database, and push a "send confirmation email" job onto Redis.

INSERT order into the database ← step 1
LPUSH job onto Redis            ← step 2

Those are two separate systems, so they can't commit together. If the process crashes between step 1 and step 2, you have an order with no email job - the customer is charged and never hears from you. Flip the order of operations and you get the opposite: an email job with no order behind it. There is no arrangement of two independent datastores that makes those two writes atomic.

This is the dual-write problem, and the usual fix is elaborate - a "transactional outbox," change-data-capture, two-phase commit. But notice what the fix is really doing: it's trying to get the queue and the business data to commit together. If they lived in the same database, they just... would.

The Queue Is Just a Table

So make the queue a table in the same file. At its core it's this:

CREATE TABLE jobs (
    id INTEGER PRIMARY KEY,
    queue TEXT NOT NULL,
    payload TEXT NOT NULL,                          -- JSON
    status TEXT NOT NULL DEFAULT 'pending',         -- pending | running | done
    attempts INTEGER NOT NULL DEFAULT 0,
    claimed_at INTEGER,                             -- unix seconds, NULL until claimed
    created_at INTEGER NOT NULL DEFAULT (unixepoch())
);

-- A partial index so workers only scan rows that are actually waiting.
CREATE INDEX idx_jobs_pending ON jobs (queue, id) WHERE status = 'pending';

Now the magic move: enqueue the job in the same transaction as the business write.

BEGIN IMMEDIATE;
INSERT INTO orders (id, total) VALUES (42, 99);
INSERT INTO jobs (queue, payload) VALUES ('emails', '{"order_id":42}');
COMMIT;
-- both rows land together, or neither does

The dual-write problem is simply gone. If the transaction commits, you have an order and its job. If anything fails, the rollback drops both. There's no window where one exists without the other, because they were never two separate writes to begin with - they were one transaction.

Claiming a Job Without Two Workers Grabbing It

The other thing a queue must do is hand each job to exactly one worker. With the jobs in a table, a single atomic statement does it. SQLite's RETURNING clause (available since version 3.35) lets you mark a job as claimed and read it back in one shot:

UPDATE jobs
SET status = 'running',
    attempts = attempts + 1,
    claimed_at = unixepoch()
WHERE id = (
    SELECT id FROM jobs
    WHERE queue = 'emails' AND status = 'pending'
    ORDER BY id
    LIMIT 1
)
RETURNING id, payload;

Because the UPDATE takes the database's write lock, two workers running this at the same time can't claim the same row - one wins the lock, marks the job running, and the other sees it's no longer pending.

When the work is done, the worker closes it out:

UPDATE jobs SET status = 'done' WHERE id = ?;
-- or DELETE it to keep the table small

That's a complete, correct, single-consumer-per-job queue in three statements and no broker.

Retries and the Worker That Died

Real workers crash mid-job. If a worker claims a job, marks it running, and then its machine reboots, that job is stuck running forever and never finishes. The standard fix is a visibility timeout: a claim is only good for a while, and a periodic sweep returns abandoned jobs to the queue.

That's why the table has claimed_at and attempts. A small reaper query, run on a timer, requeues anything that's been running too long:

-- Anything claimed more than 5 minutes ago is presumed dead - requeue it.
UPDATE jobs
SET status = 'pending',
    claimed_at = NULL
WHERE status = 'running'
  AND claimed_at < unixepoch() - 300;

attempts lets you give up gracefully: once a job has been retried, say, five times, route it to a dead-letter state instead of looping forever. That handful of columns gets you at-least-once delivery with retries - the semantics most queues actually provide.

Priorities and Scheduled Jobs, Almost for Free

Because the queue is just a table, extending it is just adding columns. Want priorities? Add a priority column and change the claim's ordering:

ALTER TABLE jobs ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;

-- ...then in the claim subquery:
ORDER BY priority DESC, id

Want delayed or scheduled jobs - "send this in an hour," or a nightly cleanup? Add a run_after column so a job isn't claimable until its time arrives:

ALTER TABLE jobs ADD COLUMN run_after INTEGER NOT NULL DEFAULT 0;

-- ...then in the claim subquery:
WHERE queue = 'emails' AND status = 'pending' AND run_after <= unixepoch()

That single column turns the queue into a scheduler: enqueue a job with a future run_after and it simply waits in the table until then. A recurring task is the same thing with a worker that re-enqueues the next occurrence when it finishes.

Features that would each be a separate Redis data structure are, here, one more column and one more clause.

Your Queue Is Queryable

A benefit that's easy to overlook: your queue is a SQL table, so you can just ask it questions. How many jobs are waiting? What's the oldest? What's failing?

SELECT queue, count(*) FROM jobs WHERE status = 'pending' GROUP BY queue;

SELECT max(unixepoch() - created_at) AS oldest_pending_secs
FROM jobs WHERE status = 'pending';

SELECT id, payload, attempts FROM jobs WHERE attempts >= 5;  -- the stuck ones

With Redis you'd reach for separate tooling to see queue depth and dig into failures. Here it's the same SELECT you already use, against the same database - and it composes with the rest of your data, so you can join jobs back to the orders that created them. Monitoring, dashboards, and one-off "why didn't this send" investigations are all just queries.

Polling Versus Waking Up

The simplest worker loop just polls: run the claim query, and if nothing comes back, sleep a bit and try again.

while True:
    job = claim_one("emails")
    if job is None:
        time.sleep(0.1)  # nothing waiting; check again shortly
        continue
    handle(job)

For most apps that's completely fine - a 100 ms poll is cheap and adds at most 100 ms of latency. It only becomes wasteful at high job rates or when you want near-instant wake-ups across many queues.

That's the one place a library earns its keep. Postgres has LISTEN/NOTIFY built in, so a worker can block until a commit touches the queue instead of polling - and tools like River or pgmq build full queues on top of it. SQLite doesn't have NOTIFY natively, but projects like Honker add Postgres-style NOTIFY/LISTEN to SQLite (as a loadable extension with bindings for several languages), so a worker wakes on commit with no polling at all. If you reach the point where polling latency matters, that's the upgrade - but you'll have shipped the table-based version long before you need it.

Making SQLite Cooperate: WAL and busy_timeout

One honest thing about SQLite: it allows only one writer at a time. For a queue with several workers all claiming and acking, you need two settings or you'll hit "database is locked" errors.

PRAGMA journal_mode = WAL;       -- readers don't block the writer, and vice versa
PRAGMA busy_timeout = 5000;      -- wait up to 5s for the write lock instead of erroring

WAL (write-ahead logging) mode lets readers and the single writer work concurrently, and busy_timeout tells a worker to wait for the write lock rather than immediately failing. With those two in place, a handful of workers on one machine claiming short transactions is perfectly happy. SQLite isn't trying to be a high-write-concurrency database - but a job queue's writes are tiny and quick, which is exactly the workload it handles well.

On Postgres, the Same Pattern Scales Further

Everything above works on Postgres too - and there it scales past SQLite's single-writer ceiling, because Postgres has a feature built for exactly this: SELECT … FOR UPDATE SKIP LOCKED. Where SQLite serializes claims on one write lock, Postgres lets many workers each grab different rows at once, skipping any row another worker has already locked:

SELECT id, payload
FROM jobs
WHERE queue = 'emails' AND status = 'pending'
ORDER BY id
LIMIT 1
FOR UPDATE SKIP LOCKED;

No two workers ever block each other or claim the same job, so you can run many consumers concurrently against one table. That's why "the queue is a table" scales much further on Postgres than on SQLite - the same idea on a writer model built for concurrency. If you're already on Postgres, you may never need a separate queue system at all.

When You Genuinely Do Still Want Redis

This isn't "Redis is bad" - it's "match the tool to the job." Reach for Redis (or a real message broker, or Postgres-as-a-queue) when you actually need what they offer:

The short version: if your app is a single service backed by one SQLite (or Postgres) database, and your job volume is in the human range - emails, webhooks, thumbnails, the long tail of "do this after the request" - the in-database queue is simpler, cheaper, and more correct than bolting on Redis. When you outgrow a single writer or a single host, that's the moment to add a dedicated system, and you'll know because you'll have a concrete bottleneck rather than a hypothetical one.

A Complete Worker, End to End

Stitched together, the whole thing is short. Here's a minimal worker in Python against the standard library's sqlite3 - open the database in WAL mode, then loop: claim a job, run it, and either mark it done or let the reaper requeue it:

import sqlite3, time, json

db = sqlite3.connect("app.db", isolation_level=None)  # autocommit; we manage txns
db.execute("PRAGMA journal_mode = WAL")
db.execute("PRAGMA busy_timeout = 5000")

def claim(queue):
    return db.execute(
        "UPDATE jobs SET status='running', attempts=attempts+1, claimed_at=unixepoch()"
        " WHERE id = (SELECT id FROM jobs"
        " WHERE queue=? AND status='pending' AND run_after<=unixepoch()"
        " ORDER BY priority DESC, id LIMIT 1)"
        " RETURNING id, payload",
        (queue,),
    ).fetchone()

while True:
    job = claim("emails")
    if job is None:
        time.sleep(0.1)  # nothing waiting; check again shortly
        continue
    job_id, payload = job
    try:
        handle(json.loads(payload))
        db.execute("UPDATE jobs SET status='done' WHERE id=?", (job_id,))
    except Exception:
        pass  # leave it 'running'; the reaper requeues it

That's a working at-least-once queue consumer: no broker, no extra process, just a loop over a table in the file your app already uses. Run several copies and the write lock keeps any two from claiming the same job. Swap the time.sleep for a notify-based wake-up later if polling latency ever matters; until then, this is the entire thing.

In Go the shape is the same. The standard pure-Go driver modernc.org/sqlite handles it fine; I personally reach for gosqlite.com (the pure-Go, CGo-free github.com/go-again/sqlite package) - it ships WAL mode, atomic-claim helpers, and RETURNING already wired, plus full-text and vector search for whatever the job handler does next. Same SQL above, just db.Exec instead of db.execute.

Start Here, Graduate Later

Most "we need Redis" moments are really "we need somewhere to put a job." When that somewhere can be the database you already run - in the same transaction as the work that created the job - you delete a whole moving part and a whole class of dual-write bug in one move.

A table, an atomic claim, a visibility-timeout reaper, and a poll loop is a real queue, and for a single service at a human-scale job rate it's simpler and more correct than bolting Redis onto the side. Build it this way first. When you genuinely outgrow a single writer or a single host you'll know, because you'll have a concrete bottleneck pointing the way rather than a hypothetical one - and that's the moment to reach for the bigger machine, not before.