Monitoring Python RQ jobs: what to watch and how to get alerted

RQ failures don't announce themselves

When a job raises, RQ moves it to the FailedJobRegistry and moves on. The worker keeps running; nothing crashes. If you're not looking at that registry, the failure is invisible - the same trap BullMQ, Celery, and every robust queue share.

So the job is to reach into the queue's state and turn it into a signal.

The four signals that matter for RQ

• Failure count / rate - jobs landing in the FailedJobRegistry over a window.
• Backlog - how many jobs are queued vs. being worked; is the worker keeping up?
• Latency - how long jobs take, and how long they wait before a worker picks them up.
• Worker liveness - are your workers actually alive and heartbeating?

Where to read them

RQ exposes queue and registry state directly:

from redis import Redis
from rq import Queue
from rq.registry import FailedJobRegistry, StartedJobRegistry

redis = Redis()
q = Queue("default", connection=redis)

queued = len(q)  # backlog
failed = FailedJobRegistry(queue=q)  # failures
started = StartedJobRegistry(queue=q)  # in-flight

print("queued:", queued)
print("failed:", len(failed))
print("started:", len(started))

Poll this on an interval and store the series - a single snapshot hides the trend, which is the part that matters.

For failures specifically, walk the registry to get the actual exceptions:

for job_id in failed.get_job_ids():
    job = q.fetch_job(job_id)
    print(job.id, job.exc_info.splitlines()[-1] if job.exc_info else "")

Two gotchas

• Group by exception, not by job. A thousand jobs failing with the same traceback is one incident. Normalize the message (strip IDs, timestamps, hosts) and group on the rest, or you'll drown in near-identical entries.
• Watch worker heartbeats. RQ workers register in Redis; if a worker dies mid-job the job can sit in StartedJobRegistry past its TTL. A rising started-but-never-finished count is a worker-health problem masquerading as a queue problem.

Turning signals into alerts

The metrics above are only useful if something evaluates them and pages a human. Options:

• A cron + your own thresholds - cheap, but you build the rate math, grouping, and delivery.
• Prometheus + Grafana - export the registry counts (e.g. rq-exporter) and alert in Grafana.
• A hosted monitor - send events out and let it handle windows, grouping, and routing.

The principle is identical across every queue: watch from outside the worker, alert on the rate (not a single failure), and group identical failures so a retry storm is one page.

Where PipeRadar fits (and doesn't, yet)

Full disclosure: I work on PipeRadar, a hosted monitor that does exactly this - failure clustering, latency percentiles, history, and rate-based alerts to Slack/PagerDuty/webhooks - for BullMQ today. RQ is on the roadmap, not shipping yet, so I'm not going to pretend you can pip install it this afternoon.

What's relevant now: PipeRadar's ingest API is queue-agnostic (it already models an adapter type per queue system), so an RQ adapter is the missing piece, not a rewrite. If RQ monitoring like this is something you'd use, the fastest way to pull it up the roadmap is to say so - you can follow progress and register interest at piperadar.dev.

In the meantime, the patterns above work with nothing but Redis and a cron. More on background-job monitoring - including a deep dive on why queue jobs fail silently (BullMQ, but the failure model is the same in RQ) - at piperadar.dev/blog.