The Reconciler Pattern: When a Queued Job Simply Never Runs

The bug that leaves no trace

Records stuck in pending. No error, no failed job, no log line.

The shape of the code:

$status = SyncStatus::create([...]); // sync_status = 'pending'
ProcessSyncStatus::dispatch($status->id);

One dispatch, at creation. Fine 99% of the time.

The 1%:

Only the last row is visible. The first three are invisible failures - dashboards green, data quietly rotting.

The bad mental model

"the queue is durable, so a dispatched job will eventually run."

Your database row is the durable thing. The queue is a delivery hint.

Retry ≠ reconcile

Retry is a queue-level mechanism. Reconciliation is a domain-level one. You need both.

The reconciler

Periodically ask the database "does anything look stuck?", and fix it.

every 15 min
|
v
+----------------------------+
| sweep pending rows         |
| updated_at < now()-15m     |  <- stale
| AND claimed_at IS NULL     |  <- never picked up by a worker
+----------------------------+
|
| touch(row)  <- so the next tick won't sweep it again
v
re-dispatch ProcessSyncStatus

Two predicates carry the weight:

• Stale: updated_at older than a threshold - fresh pending rows are just working, give the queue time.
• Unclaimed: the job stamps the row the moment a worker picks it up, so a pending-and-claimed row is in flight right now; leave it alone.

Pending and unclaimed after 15 minutes was never delivered.

final class ReconcilePendingSync extends Command
{
    protected $signature = 'sync:reconcile-pending
        {--minutes=15 : Only sweep rows untouched this long}
        {--limit=200 : Cap re-dispatches per run}
        {--dry-run}';

    public function handle(): int
    {
        $rows = SyncStatus::query()
            ->where('sync_status', 'pending')
            ->whereNull('claimed_at')
            ->where('updated_at', '<', now()->subMinutes((int) $this->option('minutes')))
            ->limit((int) $this->option('limit'))
            ->get();

        foreach ($rows as $row) {
            if ($this->option('dry-run')) {
                continue;
            }

            // ...report only
            $row->touch(); // don't sweep again next tick
            ProcessSyncStatus::dispatch($row->id);
            Log::info('reconciler: re-dispatched', ['id' => $row->id]);
        }

        return self::SUCCESS;
    }
}

Three things that are easy to get wrong

• touch() before dispatch. Otherwise the next tick sees the same stale row and dispatches it again - a job multiplier, not a reconciler.
• --limit. If something breaks for a day, the first sweep shouldn't dump 40,000 jobs into Redis.
• Idempotent job. Reconciliation will occasionally re-run something that was fine. That must be a no-op: claim the row at the top of handle(), make the work an upsert.

Tests

Queue::fake() plus a backdated updated_at covers the cases that matter:

function agePendingRow(SyncStatus $row, int $minutes): void
{
    // Bypass Eloquent timestamps to backdate updated_at.
    DB::table('sync_status')
        ->where('id', $row->id)
        ->update(['updated_at' => now()->subMinutes($minutes)]);
}

it('re-dispatches a stale unclaimed pending row', function () {
    Queue::fake();
    $stale = SyncStatus::factory()->create([
        'sync_status' => 'pending',
        'claimed_at' => null,
    ]);
    agePendingRow($stale, 60);

    $this->artisan('sync:reconcile-pending', ['--minutes' => 15])
        ->assertSuccessful();

    Queue::assertPushed(ProcessSyncStatus::class, 1);
});

it('skips a pending row already claimed by a worker', function () {
    Queue::fake();
    $claimed = SyncStatus::factory()->create([
        'sync_status' => 'pending',
        'claimed_at' => now(),
    ]);
    agePendingRow($claimed, 60);

    $this->artisan('sync:reconcile-pending', ['--minutes' => 15])
        ->assertSuccessful();

    Queue::assertNotPushed(ProcessSyncStatus::class);
});

Plus a third: a fresh pending row must be left alone. Anyone can write a sweeper that re-dispatches everything - the value is in what it refuses to touch.

Takeaway

If a database row implies "a job should be running for this", something needs to check that claim on a schedule. Dispatch-on-create is a happy-path optimisation, not a guarantee.

The rule I'm keeping: the queue delivers, the database decides. Any state that can only be advanced by a queued job needs a reconciler behind it - otherwise "eventually consistent" really means "usually consistent, and silent when it isn't."