DEV Community

Rate-Limited Job Processing in Node.js: Queue Workers, Batch Publish, Retry, Idempotency

Short answer: treat a rate limit as a scheduling constraint, claim one lane at a time with worker concurrency 1, publish bounded batches, and make the operation key durable across every retry.

The queue is only a delivery mechanism; correctness comes from the state machine around it. That distinction matters in payment and ledger backends. A process can disappear after a remote service accepts a request but before the local acknowledgement is committed. A retry that lacks the same business identity can create a second effect, while a retry that preserves the identity can converge on the first one. I design the queue, publisher, and audit record as one protocol, even when they run in separate services.

What should a rate-limited job queue do before it starts publishing?

Begin with the constrained resource. If the remote allowance belongs to an account, tenant, credential, or destination, use that value as the lane key. Worker concurrency 1 then means one active publisher per lane, not one worker for the entire installation. Independent lanes retain useful parallelism without racing against a shared allowance.

Persist an explicit lifecycle such as ready, leased, publishing, confirmed, retryable, and rejected. The claim transaction should record the lease owner, attempt number, operation key, batch identifier, and timestamps before network I/O. On restart, a publishing row whose outcome is unknown is ambiguous; it is not automatically failed.

The operation key is the durable identity of the business intent. A queue message ID is not enough because redelivery can create a new message ID. The receiver should accept the same key on every attempt, or reconciliation must query an authoritative result before another side effect is allowed. Keep the evidence needed to explain that decision, but don't copy sensitive payment payloads into ordinary logs; retention and access rules belong with the applicable legal and security owners. Keep it boring.

How can worker concurrency 1, batch publish, retry, and idempotency coexist?

The scheduler owns spacing between calls. The worker owns state transitions. Keeping those responsibilities separate makes shutdown and testing much less mysterious.

package publisher

import (
  "context"
  "errors"
  "time"
)

var ErrRateLimited = errors.New("rate limited")

type Job struct {
  ID string
  OperationKey string
  Payload []byte
  Attempt int
}

type Store interface {
  Begin(context.Context, string, int) ([]Job, error)
  Confirm(context.Context, string, string) error
  Retry(context.Context, string, time.Time, string) error
  Reject(context.Context, string, string) error
}

type Result struct {
  JobID string
  Receipt string
  Err error
}

type Publisher interface {
  Publish(context.Context, []Job) ([]Result, time.Duration, error)
}

func RunLane(ctx context.Context, lane string, store Store, out Publisher) error {
  jobs, err := store.Begin(ctx, lane, 25)
  if err != nil || len(jobs) == 0 {
    return err
  }
  results, retryAfter, err := out.Publish(ctx, jobs)
  if errors.Is(err, ErrRateLimited) {
    next := time.Now().Add(retryAfter)
    for _, job := range jobs {
      if saveErr := store.Retry(ctx, job.ID, next, "remote rate limit"); saveErr != nil {
        return saveErr
      }
    }
    return nil
  }
  if err != nil {
    return err
  }
  for _, result := range results {
    if result.Err != nil {
      if saveErr := store.Reject(ctx, result.JobID, result.Err.Error()); saveErr != nil {
        return saveErr
      }
      continue
    }
    if saveErr := store.Confirm(ctx, result.JobID, result.Receipt); saveErr != nil {
      return saveErr
    }
  }
  return nil
}

The example deliberately stores results per item, even though the transport call is a batch. A batch is a transport optimization, not an atomic transaction, unless the receiving contract says otherwise.

Imagine a batch with three items: the receiver accepts the first two, the connection closes, and the client sees no response. Replaying all three is unsafe because the first two may already have effects; treating all three as failed is equally unsafe because the third may be retried under a different state. With item-level receipts, a durable operation key, and a reconciliation query, the system can classify the first two as confirmed and leave the third as ambiguous until an authoritative answer exists.

If the response is partial, confirm the accepted items and retry only those with an unresolved outcome. If the transport fails after submission, preserve the lease and operation keys for reconciliation rather than issuing a fresh, anonymous attempt. That longer path is intentional: the audit record should explain why a later attempt was permitted, not merely report that a timer fired.

When the response is HTTP 429, honor Retry-After when present and schedule a later attempt with bounded jitter. Do not spin in the same handler. A malformed request or a rejected business rule is a terminal result, not a rate-limit retry; retain its response metadata for the audit trail.

Which queue strategy matches the failure semantics?

Labels such as FIFO and exactly once compress different guarantees. Compare the actual failure behavior instead.

Strategy Ordering scope Duplicate control Limitation Suitable use
In-process list and timer One live process Memory only Restart can lose work Disposable jobs whose loss is harmless
Durable standard queue Application-defined Consumer operation key Redelivery and reordering are expected Independent work with backpressure
Durable FIFO lane Group or lane Queue deduplication plus consumer key A hot lane limits parallelism Ordered work per tenant or account
Database outbox and scheduler Transaction-defined Unique constraint and state machine Polling and table maintenance Side effects coupled to a database commit

The catch is that a single lane is unsuitable when jobs are independent and queue age matters more than ordering. Add lanes or use a shared rate limiter. Stick with an outbox when enqueueing must be atomic with a ledger write; choose a durable queue when independent scaling is the primary concern. An in-memory timer is appropriate only when reconstruction after a restart is explicitly safe.

No queue grants exactly-once effects by itself. An exactly-once mindset means one durable business identity, observable attempts, convergence on duplicates, and a reconciliation process that can prove the final state. Delivery, HTTP submission, receiver processing, and local confirmation remain separate events.

How do you test and roll out a retryable publishing worker?

Test the boundaries where the process can vanish: immediately before publish, immediately after publish, and during item-level confirmation. Advance a fake clock beyond a lease, return a partial batch result, and run two schedulers against one lane to prove that the storage claim permits only one active lease. The invariant should be boring: one claim at a time, the same operation key on every attempt, and no silent state loss.

Measure twice. Deploy behind a lane-level switch. Start with concurrency 1 and a small batch, record intended operation keys in a shadow path that never publishes, and compare them with confirmed receipts. Migrate a small cohort only after reconciliation shows no unexplained gaps or duplicate business effects. Increase the number of lanes before increasing concurrency within a lane. Rollback means pausing new claims, allowing active leases to settle, reconciling every publishing item, and only then returning ownership to the previous consumer. Two active schedulers are not a rollback plan.

Alert on queue age, repeated attempts, expired leases, 429 rate, and reconciliation mismatches; throughput alone cannot tell an operator that money may be wrong. Your mileage may vary with the database primitive, and I am not sure every receiver exposes a durable idempotency contract. That uncertainty is a design input: document the missing guarantee, add a reconciliation check, and keep the operation key stable until the authoritative outcome is known.

References

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.