How to never pay a generative API twice: a durable task table, not a retrying job queue

Short answer: the moment you might have been charged is not the provider's response, it is the instant before the request leaves your process.

Write that instant to a database row before the HTTP call, and a worker that dies mid-request becomes recoverable without guessing: nothing on the row means nobody paid and the task can go back to the queue; a saved request id means poll and never resubmit; the boundary written with no id means a third terminal state that no automation is allowed to retry.

TL;DR Fix: own task table with a lease, and a submit_started_at column written before the paid call.

A queue retries by design; pg-boss says so in its own docs: "the job can run twice and handlers should be idempotent."

Alternative: a provider-side idempotency key, if your provider has one. Stripe does. The OpenAI and Anthropic clients ship maxRetries: 2 and send no idempotency header at all.

Trap: treating an expired deadline as a failure. A poll that ran out of patience is "I don't know", and the difference costs money every time you get it wrong.

Why a job queue is the wrong unit of recovery for a paid call

Because retrying is the entire point of a job queue, and for a paid POST a retry is a second charge.

I started by running generation steps as ordinary jobs on the queue the rest of the system already used, pg-boss, because that is where every other background task lived. That lasted until I read the queue's own documentation on what happens when a worker looks dead:

A worker that only looked dead (a network partition, a stalled event loop) keeps running its handler after its job is failed this way, so the job can run twice and handlers should be idempotent.

That is the pg-boss queue reference, and it is being honest rather than apologetic. Every durable queue has the same sentence somewhere, because a queue cannot tell a dead worker from a slow one. The contract it offers is at-least-once, and it tells you to make the handler idempotent.

For sending an email, fine. For a handler whose first line spends money at an external provider, "make it idempotent" is the hard part being handed back to you.

The defaults make it concrete. retryLimit is 2, so a failing job runs three times. expireInSeconds defaults to 15 minutes, after which an active job is retried or failed regardless of what the worker is doing.

Image generation lands in tens of seconds and video generation in minutes, so a long clip can sail past the expiration while it is perfectly healthy, get failed by the monitor, and get retried into a second charge. You can raise the expiration to 24 hours, the documented maximum, but then a genuinely dead worker sits undetected for a day.

There is a second problem that is not about money. A job that spends twenty minutes polling a remote id occupies a worker slot for twenty minutes. Concurrency on the queue is shared with everything else in the system, and a handful of slow generations will starve the short jobs that the queue exists for.

The unit of recovery had to stop being a job and become a row.

What RFC 9110 actually says about retrying a POST

It says do not, unless you can prove one of two things, and the second one is the design brief for the whole task table.

RFC 9110, section 9.2.2:

A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.

Two escape hatches. The first one, knowing the semantics are idempotent, is what a provider gives you with an idempotency key. Most generative APIs do not give you one, which I will get to.

The second, "some means to detect that the original request was never applied," is not a property of the protocol at all. It is a property of what you wrote down locally before you made the call.

The RFC also has a line about clients that skip both:

Some clients take a riskier approach and attempt to guess when an automatic retry is possible. For example, a client might automatically retry a POST request if the underlying transport connection closed before any part of a response is received, particularly if an idle persistent connection was used.

That paragraph describes the official clients of the largest paid APIs in the world, which is the subject of the next section.

Do the generative APIs give you an idempotency key?

Mostly no, and the clients retry your POSTs anyway. I went looking for this expecting a boring answer and got a much sharper one by reading the SDK sources instead of the marketing pages.

• API: Stripe / Idempotency key on a POST: Idempotency-Key, first result stored and replayed / Official client retries a POST by default: not examined
• API: OpenAI / Idempotency key on a POST: none sent by openai-node, see below / Official client retries a POST by default: yes, maxRetries: 2 on 408, 409, 429, ≥500, timeouts
• API: Anthropic / Idempotency key on a POST: no occurrence of the word in anthropic-sdk-typescript / Official client retries a POST by default: yes, maxRetries: 2 on 408, 409, 429, ≥500, timeouts
• API: Replicate / Idempotency key on a POST: no occurrence in the published OpenAPI document / Official client retries a POST by default: not examined

Stripe is the benchmark for how this is supposed to work, and its wording is worth quoting because it names the property that matters:

Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.

That is the Stripe API reference on idempotent requests. Note the retention rule in the same page: keys can be pruned once they are at least 24 hours old, and a key reused after pruning generates a new request. Even the gold standard has an expiry after which your safety net is gone, which is a good reminder that the client side still has to keep its own record.

The OpenAI side is where it gets strange. The OpenAI Node client carries the full apparatus for idempotency keys. It declares the field, it has a key generator whose default value announces where the machinery came from, and it builds the header:

protected idempotencyHeader?: string;
protected defaultIdempotencyKey(): string {
  return `stainless-node-retry-${uuid4()}`;
}
// ...later, while assembling headers:
let idempotencyHeaders: HeadersLike = {};
if (this.idempotencyHeader && method !== 'get') {
  if (!options.idempotencyKey) options.idempotencyKey = this.defaultIdempotencyKey();
  idempotencyHeaders[this.idempotencyHeader] = options.idempotencyKey;
}

idempotencyHeader is declared and never assigned anywhere in the file. The condition is therefore always false and the header is never sent. RequestOptions even exposes idempotencyKey with the comment "A unique key for this request to enable idempotency", and setting it accomplishes nothing, because there is no header name to put it in. I spent a while convinced I had missed an assignment in a subclass. There is none.

Meanwhile the retry predicate in the same file does not look at the method at all:

private async shouldRetry(response: Response): Promise<boolean> {
  const shouldRetryHeader = response.headers.get('x-should-retry');
  if (shouldRetryHeader === 'true') return true;
  if (shouldRetryHeader === 'false') return false;
  if (response.status === 408) return true; // request timeouts
  if (response.status === 409) return true; // lock timeouts
  if (response.status === 429) return true; // rate limits
  if (response.status >= 500) return true; // internal errors
  return false;
}

DEFAULT_TIMEOUT in that client is 600000, ten minutes, and a timeout goes down the same retry path. The Anthropic TypeScript client has a byte-for-byte equivalent predicate, the same maxRetries: 2 and the same ten-minute default, and zero occurrences of the string idempot in the entire file.

The default behaviour of both clients, then: send a paid POST, and if the server returns 500 or the socket goes quiet for ten minutes, send it again, up to two more times, with no way for the server to recognise it as the same request. Whether the first attempt was billed is between you and the provider's meter. The client has no way to ask, and neither do you.

This is not a scandal. It is the reasonable default for an API where most calls are cheap and a dropped connection is more likely than a silently successful one. It stops being reasonable the moment a single call costs real money and the user can see the balance.

The boundary is before the request, not after the response

The instant that matters is the last moment before the send, not the first moment after the answer. This is the one idea the whole design rests on, and it took me an embarrassing amount of staring at a state machine to see it.

I had been drawing the boundary in the obvious place: the task is safe until the provider answers, and becomes committed when it answers. That is wrong, and the reason is that the answer is exactly the thing you do not have when the worker dies. A crash between "socket write completed" and "response parsed" leaves no evidence at all on the client side, and it is the most likely crash window there is, because that window is where all the waiting happens.

Flip it. The task is safe until you begin the request, and becomes unknown the instant you do. That boundary is knowable in advance, so it can be written down in advance:

async function runImageTask(task: Task, ports: Ports): Promise<void> {
  if (task.status === "SUBMITTED" && task.remoteRequestId) {
    return pollUntilDone(task, ports); // already paid for, never resubmit
  }
  await ports.repo.markSubmitStarted(task.id); // the boundary
  const { requestId } = await ports.provider.submit(task.input);
  await ports.repo.recordSubmitted(task.id, requestId); // now it is addressable
  return pollUntilDone(task, ports);
}

Three writes, and every crash window between them maps to exactly one recovery action. markSubmitStarted is a separate round trip to the database before a call that is about to take thirty seconds, so the cost of the extra write is not worth discussing.

The one rule this imposes on the rest of the code is that nothing may sit between markSubmitStarted and the actual send. No input validation, no prompt assembly, no lazy credential fetch. Anything that can fail there produces a task parked in the unknown state for no reason, and parking a task costs a human's attention later. Build the request fully, then mark, then send.

Three recovery branches after an expired lease

A supervisor picking up a task whose lease has expired has exactly three cases, and the row tells it which one without any guessing:

• What is on the row: status = RUNNING, no submit_started_at / What happened: Died before the request went out / Recovery: Back to PENDING / Money: Nothing spent, safe to redo
• What is on the row: status = SUBMITTED, remote_request_id set / What happened: Died after the provider acknowledged / Recovery: Poll the saved id, never submit / Money: Already spent, result is retrievable
• What is on the row: status = RUNNING, submit_started_at set, no id / What happened: Died inside the request / Recovery: OUTCOME_UNKNOWN, stop / Money: Might be spent, nobody can tell

In code it is short enough to fit in a screenshot, which is the point:

type Recovery =
  | { action: "requeue" }
  | { action: "poll" }
  | { action: "park"; status: "OUTCOME_UNKNOWN" };

function recoveryFor(task: Task): Recovery {
  if (task.status === "SUBMITTED" && task.remoteRequestId) return { action: "poll" };
  if (task.submitStartedAt === null) return { action: "requeue" };
  return { action: "park", status: "OUTCOME_UNKNOWN" };
}

Order matters in that function and it is not the order I wrote first. My first version checked submitStartedAt before the request id, which sends an already-submitted task back through the submit path whenever both columns are set, which is always for a healthy submitted task. The request id is the strongest evidence on the row, so it gets tested first.

Why OUTCOME_UNKNOWN has to be terminal

Because every automatic way out of it is a bet placed with someone else's money. This was the design decision I argued with myself about longest, since a state that requires a human is an operational cost forever.

The two tempting exits both fail. Retrying assumes the request never landed, and if it did land you have paid twice and the user sees one result. Failing assumes the request never landed too, just with a nicer label: the task disappears from the pipeline, the charge does not disappear from the invoice, and the reconciliation happens weeks later when someone asks why the numbers do not match. Both are the same guess wearing different clothes.

The state is therefore terminal, the UI says the outcome could not be determined and suggests checking the provider's dashboard, and a human presses the button. In exchange, the system never spends money on its own initiative. For a paid API that is the correct side of the error to be on, and the volume is low enough that it stays correct: the branch only fires when a process dies inside the send window, which is a few seconds out of a generation that takes minutes.

The same logic applies to deadlines, and this is where I got it wrong first. Put a cap on polling, because a remote id that never resolves cannot hold a slot forever. My original code marked the task FAILED when the cap was reached. That is the same lie as before: the provider might still be working, or might have finished and billed while my poll loop gave up. An exhausted deadline is not a failure, it is an unknown outcome that happens to be slow. It goes to the same terminal state, with a different reason string.

Claiming work with FOR UPDATE SKIP LOCKED

The claim is a single statement, and PostgreSQL documents this exact use case. From the SELECT reference:

With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped. Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to a