Never mark a crypto order paid twice: confirmations, reorgs and out-of-order webhooks

A classic crypto checkout bug goes like this. A webhook comes in, the shop marks the order paid, a retry of the same webhook arrives four seconds later, and a second fulfilment job goes into the queue.

It's easy to write, easy to miss in testing, and it's a good reason to think properly about what "paid" means when the money moves on a blockchain.

Card payments trained most of us to treat a payment as a single yes/no event. On-chain payments are messier. A transfer is seen, then it gathers confirmations, and once in a while the block it landed in disappears. Customers send too little, or too much, or pay again an hour after the invoice expired. Your webhook receiver has to make sense of all of that, often out of order.

This post is about the order-system side of that problem. I'll use the event model from Coldtill, the self-hosted gateway we build, because it's the one I know in detail, but the ideas carry over to most gateways.

What happens between "sent" and "paid"

From the gateway's side, a single payment goes through roughly these stages:

• Seen. A transfer to the customer's deposit address shows up in a block (or, for Bitcoin, in the address history from an Esplora node).
• Confirming. Each new block adds a confirmation. Every network has its own required count.
• Confirmed. Once there are enough, the deposit counts toward what the customer owes.
• Reorged, occasionally. If the block holding the transfer drops off the canonical chain, the deposit isn't confirmed any more.

In Coldtill's watcher, a deposit whose block disappears in a reorg goes back to pending, and one whose transaction vanishes entirely is marked orphaned and reversed. Block hashes are re-checked as confirmations are counted, rather than just counting block heights.

The important thing for your shop: you should never have to reason about any of that yourself. If your order code is counting confirmations, the gateway is leaking a concern it should own. What you want is a payment-level state that only reaches "paid" after the chain-level work is done.

Payment states, not transfer events

Coldtill reduces all of the above to a small set of payment states:

Ship on FILLED or OVER_FILLED. Everything else is either "keep waiting" or "this order isn't going to be paid through this link".

That 1% tolerance is worth a second look. Customers paying in a volatile coin, or paying from an exchange that shaves a little off, will rarely hit the exact figure. Deciding up front how close is close enough saves a lot of support emails.

The payments list in the Coldtill dashboard, with filled, partially filled, pending and expired payments side by side

The four things that break naive receivers

1. Retries

   Webhooks are delivered at least once, not exactly once. Coldtill retries progress events after 2 and 4 seconds, and final events on a longer schedule (2 s, 4 s, 30 min, 1 h, 2 h, 4 h, 8 h, 24 h, 48 h) before marking them failed. A retry reuses the same X-Coldtill-Delivery ID, so that's your cheapest dedupe key.

2. Ordering

   Delivery order isn't guaranteed. A progress event can arrive after the final one if the first attempt of the progress event failed and was retried. The payload carries timestamp_ms, and the docs say to order by that rather than timestamp, because two events can share a second.

3. Late money

   Once a payment is closed, it stays closed. A deposit that lands after the payment is CANCELLED still counts toward the filled amount, but doesn't reopen it; you get a progress event with the new filled_amount. If you ignore everything after the final event, you'll never notice customers who paid late, and those are exactly the ones who email you angry.

4. Trusting the wrong field

   The event name header and the delivery ID header are not signed. Only the body is. So the decision "is this order paid?" has to come from the signed body's status and reference_id, never from X-Coldtill-Event.

A receiver that survives all four

Here's the shape I'd use. It's TypeScript with Express and Postgres, and it uses the SDK's verifier as shown in the webhook docs. The SQL is the part that matters; adapt it to whatever you run.

import express from 'express'
import { verifyWebhookRequest, isPaid } from '@coldtill/sdk'
import { pool } from './db'

const app = express()

app.post('/coldtill/webhook', express.raw({ type: 'application/json' }), async (req, res) => {
  let ev
  try {
    ev = await verifyWebhookRequest(req.body, req.headers, process.env.COLDTILL_WEBHOOK_SECRET!)
  } catch {
    return res.sendStatus(401)
  }

  if (!ev.event.startsWith('payment.')) return res.sendStatus(200)

  const delivery = String(req.headers['x-coldtill-delivery'] ?? '')
  const p = ev.data
  const client = await pool.connect()

  try {
    await client.query('BEGIN')

    // Dedupe retries. The delivery ID isn't signed, so it's only used to skip work,
    // never to decide anything.
    const seen = await client.query(
      'INSERT INTO webhook_seen (delivery_id) VALUES ($1) ON CONFLICT DO NOTHING',
      [delivery]
    )
    if (seen.rowCount === 0) {
      await client.query('COMMIT')
      return res.sendStatus(200)
    }

    // Keep the newest view of the payment. Older events that arrive late can't
    // overwrite newer state.
    await client.query(
      `INSERT INTO crypto_payments (reference_id, order_id, status, filled_usd, event_ms)
       VALUES ($1, $2, $3, $4, $5)
       ON CONFLICT (reference_id) DO UPDATE SET
         status = EXCLUDED.status,
         filled_usd = EXCLUDED.filled_usd,
         event_ms = EXCLUDED.event_ms
       WHERE crypto_payments.event_ms < EXCLUDED.event_ms`,
      [p.reference_id, p.invoice_id, p.status, p.filled_amount_in_usd, p.timestamp_ms]
    )

    // Fulfil once. The unique order_id on fulfilments is what actually
    // stops a second shipment, whatever happens above.
    if (isPaid(p.status)) {
      await client.query(
        `INSERT INTO fulfilments (order_id, reference_id) VALUES ($1, $2) ON CONFLICT (order_id) DO NOTHING`,
        [p.invoice_id, p.reference_id]
      )
    } else if (p.status === 'CANCELLED' && Number(p.filled_amount_in_usd) > 0) {
      await client.query(
        `INSERT INTO payment_followups (reference_id, reason) VALUES ($1, 'late or short payment') ON CONFLICT DO NOTHING`,
        [p.reference_id]
      )
    }

    await client.query('COMMIT')
    res.sendStatus(200)
  } catch (err) {
    await client.query('ROLLBACK')
    res.sendStatus(500) // let the gateway retry
  } finally {
    client.release()
  }
})

A few choices in there are deliberate:

Three layers of idempotency. The delivery table skips cheap repeats. The event_ms guard stops stale events overwriting fresh ones. The unique constraint on fulfilments.order_id is the real safety net. If the first two have a bug, the third still holds.

Everything in one transaction. If the process dies halfway, nothing is half-recorded, and the 500 makes the gateway try again.

Late money goes to a human. I don't auto-ship on a cancelled payment with money on it, because the price, the stock or the customer's intent may have changed. A follow-up queue is cheaper than guessing.

Respond fast. Coldtill wants a 2xx within 60 seconds. If fulfilment is slow, insert a job and let a worker do it.

Replacing a payment safely

One more case that surprises people. In Coldtill, each customer has at most one open payment, and creating a new one replaces the previous (REPLACED). But if the old payment has a deposit still confirming, POST /api/v1/payment returns 409 instead, so that money isn't stranded on a dead invoice.

In practice this shows up when a customer abandons checkout, changes their basket and comes back while the first transfer is still confirming. Treat the 409 as "wait and retry", not as an error page. Something like "Your previous payment is still confirming, we'll update this page in a minute" is far better than a stack trace.

Don't fulfil from the return page

The hosted checkout has a "Return to store" button, and it's tempting to mark the order paid when the customer lands back on your thank-you page. Don't. Anyone can open that URL. The docs are blunt about it: opening the return URL tells the gateway nothing. Fulfil from the verified webhook, or from GET /api/v1/payment/reference/{reference_id} if you need to poll as a fallback.

The customer's view while a payment confirms: exact amount, QR code and confirmation progress

A quick test plan

Before going live I run through this list on testnet:

• Pay the exact amount and check exactly one fulfilment row appears.
• Use Webhooks → Deliveries to resend the final event by hand and check nothing changes.
• Pay half, then the rest, and watch the order move from partially filled to filled.
• Let a payment expire, then pay it, and check it lands in the follow-up queue rather than shipping.
• Stop your receiver for a few minutes during a payment and confirm the retries catch up.

None of this is glamorous, but it's the difference between a crypto checkout you trust and one you check by hand every morning. If you want to see how the states look in a real dashboard, the live demo is linked from coldtill.com, and the webhook docs have verifier snippets for Node, PHP, Go and Python. I'm happy to answer questions in the comments.