Your Payment Webhook Is Not a Notification. It's a Distributed Systems Problem.

Why building reliable payment infrastructure requires thinking beyond HTTP 200.

Consider this scenario. A customer places an order on an e-commerce platform and pays ₦85,000. The payment provider processes the transaction successfully and sends a webhook to your backend.

Your backend receives it, updates the order to PAID, and returns HTTP 200. Everything works. Except the payment provider never receives your response because of a temporary network interruption. It retries the webhook. Your backend receives the exact same event again.

Now, depending on how you implemented your payment processing logic, you might have just credited a wallet twice, created two orders, triggered duplicate notifications, or initiated another fulfilment process.

This is not an edge case. It is a fundamental characteristic of distributed systems. And it is something every developer building payment-enabled applications should understand.

The problem with assuming exactly-once delivery

Most webhook integrations operate with an at-least-once delivery guarantee. In simple terms, the sender will keep attempting delivery until it receives an acceptable acknowledgement or exhausts its retry policy.

This means your application must assume that:

• The same event can arrive multiple times.
• Events can arrive out of order.
• A request can time out even when the operation succeeds.
• Your application can process an event but fail before acknowledging it.
• Two identical requests can execute concurrently.

The mistake is designing your application around the assumption that one request equals one business operation. HTTP does not guarantee that.

Idempotency is your first line of defence

An operation is idempotent when executing it multiple times produces the same effective result as executing it once. For webhook processing, this means identifying an event that has already been processed and refusing to repeat its business effects.

A straightforward implementation might look like this:

async function processPaymentWebhook(event: PaymentEvent) {
  const existing = await db.webhookEvent.findUnique({
    where: { eventId: event.id }
  });
  if (existing?.status === 'PROCESSED') {
    return { success: true, duplicate: true };
  }
  // Process the event...
}

However, there is a concurrency problem here. Two identical webhook requests could arrive almost simultaneously. Both might check the database before either has inserted the event. Both see nothing. Both proceed.

This is why checking for an existing record alone is insufficient. You need a database-enforced uniqueness constraint.

CREATE TABLE webhook_events (
  id UUID PRIMARY KEY,
  provider_event_id VARCHAR(255) UNIQUE NOT NULL,
  status VARCHAR(30) NOT NULL,
  received_at TIMESTAMP DEFAULT NOW(),
  processed_at TIMESTAMP
);

The unique constraint ensures that concurrent requests cannot successfully register the same provider event twice. Your application should then use an atomic insert or transaction to claim the event before executing its business effects.

Separate event receipt from business execution

Another mistake is treating webhook receipt and business processing as the same operation. They should be separate concerns.

A more resilient architecture looks like this:

• Receive the webhook.
• Verify its authenticity using the provider's signature mechanism.
• Validate the event structure.
• Persist the event with a unique provider event ID.
• Acknowledge receipt.
• Process the event asynchronously.
• Record the processing outcome.
• Retry failed processing safely.

This approach prevents a slow downstream operation from holding up the webhook response. For stronger reliability, use a transactional outbox pattern or an equivalent durable job mechanism. Simply returning HTTP 200 after placing a job in an in-memory queue is not sufficient. If the application crashes before the job is persisted, the event can be lost.

Idempotency is not just about webhooks

This is where things become particularly interesting. Imagine a customer clicking the Pay button multiple times because the page appears unresponsive. Or a mobile application retrying a request after losing connectivity. Or a merchant's POS terminal submitting a transaction again because it never received the first response. The same principle applies.

For operations that create financial or irreversible effects, introduce an idempotency key. For example:

POST /api/v1/payments
Idempotency-Key: order_123_payment_attempt_01

Your backend should associate that key with the original operation and its result. Subsequent requests using the same key should return the original result rather than initiating another payment.

Be deliberate about the key's scope, retention period, and payload consistency. Reusing the same key with a different payment amount or currency should not silently return an unrelated result.

The other problem: Out-of-order events

Suppose your payment system receives these events:

• payment.successful
• payment.pending

In that order. If your implementation blindly updates the payment status based on the latest event received, you could accidentally downgrade a successful payment to pending. Arrival order is not necessarily business-event order.

Your system needs explicit state transition rules. For example:

const allowedTransitions = {
  PENDING: ['SUCCESSFUL', 'FAILED'],
  SUCCESSFUL: [],
  FAILED: []
};

This is deliberately simplified. Real payment systems may support additional states, reversals, refunds, disputes, and provider-specific transitions. The important principle is that a state transition should be validated against the current state, not blindly accepted because an event arrived.

For financial systems, you should also maintain an immutable transaction history or ledger rather than relying exclusively on a mutable balance or status field.

Build for failure, not just the happy path

When building payment-enabled applications, I pay particular attention to questions that are easy to overlook during initial development:

• What happens when the provider processes a payment but our request times out?
• What happens when the same webhook arrives five times?
• What happens when two workers process the same event?
• What happens when the database commits but the application crashes before responding?
• What happens when our system and the provider disagree about a transaction's status?

That last question deserves special attention. You need reconciliation. Your application should periodically compare internal transaction records against the payment provider's records, identify discrepancies, and initiate a controlled resolution process. Webhooks are useful for receiving updates. They should not be your only source of financial truth.

Final thoughts

Building a payment integration is relatively straightforward. Building one that remains correct when networks fail, requests are duplicated, workers crash, and events arrive in unexpected orders is a different engineering problem entirely.

The difference is not necessarily the number of technologies you use. It is the assumptions you make about failure. A production-grade payment system should not depend on requests arriving exactly once, responses arriving on time, or external services behaving perfectly. Because eventually, they won't. And when that happens, your architecture should already know what to do.

The real test of a payment system isn't whether it works when everything goes right. It's whether it remains financially correct when everything goes wrong.