Test a Saga When Compensation Times Out and the Message Is Delivered Twice

Testing Compensation as a Distributed Operation

A saga test that stops after “payment succeeded, inventory failed, refund called” assumes compensation is reliable. It is another distributed operation, so test it under the same faults.

Sequence of Events

Use this sequence:

1. payment capture succeeds
2. inventory reservation times out
3. refund succeeds at provider
4. refund response is lost
5. compensation message is delivered again
6. late inventory-failed event arrives again

Invariant and Persistent State

The invariant is not “refund endpoint called once.” It is:

captured amount - confirmed refunded amount = final charged amount and final charged amount is never negative.

Persist an inbox record for consumed message IDs and an outbox record for each intended side effect.

Idempotency Key Design

Give the provider request a stable idempotency key derived from saga and compensation step:

{
  "sagaId": "order-42",
  "step": "refund-payment-v1",
  "idempotencyKey": "order-42:refund-payment:v1",
  "amount": 4900
}

Deterministic Simulator and Assertions

A deterministic simulator should permute duplicate delivery, delayed acknowledgement, worker crash, and out-of-order events. After every run, assert:

• one terminal order state
• at most one economic refund
• a complete evidence trail

“Already refunded” must reconcile to success only after amount and payment identity match.

Reference to AWS Guidance

AWS describes coordination choices and rollback behavior in its Saga pattern guidance.

Durable Compensation and Runbook

The implementation detail that deserves its own test is durable compensation progress: a process restart cannot erase whether the external side effect occurred.

Alert on sagas stuck in compensating, but do not let an operator click “retry” with a new key. The runbook should first query provider state, compare amount and currency, and resume the same operation identity.

Exactly-once delivery is not required to preserve money. Stable operation identity plus reconciliation is.