Your Database Said "Success." Your Message Broker Said "Try Again."

This is where distributed systems get interesting. If you've worked with backend systems long enough, you've probably written code that looks roughly like this: await using var transaction = await db.Database.BeginTransactionAsync(); order.Status = OrderStatus.Paid; db.Orders.Update(order); await db.SaveChangesAsync(); await messageBus.PublishAsync( new PaymentCompleted(order.Id) ); await transaction.CommitAsync(); At first glance, it looks reasonable. The order is updated. The event is published. The transaction commits. Done. Except there is a problem hiding in the middle. What happens if PublishAsync() fails? What happens if RabbitMQ is temporarily unavailable? What happens if the network connection drops after the broker accepted the message but before your application receives the response? What happens if the application crashes at exactly the wrong millisecond? And most importantly: What does your system believe happened? This is one of those backend problems that doesn't show up in a happy-path demo. It shows up at 3:17 AM in production. The problem with "just use a transaction" Let's simplify the system. Imagine an Order Service. It has: ┌──────────────┐ │ Order API │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Order Service│ └──────┬───────┘ │ ┌───────┴────────┐ ▼ ▼ ┌────────────┐ ┌──────────────┐ │ PostgreSQL │ │ RabbitMQ │ └────────────┘ └──────────────┘ A request comes in: POST /orders/123/pay The service needs to do two things: - Update the database. - Tell other services that the payment succeeded. For example: Order.Status = Paid + PaymentCompleted event And this is where the trouble starts. Your database and your message broker are two different systems. A database transaction can guarantee atomicity inside the database. RabbitMQ doesn't magically become part of that transaction. So this is not really: BEGIN UPDATE DATABASE PUBLISH MESSAGE COMMIT It is more like: DATABASE MESSAGE BROKER │ │ │ UPDATE │ │─────────────────────────────>│ │ │ │ COMMIT │ │ │ │ │ │ ??? PUBLISH ??? │ │ │ There is a gap. And that gap is where distributed systems become difficult. Failure scenario #1: Database first Let's say we do the sensible-looking thing: await db.SaveChangesAsync(); await messageBus.PublishAsync(message); Suppose the database succeeds. Then: Database ────────────── Order 123 Status = Paid Everything looks good. But immediately afterward: RabbitMQ ────────────── PaymentCompleted ❌ Maybe RabbitMQ is temporarily unavailable. Maybe DNS failed. Maybe the pod restarted. Maybe the process crashed. Maybe there was a network timeout. Now your database says: "Payment completed." But the rest of your system never heard about it. The Inventory Service doesn't know. The Notification Service doesn't know. The Analytics Service doesn't know. Whatever depends on PaymentCompleted doesn't know. And if you simply retry the entire HTTP request, you could create another problem. Failure scenario #2: Message first So perhaps we reverse the order: await messageBus.PublishAsync(message); await db.SaveChangesAsync(); Now imagine the message is successfully published. Then the database transaction fails. Maybe there is a deadlock. Maybe a constraint violation occurs. Maybe the database connection disappears. Now we have the opposite situation: RabbitMQ ────────────── PaymentCompleted ✅ Database ────────────── Order.Status = Pending ❌ The Inventory Service receives: PaymentCompleted But the Order Service says: Actually... no. Now we're inconsistent in the other direction. "Can we just use distributed transactions?" This is where someone usually says: "Why not use a distributed transaction?" In theory, we could try to coordinate the database and broker through a distributed transaction protocol. In practice, this introduces another set of problems. Distributed transactions can be complex, expensive, operationally awkward, and tightly couple infrastructure components. The classic transactional outbox pattern exists largely because we want the database update and the intent to publish an event to become atomic without requiring a two-phase commit across the database and broker. And this is where I think a very simple idea becomes extremely powerful. The Outbox Pattern Instead of immediately publishing the message, we save the message inside the same database transaction as the business change. Something like: ┌─────────────────────────┐ │ Transaction │ │ │ │ UPDATE Orders │ │ + │ │ INSERT OutboxMessage │ │ │ └────────────┬────────────┘ │ COMMIT │ ┌────────────▼────────────┐ │ Database │ │ │ │ Orders │ │ OutboxMessages │ └────────────┬─────────────┘ │ │ Background Worker │ ▼ ┌──────────────┐ │ RabbitMQ │ └──────────────┘ Now the important part: The application isn't trying to atomically update two systems anymore. It only has to atomically update one: the database. A simple Outbox table For example: CREATE TABLE OutboxMessages ( Id UUID PRIMARY KEY, Type VARCHAR(200) NOT NULL, Payload JSONB NOT NULL, OccurredAt TIMESTAMP NOT NULL, ProcessedAt TIMESTAMP NULL, RetryCount INT NOT NULL DEFAULT 0 ); Now our application transaction becomes: await using var transaction = await db.Database.BeginTransactionAsync(); order.Status = OrderStatus.Paid; db.Orders.Update(order); var message = new OutboxMessage { Id = Guid.NewGuid(), Type = nameof(PaymentCompleted), Payload = JsonSerializer.Serialize( new PaymentCompleted(order.Id) ), OccurredAt = DateTime.UtcNow }; db.OutboxMessages.Add(message); await db.SaveChangesAsync(); await transaction.CommitAsync(); Notice something important. We're not talking to RabbitMQ inside the transaction anymore. We're just changing the database. Either both changes succeed: Orders + OutboxMessages Or neither does. That's the part we can make truly atomic. The transactional outbox pattern specifically works by storing the outgoing message in the same database transaction and having a separate relay publish it to the broker. But now we have another problem The outbox worker has to read those messages and publish them. Something like: while (!stoppingToken.IsCancellationRequested) { var messages = await db.OutboxMessages .Where(x => x.ProcessedAt == null) .OrderBy(x => x.OccurredAt) .Take(100) .ToListAsync(); foreach (var message in messages) { await messageBus.PublishAsync( message.Type, message.Payload ); message.ProcessedAt = DateTime.UtcNow; } await db.SaveChangesAsync(); } Looks good. Until we consider this: 1. Worker reads message 2. Worker publishes message 3. RabbitMQ accepts message 4. Worker crashes 5. Worker never marks message as processed 6. Worker restarts 7. Worker publishes message AGAIN Congratulations. We've solved one consistency problem and created another. Duplicate messages. And this is one of the most important lessons in distributed systems: "Exactly once" is usually much harder than it sounds. The outbox relay itself can publish a message more than once if it crashes after publishing but before recording that it was published. The standard pattern therefore expects consumers to be able to process duplicate messages safely. Which brings us to my favorite word in distributed systems: Idempotency An operation is idempotent when performing it multiple times has the same effect as performing it once. For example: Set status = Paid is naturally easier to make idempotent than: balance += 100 because: Set Paid Set Paid Set Paid still results in: Paid But: +100 +100 +100 doesn't. Make the consumer idempotent Suppose our PaymentCompleted event reaches the Notification Service. We could create an inbox/processed-message table: CREATE TABLE ProcessedMessages ( MessageId UUID PRIMARY KEY, ProcessedAt TIMESTAMP NOT NULL ); Then: await using var transaction = await db.Database.BeginTransactionAsync(); var alreadyProcessed = await db.ProcessedMessages .AnyAsync(x => x.MessageId == message.Id); if (alreadyProcessed) { return; } await notificationService.SendPaymentConfirmationAsync( message.OrderId ); db.ProcessedMessages.Add( new ProcessedMessage { MessageId = message.Id, ProcessedAt = DateTime.UtcNow }); await db.SaveChangesAsync(); await transaction.CommitAsync(); The MessageId should also be protected by a database-level unique constraint. Why? Because this is not enough: if (!exists) { insert(); } Two instances can execute the check concurrently: Instance A Instance B │ │ │── Does it exist? ───────>│ │ │ │ │ │ │ │<────── No ───────────────│ │ │ ▼ ▼ INSERT INSERT Now both think they're the first. That's why the database should enforce the invariant. For example: ALTER TABLE ProcessedMessages ADD CONSTRAINT PK_ProcessedMessages PRIMARY KEY (MessageId); PostgreSQL's unique constraints are enforced through unique indexes, making the database itself responsible for preventing duplicate keys. This is a pattern I really like: Application logic decides what should happen. The database enforces what must never happen. But wait... there's another race condition Let's make the system more realistic. We have multiple instances: ┌───────────────┐ │ Load Balancer │ └───────┬───────┘ │ ┌──────────┼──────────┐ ▼ ▼ ▼ API Pod 1 API Pod 2 API Pod 3 │ │ │ └──────────┼──────────┘ ▼ Database Now imagine all three workers poll the outbox at the same time. They might all see: Message #123 ProcessedAt = NULL What stops them from all publishing it? This is where things get interesting. One approach is to atomically claim rows. For example, conceptually: SELECT * FROM OutboxMessages WHERE ProcessedAt IS NULL ORDER BY OccurredAt FOR UPDATE SKIP LOCKED LIMIT 100; The exact implementation depends on your database, workload and broker semantics, but the underlying idea is important: Reading a message and claiming responsibility for it are not necessarily the same operation. And once you start thinking about multiple instances, retries and crashes, the design becomes much more than: "Let's add RabbitMQ." What about retries? The worker will fail. That's normal. A production system should expect failure. So instead of: try { await Publis