Reliable Event-Driven Architecture in Spring Boot: Outbox, Inbox, Retries, and Idempotency

Event-driven architecture looks simple at first. Your application performs a business operation, publishes an event to Kafka or SQS, and another service consumes it. @Transactional public void createOrder(CreateOrderCommand command) { Order order = orderRepository.save(...); kafkaTemplate.send( "orders", new OrderCreatedEvent(order.getId()) ); } Looks reasonable. The order is saved, an OrderCreatedEvent is published, and other services can react to it. But there is a problem hiding in those few lines: What happens if the database transaction succeeds, but publishing the event fails? And on the consumer side: What happens if the same event is delivered twice? These questions lead to several patterns that become essential once event-driven systems move beyond simple demos: - Transactional Outbox - Inbox Pattern - Idempotent Consumers - Durable Retries - Multi-instance-safe processing - Observable failure handling Let's build the architecture step by step. The Dual-Write Problem Imagine an order service that needs to: - Save an order to PostgreSQL. - Publish OrderCreated to Kafka. Conceptually: Database ──────> COMMIT ✓ | Kafka ─────────> PUBLISH ✗ The database and Kafka are independent systems. A successful database commit does not guarantee a successful Kafka publish. Consider: 1. INSERT order 2. COMMIT 3. Publish OrderCreated 4. Application crashes If the application crashes between steps 2 and 3, the order exists but the event doesn't. Other services may never know that the order was created. Reversing the operations doesn't fix it: 1. Publish OrderCreated 2. INSERT order 3. Database transaction fails Now consumers may receive an event for an order that doesn't exist. This is the classic dual-write problem. The Transactional Outbox Pattern Instead of trying to atomically update the database and message broker, persist the event as part of the same database transaction as the business operation. Database Transaction ┌─────────────────────────────┐ │ │ Request ──> Business Data │ │ + │ │ Outbox Event │ │ │ └────────── COMMIT ───────────┘ | v Outbox Dispatcher | v Kafka / SQS When an order is created, we persist: ORDER + OUTBOX EVENT inside the same transaction. Either both commit or neither commits. A separate dispatcher finds pending outbox records and publishes them to the broker. A simplified lifecycle could look like: PENDING | v PROCESSING | +──── success ────> PUBLISHED | └──── failure ────> RETRY / FAILED This removes the dangerous database-and-broker dual write from the business transaction. It also gives us something extremely useful: persistent delivery state. Make the Outbox Debuggable Reliability isn't only about retrying failed operations. Eventually, someone will need to answer: What happened to this event? A useful outbox should contain enough information to answer that question: eventId eventType source correlationId payload status attempts createdAt availableAt publishedAt lastError Instead of reconstructing everything from distributed logs, an engineer can inspect the actual state: SELECT * FROM event_outbox WHERE status = 'FAILED'; This leads to an important principle: Reliability mechanisms should also improve debuggability. If an event cannot be delivered, that failure should be visible and inspectable. Reliable Publishing Is Only Half the Problem Suppose our outbox works perfectly. Every event eventually reaches Kafka. We're still not finished. Most event-driven systems operate with at-least-once delivery. That means the same event may arrive more than once. Producer | v Kafka | +──── OrderCreated #123 ────> Consumer | +──── OrderCreated #123 ────> Consumer A consumer could successfully process an event and then crash before acknowledging it. The broker delivers the event again. If the handler sends an email, perhaps the customer receives two emails. If it performs a payment operation, the consequences can be considerably worse. Consumers therefore need to assume: Every event can arrive more than once. The Inbox Pattern The Inbox Pattern provides a durable record of received events. Before processing an event, the consumer registers its unique event ID. Broker | v Inbox Registration | +── event already exists ──> DUPLICATE | └── new event | v RECEIVED | v PROCESSING / \ v v PROCESSED FAILED If the same eventId arrives again, the consumer knows that it has already seen it. The event ID becomes an idempotency boundary. Instead of depending on exactly-once delivery, we make duplicate delivery safe. The Inbox Is More Than a Deduplication Table A minimal inbox could contain nothing more than processed event IDs. In a production system, however, it can become a durable history of event processing. Consider storing: eventId eventType source correlationId payload status attempts receivedAt processedAt availableAt lastError Now imagine an incident: Order 123 was created, but the downstream action never happened. You can inspect the inbox. Was the event received? Did processing start? Did the handler fail? How many attempts were made? When is the next retry? What was the last error? Those questions become much easier to answer when processing state is explicit. Retries Should Survive Application Restarts Spring provides excellent retry mechanisms. For example: @Retryable public void handle(OrderCreatedEvent event) { ... } This can be perfectly appropriate for short-lived transient failures. But durable event processing introduces another question: What happens if the JVM dies? Event processing fails | v Retry scheduled in memory | v Application restarts If retry state exists only in memory, it disappears with the process. For critical event processing, retry state can instead be persisted: FAILED | | availableAt PROCESSED | └──── failure ────> FAILED | + attempts++ + availableAt = next retry A scheduler periodically finds events whose retry time has arrived. Because the state lives in the database, restarting the application doesn't destroy the retry information. Back Off Instead of Hammering a Failing Dependency Retrying continuously can make an outage worse. If a downstream service is unavailable, thousands of failed events retrying as quickly as possible only add pressure. A better strategy is exponential backoff: Attempt 1 → immediate Attempt 2 → +1 second Attempt 3 → +2 seconds Attempt 4 → +4 seconds Attempt 5 → +8 seconds Eventually, the configured retry limit is exhausted. At that point, the event can remain explicitly failed: status = FAILED availableAt = null Automatic retries stop. But importantly, the failure doesn't disappear. The event remains available for investigation and operational recovery. What About Dead-Letter Queues? Dead-letter queues are useful, particularly for broker-level failures. But the broker's DLQ doesn't necessarily need to become the application's primary record of processing failure. There is a useful distinction: Broker concern Application concern Delivery failure Processing failure Malformed message Business handler failure Transport problem Retry exhaustion | | v v DLQ INBOX The two mechanisms can coexist. A database-backed inbox gives the application direct visibility into its own processing state, while a DLQ remains available for appropriate broker- and transport-level failures. Then You Deploy Multiple Pods Everything becomes more interesting once the application runs more than one instance. OUTBOX | pending event | ┌──────────┴──────────┐ v v Pod A Pod B Dispatcher Dispatcher Both instances may discover the same pending event. Without concurrency control, both may attempt to process it. Production implementations therefore need a concept of claiming or locking. For example: status lockOwner lockedAt An instance claims records transactionally before processing them. Other instances can then determine that those records are already being handled. But this creates another question: What happens if a pod claims an event and then dies? The system needs a deterministic mechanism for recovering stale claims after an appropriate timeout. At this point, the outbox is no longer just a database table plus a scheduled query. It has become infrastructure. Even the Scheduler Can Fail There is another failure mode that is surprisingly easy to overlook. Suppose the dispatcher completes successfully and schedules its next execution: Dispatcher completes | v schedule(nextRun) | X TaskScheduler rejects the task If the scheduler continues reporting itself as running, the application has entered a dangerous state. Everything appears healthy. But no future dispatch will happen. Events can quietly accumulate in the outbox. A reliable scheduler therefore benefits from an explicit lifecycle: STOPPED | v STARTING | v RUNNING | +──── scheduling failure ────> FAILED A useful invariant is: A scheduler must not report itself as running if no task is scheduled and no work is currently executing. Scheduling infrastructure itself needs observable failure semantics. Observability Is Part of Reliability Imagine receiving a production incident at 2 AM: We created the order, but the downstream system didn't process it. Ideally, you should be able to follow the event: Order | v Outbox Event | +── created +── claimed +── publish attempts +── published | v Broker | v Inbox Event | +── received +── processing attempts +── failure reason +── retry schedule +── processed Correlation metadata should connect the pieces. Payloads should be readable. State transitions should be explicit. Failures should remain inspectable. Logs should explain what the infrastructure is doing without becoming the only source of truth. A reliable system isn't only one that recovers from failures. It is one that helps engineers understand those failures. Putting It All Together Once these pieces are combined, the architecture starts looking like this: Business Transaction | v Transactional Outbox | v Outbox Dispatcher | v Kafka / SQS | v Inbox | v Idempotency Check | v Event Handler | +── Success | └── Durable Retry Unde