Designing a Flash-Sale Seat Reservation System in AWS (Part 1): The Architecture

TL;DR: A national certification exam had 20,700 seats across five levels. When registration opened, far more people tried to book than there were seats. The previous system crashed under that load. It also sold the same seat twice. The new design follows three rules. First, one atomic Redis script decides who gets a seat. (Atomic means the whole check-and-count runs as one step that nothing can interrupt.) No database takes part in that decision. Second, the request path does only three things: admit the booking in Redis, create a payment, and put a record on a queue. Everything else happens later, in the background. Third, everything that can be a static file is served from the edge (the CDN, close to the user). CloudFront, S3 and an AWS WAF web ACL handled about three out of four requests without touching a server. We sized the API fleet from load tests and added servers before the event. We did not use autoscaling. In the first hour it served 1.5M+ requests at the origin, peaked at 54.8K requests per minute, kept median latency under 10 ms, returned a server error on fewer than 0.01% of requests, and sold every seat with zero double bookings. This is Part 1 of a three-part series. It is written as a design reference: the challenges, the options we compared, and why we chose what we chose. - Part 1 (this post): the problem, the architecture, the edge, and capacity. - Part 2: never selling a seat twice, covering the admission script, retries, and failover. - Part 3: holds, payments, and the slow path, covering the hold lifecycle, payment signals we cannot trust, and durable storage that runs in the background. The problem What made this hard? The work itself looks simple: a form with a candidate code, an email address and an exam level, then a payment. The traffic pattern is what makes it hard. Registration opens at an announced minute, so almost everyone arrives in the same few seconds. They keep refreshing the page until they get a seat or the seats run out. The previous system crashed at exactly that moment. It also sold some seats twice. The client's requirement was simple and direct: stay up, and never book a seat twice. Written out as requirements: | Requirement | Detail | |---|---| | Hard caps | A cap (a maximum) per level, for five levels, and a global cap of 20,700. Never exceeded, not even by one seat. | | One booking per candidate | A candidate code can hold at most one live booking. | | Eligible candidates only | The code must be on an allowlist loaded in advance, and the email must match the one we have stored for that code. | | Pay to keep the seat | A seat is held for 10 minutes while the candidate pays through the payment gateway. If the payment is not completed, the seat returns to the pool of free seats. | | Money-safe | Nobody is charged for a seat they did not get, and nobody who paid loses their seat. | | Available under the burst | The page must load and answer quickly, even when everyone arrives at once. | | First come, first served | No lottery. Whoever is admitted first gets the seat. | "Never exceed the cap" and "one booking per candidate" are the two rules that define correctness. Part 2 is entirely about them. The rest of this post is about surviving the burst while keeping those rules true. The shape of the load What does a flash sale look like in the traffic numbers? Figure 1 shows the real traffic around opening time, grouped into five-minute buckets from CloudWatch. Three things stand out, and each one shaped the design: - The ramp is a cliff. In other words, traffic does not grow slowly; it jumps almost straight up. Origin traffic went from near zero to about 12K requests per minute in the first five minutes after opening, and to more than 50K per minute within twenty minutes. Anything that reacts to load, such as an Auto Scaling policy, a function with a cold start, or a database that scales its capacity, reacts too late. The moment that matters has already passed. - Most requests are not bookings. People reload the page and check availability long before and long after they submit the form. In the first hour CloudFront saw about 6.2M requests. Only about 1.5M of them reached the servers. Every request the edge answers is one that the booking path never has to handle. - Most of the origin traffic is rejections. Once seats run low, most /book calls end with a fast "no": duplicate, level full, or closed. A rejection must cost almost nothing, because there are far more rejections than admissions. Challenge 1: where do we decide who gets a seat? What is the single hardest decision in this system? Every booking asks the same question: is there still a seat at this level, and has this person already booked one? Thousands of requests ask it at the same instant. The answer must be exactly right for every one of them. Whatever component answers that question is the bottleneck of the whole system, so we chose it first. We considered four options. | Option | How it works | Why not (or why) | |---|---|---| | A. Relational database row lock | SELECT ... FOR UPDATE on a seats_left row, then insert the booking | Correct, but every booking must wait its turn on one row. Lock waits build up, connection pools run empty, and everything times out at the same time. This is the classic failure of older systems of this kind. | | B. DynamoDB conditional counters | UpdateItem with seats_left > 0 on a counter item, plus a transaction for the duplicate check | Correct and serverless, but a single counter item is a hot key (one item that every request writes to). One partition limits how many writes per second it can take. Adding a duplicate check means a transaction on every attempt, and even a rejection still costs a write. | | C. Virtual waiting room | Put everyone in a queue and let people in at a fixed rate | Protects the backend, but it is a whole product to build or buy. Users wait for minutes, and you still need an atomic counter at the end of the queue. | | D. One atomic script in Redis | A Lua script checks for a duplicate, checks the level cap and the global cap, and records the admission, all in one call | Redis runs a script from start to finish before it runs anything else, so no lock is held while waiting on the network. It costs one round trip of microseconds, and a rejection is only a read. The open questions, durability and failover, are answered in Part 2. | We chose D. The deciding argument was this: Redis runs commands on a single thread, and that single thread is the lock. It is held only for the microseconds a script runs, never for a network round trip. The same property makes rejections nearly free. That matters because most of the traffic is rejections. The cost is that the source of truth for "who has a seat" now lives in memory. That creates three problems that Part 2 has to solve: a reply lost on the network, a process crash between steps, and a failover that loses the last few writes. We accepted those problems with open eyes. The other options had problems we could not solve within a single burst. We also chose the smallest possible Redis setup: one shard (cluster mode disabled), one primary and one replica in different Availability Zones, with automatic failover. The script touches several keys at once. On one shard, that needs no extra work to keep the keys together (no hash tags). As Figure 4 shows later, a single cache.r6g.large primary never went above 10% engine CPU. Challenge 2: keep the hot path short What exactly happens inside a booking request? The previous system did all its real work inside the request. It wrote the booking to the database, sent the email and updated the counters, all while the user waited. Under a burst, each of those steps becomes a waiting line, and the slowest one sets the latency for everyone. We split the work into a hot path (the steps the user waits for) and a slow path (the steps that run after the response is sent). On the hot path, POST /book does exactly three things: - Admit the booking in Redis with the atomic script. - Create a payment with the payment gateway and get back a payment URL. - Enqueue a HELD record, which means send it to an SQS FIFO queue. It then returns 202 Accepted with the payment URL, and the browser redirects to the gateway. If step 2 or step 3 fails, the API undoes the admission before it responds. A failure never leaks a seat (leaves a seat counted with nobody holding it). The database is deliberately missing from this list. A worker reads the queue and writes the records into DynamoDB a few seconds later. Payment confirmations, expiry, emails and the back-office sync all run on the slow path. Part 3 covers that path in detail, including why it is safe for the database to be a few seconds behind the seat count. The payment gateway call is the one dependency on the hot path that we do not control. It is the only reason the hot path is not a pure in-memory operation. It is also the only slow step, and it shows in the slowest requests: p99 latency was 4.1 s in the first five minutes, when every admitted user was creating a payment at the same time. After that it settled to about 0.3 s. The median stayed at 2 to 5 ms the whole time, because most requests were fast rejections that never reached the gateway. Placing the gateway call after admission is what makes this work: rejected requests never wait for it. Challenge 3: keep traffic off the servers How do you make three out of four requests never reach your code? The cheapest request is the one your servers never see. We put everything we could behind CloudFront. - The page itself is a static HTML file in a private S3 bucket, served through CloudFront with Origin Access Control. Page loads never touch a server. - Availability counts (seats left per level, and whether booking is open or closed) are a small status.json document in the same bucket. The API rewrites it as the counts change. The browser reads it once when the page loads. After that, the page reacts