Seven Ideas That Keep Distributed Systems From Falling Over

Hello, I'm Maneshwar, and I'm building LiveReview - a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. How does Amazon stay up during Black Friday when a normal server would be on fire by 9am? Why do banks rarely lose a transaction even when a data center loses power mid-transfer? The honest answer is not "they don't have failures." Everyone has failures. Networks partition, disks die, servers get evicted by a bored Kubernetes scheduler for no reason anyone can explain. Reliability isn't about preventing that. It's about the system still doing the right thing while it's happening. There are seven ideas that keep showing up whenever you dig into how large systems actually stay reliable. I went through and a couple of them connect straight back to Kademlia and XOR distance, which I wrote about a while back. Small world. 1. The CAP Theorem: pick your poison CAP says a distributed system can give you at most two of these three, at the same time: - Consistency: every node sees the same data at the same time - Availability: every request to a non-failing node gets a response - Partition tolerance: the system keeps working even when the network splits Here's the part people skip past: partitions are not optional. Cables get cut, switches die, cloud regions lose connectivity to each other. Partition tolerance isn't a feature you choose, it's a fact of networked life. So in practice, CAP quietly becomes "choose consistency or availability, for the duration of the partition." Two real systems, two real answers: - Google Spanner picks consistency. It uses atomic clocks and GPS-synced time (the actual TrueTime API) to keep transactions linearizable across continents. During a partition, the majority side keeps serving reads and writes, the minority side drops to read-only until things heal. - Amazon DynamoDB picks availability. It keeps accepting writes during a partition and resolves conflicting updates afterward using timestamps. You always get a response. Sometimes it's a slightly stale one. Neither is wrong. A bank ledger wants Spanner's paranoia. Your Twitter feed does not need to be linearizable, it needs to not go down. Here's what "consistency" actually looks like on the wire: every node holding the exact same value at the exact same time, no stragglers, no stale reads sneaking through. That guarantee is expensive, which is exactly why not everyone pays for it. Now flip to availability. Here the system cares more about answering something than answering the latest thing. Notice the older values (t-1, t-2) still floating around some nodes below. That's the trade Dynamo makes on purpose, every request gets a response, even if it's a slightly out-of-date one. And then there's the partition itself, the thing that forces the choice in the first place. Below, a chunk of the ring has gone unreachable (the red X nodes). The surviving majority partition keeps talking to itself in teal, while the minority is cut off entirely, that's the moment Spanner would start rejecting writes and Dynamo would keep serving them. Put those three together and here's the decision every single node is quietly making the instant a partition happens, boiled down to one flowchart. If it can reach a quorum, life goes on as normal. If it can't, it has to pick a lane, reject the request or serve something possibly stale, and that one branch is the entire CAP theorem in practice. 2. Eventual consistency: it'll get there, chill Eventual consistency makes a deceptively small promise: if you stop writing, every replica will eventually agree. That's it. No promise about when. That sounds sloppy until you realize what it buys you. A write can return immediately without waiting for every replica to confirm, which is why Amazon's shopping cart lets you add an item even if a couple of backend servers are having a bad day. Your cart write doesn't block on all of them. The obvious follow-up question: what happens when two replicas get conflicting updates? Three common answers: - Last write wins (LWW): pick whichever update has the newer timestamp. Simple, and simple to lose data with, since "newer" depends on whose clock you trust. - CRDTs (conflict-free replicated data types): data structures that are mathematically guaranteed to converge no matter what order updates arrive in. No coordination needed, the math just works out. - Application-defined merges: you write the merge logic yourself, using whatever your business actually needs (e.g. "cart merge = union of items, not last-write-wins"). def merge_last_write_wins(local, remote): # simplest possible conflict resolution: newer timestamp survives return remote if remote.timestamp > local.timestamp else local DynamoDB typically converges within milliseconds under normal load, which is why "eventual" feels instant almost all the time and only bites you during actual network weirdness. Here's the shape of a typical eventually-consistent write path: client talks to a server, server talks to a primary, and the primary is the one source of truth everything else copies from. Nothing below the primary is guaranteed to be caught up the instant you write, it's guaranteed to get there. Zoom into that last hop and you can see the replication itself happening as four discrete steps: the write lands on the primary, gets acknowledged back to the caller, and only then gets pushed out to the replicas sitting behind it. That gap between step 2 (caller gets its answer) and steps 3-4 (replicas actually catching up) is the entire "eventual" in eventual consistency, and it's usually measured in milliseconds, not minutes. And yes, on a bad day that gap can stretch a lot further than milliseconds, which is basically this entire meme. 3. Load balancing: the bouncer at the door Load balancers spread incoming requests across servers, and the "simple" part of that sentence is doing a lot of lying. There are two layers to know: - Layer 4 balancers route on IP address and TCP/UDP port. Fast, because they never look inside the packet. - Layer 7 balancers read HTTP headers, URLs, even the request body, and route based on that. Smarter, more expensive. And routing algorithms have gotten past plain round robin: - Least connections: send the new request to whichever server currently has the fewest active connections - Least response time: same idea, but also factors in how fast each server has been responding recently, so a technically-idle-but-slow server doesn't get flooded Load balancers themselves need to not be a single point of failure, so they're usually deployed as a primary/secondary pair with a heartbeat, failing over in milliseconds if the primary drops. Consistent hashing (up next) is often what keeps a given client landing on the same backend server every time, which matters a lot if that server is holding session state in memory. At its simplest, this is the whole picture: two clients, one load balancer, three servers, and a routing decision made per request. Peek inside the load balancer itself and it's just a process terminating TCP/UDP connections and forwarding them onward, usually bound to one IP and port that every client hits. And here's round robin specifically doing its thing over a few requests, cycling evenly through Server A, B, and C regardless of how loaded any of them actually are. That "regardless of load" part is exactly why least-connections and least-response-time exist as smarter alternatives. 4. Consistent hashing: don't reshuffle the whole deck Quick problem statement: you've horizontally scaled your data across N nodes. Now you want to add or remove a node without moving nearly all of your data around. Plain modular hashing (node = hash(key) % N ) fails at this spectacularly. Change N by one, and almost every key maps to a different node. That's a full data migration triggered by adding a single server. Consistent hashing fixes this with a neat trick: put both the nodes and the keys on the same circular hash ring. - Hash each node to get its position on the ring - To find where a key lives, hash the key and walk clockwise until you hit the first node - Replicate to the next N-1 nodes clockwise, for redundancy Add a node, and it only takes over keys from its immediate neighbor. Remove one, and its keys shift to the next node over. Instead of remapping practically everything, you move roughly K/N keys, where K is total keys and N is node count. DynamoDB and Cassandra both lean on exactly this. If "hash space" and "ring" and "closest node wins" sound familiar, that's because Kademlia is solving a strikingly similar problem for peer discovery, just with XOR distance instead of clockwise distance on a ring. Different metric, same underlying move: stop routing through a central authority, let structure do the work. Here's the ring itself: every server hashed onto a fixed position, and every key just walking clockwise until it finds a server to land on. And here's the actual assignment happening for a handful of keys across three nodes, with the K/n math spelled out, four keys, three nodes, so each node ends up owning roughly one and a third keys' worth of the ring. 5. Circuit breakers: fail fast, on purpose Here's a failure mode that's sneakier than it sounds: one slow service starts a cascade. Service A calls Service B, B is struggling, so A's requests start piling up waiting on B, A's own threads exhaust, and now A is down too, even though A's own code was fine. Circuit breakers stop this by giving up on purpose, fast. Three states: - Closed: requests flow through normally - Open: requests fail immediately, without even attempting the call, buying the failing service time to recover - Half-open: a few test requests are let through to check if the service has recovered if failure_rate > threshold: state = OPEN # stop calling, fail fast elif state == OPEN and timeout_elapse