Failure Engineering Explained by Uncle to Nephew - Episode 2: Types of Failures
Part 1 - Hardware Failure
👦 Nephew: This one's obvious anyway - I deploy to AWS. The cloud hides hardware failure from me.
👨🦳 Uncle: Does it?
👦 Nephew: ...doesn't it? That's the whole point of paying for EC2 instead of buying a server.
👨🦳 Uncle: Let's trace it. Your app sits on an EC2 instance. What's underneath the instance?
👦 Nephew: Virtual machine stuff, I guess?
👨🦳 Uncle: And underneath that?
👦 Nephew: ...an actual physical machine somewhere. In a data center.
👨🦳 Uncle: There it is.
Your app
|
| "Virtual" server (EC2/Droplet)
|
| ACTUAL physical hardware somewhere in a data center
|
| Still capable of failing - just less visible to you
👦 Nephew: So it's not hidden. It's just one layer further away than I thought.
👨🦳 Uncle: Exactly. AWS absorbs a lot of it - that's part of what you're paying for - but disks still fail, instances still get abruptly terminated, whole availability zones still go down. That's Hardware Failure.
Hardware Failure Examples
- SSD/HDD fails, data on that disk is unreadable
- RAM stick develops a fault, random crashes
- A physical server loses power
- A whole data center rack fails
👦 Nephew: This is the Chaos Monkey thing from last time.
👨🦳 Uncle: That's exactly it. Chaos Monkey exists because Netflix can't opt out of hardware failing. They just refused to be surprised by it.
Part 2 - Software Failure
👨🦳 Uncle: Look at this and tell me what's wrong.
app.get('/user/:id', async (req, res) => {
const user = await db.findUser(req.params.id);
res.json(user.profile);
});
👦 Nephew: If user is null, user.profile crashes.
👨🦳 Uncle: Right. What category is that?
👦 Nephew: Easy - software failure. My bug.
👨🦳 Uncle: Now - remember the memory leak we covered in the Node Internals series? Something holding a reference so GC can't collect it, and the process slowly OOMs?
👦 Nephew: Yeah.
👨🦳 Uncle: Same category. So is a null check crash the same severity as a slow memory leak that kills your process three days later?
👦 Nephew: ...no. Not even close.
👨🦳 Uncle: Right. Same label. Wildly different scale. A missing null check takes down one request, maybe your process if nobody caught it. A leak takes down the whole thing, just slower. Worth remembering when you're deciding how much time a fix actually deserves.
Software Failure Examples
- Null/undefined reference crashes a function
- Infinite loop consumes 100% CPU
- A dependency's new version silently breaks your code
- Unhandled promise rejection kills the process
- Memory leak grows until the process OOMs
Part 3 - Network Failure
👦 Nephew: My request timed out yesterday, on the Pilooopu admin panel. Was that the server being down?
👨🦳 Uncle: Was it?
👦 Nephew: I mean... I assumed so. It just hung.
👨🦳 Uncle: Let's trace what "hung" actually means. Client sends a request. What has to happen before the server even sees it?
👦 Nephew: DNS resolves the domain, then... it goes through routers, I guess, to reach the server.
👨🦳 Uncle: And if one of those routers drops the packet?
👦 Nephew: The server never even gets the request.
👨🦳 Uncle: So from where you're sitting -
👦 Nephew: - it looks exactly the same as the server being down. A timeout either way.
Client Network Server
| | |
| ---- request --------->| |
| |X packet dropped here |
| | |
| (client sees a timeout, server never even saw the request)
👨🦳 Uncle: That's Network Failure - and that's exactly why it's miserable to debug. Same symptom, four possible systems responsible, and none of them are the one you're staring at.
Network Failure Examples
- DNS lookup fails or times out
- Packet loss between your server and the database
- A load balancer misroutes or drops connections
- TLS handshake fails
- Latency spikes without an outright failure
Part 4 - Database Failure
👦 Nephew: Speaking of hanging - remember my app just froze under load a few months back? No errors, nothing in the logs?
👨🦳 Uncle: I remember. What did you eventually find?
👦 Nephew: Connection pool, I think. But I never really confirmed why.
👨🦳 Uncle: Let's confirm it now. Pool size ten. Ten requests come in and hold their connections a bit too long. Eleventh request arrives -
👦 Nephew: - no connection left for it. It just waits.
Connection Pool (size: 10)
+----------------------------------+
| [used][used][used]...[used]| ← all 10 in use
+----------------------------------+
| Request #11 arrives
| No free connection available
| Request waits... and waits... eventually times out
👦 Nephew: So that means MySQL actually crashed, right? That's what caused it?
👨🦳 Uncle: No - that's exactly the mistake everyone makes. MySQL was probably sitting there completely healthy the entire time.
👦 Nephew: Then what was actually broken?
👨🦳 Uncle: Your app. Holding connections too long, not releasing them properly. Database Failure, but the database itself is often innocent.
Database Failure Examples
- Connection pool exhausted - no free connections left
- A slow query locks a table, backing up every other query
- Replication lag - replica serves stale data
- Disk fills up, writes start failing
- Primary node crashes, no failover configured
👦 Nephew: So the database's failure was never the database's fault.
👨🦳 Uncle: Almost never is.
Part 5 - Third-Party Failure
👦 Nephew: Here's a theory. If I didn't write the code - Stripe, SendGrid, whatever - it's not really my problem when it breaks. That's on them.
👨🦳 Uncle: Is it?
👦 Nephew: ...I feel like you're about to ruin this theory.
👨🦳 Uncle: Let's trace it. Razorpay goes down. Your server calls Razorpay and waits. What does "waits" mean for the requests behind it?
👦 Nephew: They queue up behind the one that's stuck.
👨🦳 Uncle: And if Razorpay takes ninety seconds to time out?
👦 Nephew: Then every user behind that one call is stuck for ninety seconds too. Even the ones who have nothing to do with payment.
Your Server --- calls ---> Third-Party API
| |
| Third party goes down |
| |
| Your server, if not careful, hangs waiting on that call too
| |
| Your OWN healthy users start getting stuck behind it
👦 Nephew: So their outage becomes my outage. For free.
👨🦳 Uncle: That's Third-Party Failure. You genuinely can't fix their five minutes of downtime - but you control whether it becomes your five minutes too. That's the entire reason Circuit Breaker exists, which we'll get to in Module 3.
Third-Party Failure Examples
- Payment gateway (Stripe/Razorpay) times out mid-transaction
- Email service (SendGrid) rate-limits or goes down
- A third-party auth provider (Google/Facebook login) is slow
- An external weather/maps/data API changes its response format
👦 Nephew: Great. So now even my database has trust issues, and so does everything outside it.
Part 6 - Human Error
👦 Nephew: This category feels unfair to even call a "type of failure." It's not the system. It's just someone screwing up.
👨🦳 Uncle: Is it, though? Say someone on your team pushes a bad env variable to production by accident. Who's at fault?
👦 Nephew: ...the person who pushed it?
👨🦳 Uncle: Or - was there no review step that would've caught it?
👦 Nephew: ...there wasn't, actually.
👨🦳 Uncle: Then who's really at fault - the person having a bad Friday at 5 PM, or the process that let one bad Friday reach production?
Junior framing
↓
Someone made a mistake, fire them.
Senior framing
↓
The SYSTEM allowed one honest mistake to cause this much damage. Fix the system.
👦 Nephew: So the fix isn't "be more careful."
👨🦳 Uncle: "Be more careful" isn't a fix. It's a hope. The real fix is code review, staging environments, a confirmation step before anything destructive.
Human Error Examples
- Wrong environment variable in production
- Accidentally deployed to the wrong environment
- A bad migration deletes/corrupts data
- Force-pushed over someone else's changes
- Misconfigured a firewall rule, blocked legitimate traffic
Part 7 - Resource Exhaustion
👦 Nephew: Isn't this basically the event loop thing from Node Internals? Getting overwhelmed under a traffic burst?
👨🦳 Uncle: Close. Let's actually trace where it breaks, because "overwhelmed" is doing a lot of work in that sentence. Traffic climbs. What runs out first?
👦 Nephew: Memory? CPU?
👨🦳 Uncle: Could be either, or disk, or file descriptors, or even a rate limit on an API you call. Point is - it's rarely instant.
Normal: memory/CPU usage stays flat
Rising load: usage climbs steadily
Exhaustion: usage hits the ceiling → OS kills the process, or the event loop can't keep up with new requests
👦 Nephew: So this is the category where success is what kills you. Too many users show up, and that's the failure.
👨🦳 Uncle: That's Resource Exhaustion, and exactly why it's last on this list - and honestly the one worth planning for earliest, the moment your product starts actually growing.
Resource Exhaustion Examples
- Out of memory (OOM) - process gets killed by the OS
- CPU pegged at 100%, event loop can't keep up
- Disk full - logs or temp files fill the disk
- Too many open file descriptors / socket connections
- Rate limit hit on an external API you depend on
Part 8 - Putting the Seven Together
👨🦳 Uncle: Try something. Razorpay goes slow on you - third party failure. Walk me through what happens next, using what you now know.
👦 Nephew: Okay... my requests to Razorpay start piling up. That eats into my connection pool, so - database failure?
👨🦳 Uncle: Keep going.
👦 Nephew: All those pending requests are sitting in memory, waiting. So memory climbs. Resource exhaustion.
👨🦳 Uncle: And then?
👦 Nephew: ...the process just dies. Software failure, technically, since the crash itself is code failing to handle it.
Example chain:
Third-party payment API is slow (Third-Party Failure)
|
| Your requests to it start piling up
|
| Connection pool gets exhausted (Database Failure)
|
| Memory usage climbs holding pending requests (Resource Exhaustion)
|
| Process crashes (Software Failure)
👨🦳 Uncle: You just traced a real incident shape without me saying a word. One small external failure, left unhandled, cascades through your own system until it takes the whole thing down. Timeout, Circuit Breaker, Bulkhead - Module 3 - each one exists to break that chain at a different link.
👦 Nephew: Basically everything hates me.
👨🦳 Uncle: Welcome to production.
What we covered in Episode 2
- Hardware Failure - physical machines still fail, even in the cloud
- Software Failure - bugs, unhandled errors, memory leaks; broadest category, variable blast radius
- Network Failure - the path between two healthy machines can still break
- Database Failure - often the app's fault (connection pool misuse), not the database's
- Third-Party Failure - you can't control it, but you control your reaction to it
- Human Error - reframed as a process/guardrail failure, not a personal one
- Resource Exhaustion - memory, CPU, disk, file descriptors; usually a ramp, not a cliff
- How a real incident often chains across several categories at once
Next up - Episode 3: "Failure Detection" - timeouts, health checks, heartbeats, logs, and monitoring: how a system finds out something broke before a user has to tell you.
Comments
No comments yet. Start the discussion.