Beyond Happy Path Engineering: the Network

What happens when network calls stop behaving like clean request/response interactions

Timeouts, retries, duplicate side effects, idempotency, backoff, circuit breakers, load shedding, degraded states, observability, etc.

The Problem Space

Network calls are inherently unreliable. Unlike local function calls, they can fail in ways that leave the caller uncertain about whether the operation succeeded or not. This uncertainty cascades through the system.

Key Concepts

• Timeouts – A hard limit on how long to wait for a response. Without them, a single slow dependency can tie up resources indefinitely.
• Retries – Repeating a failed request, often with backoff. Retries amplify load on already-stressed systems if not carefully managed.
• Duplicate side effects – The same operation executed more than once due to retries or network duplication. This is the core danger of retries.
• Idempotency – Designing operations so that executing them multiple times produces the same result as executing them once. The primary defense against duplicate side effects.
• Backoff – Waiting an increasing amount of time between retries. Common strategies include fixed, incremental, and exponential backoff with jitter.
• Circuit breakers – A pattern that stops making requests to a failing dependency once a failure threshold is reached, allowing it time to recover.
• Load shedding – Deliberately dropping requests when the system is under too much load, rather than attempting to serve them and making things worse.
• Degraded states – Continuing to operate with reduced functionality when dependencies are unavailable, rather than failing entirely.
• Observability – The ability to understand what is happening inside the system through logs, metrics, and traces, especially during failures.