Kubernetes Networking [Level-2: Pod-to-Pod Communication]

This is Level 2 of our Kubernetes networking series. In Level 0, we covered core Linux networking primitives, and in Level 1, we learned how a Pod gets its own network namespace, eth0 interface, and IP address. Now it's time to answer one of the most important questions in Kubernetes networking: How does Pod A actually communicate with Pod B? We'll start with the simplest possible case and work our way up to a full hop-by-hop trace of cross-node Pod traffic - including routing, overlay networking, VXLAN, and BGP. Table of Contents - Our Scenario - The Kubernetes Networking Expectation - First Case: Same-Node Communication - How Pod A Knows Where to Send the Packet - Crossing the Node Boundary - The Node Routing Table - Who Creates These Routes? - Two Major Approaches: Routing vs Overlay - Overlay Networking: A Simple Analogy - Why We Need Encapsulation - VXLAN in a Nutshell - Routing Without an Overlay - Why BGP Shows Up Here - The Complete Cross-Node Packet Journey - Pod IP vs Node IP - Why You Can't Just Use Pod IPs on the Internet - Kubernetes Doesn't Mandate an Overlay - Same-Node vs Cross-Node: A Quick Reference - The Big Problem We've Just Created - What's Next: Kubernetes Services Our Scenario Let's set up a concrete example. We have two Pods: Pod A IP: 10.244.1.5 Node: node-1 Pod B IP: 10.244.2.5 Node: node-2 Visually: Kubernetes Cluster Node 1 Node 2 ┌───────────────┐ ┌───────────────┐ │ Pod A │ │ Pod B │ │ 10.244.1.5 │ │ 10.244.2.5 │ └───────┬───────┘ └───────▲───────┘ │ │ └──────────── Network ─────────────┘ Pod A wants to send traffic to Pod B: 10.244.1.5 → 10.244.2.5 . The Kubernetes Networking Expectation Kubernetes enforces an important networking model: Every Pod should be able to communicate with every other Pod, without anyone needing to manually manage routes for each Pod. Ideally, from the application's point of view: Pod A (10.244.1.5) → HTTP → Pod B (10.244.2.5) The application shouldn't need to know: - Is Pod B on Node 1 or Node 2? - Is the traffic going over VXLAN? - Is it using BGP? - Is it using eBPF? All of that complexity belongs to the networking layer, not the application. First Case: Same-Node Communication Let's start with the easier scenario: both Pods live on the same Node. Node 1 Pod A: 10.244.1.5 Pod B: 10.244.1.6 Conceptually, traffic flows like this: Pod A ↓ eth0 ↓ veth ↓ Node networking ↓ veth ↓ eth0 ↓ Pod B The exact implementation depends on the CNI plugin, but the core idea holds: Pod A → Node networking → Pod B How Pod A Knows Where to Send the Packet Remember ip route from Level 0? Every Pod has its own routing table. For example: default via 10.244.1.1 dev eth0 10.244.1.0/24 dev eth0 When Pod A wants to reach 10.244.1.6 , Linux checks: do I have a route for this? Yes - it falls within 10.244.1.0/24 . So Linux sends the packet out through eth0 . Crossing the Node Boundary Now the interesting case: Pod A and Pod B are on different Nodes. Pod A (10.244.1.5) on Node 1 ↓ Pod B (10.244.2.5) on Node 2 Pod A sends a packet with: Source: 10.244.1.5 Destination: 10.244.2.5 The packet leaves the Pod through eth0 , crosses the veth pair, and lands on Node 1's networking stack. Now Node 1 has to answer a critical question: Where is 10.244.2.5? The Node Routing Table Node 1 needs to know how to reach the Pod network living on Node 2. Conceptually, it might have a routing table like: 10.244.1.0/24 → local 10.244.2.0/24 → Node 2 In plain terms: 10.244.1.x → my local Pods 10.244.2.x → Pods on Node 2 So the flow becomes: Pod A → Node 1 → Route → Node 2 → Pod B This is the fundamental shape of all cross-node Pod traffic in Kubernetes. Who Creates These Routes? This is exactly where the CNI implementation becomes important. Different networking plugins solve this problem differently, using mechanisms such as: - Routing - Overlay networking - VXLAN - Geneve - BGP - Cloud-native routing - eBPF We'll dig into specific CNI implementations properly in Level 7. For now, just remember: CNI's job is to make sure the network knows how to reach every Pod IP. Two Major Approaches: Routing vs Overlay There are two broad strategies Kubernetes networking implementations use to make cross-node Pod traffic work. Approach 1 - Routing The underlying physical (or cloud) network is taught how to reach each Pod network directly: Pod A → Node 1 → Router → Node 2 → Pod B Approach 2 - Overlay The Pod's packet gets encapsulated inside another packet addressed between Nodes: Original packet: 10.244.1.5 → 10.244.2.5 ↓ encapsulation Outer packet: Node1-IP → Node2-IP | | carries ↓ 10.244.1.5 → 10.244.2.5 We'll explore both approaches in more depth below. Overlay Networking: A Simple Analogy Imagine sending a letter between two buildings, but the postal system only understands building addresses - not individual recipients. So you put your letter (addressed to a specific person) inside an outer envelope addressed to the building itself: Outer envelope: Node 1 → Node 2 Inside: Pod A → Pod B That's essentially what packet encapsulation does: Outer packet ┌──────────────────────────────┐ │ Node 1 → Node 2 │ │ │ │ Inner packet │ │ Pod A → Pod B │ └──────────────────────────────┘ Why We Need Encapsulation Suppose the physical network only knows about Node IPs, like 192.168.1.10 and 192.168.1.20 - it has no idea what 10.244.1.5 or 10.244.2.5 even are. The physical network understands Nodes, not Pods. So the trick is to communicate at the level the network does understand (Node to Node), while carrying the Pod-level conversation inside: Node 1 (192.168.1.10) → Node 2 (192.168.1.20) carries Pod A (10.244.1.5) → Pod B (10.244.2.5) VXLAN in a Nutshell VXLAN is one of the most common technologies used to implement overlay networking. You don't need to master every detail yet - just the high-level flow: Pod packet (10.244.1.5 → 10.244.2.5) ↓ VXLAN encapsulation ↓ Node 1 → Node 2 ↓ VXLAN decapsulation ↓ Pod B Or more generally: Pod → Overlay → Node → Network → Node → Overlay → Pod Routing Without an Overlay An alternative approach is to make the underlying network itself understand Pod networks directly - no encapsulation required. Node 1 - Pod CIDR: 10.244.1.0/24 Node 2 - Pod CIDR: 10.244.2.0/24 If the network knows: 10.244.1.0/24 → Node 1 10.244.2.0/24 → Node 2 ...then traffic can be routed directly, with no wrapping/unwrapping step: Pod A → Node 1 → Router → Node 2 → Pod B Why BGP Shows Up Here You'll eventually run into BGP (Border Gateway Protocol) in Kubernetes networking discussions - don't let the name intimidate you. At a high level, BGP is simply a protocol for distributing routes across the network. For example: Node 1 announces: "10.244.1.0/24 is reachable through me" Node 2 announces: "10.244.2.0/24 is reachable through me" Once these routes propagate, the network can route Pod traffic directly - no overlay needed. Some CNI implementations use BGP specifically for this purpose. The Complete Cross-Node Packet Journey Let's trace a full request from Pod A (10.244.1.5 ) to Pod B (10.244.2.5 ) step by step. Step 1 - Application: Pod A's application sends GET /api to 10.244.2.5:8080 . Step 2 - Pod network namespace: Linux sees source 10.244.1.5 , destination 10.244.2.5 . Step 3 - Pod eth0: The packet leaves the Pod via eth0 . Step 4 - veth: The packet crosses the veth pair from the Pod namespace into the Node namespace. Step 5 - Node routing: Node 1 asks "where is 10.244.2.5?" and finds that 10.244.2.0/24 → Node 2 . Step 6 - Cross-node network: Depending on the CNI implementation, this hop uses routing, an overlay, eBPF, or cloud-native networking. Step 7 - Node 2: The packet arrives and Node 2 determines that 10.244.2.5 belongs to Pod B. Step 8 - Pod B: The packet travels Node 2 → veth → Pod B network namespace → eth0 → application . Putting it all together: Pod A (10.244.1.5) ↓ eth0 → veth → Node 1 ↓ Routing / Overlay / eBPF ↓ Node 2 → veth → eth0 ↓ Pod B (10.244.2.5) Pod IP vs Node IP This distinction trips up a lot of people, so let's be explicit. Suppose: Node 1: 192.168.1.10 → Pod A: 10.244.1.5 Node 2: 192.168.1.20 → Pod B: 10.244.2.5 These live in two entirely separate networks: Node network: 192.168.1.0/24 Pod network: 10.244.0.0/16 A Pod's IP is not the same as, or derived from, its Node's IP. Keep this mental separation clear - it will save you a lot of confusion later. Why You Can't Just Use Pod IPs on the Internet A private Pod IP like 10.244.1.5 isn't reachable directly from the public internet. Instead, traffic typically flows like this: Internet ↓ Load Balancer / Ingress ↓ Kubernetes ↓ Service ↓ Pod This is precisely why Kubernetes has dedicated abstractions - Services, Ingress, and Gateway API - which we'll cover soon. Kubernetes Doesn't Mandate an Overlay Here's an important point, especially if you're prepping for interviews: it's a common misconception that "Kubernetes networking uses an overlay network." That's not universally true. Kubernetes defines a networking model (every Pod can reach every other Pod by IP), but it doesn't dictate how that model is implemented: Kubernetes networking model | ┌─────┼─────┐ ↓ ↓ ↓ Routing Overlay eBPF The specific CNI plugin you choose determines the actual implementation. Same-Node vs Cross-Node: A Quick Reference | Traffic | What Happens Conceptually | |---|---| | Pod → Pod, same Node | Node-local networking | | Pod → Pod, different Nodes | Cross-node routing/overlay/dataplane | | Pod → Internet | Node/network egress path | | Internet → Pod | Usually LoadBalancer/Ingress/Gateway + Service | | Pod → Service | Service dataplane selects a backend Pod | That last row is exactly the problem we tackle next. The Big Problem We've Just Created So far, Pod A → Pod B works fine. But there's a serious practical problem lurking here. Suppose we have three backend Pods: Backend Pod 1: 10.244.1.5 Backend Pod 2: 10.244.2.5 Backend Pod 3: 10.244.3.5 The frontend needs to call the backend - but which IP should it use? And what happens when Backend Pod 1 dies and Kubernetes repla