Kubernetes Networking Was Easy - Until Production Got Real

When I first started managing microservices at scale, Kubernetes felt like magic - until it didn’t. Pods restarted, requests vanished into the void, and tracing cross-namespace traffic became a weekend sport. At first, the networking model seemed almost too simple. A Pod gets an IP. A Kubernetes Service gives that Pod a stable virtual endpoint. CoreDNS handles service discovery. The CNI provides pod-to-pod connectivity. kube-proxy handles service traffic using mechanisms such as iptables or IPVS. For a small cluster, that model works remarkably well. Then you have 150 services. Then 500. Then teams start deploying independently, services communicate across namespaces, traffic crosses availability zones, a few workloads become extremely chatty, and somebody introduces three layers of retries because the network is sometimes flaky. That's when the networking stops being infrastructure you can ignore. You start asking questions that Kubernetes Services alone don't really answer: - Which service is calling this endpoint? - Why did this request get a 503? - Which version received it? - Is the connection failing, or is the application returning the error? - Why did latency jump only for traffic crossing namespaces? - Is this workload actually talking to the service it thinks it is? This is where a service mesh starts making sense. And this is also where you discover that a service mesh isn't free. The Architectural Shift: From Kubernetes Services to a Service Mesh Kubernetes gives you the basic network primitives. The CNI establishes the network between Pods. A Service provides a stable virtual IP and load-balancing abstraction. kube-proxy programs the node networking rules required to direct Service traffic toward backend Pods. That's enough to answer: How does service-a reach service-b? But production systems eventually ask: How should service-a reach service-b? Those are very different questions. Suppose payments has two versions: payments-v1 → 90% payments-v2 → 10% Kubernetes Services don't natively give you application-aware traffic splitting based on HTTP headers, cookies, weights, or request properties. You can create separate Services, manipulate Deployments, introduce an ingress controller, or build application-level routing logic. But now routing logic starts leaking into multiple layers. Istio moves much of that policy into the networking layer. The basic architecture becomes: Istio Control Plane │ Configuration / Certificates │ ▼ ┌─────────────┐ │ Envoy │ │ Proxies │ └─────────────┘ │ ┌──────────────────────┼──────────────────────┐ ▼ ▼ ▼ service-a service-b service-c + Envoy + Envoy + Envoy The important distinction is between the control plane and data plane. The Istio control plane manages configuration and security material. The Envoy proxies sit in the traffic path and actually handle requests. That distinction matters enormously when debugging. The control plane can be struggling while existing Envoy configuration continues serving traffic perfectly well. Understanding that separation saves a lot of unnecessary panic during incidents. The Kubernetes Layer Still Matters One mistake I see repeatedly is treating Istio as if it replaces Kubernetes networking. It doesn't. Your underlying Kubernetes networking still needs to work. A typical request might look like: Pod A │ ▼ Envoy sidecar │ ▼ ClusterIP │ ▼ Kubernetes networking │ ▼ Envoy sidecar │ ▼ Pod B The mesh adds another layer of behavior on top of Kubernetes networking. That means there are now more places where things can go wrong. DNS → Service → CNI → routing → Envoy listener → Envoy cluster → application When someone says: The service is reachable. That statement is almost meaningless without knowing from where, through what path, and at which layer. Ingress Isn't Service-to-Service Networking Another production lesson is keeping north-south and east-west traffic conceptually separate. North-south traffic is traffic entering or leaving the cluster. East-west traffic is traffic between workloads inside the cluster. An Istio Ingress Gateway handles the former. For instance: Internet │ ▼ Load Balancer │ ▼ Istio Ingress Gateway │ ▼ VirtualService │ ▼ service-a Inside the cluster: service-a │ ▼ Envoy │ ▼ service-b │ ▼ Envoy These are different traffic-management problems. The ingress gateway is your controlled entry point. The sidecars or ambient data plane handle service-to-service communication. Mixing those responsibilities makes architecture and troubleshooting unnecessarily difficult. 1. Securing the Perimeter: Ingress and Egress A production cluster shouldn't have every workload freely reaching the internet. Direct Pod egress looks convenient: Pod │ ├── api.example.com ├── payment-provider.com ├── random-third-party.com └── anything-else The problem isn't simply security. It's control. When something goes wrong, you want to know: - Which workload made the connection? - Where did it connect? - Was the destination approved? - What protocol was used? - Can we block it centrally? - Can we observe the traffic? An Istio Egress Gateway gives you a controlled exit point: Pod │ ▼ Envoy │ ▼ Egress Gateway │ ▼ External Service Now outbound traffic can be governed at a predictable boundary. That doesn't mean every organization needs to force every packet through an egress gateway. You pay for centralized inspection with additional hops, infrastructure, configuration, and failure modes. The right question isn't: Can we put everything through the egress gateway? It's: Which external traffic actually needs centralized policy and visibility? That distinction matters. 2. mTLS: Zero Trust Without Breaking Everything One of Istio's strongest capabilities is mutual TLS. Instead of: service-a ───── HTTP ─────> service-b you can have: service-a │ Envoy │ │ mTLS ▼ Envoy │ service-b The application doesn't necessarily need to manage certificates itself. The mesh handles identity and encryption between workloads. But switching an existing production environment directly to: mode: STRICT can turn a quiet Tuesday into a very long night. Why? Because not everything is necessarily inside the mesh. You might have: mesh workload │ ▼ legacy service │ X TLS required The legacy workload doesn't have an Envoy sidecar and therefore cannot participate in mesh mTLS in the same way. This is where PeerAuthentication modes matter. PERMISSIVE Accept both plaintext and mTLS. Useful during migration. mTLS ────────┐ ├──> workload plaintext ───┘ STRICT Require mTLS. plaintext ───> rejected mTLS ────────> accepted A safer migration looks like: Phase 1 PERMISSIVE ↓ Inject sidecars ↓ Verify workload communication ↓ Identify legacy clients ↓ Migrate dependencies ↓ STRICT Don't turn on STRICT because the architecture diagram says everything is meshed. Production traffic doesn't care what the architecture diagram says. 3. Traffic Shifting Without DNS Games Canary deployments are another area where Istio becomes extremely useful. Without mesh-level routing, teams sometimes create: payments-v1.example.com payments-v2.example.com and manipulate DNS or load balancers. DNS isn't designed to provide precise request-level traffic control. Caching, TTLs, resolvers, client behavior, and connection reuse all get involved. Istio lets you shift traffic directly at the request-routing layer. A VirtualService can express something conceptually like: payments │ ├── v1 → 90% │ └── v2 → 10% Then: 90% → payments-v1 10% → payments-v2 You can gradually move: 100 / 0 ↓ 95 / 5 ↓ 90 / 10 ↓ 75 / 25 ↓ 50 / 50 ↓ 0 / 100 The interesting part isn't the YAML. It's what you can do without changing application code or DNS. But traffic splitting introduces another responsibility: knowing what you're actually measuring. If v2 receives 10% of traffic but happens to receive the most expensive customer requests, raw request percentages can become misleading. Traffic management is easy. Traffic management with meaningful telemetry is the real engineering problem. DestinationRules: Where Routing Gets Interesting A VirtualService describes how requests should be routed. A DestinationRule describes policies applied to traffic going toward a destination. This is where things such as subsets, connection pools, circuit breakers, outlier detection, and TLS behavior start becoming relevant. For instance: payments │ ├── subset: v1 │ └── subset: v2 VirtualService: route 90% → v1 route 10% → v2 DestinationRule: v1 → connection policy v2 → connection policy That separation becomes valuable once routing rules grow beyond simple send traffic here. It also becomes a source of configuration complexity. We'll get to that. The Battle Scars: The Istio Tax Let's talk about the part nobody gets excited about during the architecture presentation. Every sidecar consumes resources. One proxy doesn't sound like much. Now multiply it. 500 workloads × 1 Envoy proxy = 500 additional processes At low traffic, this can look harmless. Under high concurrency, it isn't. Envoy maintains connections, buffers data, processes HTTP, performs TLS operations, tracks metrics, handles filters, and maintains configuration. Memory consumption can become particularly painful. Imagine a cluster where application Pods were sized carefully: Application: 500Mi memory Envoy: 250Mi memory Suddenly your 500 MiB application workload isn't a 500 MiB workload anymore. It's closer to: 750MiB+. And that changes: node packing, autoscaling, eviction pressure, cluster cost, and pod startup behavior CPU can also spike under heavy request rates, TLS operations, logging, or complex filters. This is why blindly enabling sidecars everywhere is dangerous. Control the scope Istio's Sidecar resource can be used to constrain the configuration visibility available to workloads. That's important in larger environments. A proxy doesn't necessarily need configuration for every service in the cluster. If a workload can only communicate with payments, orders, and identity there's little reason for its E