❓ Episode 10 new router who dis

Hey all 👋

Last episode ended with a victory lap: a two-node, git-defined, nightly-backed-up, tunnel-fronted platform, and a sentence I stand by: the router remains the weakest member of the team and has been architected around accordingly.

Reader, I changed the router. The old one had been established, across several episodes, as this saga's least reliable character. The new one arrived, got plugged in, and did what every new router does on its first day: it handed out addresses from a different subnet.

My whole network moved from 192.168.68.x to 192.168.0.x:

A note on format: this is my first RCA-style post on this blog. Real RCAs have an impact statement, a clean timeline, and one tidy root cause, written by someone calm. Mine has an impact statement ("kubectl said EOF"), a timeline ("evening, then more evening"), and a root cause that kept growing a plus-one every time I fixed the previous one. I am calling it an RCA anyway, because "Seven Root Causes and a Port Number" did not fit in the title bar.

Here is what I learned about my own architecture. It had been carefully built around the router's flaws. It had never once been built around the router's address range. Episode 3 me even wrote this down, in the lessons list, in plain text: reserve the address at the router level, not just on the box. Episode 3 me was talking to a wall, and the wall was me.

By the end of the night I had found seven places where my cluster had the old address memorized, and none of them had been told about the move. Here they are, in the order I found them.

🚨 Symptom One: kubectl Says "Unexpected EOF"

$ kubectl get nodes -o wide
Unable to connect to the server: unexpected EOF

My first theory was certificates, because it is everyone's first theory. I added the new IP to csr.conf and csr.conf.template on msi, ran microk8s config, and pasted the output over my workstation's kubeconfig. Same error, delivered with the same calm.

That should have told me something. A certificate mismatch is not shy: it says x509: certificate is valid for X, not Y, out loud, in a full sentence. An EOF is the sound of a door closing.

So I started ruling things out, from the outside in:

• The kubeconfig server: line already pointed at https://192.168.0.201:16443.
• No KUBECONFIG override, no proxy variables.
• ip -4 addr on msi showed 192.168.0.201.
• nc -vz 192.168.0.201 16443: open. kubelite was listening on it.

And the one that confused me most:

$ curl -vk https://192.168.0.201:16443/version
... TLS handshake completes ...
< HTTP/2 401
{ "kind": "Status", "reason": "Unauthorized", "code": 401 }

A clean TLS handshake and a polite 401. The API server was awake, reachable, and well-mannered. This is Episode 7's trap with the sign reversed: a healthy-looking API server proves the doorman is on shift, not that anyone is home.

The decisive test was boring. I ran the same command on msi itself:

$ sudo microk8s kubectl get nodes
^C

It hung. A command that hangs identically on the machine itself has stopped being a networking problem and become a cluster problem.

So: logs. kubelite first:

awaiting required bookmark event for initial events stream, no events received for 30s

The API server was waiting for events from its datastore, and the datastore was not sending any. So the datastore's own logs:

[go-dqlite] attempt 10386: server 192.168.68.210:19001: dial: dial tcp 192.168.68.210:19001: i/o timeout

Attempt ten thousand, three hundred and eighty-six. dqlite was trying to reach itself at the old address, patiently, thousands of times.

Both of its state files agreed with the logs:

# /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml
- ID: 3297041220608546238
  Address: 192.168.68.210:19001
  Role: 0

info.yaml said the same thing. I should have seen this coming, because Episode 5 explains exactly why: the moment the Dell joined, dqlite stopped listening on loopback and bound itself to the LAN address so the other node could reach it. A single-node MicroK8s is structurally immune to network changes. Mine lost its immunity the day it made a friend.

With its only member unreachable, dqlite couldn't elect a leader, and an API server without a datastore is a very confident receptionist for an empty building.

The fix is the single-node version of the documented dqlite recovery: stop MicroK8s, back up the backend directory (the one step here that can't be redone), swap the address in the state files, reconfigure dqlite with the bundled tool, start.

sudo microk8s stop
sudo tar -cvf ~/microk8s-backend-backup.tar /var/snap/microk8s/current/var/kubernetes/backend
cd /var/snap/microk8s/current/var/kubernetes/backend
sudo sed -i "s/192.168.68.210/192.168.0.201/g" info.yaml cluster.yaml
[ -f localnode.yaml ] && sudo sed -i "s/192.168.68.210/192.168.0.201/g" localnode.yaml
sudo /snap/microk8s/current/bin/dqlite \
  -s 127.0.0.1:19001 \
  -c /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt \
  -k /var/snap/microk8s/current/var/kubernetes/backend/cluster.key \
  k8s ".reconfigure /var/snap/microk8s/current/var/kubernetes/backend/ /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml"
sudo microk8s start
sudo microk8s kubectl get nodes

msi: Ready. Nobody clapped, because the Dell was still very quiet.

🧟 Symptom Two: The Dell Stops Answering the Phone

NAME   STATUS     ROLES    AGE   VERSION
dell   NotReady   <none>   59d   v1.35.6
msi    Ready      <none>   59d   v1.35.6

The Dell is a worker, which means it isn't part of dqlite at all. It talks to the control plane through a local apiserver-proxy service, and the list of control plane endpoints that proxy forwards to lives in args/traefik/provider.yaml. Mine had exactly one entry:

services:
  kube-apiserver:
    loadBalancer:
      servers:
        - address: 192.168.68.210:16443

One endpoint, and it was the old address of msi. The proxy normally refreshes that list from the cluster, but it can't ask the cluster anything when its only way of asking has moved house. A worker keeps its own copy of where the control plane lives, and nobody tells it when that changes.

sudo sed -i 's/192\.168\.68\.210/192.168.0.201/g' /var/snap/microk8s/current/args/traefik/provider.yaml
sudo microk8s stop
sudo microk8s start

While grepping the Dell's config for the old subnet I found one more tenant: can-reach=192.168.68.210 in args/cni-network/cni.yaml. That is Calico's hint for which network interface to use for pod traffic, pointed at an address that no longer exists. The live value sits on the calico-node DaemonSet, so that is the place to check:

kubectl -n kube-system get ds calico-node -o yaml | grep -B1 can-reach

🔐 Sidequest 1: The Pod That Remembered the Old Address

The Airflow API server pod, which gets scheduled on the Dell, hit the same species of problem at startup: a certificate that only knew the old addresses. The fix was the usual ritual. Add the new IPs to the CSR template on the Dell, regenerate the certificates from msi, and the pod came up healthy.

Notice how many fixes in this episode are "find the file where the old address lives and tell it about the move." That is the episode.

🪪 Sidequest 2: kubectl logs and the Other Certificate

With the nodes healthy, I tried to read the logs of a cloudflared pod running on the Dell:

Error from server: Get "https://192.168.0.202:10250/containerLogs/cloudflared/cloudflared-.../cloudflared": tls: failed to verify certificate: x509: certificate is valid for 192.168.68.111, not 192.168.0.202

This is not about cloudflared at all. kubectl logs, exec and port-forward take a different road from every other command:

kubectl -> API server (msi) -> kubelet on dell :10250 -> container

The API server connects to the kubelet's HTTPS port on the node, and the address it uses must appear in the kubelet's own serving certificate. Every node has one, and it is a different certificate from the API server's, which I had already fixed and was feeling smug about.

The Dell's had been issued when it joined the cluster:

subject=CN=system:node:dell, O=system:nodes
issuer=CN=10.152.183.1
X509v3 Subject Alternative Name:
  DNS:dell, IP Address:192.168.68.111

Instead of re-joining the node, I asked the kubelet to request a fresh serving certificate from the cluster CA, which MicroK8s doesn't turn on by default. On the Dell:

echo "--rotate-server-certificates=true" | sudo tee -a /var/snap/microk8s/current/args/kubelet
sudo snap stop microk8s && sudo snap start microk8s

The kubelet filed a certificate signing request. Serving CSRs are not auto-approved, which is a sensible default and a mild inconvenience:

$ kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR           CONDITION
csr-bgrrf   19s   kubernetes.io/kubelet-serving   system:node:dell    Pending
$ kubectl certificate approve csr-bgrrf
certificatesigningrequest.certificates.k8s.io/csr-bgrrf approved

(Approve only a CSR you expect, from the node you expect.)

Then I asked the kubelet what it was actually serving:

$ echo | openssl s_client -connect 192.168.0.202:10250 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
  DNS:dell, IP Address:192.168.0.202

The old kubelet.crt on disk still lists the old IP, but the kubelet no longer serves it, so I left the file alone like a souvenir. kubectl logs worked again.

The fine print: with rotation on, the next IP change will produce a new CSR, and logs and exec will stay broken until I approve it. A manual step reintroduced, written on the checklist, where future-me will find it and sigh.

🚇 Sidequest 3: The Tunnel Route

Episode 9 had some fine print, and I am quoting my past self: remotely managed means the hostname routing lives in Cloudflare's dashboard, not in my repo, which makes it the one thing a cluster rebuild can't restore. This episode, that fine print sent me an invoice.

The route in the dashboard still pointed at an old node address. The tunnel itself was fine. The cloudflared pods run inside the cluster, so they didn't care about the router, and the logs showed four registered connections. The only broken thing was where the tunnel was sending traffic.

The robust fix is to stop sending it to a number at all. cloudflared runs in the cluster, which means it can resolve in-cluster Service names, and every Service gets <service>.<namespace>.svc.cluster.local. So I pointed the route at the Airflow API server Service by name, and kept the port from the old route:

http://airflow-api-server.airflow.svc.cluster.local:30080

The tunnel logs objected immediately:

ERR error="Unable to reach the origin service ... dial tcp 10.152.183.41:30080: i/o timeout"

Look at that address again. 10.152.183.41 is the Service's ClusterIP, so the name resolved fine and DNS was innocent. The port was the problem. 30080 is the NodePort I pinned for the Envoy gateway back in Episode 4, and a NodePort exists on the nodes' own IPs, nowhere else. The ClusterIP listens on the Service's regular port, which for the Airflow API server is 8080. I had combined a perfectly good name with a port that only exists on a different kind of address, which is like mailing a letter to the right house with the apartment number from the old one.

I changed 30080 to 8080 in the dashboard. Instant. No restart. The tunnel now goes straight to the Airflow API server Service, which also means the Envoy gateway from Episode 4 is no longer in the tunnel's path. And because the route is a name, the next router can do whatever it likes.

🗺️ The Census: Where My Cluster Kept the Old Address

🧠 What This Taught Me

• A cluster's address is not one fact. Mine lived in seven places, in seven dialects: datastore state files, a worker's proxy list, a CNI hint, certificates, a kubeconfig, a tunnel route. Changing the IP is a census, not an edit.
• unexpected EOF is not a certificate error. Certificates complain in full sentences. EOF means the door closed. Curl the endpoint, then run the same command on the node itself, to split client problems from cluster problems.
• A polite 401 proves the API server is awake, not that the control plane is. Episode 7, with the sign reversed.
• When the API server hangs, read the datastore logs. dial ... i/o timeout to your own old IP is a very specific kind of confession.
• Workers keep their own copy of where the control plane lives. provider.yaml does not update itself when its only entry stops existing.
• logs, exec and port-forward go through the kubelet's certificate. A different certificate, with its own list of addresses, fixed by its own CSR.
• NodePort and Service port are not interchangeable. Node IP plus NodePort, or Service name plus Service port. Never one from each column.
• Names beat numbers, and reservations beat both. Route to Service DNS, and tell the router who gets which address. Episode 3 said so, and I intend to start reading Episode 3.
• The one thing git can't restore is the thing living in the dashboard. Episode 9's fine print came due right on schedule.

📋 Quick Reference (For Skimmers)

| dqlite's idea of its own address | `sudo cat /var/snap/microk