The Kernel Underneath Kubernetes: nftables, netfilter, and Why svclb-traefik Kept Crash-Looping

A few weeks ago, I hit a bug that sent me further down the Linux networking stack than I'd gone in years: an NVIDIA Jetson device, running k3s at the edge, with svclb-traefik stuck in a CrashLoopBackOff that made no sense from the Kubernetes side. Pods scheduled fine. Images pulled fine. The manifests were correct. The problem was three layers below anything kubectl describe could tell me, in the kernel's packet-classification engine. The fix ended up being: blacklist nf_tables . But understanding why that line worked meant actually learning what nftables is, how it relates to the iptables tools everyone's used for twenty years, and where eBPF fits into all of this - because the honest answer is that most of us run this stack daily without ever needing to know what's underneath it, until a vendor kernel breaks the abstraction. This is that deep dive. Netfilter: the actual foundation Before nftables, before iptables, there's netfilter - the kernel subsystem that makes any of this possible. Netfilter itself doesn't know what a "rule," "chain," or "table" is. It's simpler and older than that: it's a set of five hook points wired into the network stack where kernel code can intercept a packet and hand back a verdict. PREROUTING → [routing decision] → INPUT → (local socket) → FORWARD → POSTROUTING → (out the interface) OUTPUT (locally-generated packets) → POSTROUTING - PREROUTING - packet has just arrived, before the kernel has decided whether it's for this host or being forwarded - INPUT - packet is headed to a local process - FORWARD - packet is being routed through this host - OUTPUT - packet originated locally - POSTROUTING - packet is about to leave an interface At each hook, netfilter calls whatever's registered there, in priority order, and each callback returns a verdict - ACCEPT , DROP , QUEUE , and a few others. That's the entire contract. Everything else - iptables, nftables, connection tracking, NAT - is a tenant of this hook system, not part of it. Packet classification, defined properly Once you have hooks, you need something to actually decide what happens to a packet at each one. That's packet classification: given a packet's header fields - source/destination address, protocol, ports, TCP flags, interface - find which rule(s) apply and execute the associated action. There are, broadly, two ways to implement this: - Linear evaluation - walk an ordered list of rules top to bottom, stop at first match. Simple to reason about, but the cost is O(n) per packet. A table with 10,000 rules can mean 10,000 comparisons for one packet in the worst case. - Indexed evaluation - use hash tables, sets, or interval trees so a packet jumps close to O(1) or O(log n) to the relevant rule instead of scanning linearly. This distinction is the entire reason nftables exists. Classic iptables is a linear-evaluation engine implemented as a chain of kernel modules. nftables is an indexed-evaluation engine implemented as a small in-kernel virtual machine that interprets a compact bytecode program against structured data - sets, maps, concatenated key lookups - instead of walking modules one at a time. Why there used to be four separate tools If netfilter's hooks are protocol-agnostic in principle, why did we end up with iptables , ip6tables , arptables , and ebtables as four completely separate tools? Because each protocol family has a different header shape, and historically each got its own userspace binary, its own kernel module family, and its own rule storage - with zero sharing between them. | Tool | Operates on | Header fields classified | |---|---|---| iptables | IPv4 | source/dest IP, protocol, TTL | ip6tables | IPv6 | 128-bit addresses, extension headers, no in-header fragmentation | arptables | ARP | opcode, hardware/protocol addresses | ebtables | Ethernet bridging (L2) | MAC addresses, EtherType, VLAN tags - inside the bridge code path, not the routed IP path | Each tool has its own binary, its own kernel modules (ip_tables , ip6_tables , arp_tables , ebtables ), its own match/target extension modules, and - critically - no shared rule-set with the others. A single bridged-and-routed packet in a Kubernetes CNI setup might conceptually need to be evaluated by both the IP-layer rules and the bridge-layer rules, with no shared engine, syntax, or coordination between them. That fragmentation, multiplied across dual-stack IPv4/IPv6 clusters with bridge-mode CNIs, is exactly what nftables was built to collapse into one engine with multiple table families (ip , ip6 , inet , arp , bridge , netdev ) instead of four unrelated tools. Anatomy of a rule, and a packet walking through one The classic model - shared by iptables, ip6tables, and arptables - is table → chain → rule . - Tables group rules by purpose: filter (accept/drop decisions),nat (SNAT/DNAT/MASQUERADE),mangle (header rewriting),raw (conntrack exemptions). - Chains are ordered rule lists. Built-in chains map directly onto netfilter's five hooks. User-defined chains are only reached via a jump from a built-in one. - Rules are match conditions (ANDed together) plus a target - a verdict that either terminates evaluation ( ACCEPT ,DROP ) or lets it continue (LOG ,MARK ). Here's a concrete trace. A client sends a TCP SYN to port 22: IPv4: src=203.0.113.50 dst=198.51.100.10 proto=6(TCP) ttl=54 TCP: sport=51422 dport=22 flags=SYN Against this INPUT chain: iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j DROP iptables -A INPUT -j DROP - Conntrack, running at PREROUTING, tags this as a NEW connection (no prior entry for this 4-tuple). - Rule 1: -i lo checks the packet's incoming-interface metadata. Arrived oneth0 , notlo → no match. - Rule 2: -m state ESTABLISHED,RELATED queries conntrack. State isNEW → no match. - Rule 3: -p tcp matches (protocol field is 6), but-s 10.0.0.0/8 fails - the source isn't in that range. One failed AND-condition fails the whole rule. - Rule 4: -p tcp --dport 22 - both match. DROP fires. Evaluation stops. The packet is silently discarded. Notice what determined the outcome: rule order, not rule specificity. If rules 3 and 4 were swapped, an authorized 10.x.x.x client would hit the blanket DROP before ever reaching its allow rule. There's no automatic "most specific wins" resolution - whichever rule the packet reaches first and fully matches, wins. This is the sharpest edge of linear packet classification, and it's the source of a large fraction of "why isn't my rule working" debugging sessions. Where nftables changes the mechanics The same rule-set in nftables doesn't have to be a flat list. -s 10.0.0.0/8 and a destination port can be expressed as set-membership tests and verdict maps, so the kernel's bytecode VM does a hash or interval lookup instead of repeating a linear CIDR comparison per rule. For five rules, this makes no practical difference. For five thousand rules covering thousands of allowed subnets - which is not a hypothetical in any reasonably-sized cluster, iptables degrades linearly while nftables' set-backed lookup stays close to constant time. This is also, not coincidentally, why most modern distros quietly turned the iptables binary itself into a compatibility shim: iptables-nft translates classic iptables syntax into nftables rules under the hood, so most users get nftables' engine without ever typing nft directly. The legacy path - iptables-legacy - still exists, talking straight to the old ip_tables kernel module, bypassing nftables entirely. Where the Jetson actually broke My first instinct was that this was an update-alternatives problem - that iptables was somehow resolving to the wrong binary. It wasn't: $ update-alternatives --display iptables iptables - auto mode link currently points to /usr/sbin/iptables-legacy Already pinned to legacy at the host level. Dead end, but an instructive one, because it meant the failure was happening somewhere the host's alternatives system doesn't reach. The actual answer was in the crash logs of the svclb-traefik pod itself: + lsmod + grep -qF nf_tables + '[' 0 '=' 0 ] + mode=nft + ln -sf xtables-nft-multi /usr/sbin/iptables + iptables -t filter -I FORWARD -s 0.0.0.0/0 -p TCP --dport 80 -j ACCEPT Warning: Extension tcp revision 0 not supported, missing kernel module? iptables v1.8.11 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain FORWARD The non-obvious part - and the reason pinning the host's alternative wasn't enough, is that the svclb-traefik container's own entrypoint script does an independent lsmod | grep nf_tables check at container startup, entirely separate from whatever the host has configured. If the module is loaded anywhere on the system, the container self-selects nft mode and symlinks its own iptables to xtables-nft-multi internally, silently overriding the host-level legacy setting. It then tries to insert a FORWARD rule through that nft path, and NVIDIA's L4T/tegra kernel, which doesn't track mainline closely and ships an incomplete nf_tables implementation, rejects the insert outright. So the fix isn't at the alternatives layer at all. It's at the kernel module layer, applied and verified live: sudo modprobe -r nf_tables lsmod | grep nf_tables # confirm empty, and check nothing else (e.g. a WireGuard mesh) depends on it first sudo systemctl restart k3s kubectl get pods -n kube-system -w | grep svclb-traefik With the module gone, the container's startup check finds nothing, stays in legacy mode, and the FORWARD rule inserts cleanly - pod goes to 2/2 Running . Made permanent with: echo "blacklist nf_tables" | sudo tee /etc/modprobe.d/nf_tables-blacklist.conf sudo update-initramfs -u The lesson generalizes past this one bug: a host-level setting (the alternatives symlink) can be quietly overridden by a container that does its own environment detection at startup. Fixing the layer you can see