I Run Bare-Metal Kubernetes on $200 of Scrap Hardware (And Why I Burned 3 SD Cards Learning)

From a bricked Raspberry Pi to a 4-node Talos Linux cluster running Cilium eBPF, ArgoCD, and Longhorn - every failure, every fix, and the homelab that now hosts my production side projects.

The Setup

It started with a Dell OptiPlex I found in a recycling pile. Intel i5-6500, 16GB DDR4, a 256GB SATA SSD. Someone's discarded office PC. I wiped the Windows 10 sticker off the case, installed Proxmox, and told myself: "This is going to be my cloud."

Three months later, that OptiPlex plus three Raspberry Pi 4s, a second-hand managed switch, and a drawer full of dead SD cards became a 4-node bare-metal Kubernetes cluster running:

• Talos Linux - immutable, API-driven OS built for K8s
• Cilium - eBPF-based CNI replacing kube-proxy and providing cluster mesh
• ArgoCD - GitOps continuous delivery
• Longhorn - distributed block storage across nodes
• MetalLB - bare-metal load balancing with BGP
• Prometheus + Grafana - monitoring stack pulling metrics from the cluster and my ISP's router

Total cost: ~$220 (OptiPlex was free, Pis were ~$60 each, switch was $35 on eBay).

This is not a tutorial. This is an autopsy of every mistake I made building it.

Why Not Just Use EKS?

I run AWS infrastructure professionally. I can spin up an EKS cluster in 20 minutes with Terraform. I know what managed Kubernetes costs at scale. But I was hitting a wall:

1. You can't break AWS
   In production, I follow well-architected patterns. VPCs, private subnets, IAM roles with least privilege, GuardDuty, Security Hub. Everything is safe. Everything is boring.
   In my homelab, I can delete the CNI and watch the cluster melt. I can spin up a privileged container with hostNetwork: true and see what breaks. I can push a bad Cilium network policy and lock myself out of every node - which I did. Twice.

2. The gap between "I passed CKA" and "I understand Kubernetes"
   Certifications teach you how to use the API. They do not teach you what happens when etcd runs out of disk because Longhorn replicas ballooned and you didn't set volume quotas. They do not teach you that Cilium's eBPF kube-proxy replacement silently breaks certain firewall rules in ways that are invisible to kubectl.

3. If I can't self-host it, do I really know it?
   This is my rule. I don't put a technology on my CV unless I've operated it without a managed control plane. Talos, Cilium, ArgoCD - I wanted to feel them fail.

The Architecture

┌─────────────────────────────────────────────────────────────┐
│DELL OPTIPLEX (Control Plane)                                │
│Talos Linux · x86_64 · 16GB RAM                             │
│┌─────────────┐┌─────────────┐┌─────────────────────┐│
││etcd (q=1)   ││ API Server  ││Controller Manager    ││
│└─────────────┘└─────────────┘└─────────────────────┘│
│┌─────────────┐┌─────────────┐┌─────────────────────┐│
││Scheduler    ││ Cilium Agent││Talos API Server      ││
│└─────────────┘└─────────────┘└─────────────────────┘│
└─────────────────────────────────────────────────────────────┘
           │
   ┌───────┼───────────────┐
   ▼       ▼               ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ RPI-01 (arm64)│ │ RPI-02 (arm64)│ │ RPI-03 (arm64)│
│4GB RAM · 64GB │ │4GB RAM · 64GB │ │4GB RAM · 64GB │
│ USB-SSD boot  │ │ USB-SSD boot  │ │ USB-SSD boot  │
│┌────────────┐│ │┌────────────┐│ │┌────────────┐│
││ Cilium Agent││ ││ Cilium Agent││ ││ Cilium Agent││
││ kubelet     ││ ││ kubelet     ││ ││ kubelet     ││
││ Longhorn    ││ ││ Longhorn    ││ ││ Longhorn    ││
││ (replica)   ││ ││ (replica)   ││ ││ (replica)   ││
│└────────────┘│ │└────────────┘│ │└────────────┘│
└──────────────┘ └──────────────┘ └──────────────┘
                │ │ │
                └───┼───────────┐
                    ▼           ▼
            ┌───────────────────┐
            │Netgear GS308      │
            │8-Port Gigabit     │
            └───────────────────┘

Design Decisions

The Failures (And What Each One Taught Me)

Failure 1: I Burned 3 SD Cards in 6 Weeks

My first cluster booted from 64GB SanDisk Ultra SD cards. They were cheap. They were fast - for the first week.

Then etcd started crashing. The logs showed mvcc: database space exceeded. I ran etcdctl defrag and it helped for a day. Then the Pi locked up. I rebooted. The SD card wasn't recognised anymore. Corrupted beyond repair.

The second SD card died the same way. The third one survived longest because I moved /var/lib/longhorn to an NFS share - but the card still degraded after 5 weeks.

Root cause: Etcd does frequent small writes. SD cards use wear-leveling algorithms designed for photos (large sequential writes), not databases (random small writes). The flash translation layer dies.

The fix: Every node now boots from a $25 USB3 SSD. Talos supports this natively - you flash the installer ISO, boot from USB, and install to the SSD. The difference is staggering. I've been running the current cluster for 8 months with zero storage-related crashes.

Lesson: Storage media is not interchangeable. Understand your write patterns before you pick hardware.

Failure 2: A Bad CiliumNetworkPolicy Locked Me Out of Every Node

I was experimenting with zero-trust network policies. I applied a CiliumNetworkPolicy that allowed ingress only from pods with label app=frontend. I forgot to include the kube-system namespace. I forgot to include Cilium's own health checks.

I applied it with kubectl apply -f . Within 30 seconds, my SSH session to the control plane hung. kubectl timed out. The Pi nodes showed as NotReady. I had physically unplugged and rebooted everything before I could get back in.

What happened: Cilium enforces policies at the eBPF level - before packets even reach iptables or the host network stack. A bad policy doesn't just drop traffic; it drops it invisibly. No logs in dmesg. No iptables -L entries. Just silence.

The fix: Talos has a recovery mechanism. You boot the node with a kernel parameter talos.config=none and it starts without the network stack enabled. From there, you can mount the Talos EPHEMERAL partition, edit the CNI state, and reboot. But I didn't know this at the time.

The real fix is procedural:

• Always test network policies with cilium policy trace before applying
• Always keep a default-allow-all policy in Git, ready to apply
• Never apply network policies on Friday evening (learned the hard way)

Lesson: eBPF is powerful because it's invisible. That same invisibility makes debugging a nightmare. Test before you enforce.

Failure 3: Longhorn Filled My Disks and kubelet Evicted Everything

I deployed a PostgreSQL database for MarketPulse with a 10Gi Longhorn volume. Longhorn replicates volumes across nodes for HA. I set replica count to 3.

What I didn't know: Longhorn stores replicas as sparse files that grow as data is written. It also keeps snapshots for backups. My "10Gi" database had 3 replicas × 10Gi × 2 snapshots = 60Gi actual usage.

Each Pi had a 120Gi SSD. Longhorn consumed 60Gi. Talos system overhead consumed 20Gi. The remaining 40Gi filled up with container images and logs.

One morning, every pod was Evicted. kubelet had hit the 85% disk pressure threshold and started killing pods to reclaim space. But Longhorn volumes are block devices - kubelet can't delete them. The cluster entered a death spiral.

The fix: I now enforce:

• Volume quotas in Longhorn (max 2 replicas for non-critical workloads)
• Automatic snapshot cleanup (retain 2, delete older than 7 days)
• Node disk alerts in Prometheus at 70% usage, not 85%
• Container image garbage collection: imageGCThreshold: 70

Lesson: Distributed storage is not free storage. Replicas × snapshots × logs = real disk consumption. Do the math before you deploy.

Failure 4: MetalLB Gave Out IPs Already Used by My Router's DHCP Pool

I installed MetalLB in Layer 2 mode and assigned it the range 192.168.1.100-192.168.1.150. I exposed a LoadBalancer service. It got 192.168.1.105. I could curl it from my laptop. Success!

Three hours later, my wife's laptop couldn't reach the internet. It had been assigned 192.168.1.105 by the router's DHCP pool. Same IP. Two MAC addresses. Chaos.

The fix: I logged into my ISP router (which I hate but cannot replace), found the DHCP range, and moved MetalLB to 192.168.1.200-192.168.1.250 - outside the DHCP pool.

Better fix: I later configured MetalLB in BGP mode. My router (running OpenWrt on a separate access point) peers with MetalLB and learns routes dynamically. No IP conflicts. No L2 ARP spoofing. True routing.

Lesson: IP address management is not optional on bare metal. Your homelab is a real network. Treat it like one.

Failure 5: I Upgraded Talos and Lost the Cluster Because I Didn't Backup etcd

Talos makes OS upgrades trivial: talosctl upgrade --image ghcr.io/siderolabs/installer:v1.7.0. One command, rolling reboot, cluster stays up.

Except I upgraded from v1.6.0 to v1.7.0 while the cluster was under disk pressure (see Failure 3). The control plane rebooted. etcd failed to start because the WAL files were corrupted from the previous disk-full event. The cluster had no quorum. I had no backup.

I spent 6 hours trying to recover etcd with etcdctl snapshot restore. The snapshot I had was 3 weeks old. I lost:

• All ArgoCD application definitions (I had them in Git, so I re-synced)
• All Prometheus metrics history
• All Longhorn volume states (the PVCs survived, but the volume attachments were inconsistent)

The fix:

• Automated etcd snapshots every 6 hours to S3 (yes, I back up my homelab to AWS - the irony is not lost on me)
• Pre-upgrade health checks: disk <70%, all nodes Ready, all Longhorn volumes healthy
• I wrote a tiny Go CLI that wraps talosctl upgrade with these checks. It lives in my homelab-tools repo.

Lesson: Immutable infrastructure doesn't mean unbreakable infrastructure. Back up your control plane state. Or learn etcd disaster recovery at 2 AM like I did.

What's Running on It Now?

The OptiPlex sits at ~60% CPU under normal load. The Pis hover at 40-70%. I can burst higher, but I know the limits.

3 Lessons That Transfer to Production

1. Resource quotas are not bureaucracy
   In my homelab, I have no cost optimization team. When Longhorn eats all disk, I pay with crashed pods and 2 AM debugging. Setting requests and limits, configuring PVC sizes, defining retention policies - these are not "enterprise overhead." They are survival mechanisms.

2. Observability first, workload second
   I deployed Prometheus and Grafana before I deployed any application. Not because I'm disciplined - because I learned that debugging a bare-metal cluster without metrics is like performing surgery in the dark. Hubble shows me every dropped packet. Node exporter shows me disk I/O patterns I never would have guessed. In production, I'd push this to Thanos or Cortex. In my homelab, I run a 30-day retention and call it enough.

3. GitOps is the only way I trust myself
   I have root access to every node. I can kubectl exec into any pod. Without ArgoCD, I would absolutely make manual changes at 11 PM and forget them by morning. With ArgoCD, the cluster state is in Git. If I change something manually, the UI shows it as OutOfSync and I feel shame. That shame keeps my cluster declarative.

The Hardware List (If You Want to Replicate)

The Pi 5 exists now and is faster. I'm staying on Pi 4 because arm64 compat is tested and I'm lazy.

TL;DR

I built a 4-node bare-metal Kubernetes cluster on $220 of scrap hardware and Raspberry Pis. I use Talos Linux, Cilium eBPF, ArgoCD, Longhorn, and MetalLB. I burned SD cards, locked myself out with network policies, filled disks with storage replicas, caused IP conflicts, and corrupted etcd during an upgrade. Every failure taught me something I use in production. The cluster now runs my public projects, my monitoring, and my experiments.

If you're learning Kubernetes, break it at home before you break it at work.

What's your homelab stack? Drop it in the comments - always looking for new failure modes to learn from.