Upgrading a Locked-Down Linux Fleet with Nothing but kubectl

Say you're running a fleet of machines on an immutable operating system - root filesystem read-only, nothing drifts, and you upgrade the whole OS as a single artifact instead of patching it in place. Great for reliability. But now you need to do something mundane: rotate a login password across every node, or roll out a new OS version. The obvious move is to SSH into each box and do it by hand. Except that's exactly the kind of ad-hoc, drift-inducing change the immutable model exists to prevent - and on a real fleet, "SSH into each box" doesn't scale past your patience. So here's the setup I landed on, and the thing I actually want to show you: I can change a password or push a full OS upgrade to the whole fleet by applying a single Kubernetes ConfigMap. No SSH, no per-node scripts, no drift. # Rotate the password everywhere: kubectl get configmap kairos-config -n kube-system -o yaml | \ sed 's/passwd: kairos/passwd: something-better/' | \ kubectl apply -f - # Roll out a new OS image to the whole fleet: kubectl create configmap kairos-upgrade \ --from-literal=image=docker.io/you/kairos-rhel96:v2 \ --from-literal=version=v2.0.0 \ -n kube-system --dry-run=client -o yaml | kubectl apply -f - That's the entire operator experience. Below is how it works - and, more interestingly, how I made it do that without handing a compromised container the keys to every machine. The one-paragraph background The OS here is Kairos, an open-source immutable Linux distribution built for running Kubernetes at the edge. The mental model: you build the OS like a container image, ship it as an artifact, and replace it wholesale to upgrade. Upgrades write the new image to an inactive A/B partition and flip the bootloader to it - so a bad upgrade rolls back instead of leaving you with a half-patched box. My image is built on RHEL 9.6 with k3s baked in. If you've never touched Kairos, that's all you need to follow along. The interesting part isn't Kairos itself. The interesting part: how a kubectl apply becomes a change on the host The gap I had to bridge is this: editing a ConfigMap happens inside Kubernetes, but flipping a GRUB partition, running an OS upgrade, or rebooting a node happens on the host, as root, outside the cluster's world. Something has to cross that boundary. The naive way to cross it is a privileged DaemonSet - a pod with the host's PID namespace, the host root filesystem mounted in, and all Linux capabilities intact. It works, and it's a trap. A pod with that much power is one of the cleanest paths on a whole cluster from "attacker gets into a container" to "attacker has root on every node." A bad base image, one poisoned dependency, a sloppy RBAC rule - any of those, and a privileged DaemonSet hands the whole fleet over. So I split the job in two, along the privilege boundary: In the cluster: an unprivileged DaemonSet pod. All capabilities dropped, allowPrivilegeEscalation: false , and its only reach into the host is two narrowhostPath mounts -/oem and/usr/local . It can write two files. That's the entire blast radius. It reads the ConfigMaps you apply and drops the requested change into those files.On the host: a plain root-owned systemd service (kairos-agent-watcher ) that loops every 15 seconds, reads only those same two files, and does the actual privileged work - applying config withkairos-agent run-stage , flipping the A/B slot, running an upgrade, rebooting. The payoff is the security property: the pod that faces the cluster has almost no power, and the process that has all the power never faces the cluster. If someone fully owns the DaemonSet pod, the worst they can do is write a config change or point at an upgrade image. They still can't run an arbitrary command as root on the node, because nothing in the pod ever executes privileged work - the privilege lives in a service that only ever reads two files. That's the idea I'd take to any host-level-ops-from-Kubernetes problem, Kairos or not: don't grant the network-facing thing the privilege; let it express intent into a narrow channel, and keep the privilege on the other side of that channel. The flow, end to end: A scar worth sharing: the disk that installed itself across two disks One more piece runs earlier, at install time - a small service that picks which disk the OS lands on. On a multi-disk box you don't want the OS installing over your data disk or your install media, so the script picks the smallest attached disk over 20GB that isn't the boot media. That sounds simple. It was not simple, and I want to be honest about how I learned that, because it's the kind of thing you won't find in the docs. The one that actually hurt showed up after we shipped it to a client. On their hardware, the install would sometimes end up with the OS partitions split across two different disks - some on one, some on another. It took a bunch of back-and-forth with my team to work out why: those disks had been used for a previous Kairos install, and the leftover Kairos partition labels (COS_GRUB , COS_OEM , COS_STATE , and friends) were still on them. Kairos finds its partitions by label. With stale labels lying around on a second disk, the installer happily adopted them, and the install smeared itself across both. The fix is unglamorous and now lives in the script: before install, scrub any stale Kairos labels off the candidate disks (wipefs , sgdisk --zap-all ) so nothing gets adopted by accident. But the lesson is the transferable bit: label-based partition discovery trusts labels, so an install onto previously-used hardware has to assume those labels are lying. That's a sentence I wish I'd read before shipping, not after. (That scrub step is destructive by design - it's meant for lab and disposable-disk hardware. Read it before you point it at anything you care about.) The whole thing is on GitHub The full working setup - the Dockerfile, both systemd services, the DaemonSet and ConfigMaps, the disk-select script, and step-by-step commands to build the image, produce an ISO, spin up a demo VM, and test both a password rotation and a live OS upgrade end to end - is here: → https://github.com/sidharth48-hub/kairos The README walks through it in order and calls out what's demo-only (default passwords, the destructive disk scrub) versus what's safe to reuse. If you're running immutable Linux under Kubernetes and you've been SSHing into nodes for day-2 changes, I think the pod-writes-intent / host-holds-privilege split is worth stealing - even if you never touch Kairos. Happy to answer questions in the comments - especially if you've solved the host-ops-from-a-cluster problem a different way. I'd like to hear it. Top comments (0)