The Shoemaker's Children Aren't Barefoot

This article covers the architecture and the design decisions behind Home Automated Infrastructure, an open source project that runs a smart home with GitOps and Infrastructure as Code. Everything sits on a Kubernetes cluster of four Raspberry Pi 4B boards. Talos Linux is the operating system, ArgoCD keeps the manifests in sync, Terraform provisions the cloud resources, and Home Assistant runs the automation. The point I want to make is simple. The DevSecOps practices we use at work hold up fine in a house, and they let me stay off proprietary home automation platforms. How the house got here We bought the house in 2021. I wanted the freedom to make it as self-sufficient as I could, and I forget things constantly, so anything I can automate, I automate. At the back there was an edícula, an outbuilding nobody used. It held leftover construction material and nothing else. And yes, that thing hanging from the ceiling was the server rack. I turned the outbuilding into the hub. Every floor got Cat5e cabling and one UniFi AP, and the PoE switch in the outbuilding feeds them data and power. Next to the switch sits a Cloud Gateway Ultra that shapes traffic and balances the internet connections. I keep two separate ISPs for work and fail over between them at the router. To power all of that, plus the rest of the house, I put in solar panels and an inverter. What I ended up with was a house made of separate subsystems: network, solar, IoT devices. Each one had its own dashboard and its own logic. Checking three interfaces to answer one question gets old fast, so I needed something in the middle. That something is Home Assistant. Solution architecture The architecture has three layers: edge hardware, container orchestration, and cloud infrastructure. They talk to each other declaratively, and the Git repository holds the single source of truth for everything above the network. The network gear is the exception, and it is worth naming up front rather than letting you find it later. The UniFi controller, the VLANs and the ISP failover are configured by hand, in a UI. Everything else in this post is checkable against the repository; that part isn't. Separation of responsibility layers The cluster is called skynet. It has four Raspberry Pi 4B boards, three control plane nodes and one worker. The nodes are named after the Three Musketeers, so athos, porthos and aramis run the control plane and dartagnan does the work. The Pi 4B is a compromise between price, power draw and CPU. For what runs here, automation, monitoring and ingress, it is enough. None of it is critical enough to justify more. The Talos config pins the generic ARM64 image rather than a board revision, so if you try this on a Pi 5 expect the install disk path to be the first thing that breaks. Talos Linux Talos Linux is an immutable operating system built only for Kubernetes. It has no interactive shell and no SSH. You configure it through an API, and I keep that configuration in Terraform. That buys me two things. There is almost nothing installed on the box for an attacker to use, and updates either apply completely or roll back. Anything that drifts from the declared config gets corrected on the next deploy, so manual tweaks can't pile up over months the way they do on a normal server. general_patch = yamlencode({ cluster = { proxy = { extraArgs = { "metrics-bind-address" = "0.0.0.0:10249" } } } machine = { features = { kubePrism = { enabled = true port : 7445 } } install = { image = "factory.talos.dev/nocloud-installer/${local.schematic_id}:${var.talos_version}" disk = "/dev/mmcblk0" } network = { interfaces = [{ interface = "eth0" dhcp = true }] } } }) That block is the whole common configuration, shared by the control plane nodes and the worker. Since the code defines the entire machine, replacing a board is a rebuild, not an investigation. Home Assistant on Kubernetes The normal way to run Home Assistant is Home Assistant OS, on a mini PC or on its own Raspberry Pi. That is the supported install, with the Supervisor, the add-on store and one-click backups. Running it inside Kubernetes is more work by any measure, so let me explain why I did it anyway. Home Assistant is not the system. It is one application inside a system. The house already ran on a managed network, solar generation, DNS, certificates and monitoring, and all of that already lived in Git. Leaving automation outside that flow would give me two ways to change the house: one versioned and reviewed, and one where I click around a UI and hope I remember what I did. Containers are not the win here. The win is that Home Assistant gets everything the cluster already does. ArgoCD deploys it, the External Secrets Operator pulls its secrets from SSM, Traefik gives it valid TLS, Prometheus scrapes it, and anything that drifts gets put back. The price for that is real. No Supervisor, no add-ons. The container image is not Home Assistant OS. MQTT, ESPHome and Zigbee2MQTT stop being plugins and become applications in the cluster, each with its own chart, its own resources and its own lifecycle. Under GitOps that is an improvement, since every piece becomes code someone can review instead of settings buried in a UI. It is also work that Home Assistant OS would have done for me. Discovery does not cross the pod network. Plenty of integrations depend on mDNS, SSDP or DHCP broadcast, and those packets never reach a pod on the overlay network. Most of my devices have static IPs, including the FoxESS inverter, the UniFi APs and the gateway, so I don't lean on automatic discovery much. State is the hard part. Home Assistant keeps its configuration and history in /config, so it needs a persistent volume. With local-storage the pod lands on whichever node holds that disk. The PV pins by node role rather than by name, so it resolves to dartagnan only because dartagnan is the only worker, but the effect is the same: the cluster's high availability stops there. Three control plane nodes do not make Home Assistant highly available. And the other half of that sentence deserves saying too, because I nearly wrote the comfortable version. There is no backup. No Velero, no restic, no cron job copying /config anywhere. If that SD card dies I can rebuild the node in minutes and Home Assistant's entire configuration and history are gone with the card. Rebuild is not restore. It is the largest hole in this setup and the only item on the list that would actually cost me something. The house is production. A bad deploy in a lab costs me an evening. A bad deploy here turns off someone's lights, and that someone lives with me. It forces a discipline a homelab usually doesn't, which is the main reason the project has taught me so much. Distribution of responsibilities Two systems keep the repository and the running cluster in agreement. ArgoCD watches the /k8s/ directory and syncs any difference to the cluster. The ApplicationSet generator reads two levels deep, k8s/ / , and that directory convention is what does the work - drop a chart in the right place and it deploys itself. Self heal and prune are on, so what is in Git is what is running. Terrateam picks up pull requests that touch /terraform/, runs terraform plan and posts it as a PR comment. It also runs Infracost against the plan, so every pull request tells me what the change costs in BRL before I merge it, which is a strange and useful thing to have on a homelab. I never run Terraform by hand, and no credentials sit in an external CI pipeline, because the actions that run the plan execute inside the cluster. Example of an action performed by Terrateam. Why Terrateam and not Atlantis? - The permissions for infrastructure secrets go into the GitHub Action itself instead of collecting in a long-running service, which is what happens with Atlantis. I get tighter separation, and I can run it on my own hardware with the Actions Runner Controller. - The free tier covers me, and only the workers run in my environment. - Their drift detection works, which keeps stray manual changes from surviving in my account. GitOps and the development flow GitOps means the Git repository holds the desired state of the system. Every change to infrastructure or applications goes in as a commit, so the history of the house is the history of the repository, and I can go back to any point in it. Here that comes down to one rule: no manual changes with kubectl, none in the AWS console. If a setting is not in Git it does not exist, and the next sync should remove it. The rule covers the cluster and the AWS account, which is to say everything except the network gear I flagged earlier. Kubernetes applications - Modify Helm charts and values under /k8s/ - Commit and push to the main branch - ArgoCD notices the difference between Git and the cluster within 3 minutes - ArgoCD syncs the changes - The cluster converges on the state declared in Git Terraform infrastructure - Create a branch and modify files under /terraform/ - Open a pull request on GitHub - Terrateam runs terraform plan and posts the result as a comment - A reviewer reads the plan and approves it or asks for changes - Apply is not automatic - I trigger it once the plan looks right, and Terrateam merges the pull request and deletes the branch after it succeeds Initial bootstrap Cluster initialization follows a dependency order. The modules in terraform/bootstrap/ run in this sequence: - state creates the S3 bucket and the DynamoDB table that hold Terraform state remotely, with locking - cluster provisions the Talos Linux cluster across the four boards - machines applies the init-data and boots the nodes - argocd installs ArgoCD - argocd-applicationset configures the ApplicationSet that watches /k8s/ and syncs every application Security and DevSecOps Least privilege Every AWS IAM role gets the smallest set of permissions that function needs. The self-hosted runners assume roles through their ServiceAccount, and Terrateam authenticates with a