GPUs keep falling off the PCIe bus, and standard node health does not notice

The failure the default node model was never taught to see

If you have only ever run CPU workloads on Kubernetes, node health reads as a solved problem. The kubelet reports back to the control plane, the node condition transitions to NotReady when something is wrong, the controller manager evicts pods after a grace period, the scheduler places them elsewhere. That loop assumes hardware is either present or absent. A machine that is up, reachable, and running a kubelet is a machine you can send work to.

GPU nodes break the assumption. The host stays up. The kubelet keeps sending heartbeats. The CPU cores are fine. What has changed is that one of the GPUs the pod actually needs is no longer visible on the PCIe bus, or the driver has hung, or a memory error has taken a region out from under the workload. From above, the node still shows Ready. From inside the pod, the job just failed a CUDA call and will keep failing every retry until someone drains and recycles the host.

Why the writeup exists

The piece frames itself as lessons learned from operating this at Amazon EKS scale. The subtext is straightforward: at fleet size, "someone will notice and cordon it" is not an operational plan. Every hour a broken GPU stays in the schedulable pool is another job sent to it, another failed run, another retry that eats a slot behind a queue.

The economic consequence differs from a CPU incident too. A GPU host is not fungible with a general-purpose node. The pod that was on it cannot just reschedule anywhere; it has to reschedule onto another GPU host, which is a smaller and contended pool. A silent failure in that pool costs more than a loud failure in a general one.

What a GPU-aware health loop has to do

Making GPU nodes self-heal without operator input means adding signals the default node condition vocabulary does not carry.

• Something on the host has to watch the GPU directly: driver liveness, hardware presence on the bus, memory errors, thermals.
• Something has to translate those signals into a form the control plane already respects, usually a taint or a node condition the scheduler will honour.
• And something has to close the loop by draining the workload and taking the host out of the pool before the retry storm starts.

None of those pieces are novel on their own. The work is in the plumbing between them, and in deciding how conservative the auto-remediation gets. Cordon and drain on the first error and you burn expensive capacity on transient faults. Wait for a threshold and you leak failed jobs into the queue while you count. The AWS piece is a data point on where a large operator has landed on that trade-off, from the team that has to answer for it.

The residual caveat

Two things carry beyond AWS's own environment.

First, if you are running GPUs on Kubernetes and have not written down what "the GPU is gone but the node is up" looks like in your monitoring, the default node condition model will not close that gap for you.

Second, self-healing here is a euphemism for a pipeline: detect on the host, surface as a taint the scheduler already understands, drain and replace before the next batch of retries lands. Read the article for how one team wired it; the failure mode it names is the one every GPU fleet eventually meets.