What a Kubernetes controller actually does when you break something

What a Kubernetes controller actually does when you break something

⚡ TL;DR Four things about controller mechanics are widely half-understood: what Reconcile receives, where its work comes from, what a periodic resync is, and what a predicate turns off. I built an operator, broke it five ways, and measured each mechanism directly. The reconcile function runs in 2.71ms mean, 77/77 under 25ms, a short resync period costs zero additional API requests, and GenerationChangedPredicate cut steady-state reconciles by 48.5% without touching live repair at all. That last combination is the one that matters at scale. Repo, raw data, and harness: kirPoNik/k8s-drift-operator.

🧩 The four barriers

Everyone who runs Kubernetes knows the platform repairs itself. Delete a pod, it comes back. Scale a Deployment by accident, something puts it back. Almost nobody who relies on that property can say how it works, and the gaps are specific and consequential.

I keep meeting the same four:

• People think a controller is told what changed. It is not, and the reason it is not is the single most important design decision in Kubernetes.
• People think a controller polls the API server. It does not, and knowing what it does instead tells you where your API load actually comes from.
• People think a resync is a re-check against the cluster. It is not, which is why a short resync period is nearly free - and why the number that is expensive sits somewhere else entirely.
• People treat a predicate as a pure optimisation. It is a filter with a silent cost, and the cost is not the one the documentation warns you about first.

So I built the smallest system that has the self-healing property, broke it on purpose ten times per failure mode, and instrumented each of those four mechanisms until I could state what it does rather than what it is said to do.

What I built

• One CRD called Echo, holding an image, a replica count, and a greeting.
• A controller keeps three child objects in sync with it - a Deployment, a Service, and a ConfigMap holding the greeting - with owner references on all three so Kubernetes garbage-collects them when the Echo goes away.
• Observed state written back to Echo.status.
• Plus a Python harness that injects drift and measures repair.

Environment, because every number depends on it: macOS on Apple Silicon, Go 1.27.0, kind v0.32.0 with node image kindest/node:v1.36.1 (Kubernetes 1.36.1), kubebuilder v4.15.0, controller-runtime v0.24.1, demo image hashicorp/http-echo:1.0.0.

🎯 Barrier 1: Reconcile has no idea what changed

Here is the whole thing, and it explains more of Kubernetes than any diagram:

func (r *EchoReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // function body
}

req is a namespace and a name. That is the entire input. Not a diff. Not "replicas changed from 1 to 3." Not a copy of the object.

Every invocation re-reads the desired spec from scratch, recomputes what all three children should look like, and hands that to controllerutil.CreateOrUpdate to diff against whatever exists right now. Identical work whether the trigger was a kubectl command a millisecond ago, a timer firing with nothing wrong, or a process that just booted after eight hours down.

This is level-triggered control: act on the current state of the world, not on the transition that produced it. Edge-triggered is the opposite - act on the delta the event carries.

Edge-triggered is the more intuitive model coming from event-driven systems, and it looks cheaper. Why rebuild an entire Deployment spec when you already know exactly what changed? Because the cheaper model requires the event stream to be perfect, and it never is. Kubernetes settled this early and wrote it down. The architecture principles in the design proposals archive state that functionality must be level-based: correct behaviour given desired state and observed state, regardless of how many intermediate updates were missed. Edge-triggered behaviour, in their words, "must be just an optimization."

The consequence is the entire reason the model holds up. A missed watch event is not a lost repair. It is a delayed one. There is no catch-up code path you could have forgotten to write, because there is no catch-up code path. There is one path, and it runs constantly.

This is also why the API surface looks the way it does. Reconciling from a name rather than a diff is what lets a Deployment controller, a ReplicaSet controller, and your own operator all act on the same objects without coordinating, and lets any of them be restarted, upgraded, or briefly killed without a recovery procedure.

🧵 Barrier 2: Where the work comes from, and what it costs

A controller does not poll. Four pieces do the work, and keeping them separate in your head is what makes the rest of this legible - because the interesting failures live in the gaps between them.

• Watch. A long-lived HTTP connection to the API server that streams changes for one resource type. Established once at startup and held open. Not a request per check.
• Informer cache. A local in-memory copy of every watched object of that type, kept current by the watch. This is the piece with the largest practical consequence: reads inside Reconcile are served from local memory and cost no API request. A reconciler that reads ten objects and writes nothing generates zero API traffic. Writes always go to the API server - the cache is a read path only.
• Work queue. Holds keys, not events: namespace/name strings. It deduplicates. If the same key is enqueued five times before a worker picks it up, one Reconcile runs, not five. This is why the reconciler never learns which watch woke it, and why it does not need to.
• Predicates. Filters that sit in front of the queue and decide which events are allowed to enqueue a key at all. Hold onto that position - barrier 4 is entirely about it.

Wiring, four watches into one queue:

return ctrl.NewControllerManagedBy(mgr).
    For(&labv1alpha1.Echo{}).
    Owns(&appsv1.Deployment{}).
    Owns(&corev1.Service{}).
    Owns(&corev1.ConfigMap{}).
    Named("echo").
    Complete(r)

For watches the custom resource. Each Owns watches a child type and maps any event on it back to the owner's key through the owner reference. Four event sources, one key, one reconciler. One reconciler box, not four handlers. That is the point of the diagram.

⏱️ Barrier 3: What "fast" means, and how to measure it without measuring yourself

Five drift types, ten runs each. Timing starts when the drift command returns and stops when the affected object matches desired state again - not when the pod serves traffic, since pod startup is the kubelet's problem, not this loop's.

Those four sub-100ms figures are upper bounds set by my measurement harness, not durations of the reconcile loop. Here is how I know, because the reasoning generalises to any convergence measurement you build from the outside.

The poller is four lines, and the ordering is the whole story:

def wait_until(check, timeout):
    start = now()
    while True:
        if check():          # runs immediately, before any sleep
            return now() - start
        if now() - start > timeout:
            return None
        time.sleep(0.1)

check() shells out to kubectl get deployment -o json: fork, exec, TLS, API round trip, JSON parse. Roughly 50 to 60 milliseconds. A run needing a second look would therefore report at least 0.055 + 0.100 + 0.055 ≈ 0.21s.

The largest value anywhere in my data - all 60 timed runs, including the in-cluster comparison below - is 0.1091s. Nothing approaches 0.21s. Every run resolved on its first observation. The sleep(0.1) never executed once, in any configuration.

Each figure in that table is the wall time of one kubectl get, and the repair had finished before that kubectl returned. The stdev column describes how consistently my laptop spawns a subprocess.

I caught this because a different number looked impossible first. My original D4 script started its timer after the manager subprocess had booted, which excluded the exact cost D4 exists to measure. It reported 0.054s - a cold process booting, connecting, listing and reconciling in 54 milliseconds, indistinguishable from a warm watch-triggered repair. That implausibility is the only reason a wrong number is not in the table above. Re-measured correctly, D4 is 0.367s. Run one sample and sanity-check it against physics before committing to ten.

📊 The instrument that can see it

controller-runtime already exposes a histogram of Reconcile call duration: controller_runtime_reconcile_time_seconds. No polling in the measurement path at all.

So I stopped timing from outside and read the histogram's delta across exactly ten D1 injections. A Prometheus histogram counts observations into cumulative buckets - le="0.005" holds every reconcile at or under 5ms, le="0.01" at or under 10ms. Cumulative buckets give you real bounds instead of a mean that hides its own tail, which is exactly what you want when the question is "how bad does this get."

The claim that carries weight is 77/77 under 25ms. The 77 reconciles are a mixed population - ten repairs plus the settling and status-write passes each triggers - and no-op passes are cheaper than repairs, so the mean is diluted downward by design.

The histogram measures time inside the function, including its own write round-trips, excluding watch delivery and queue wait. So: bounded above by the polling harness at ~55ms, resolved below by the histogram at single-digit milliseconds. Both instruments correct, measuring different things.

🏠 The controller-down case is a different shape entirely

D4 is the only row where the harness resolved a real duration: 0.3673s, stdev 0.0038s. Six to seven times the live cases, and far more consistent. That shape is a fixed cost. The delay is process boot plus informer cache warm-up, and the tight stdev is what a fixed cost looks like next to variable watch and queue timing.

The mechanism is worth internalising, because it is the direct payoff of barrier 1. The controller does not "catch up." On startup, the manager lists everything that exists and reconciles all of it through the same code path as every other trigger. Down for five minutes or five hours produces an identical repair, because no separate downtime-recovery mechanism exists to degrade.

I also repeated D1 with the controller deployed in-cluster instead of running on the host:

Both rows sit at the harness floor, so nearly identical p50s establish that both configurations are faster than this instrument can resolve. If one were 2ms and the other 8ms, this table would look the same. The tail is real - max doubles, stdev goes up roughly sixfold - and it is the tail of the observation.

In-cluster, the controller Pod shares a node with the API server, etcd, kube-proxy and CoreDNS, and a loaded node's API server answers any client's GET more slowly, mine included. That accounts for a slower kubectl get. Attaching it to reconcile latency requires the histogram in-cluster, which is a redeploy I have not run.

🕳️ Barrier 4: What a predicate actually turns off

There is a known trap in controller-runtime: add GenerationChangedPredicate to an owned-type watch and lose your self-healing safety net, silently, with no error anywhere. I set out to demonstrate it - predicate on the Deployment watch, repeat the scale-to-zero drift, watch it fail to converge. It converged. Ten out of ten, same speed as without.

That result is the barrier, and resolving it requires two Kubernetes fields that are constantly conflated:

• resourceVersion changes on every write to an object, including status writes.
• generation changes only when .spec changes. It is the API server saying "someone asked for something different."

GenerationChangedPredicate compares .metadata.generation on the old and new object and drops the event when they match. A kubectl scale is a genuine spec write, so generation bumps, so the predicate passes the event and the reconciler heals it exactly as if no predicate existed.

The test was aimed at the wrong class of event. The class it actually drops is a resync: a local timer inside each informer that periodically re-emits every object it holds as a synthetic update event, where old and new are literally the same cached object. Same generation, by construction, because nothing happened. Not inference - the Options.SyncPeriod doc comment in the exact version pinned here, v0.24.1, states it.

A resync locally triggers an artificial update event with the same object as both old and new, and predicates expecting those to differ (it names GenerationChangedPredicate) "will filter out this event."

It also states that a resync does not sync between the local cache and the server, which is barrier 3's answer and the next section's foundation.

🔍 Isolating it

So I measured the thing the predicate actually suppresses: steady-state reconcile rate with no drift at all.

A 48.5% drop. No error, no log line, no externally visible difference. The only way to detect it is to notice a rate that should exist and does not.

That drop is well clear of measurement noise: a separate, identically configured 10-second