Kubernetes Under the Hood: What Actually Happens When You Run kubectl apply
Open a terminal on your workstation, point your environment at a production cluster, and execute a single declarative command:
$ kubectl apply -f deployment.yaml
deployment.apps/web-service configured
Within four seconds, three containers running your application have materialized across three separate physical servers in different availability zones. They have private IP addresses, health checks, CPU limits, shared storage volumes, and internal load-balancer endpoints routing traffic to them.
To the developer typing in the terminal, it feels like an imperative remote procedure call: Kubernetes, go spin up my containers right now. The reality is fundamentally different. There is no central orchestrator pushing containers onto servers. The kubectl binary does not talk to your worker nodes. The servers running your workload never received a command telling them to start your app.
In fact, if every single control plane node suddenly lost power thirty seconds after you executed that command, your containers would continue running completely uninterrupted.
Kubernetes is not an imperative management tool. It is an asynchronous distributed state engine built on declarative consensus, optimistic concurrency control, and continuous reconciliation loops. When you run kubectl apply, you are not starting software. You are recording a desire in a distributed database, and triggering a cascade of autonomous control loops that continuously drag reality toward your declared intent.
Here is the mechanical breakdown of every step that unfolds between your keystroke and a live container.
The End-to-End Lifecycle: An Overview
Before diving into each subsystem, examine the complete path your manifest travels across the cluster:
The journey divides into five distinct stages:
- Client Preparation:
kubectlperforms schema discovery and computes a 3-way strategic merge patch before sending an HTTP request. - The API Server Gauntlet:
kube-apiserverruns the payload through authentication, authorization, mutating webhooks, schema validation, and validating webhooks before committing toetcd. - Controller Reconciliation: The Deployment Controller detects the updated spec and creates a ReplicaSet, which then generates individual Pod manifests with unassigned node names.
- Scheduling: The
kube-schedulerwatches for pending Pods, filters out unsuitable nodes, scores the survivors, and writes a binding back to the API server. - Node Execution: The
kubeletdaemon on the target worker node observes the assignment, directs the Container Runtime Interface (CRI) to launch a Pause container, configures networking via CNI, and starts your application containers.
Let us inspect the gears of each stage.
Client-Side Mechanics: The 3-Way Strategic Merge Patch
Long before any network packet leaves your laptop, kubectl executes critical validation logic. When you execute kubectl apply, the CLI does not blindly upload your YAML file.
First, it queries the cluster Discovery API (/openapi/v2 or /openapi/v3), caching the cluster's OpenAPI schema locally in ~/.kube/cache/discovery/. It validates that your fields exist, your indentation is valid, and your types match the cluster version.
Next, kubectl solves a fundamental distributed systems dilemma: concurrent modifications. Imagine you apply a deployment with replicas: 3. An hour later, a Horizontal Pod Autoscaler (HPA) scales the deployment to replicas: 10. If you subsequently edit the container image in your local YAML and run kubectl apply, should your command overwrite replicas back to 3, or should it preserve the HPA's 10 replicas?
To solve this without locking resources, Kubernetes implements a 3-Way Strategic Merge Patch using three distinct data sources:
- The Local Configuration: The YAML file on your workstation disk.
- The Live Cluster State: The current object as recorded in the cluster database.
- The Last-Applied Configuration: A serialized JSON snapshot stored inside an annotation on the live object:
kubectl.kubernetes.io/last-applied-configuration.
[ 1. Local YAML ]
│ ├── (Diff) ──┐
│▼
[ 3. Last-Applied JSON ] ───► [ Strategic Merge Patch Engine ]
│▲
├── (Diff) ──┘
│
[ 2. Live State ]
The patch engine calculates two separate diffs: What changed between your previous local file and your current local file? What changed between your previous local file and the live cluster state?
If a field (like replicas) was modified in the live cluster but was not modified in your local file, kubectl preserves the live cluster value. If a field was modified in your local file, kubectl updates it. If a field was deleted from your local file that existed in the last-applied annotation, kubectl removes it from the cluster.
Once the patch is computed, kubectl serializes it into a JSON Patch payload and issues an HTTP PATCH request to the API server endpoint:
PATCH /apis/apps/v1/namespaces/default/deployments/web-service
Content-Type: application/strategic-merge-patch+json
Inside kube-apiserver: The Admission Control Gauntlet
The kube-apiserver is the front door of Kubernetes. It is also completely stateless. It does not run controllers, it does not schedule pods, and it does not manage containers. Its sole responsibility is acting as a secure, transactional gateway to the cluster database.
Every request that reaches kube-apiserver must survive six sequential gates:
Stage 1: Authentication (Who are you?)
The API server inspects the incoming TLS handshake and HTTP headers. It passes the credentials through a chain of configured authenticators:
- X.509 Client Certificates: Often used for cluster administrators (extracting the Common Name as username and Organization as group).
- OpenID Connect (OIDC): Bearer tokens issued by providers like Okta, Google, or Keycloak.
- ServiceAccount Tokens: Signed JWTs issued by the cluster token controller.
The first authenticator that successfully validates the request attaches an identity struct (User: alice, Groups: ["devs", "system:authenticated"]) to the request context. If all authenticators fail, the request terminates immediately with 401 Unauthorized.
Stage 2: Authorization (Are you allowed to do this?)
Next, the identity passes to the Role-Based Access Control (RBAC) engine. The authorizer extracts four attributes:
- User/Group: Who is making the request?
- Verb: What action are they taking? (
get,list,create,update,patch,delete) - Resource: Which object type is targeted? (
deployments,pods,services) - Namespace: In which administrative boundary?
The engine evaluates all RoleBinding and ClusterRoleBinding objects. If no rule grants explicit permission for that user, verb, and resource in that namespace, the request is rejected with 403 Forbidden.
Stage 3: Mutating Admission Webhooks (Modifying the object)
If authorized, the request enters the admission control chain. This phase begins with Mutating Admission Controllers. These plugins can modify the object before it is stored. For example:
- Setting default values for omitted fields (e.g., setting
imagePullPolicy: IfNotPresent). - Injecting service mesh sidecars: When you deploy an application into an Istio-enabled namespace, a mutating webhook catches your Pod spec and injects an Envoy proxy container alongside your application container.
Because mutating webhooks can modify the object, they run in a loop until the object stabilizes.
Stage 4: Object Schema Validation
Once mutations settle, the API server performs strict schema validation against the internal Golang struct definitions of the target resource. If a field contains an invalid type (such as a string where an integer is expected) or violates regex constraints, the request fails with 422 Unprocessable Entity.
Stage 5: Validating Admission Webhooks (Policy enforcement)
The modified, validated object now reaches Validating Admission Controllers. Unlike mutating controllers, validating controllers are strictly read-only decision gates. Tools like Open Policy Agent (OPA) Gatekeeper or Kyverno inspect the final object against organization security policies:
- Does this container try to run as root (
runAsNonRoot: false)? - Does the container image originate from an approved corporate registry?
- Did the developer declare mandatory CPU and memory resource limits?
If any validating webhook rejects the object, the entire transaction aborts.
Stage 6: Persistence in etcd
If the object passes all five gates, the API server serializes it into Protocol Buffers (protobuf) format and commits it to the distributed key-value store, etcd. The API server responds to your terminal with HTTP 200 OK or HTTP 201 Created.
At this exact moment, your kubectl apply command exits with a success message. Yet on your servers, not a single container has been launched yet.
The Source of Truth: etcd, Raft Quorum, and Resource Versions
Every piece of state in a Kubernetes cluster lives in etcd. If etcd is healthy, the cluster can rebuild itself from scratch. If etcd is lost without backups, your cluster state is gone forever.
etcd is a strongly consistent, distributed key-value store that coordinates using the Raft consensus algorithm.
[ etcd Node 1 ] (Leader) ├── Raft Heartbeat (RPC) ──► [ etcd Node 2 ] (Follower) └── Raft Heartbeat (RPC) ──► [ etcd Node 3 ] (Follower)
In a production 3-node or 5-node etcd cluster, one node is elected Leader. When the API server writes your Deployment object, the Leader appends the entry to its Raft log and replicates it to the follower nodes. The write is only committed once a quorum (a majority: $(N/2) + 1$) acknowledges the log entry. This is why production clusters always run an odd number of control plane nodes (3 nodes tolerate 1 failure, 5 nodes tolerate 2 failures).
Optimistic Concurrency Control (resourceVersion)
Databases like PostgreSQL handle concurrent updates with row-level locks or transaction isolation levels. In a distributed control plane where dozens of controllers inspect the same object simultaneously, database locks would cause crippling bottlenecks. Kubernetes avoids locks entirely using Optimistic Concurrency Control (OCC).
Every object stored in etcd has a mandatory metadata attribute: metadata.resourceVersion. This is a monotonically increasing integer tracked by etcd.
metadata:
name: web-service
namespace: default
resourceVersion: "4829104"
When a controller reads an object, it records the resourceVersion. When it sends an update, the API server checks if the incoming resourceVersion matches the current version in etcd. If another controller updated the object in the intervening millisecond, the version numbers do not match. The API server immediately rejects the write with 409 Conflict. The rejected controller does not crash; it simply re-reads the fresh state, re-evaluates its calculation, and retries.
The Watch API: Eliminating Polling
In early distributed systems, clients constantly polled the database to check for updates (GET /pods every 2 seconds). With thousands of pods, polling rapidly saturates CPU and network bandwidth. Kubernetes replaces polling with the etcd Watch API, exposed through kube-apiserver over persistent HTTP/2 gRPC streams.
When a controller connects, it requests a Watch stream starting at a specific resourceVersion:
Watch ( resource = "pods" , startingResourceVersion = "4829104" )
The API server holds this connection open and pushes compact change events (ADDED, MODIFIED, DELETED) in real time as commits occur in etcd.
The Controller Manager: Level-Triggered Reconciliation Loops
If the API server is just a database gateway, what actually makes things happen? The answer is kube-controller-manager: a single binary executing dozens of independent, concurrent reconciliation loops.
A reconciliation loop follows a simple, infinite loop:
$$\text{Observe Current State} \longrightarrow \text{Compare with Desired State} \longrightarrow \text{Act to Eliminate the Diff}$$
Level-Triggered vs Edge-Triggered Systems
To understand why Kubernetes is exceptionally reliable, you must understand the difference between edge-triggered and level-triggered architectures:
- Edge-Triggered (Message-Driven): Reacts to state transitions. If a message says "Add 1 Pod," you add 1 pod. If a network blip drops that message, your system falls out of sync permanently.
- Level-Triggered (State-Driven): Reacts to current levels. It does not care what happened in the past; it only cares about the current snapshot. It asks: "The user declared 3 pods. Right now, there are 2 pods. Therefore, I will create 1 pod."
Because Kubernetes is level-triggered, it is naturally self-healing. If a network partition drops twenty events, the controller does not get confused. On the next reconciliation pass, it looks at the live level, spots the discrepancy, and repairs it.
The client-go Informer Pipeline
Controllers never read directly from etcd. If fifty controllers queried etcd on every loop iteration, etcd would collapse under read pressure. Instead, every controller uses the client-go Informer library:
- Reflector: Establishes a ListWatch connection to the API server. On startup, it lists all objects once, then watches for updates.
- DeltaFIFO: A queue that receives change events and deduplicates them.
- Indexer (Local Cache): An in-memory cache holding the controller's view of cluster objects. When a controller needs to read an object, it reads from this local RAM cache in microseconds with zero network calls.
- WorkQueue: A thread-safe, rate-limiting queue that holds the keys (
namespace/name) of objects that need reconciliation.
The Controller Cascade
When you apply a Deployment, a sequence of controllers execute like dominoes:
Deployment Controller: Watches Deployments and ReplicaSets. It spots your new Deployment in its Indexer cache. It calculates that no ReplicaSet exists matching the new pod template hash. It calls the API server: POST /apis/apps/v1/namespaces/default/replicasets.
ReplicaSet
Comments
No comments yet. Start the discussion.