Part II — The control plane and the loop

The Control Plane: One Source of Truth

Lesson 02 gave you the Pod (the smallest schedulable unit) and the Node, where a kubelet runs the reconcile loop for the Pods assigned to it and reports the actual state of those Pods back up. But back up to what? Two facts are now floating without a home: the desired state a user declares ("run 3 replicas of this Pod") and the actual state each kubelet observes ("Pod-7 is dead on node-2"). If each actor keeps its own copy, they drift, disagree, and fight. This lesson derives the single durable, consistent place all of that state lives — and the one gatekeeper that is allowed to write to it.

1 · Why split state self-destructs

Picture three actors, each with its own notebook. A user wants 3 replicas, so she writes "3" in her notebook. The scheduler decides Pod-A→node-1, Pod-B→node-2, Pod-C→node-1, and records that in its notebook. Each kubelet writes down which Pods it is actually running. Now node-2 reboots. Its kubelet's notebook is wiped; it comes back believing it runs nothing. The scheduler still thinks Pod-B lives there. The user still thinks 3 are up. Three notebooks, three truths, and no procedure to say which one is authoritative.

The failure is not a bug in any one actor — it is structural. Whenever the same fact is stored in N places, you have N copies that can diverge, and reconciling N copies is its own distributed-systems problem. The only escape is to collapse N to 1: one record that is, by definition, the truth. Desired state and actual state both live there, every actor reads and writes through it, and no actor caches a private authoritative copy. Building that single source of truth correctly is what the entire control plane is for.

2 · Choosing the store: consistent, replicated, watchable

"One record" cannot mean one file on one machine — that machine dies and the cluster's brain dies with it. The store must survive node loss, so it is replicated across several machines (typically 3 or 5). But replication reopens the very problem we set out to kill: replicas can disagree. So we need not just replication but strong consistency — every reader sees the same committed value, and there is never a moment where two replicas would answer the same question differently.

That is exactly what etcd provides. etcd is a distributed key-value store: keys look like filesystem paths (/registry/pods/default/web-0) and values are the serialized object. It achieves consistency with the Raft consensus algorithm — the replicas elect a leader, every write is proposed to the leader, and the write is only acknowledged once a majority (a quorum) has durably logged it. With 3 replicas the quorum is 2, so the cluster tolerates 1 failure; with 5 replicas the quorum is 3 and it tolerates 2. (This is why etcd clusters are sized in odd numbers — an even count buys no extra fault tolerance but doubles the chance of a split vote.)

Two more properties earn etcd the job. First, every change bumps a single, cluster-wide, monotonically increasing revision number — a global logical clock you can compare and order. Second, etcd supports watch: a client can ask "tell me about every change to keys under this prefix, starting from revision R," and etcd streams events as they commit. Watch is the feature that lets the rest of the system stop polling — we lean on it hard in §5.

3 · One writer in front: the API server

etcd is now the single store — but should every component connect to it directly? No, and the reason is the same instinct that built etcd in the first place. etcd is dumb on purpose: it stores bytes and orders them. It does not know that a Pod's name must be a valid DNS label, that this user is allowed to delete that Secret, or that a field is read-only. If 30 different programs all wrote to etcd directly, those rules would be re-implemented (and broken) 30 times, the etcd credentials would be everywhere, and a single buggy client could write a malformed object that crashes every reader.

So we put exactly one program in front of etcd: the kube-apiserver. It is the only component permitted to talk to etcd. Everything else — kubelets, controllers, the scheduler, and your kubectl — talks only to the API server, over a REST API (and a watch API). Every write runs the same gauntlet before a single byte reaches etcd:

This is the hub-and-spoke design: the API server is the hub, etcd is the vault directly behind it, and every other component is a spoke. The payoff is enormous — the validation, security, and defaulting rules live in exactly one place; etcd's credentials never leave the hub; and the rest of the cluster is decoupled from the storage format entirely (the API server can re-encode or even migrate the backing store without any spoke noticing). The API server is otherwise stateless: it holds no authoritative data of its own, so you can run several replicas behind a load balancer for availability, all fronting the same etcd.

4 · The resource model: spec, status, and resourceVersion

Now the data model that makes the gap measurable. Every object the API server stores has the same shape: a spec (the desired state, written by the user) and a status (the actual state, written by the controller/kubelet that owns reality for that object). A Deployment's spec says "3 replicas"; its status says "2 available." A Node's spec is nearly empty; its status carries the kubelet's report of allocatable CPU and memory. The single most important convention in Kubernetes is this spec/status split — it is desired-vs-actual from §1, baked into every object, so the gap is always a field you can read.

That raises a concurrency problem. Many actors edit the same object: a user scales a Deployment while its controller updates the same object's status, all within the same instant. With naive last-write-wins, the controller's GET-then-PUT could silently overwrite the user's change with stale data. Kubernetes solves this with optimistic concurrency, not locks. Every object carries a resourceVersion — an opaque token (backed by etcd's revision) that changes on every write. The contract is:

This is optimistic because it assumes conflicts are rare and pays nothing in the common case — no lock to acquire, no lock to release, no deadlock, no actor stuck waiting on a crashed lock-holder. It only costs a re-read on the rare collision. The widget below makes the race concrete: two writers both read rv=42, both try to write, exactly one wins, and the loser must re-read and retry against the new version.

Note what the spec/status split buys here: writer A edits spec.replicas and writer B edits a status field, so even when they collide the loser's retry re-applies its change to a different field — they are not fighting over the same data, just over the same object's version token. (The API server even exposes spec and status as separate subresource endpoints, but the resourceVersion discipline is identical.)

5 · Hub-and-spoke and watch, not poll

With one store and one writer, the architecture is a wheel. Every component is a spoke that connects inward to the API server; etcd sits behind the hub and is invisible to the spokes.

                    desired state in, actual state out
                                 |
            kubectl ----\        v        /---- scheduler
                         \   +---------+  /     (assigns Pods
                          \  |  kube-  | /       to nodes -> L10)
                           \ | apiserver| /
   controllers -------------+----+-----+-------------+ kubelets
   (close desired/actual    |    | (only writer)     | (run Pods,
    gap -> L04)             |    v                   |  report status)
                           authn -> authz -> admission -> validate
                                       |
                                       v
                              +-----------------+
                              |      etcd        |   Raft quorum,
                              |  /registry/...   |   global revision,
                              | (single source   |   WATCH stream
                              |   of truth)       |
                              +-----------------+

   rule: every spoke talks ONLY to the apiserver. nobody touches etcd directly.

The spokes do not poll. Polling — "ask the API server every 2 seconds whether anything changed" — wastes CPU and network when nothing changes and still reacts up to a full interval late when something does. Instead each spoke opens a watch: a long-lived connection that says "stream me every change to Pods I care about, starting from the resourceVersion I last saw." The API server (backed by etcd's watch) pushes an event the instant an object changes. A new Pod is created → the scheduler's watch fires immediately → it binds the Pod to a node → that write fires the target kubelet's watch → the kubelet starts the containers. Reactions are event-driven and near-instant, not interval-bound.

One subtlety the resourceVersion makes possible: a watcher that disconnects (network blip) does not have to re-fetch the world. It reconnects and asks to watch "from resourceVersion R" — the last revision it processed — and the API server replays everything since. If R is too old to be retained, the watcher does a fresh list (to get a consistent snapshot and its current resourceVersion) and re-watches from there. List-then-watch is the standard pattern every controller uses; you will see it reappear in lesson 04 as the heart of the controller loop. And because watch delivers the current object (not a "field X incremented" delta), the whole system stays level-triggered: a spoke that wakes from any starting point simply reads the current truth and reconciles toward it — exactly the engine from lesson 00, now wired through a single hub.

Failure modes & checklist

Checkpoint exercise

Where this points next

You now have a single, durable, consistent truth (etcd) and one disciplined gatekeeper in front of it (the API server), with every actor reading and writing through watches over one hub. But notice the hole this leaves: the API server only stores state. It will record "desired: 3 replicas" and, when a node dies, faithfully record "actual: 1 Pod left, 2 gone" — and then do absolutely nothing about it. A successful write is a recorded intent, not an executed action. The gap between desired and actual is now perfectly visible in one place, and yet no component is watching that gap and acting to close it. That missing actor — a program that watches a spec, compares it to reality, and continuously drives the difference to zero — is the controller, and figuring out which Pods are "its" Pods forces the label/selector model. That is lesson 04, Controllers & Labels.

Interview prompts

• Why does Kubernetes need a single source of truth, and what goes wrong without one? (§1 — if desired and actual state live in separate per-actor copies, a reboot or race makes two actors believe two different truths and the reconcile loop converges toward whichever it read; collapsing N copies to 1 trusted record is the only structural fix.)
• Why etcd specifically — what would a Redis cache or a single SQL box fail at? (§2 — the store must be both replicated (survive node loss) and strongly consistent (no two replicas disagree); etcd gives Raft quorum consistency, durability, a global revision, and a native watch stream; a default cache isn't durable/consistent under partition and a single SQL box is a single point of failure with no cheap watch.)
• Why is the API server the only thing allowed to write to etcd? (§3 — etcd is intentionally dumb (stores bytes); the API server centralizes authn, authz, admission, validation, and defaulting in one place, keeps etcd credentials off every client, and decouples spokes from the storage format. Direct etcd writes bypass every safeguard.)
• What is the spec/status split and why does it matter? (§4 — every object carries spec (desired, user-written) and status (actual, written by the owning controller/kubelet); it bakes desired-vs-actual into every object so the gap the system reconciles is always a readable field.)
• Explain optimistic concurrency and resourceVersion; what is a 409? (§4 — each object has a resourceVersion token; you GET it, modify, and PUT only if the store is still at that version; a concurrent write makes your PUT stale and the API server returns HTTP 409 Conflict, so you re-GET, re-apply, and retry — no locking, cheap in the common case.)
• Why watch instead of poll, and how does a watcher recover after a disconnect? (§5 — watch pushes a change on commit (ms latency, ~free when idle) versus polling's interval lag and constant cost; on reconnect a watcher resumes "from resourceVersion R," and if R is too old it does a fresh list to get a consistent snapshot then re-watches — list-then-watch.)
• If the API server accepted my Pod, is it running? (§3, §5, handoff — no; the API server only stored the intent in etcd. A separate actor (scheduler, then a controller/kubelet) must observe the stored gap via watch and act; a stored desired state with nothing closing the gap stays Pending forever — which is why controllers come next.)