Part II — The control plane and the loop

Controllers & Labels: the Loop Made Real, and Its Glue

Lesson 03 built the cluster's single source of truth: etcd holds all state, and the API server is the one process that reads and writes it, serving a watchable REST API to everyone else. But that lesson ended on a quiet betrayal. The API server only stores desired state. Create a Pod, the spec lands in etcd, the kubelet on the assigned node runs it — and then the node's power supply dies. The kubelet stops reporting, the Pod is gone from the world, yet its record still sits placidly in etcd marked as something that should exist. Storing the wish does not grant it. Something has to watch the gap between wish and reality and close it — and that something is the engine from lesson 00, finally written as real code. This lesson builds the controller, the ReplicaSet, and the one structural trick — the label selector — that lets a controller find its own Pods without ever naming them.

The previous step left this broken

The API server is a passive store. It validates and persists desired state and reports actual state, but it never acts on the difference between them. A Pod whose node has died is still recorded as wanted; nothing restores the count. Reconciliation — the observe → diff → act loop from lesson 00 — has been described but never instantiated. And there is a deeper missing piece hiding inside "restore the Pod": whatever does the restoring must know which Pods are its Pods, and Pod names are too brittle to be that anchor.

Linear position

Forced by: the API server stores desired state but does not reconcile it; a dead Pod stays "wanted" forever with nothing to recreate it.
New capability: a controller — a program that watches a spec and the real set and drives one toward the other — made concrete as the ReplicaSet (keep N matching Pods alive, level-triggered and idempotent), with labels + selectors as the loose-coupling-by-query mechanism that lets it own the right Pods.
Forces next: a ReplicaSet pins exactly one Pod template (one image version); shipping v2 means replacing the whole set without downtime — and being able to undo it.

The plan

Five moves. (1) Turn the lesson-00 loop into a real program: what a controller actually is, where it runs, and how it watches. (2) Build the ReplicaSet and prove it is level-triggered and idempotent with replica arithmetic. (3) Pose the ownership problem — "which Pods are mine?" — and reject naming. (4) Derive labels and selectors as coupling-by-query, and show the same idea wiring Services, scheduling, and policy. (5) Failure modes, a checkpoint, and the hand-off to Deployments.

1 · The loop, written as a program

Lesson 00 described an engine: observe the actual state, diff it against the desired state, act to shrink the difference, repeat forever. A controller is that engine compiled into a real process. It is a control loop that, for one kind of resource, runs roughly this body without end:

→observe → read the desired spec and the current real objects from the API server (never from etcd directly — that rule from lesson 03 holds for controllers too)

→diff → compute the difference: desired says 3, I see 2 alive → I am short one

→act → issue the minimal API calls to close the gap: create one Pod (or delete an extra one)

→repeat → block on a watch until something changes, then run the body again

Where does it run? In the kube-controller-manager — a single control-plane binary that hosts dozens of these loops (the ReplicaSet controller, the Deployment controller, the Job controller, the node controller, and more), each independent. It does not poll on a timer in a hot loop; it uses the API server's watch (lesson 03): a long-lived stream that pushes a notification the instant a relevant object is created, updated, or deleted. A controller keeps a local cache of the objects it cares about, kept fresh by that watch, and re-runs its reconcile body whenever the cache changes (plus a slow periodic resync as a safety net against missed events).

Two properties of this loop are not decoration — they are the whole reason Kubernetes survives chaos. The loop is level-triggered: it reacts to the current state of the world (the "level"), not to the events that produced it (the "edge"). An edge-triggered system would listen for "Pod died" messages and recreate one per message — and if it missed three death messages during a network blip, it would be permanently three short. A level-triggered controller just re-reads "I want 3, I see 0" and creates 3, no matter how many deaths it did or didn't hear about. The loop is also idempotent: running it twice when nothing changed does nothing the second time, because the diff is empty. Convergence, not commands.

2 · The ReplicaSet: keep N Pods alive

The first and most fundamental controller is the ReplicaSet. Its desired spec is tiny: a replica count (how many Pods you want), a Pod template (the spec to stamp out copies from — image, ports, resource requests), and a selector (which Pods it owns — §4). Its job is one sentence: make the number of matching, running Pods equal the replica count.

   ReplicaSet "web"  (desired: replicas = 3)
        |
        |  reconcile loop, level-triggered
        v
   observe matching Pods ──► count = 2   (one node just died)
        |
        diff: desired 3 − actual 2 = +1
        |
        act: create 1 Pod from the template
        |
        v
   count = 3   →  diff = 0  →  idle (idempotent: does nothing more)

Walk the arithmetic, because the behavior is the point. You set replicas = 3. Three Pods run. A node hosting one of them loses power; its kubelet stops reporting and the node controller eventually marks that Pod gone. The ReplicaSet's watch fires, it re-observes actual = 2, computes 3 − 2 = +1, and creates one Pod. Now suppose something far worse: a whole rack fails and 3 Pods vanish at once. An edge-triggered "restart on crash" cron would be hopeless. The ReplicaSet simply observes actual = 0, computes 3 − 0 = +3, and creates exactly 3. Kill 3, it makes 3. Kill 1, it makes 1. Add an extra Pod by hand that happens to match its selector, and it observes actual = 4, computes 3 − 4 = −1, and deletes one. The count is driven from both directions to exactly N.

replicas (desired)

The integer you want. The single knob a human sets. Change it from 3 to 10 and the loop creates 7 more on its next pass — no orchestration script, just a new diff.

Pod template

The stamp. Every Pod the ReplicaSet creates is minted from this spec. It is fixed — a ReplicaSet describes exactly one version of the app. (This single fact is what forces Deployments in lesson 05.)

level-triggered

Reacts to current state, not to events. Miss every "Pod died" notification and it still converges on the next resync, because it re-reads the level "I see 0, I want 3."

idempotent

Reconcile when the diff is empty and nothing happens. Safe to run a thousand times a second; safe to run after a controller-manager restart. Convergence has no side effects once converged.

This is lesson 00's engine made of real code: a watch keeps a local cache fresh, a reconcile function computes desired-minus-actual, and a handful of create/delete calls to the API server close the gap. The API server still only stores; the ReplicaSet controller is the actor that makes the stored wish come true. But everything above quietly assumed a hard question was already answered.

3 · The hard question: which Pods are mine?

"Observe the matching Pods, count them" — matching how? The ReplicaSet needs to look at the cluster's Pods and decide which ones are the ones it is responsible for. Get this wrong and a controller either ignores Pods it should manage (so it over-creates) or grabs Pods that belong to someone else (so it deletes them). Ownership has to be exact.

The obvious answer is to track Pods by name: the ReplicaSet remembers it created web-a1b2, web-c3d4, web-e5f6, and watches those three names. This is brittle in three independent ways, and each is fatal:

Naming Pods directly	Why it breaks
Names churn	A recreated Pod gets a new random name (Pod names must be unique and a dead Pod's name is tombstoned). The controller's list of names is stale the moment it acts; it would have to rewrite its own spec on every recreate.
You can't address a set	"My Pods" is a moving population that grows, shrinks, and rolls. A static list of names cannot describe "however many Pods currently wear this identity." You want to talk about the set, not its current members.
No one else can join the query	Later, a Service needs the same answer ("which Pods are the web backends right now?"), and so does a NetworkPolicy. If ownership lives as a private name-list inside the ReplicaSet, nothing else can ask the question.

Reference-by-name is tight coupling: the referrer holds a direct pointer to specific instances, and the moment those instances change identity the pointer dangles. What we want is the opposite — a way to describe a kind of Pod and let membership be computed continuously from reality.

4 · Labels & selectors: coupling by query

The answer is the single most under-appreciated idea in Kubernetes. Every object can carry labels: arbitrary key=value string pairs you attach, like app=web, env=prod, tier=frontend, version=v1. Labels are metadata you choose — Kubernetes assigns no meaning to them; they exist purely to be queried. A controller then declares a label selector: a predicate over labels, like app=web,env=prod (the comma means and). The rule is:

The structural idea

A controller owns every object that matches its selector — right now, computed live, by query. It never holds a list of names. "My Pods" is not a stored reference; it is the result of a query re-evaluated continuously against the real set. This is loose coupling by query, not by reference.

Now the brittleness dissolves. A recreated Pod is minted from the template, which stamps it with the labels app=web,env=prod — so it matches the selector automatically, with no name to update. The selector describes the set, not its members, so it stays correct as Pods are born and die. And because the selector is just a public query, anything else can ask the same question independently: a Service selects app=web to find load-balancing targets (lesson 06), the scheduler reads labels to spread or co-locate Pods (lesson 10), a NetworkPolicy selects tier=frontend to decide what may connect (lesson 12). One mechanism, four subsystems, zero shared pointers.

→label → you stamp Pods with app=web,env=prod,tier=fe via the template; arbitrary, meaningful only to you

→select → a controller declares selector: app=web,env=prod (an AND over equalities; set-based ops like in/notin also exist)

→own → the matching set is its responsibility, recomputed on every reconcile — membership is a verb, not a list

→reuse → Services, the scheduler, and policy run their own selectors over the same labels, fully decoupled from the ReplicaSet

The one discipline this demands: selectors must not overlap across controllers. If two ReplicaSets both select app=web, they will fight over the same Pods — each sees the other's Pods as its own and the counts thrash. (Kubernetes mitigates this with an ownerReference on each Pod recording its controller, used to break ties and cascade deletes, but you are still expected to keep selectors disjoint.) Choose label schemes so each controller's selector carves out a private slice. Get that right and the whole cluster coordinates through a shared, queryable label space instead of a tangle of direct references.

Label-selector playground — own the set, not the names

Each card is a Pod with toggleable key=value labels (click a label to flip it on/off). Type a selector below as comma-separated equalities — e.g. app=web,env=prod (comma = AND). Matched Pods light up green and the ReplicaSet's count updates live. Hit kill a matched Pod: the count drops, then the level-triggered loop notices the gap and recreates one (with the right labels, so it re-matches). The ReplicaSet never knew a single Pod's name.

selector:

ReplicaSet desired

ReplicaSet sees (matched)

—

Diff (desired − actual)

—

5 · Failure modes & checklist

Failure modes

Overlapping selectors. Two controllers select the same labels and fight over the same Pods, deleting each other's work. Signal: Pod counts oscillate, Pods get created and immediately deleted, churn in the event log with no human change.
Editing a live ReplicaSet's template expecting a rollout. Changing the template does not touch existing Pods — only newly created ones use the new template, so you get a mixed fleet. Signal: half the Pods on the old image after a "deploy" that was really a template edit. (This is exactly the gap Deployments close.)
Treating a deleted Pod as a bug. You hand-create a Pod with matching labels; the ReplicaSet sees actual > desired and deletes one. Signal: "Kubernetes keeps killing my Pod" — it is reconciling to N, as designed.
Mutating a Pod's labels at runtime. Strip the label a ReplicaSet selects on and the Pod is orphaned; the controller sees the count drop and creates a replacement, leaving the orphan running. Signal: one extra unmanaged Pod lingering; useful as a deliberate debugging trick, surprising by accident.
Confusing labels with annotations. Putting queryable identity in annotations (which are not selectable) so selectors match nothing. Signal: ReplicaSet sees 0, creates N forever, ignores your existing Pods.

Checklist

Give every workload a coherent label scheme up front — at minimum app, plus env and tier as needed; keep each controller's selector disjoint.
Remember the loop is level-triggered: design for "converge to N," never for "react to each death."
Never depend on a specific Pod name; address Pods only through selectors (and, later, Services).
To change the app's version, change desired state through the Deployment (lesson 05), not by editing a ReplicaSet template in place.
Verify ownership with kubectl get pods -l app=web,env=prod — if that query disagrees with what the controller manages, your labels are wrong.
Keep selectors immutable once a controller is live; changing a selector can orphan or capture Pods unexpectedly.

Checkpoint exercise

Try it

In the widget, set the selector to app=web,env=prod and note how many Pods the ReplicaSet sees. Now click kill a matched Pod and watch the diff jump to +1, then settle back to 0 as the loop recreates one — confirm the new Pod re-matches without you touching any names. Next, on one matched Pod toggle env=prod off: predict before you look — does the ReplicaSet's count drop, and what does the loop do? (It should orphan that Pod and create a replacement, because the selector no longer matches it.) Finally, change the selector to just app=web and explain in one sentence why more Pods now match, and why two ReplicaSets that both selected app=web would be a disaster.

Where this points next

The ReplicaSet keeps N Pods alive and finds them by query — robust against node death, racks failing, and stray hand-made Pods. But re-read the topic card on the Pod template: it is fixed. A ReplicaSet describes exactly one version of your app. To ship v2 you would have to delete the v1 ReplicaSet and create a v2 one — which means a window with zero Pods running (downtime), and no record of v1 to fall back to if v2 is broken (no undo). Changing desired state is currently an all-or-nothing, irreversible swap. Lesson 05, Deployments, introduces a controller of ReplicaSets: it stands up a new ReplicaSet for v2 and shifts replicas from old to new under a strategy (maxSurge/maxUnavailable), keeps revision history for one-command rollback, and turns "deploy" from a dangerous manual swap into a safe, reversible change of desired state.

Takeaway

The API server only stores desired state, so a dead Pod stays "wanted" with nothing to restore it — that gap forces the controller, the lesson-00 loop written as real code (observe → diff → act → watch → repeat) running in the kube-controller-manager. The ReplicaSet is the foundational controller: keep the number of matching Pods equal to replicas, level-triggered (kill 3, it makes exactly 3, because it re-reads the current level rather than counting death events) and idempotent (reconciling a converged set does nothing). The deep idea is how it finds its Pods: not by name — names churn, can't describe a moving set, and can't be reused by others — but by label selector. Pods carry arbitrary key=value labels; a controller owns every Pod matching its selector, recomputed live. This is loose coupling by query, not reference, and it is the single structural mechanism that later also wires Services, the scheduler, and policy over the same shared label space. The catch that forces lesson 05: a ReplicaSet pins one fixed template, so shipping v2 means swapping the whole set — downtime, no undo.

Interview prompts

What is a controller, and where does it run? (§1 — a control loop (observe → diff → act → repeat) that drives actual state toward a desired spec; it runs in the kube-controller-manager, talks only to the API server, and uses watch + a local cache rather than tight polling.)
What does "level-triggered" mean and why does it matter? (§1, §2 — the loop reacts to the current state (the level "I see 2, want 3"), not to events (edges like "a Pod died"). It is robust to missed events: kill 3 Pods at once and it still recreates exactly 3, because it re-reads the level rather than counting deaths.)
Why can't a ReplicaSet just track its Pods by name? (§3 — names churn (recreated Pods get new names), a name-list can't describe a moving set, and a private name-list can't be reused by Services or policy; reference-by-name is tight coupling that dangles the moment instances change identity.)
How does a controller actually decide which Pods are its Pods? (§4 — by label selector: Pods carry arbitrary key=value labels, the controller owns every Pod matching its selector, recomputed live on each reconcile. Membership is a query result, not a stored reference — loose coupling by query.)
Why is the label/selector idea called the most under-appreciated in Kubernetes? (§4 — the same queryable label space decouples and wires four subsystems with zero shared pointers: ReplicaSets own Pods, Services find endpoints (06), the scheduler spreads/co-locates (10), and NetworkPolicy gates traffic (12) — all by independent selectors over the same labels.)
What happens if two ReplicaSets have overlapping selectors? (§4, §5 — they fight: each sees the other's Pods as its own and reconciles toward its own count, causing oscillation/churn. ownerReference mitigates but you must keep selectors disjoint.)
You edit a running ReplicaSet's Pod template to a new image — what happens to existing Pods? (§2, §5 — nothing; the template only governs newly created Pods, so you get a mixed-version fleet. Reconciling to N never re-stamps live Pods. This is exactly why Deployments (05) exist.)