all_lessons/kubernetes/05lesson 6 / 14

Part II — The control plane and the loop

Deployments: Changing Desired State Safely

Lesson 04 gave you a ReplicaSet: a controller that watches a label selector and keeps exactly N matching Pods alive, level-triggered and idempotent. But a ReplicaSet pins one Pod template — one image, one version. To ship v2 you would have to edit the template in place (the ReplicaSet will not re-roll existing Pods) or delete the v1 set and create a v2 set. The first does nothing; the second tears down every v1 Pod at once — a hard outage — and if v2 crashes on boot, there is no button to bring v1 back. This lesson derives the Deployment: a controller of ReplicaSets that changes desired state safely, with bounded capacity loss and one-command undo.

The previous step left this broken
A ReplicaSet reconciles a Pod count against one frozen template. It has no concept of "the next version," no memory of the previous one, and no strategy for moving from one to the other. Releasing v2 by hand means either an all-at-once delete-and-recreate (downtime for every user during image pull + boot) or a fragile manual dance of scaling two ReplicaSets in lockstep — and either way, a bad v2 leaves you with no recorded prior state to roll back to.
Linear position
Forced by: a ReplicaSet holds one template, so a version change is a destroy-and-replace with no undo.
New capability: a Deployment owns a set of ReplicaSets, creates a new one per version, shifts replicas old→new under a strategy (RollingUpdate or Recreate) that bounds capacity loss, and keeps revision history for one-command rollback.
Forces next: rolling Pods are deleted and recreated, so they get new IPs every time — a client can never be pointed at a Pod that keeps moving.
The plan
Six moves. (1) Why a ReplicaSet alone cannot release safely. (2) The Deployment as a controller of ReplicaSets — the three-level ownership chain. (3) The two strategies: Recreate vs RollingUpdate. (4) The rollout arithmetic — maxSurge / maxUnavailable and the minimum available capacity, worked for replicas=10. (5) Revision history and rollback; readiness gating the roll; how selectors keep traffic flowing. (6) Failure modes, checkpoint, and the IP-churn problem that forces Services.

1 · Why a ReplicaSet cannot release

A ReplicaSet's whole job is one equation: observed Pods matching selector == desired count. It does not own the idea of an upgrade. If you change its template (say, bump the image from app:v1 to app:v2), the ReplicaSet does not touch the Pods it already has — they still run v1. The new template only applies to Pods it creates from now on. So you would have to manually delete v1 Pods to force replacements, with no control over the rate, no health-gating, and no record of what v1 even was.

The blunt alternative — delete the v1 ReplicaSet, create a v2 ReplicaSet — drops your replica count to 0 the instant the old Pods terminate, then climbs back up only after v2 images pull and boot. For a 10-replica service that is a full outage lasting the pull-plus-boot time (tens of seconds to minutes for a large image). And if v2 segfaults on startup, you are now staring at 10 crash-looping Pods with the v1 template gone from the cluster. You need a controller that remembers v1, introduces v2 gradually, and can reverse course. That is a Deployment.

2 · The Deployment: a controller of ReplicaSets

A Deployment is a higher-level controller whose desired state is "N Pods of this template, reached via this strategy." It does not manage Pods directly. Instead it manages ReplicaSets, each of which manages Pods — the same ownership-by-selector idea from lesson 04, stacked one level higher.

  Deployment  app=web, replicas=10, strategy=RollingUpdate
      │   (owns, by template hash)
      ├── ReplicaSet  app=web,pod-template-hash=v1   desired 0   ← old revision (kept)
      │       └── (Pods, draining to 0)
      └── ReplicaSet  app=web,pod-template-hash=v2   desired 10  ← new revision
              └── Pod  Pod  Pod  Pod  Pod  Pod  Pod  Pod  Pod  Pod

When you change the Pod template, the Deployment controller hashes the new template into a pod-template-hash label, creates a brand-new ReplicaSet for that hash, and then drives a transfer: it scales the new ReplicaSet up and the old one down, a few replicas at a time, until the new set holds all the replicas and the old set holds zero. Crucially, it does not delete the old ReplicaSet — it leaves it scaled to 0 as a recorded revision. That retained, scaled-to-zero ReplicaSet is exactly what makes rollback a one-liner (§5).

The Deployment is the engine from lesson 00 applied to ReplicaSets instead of to Pods: observe (which ReplicaSets exist, at what scale) → diff (do they match the desired template + strategy?) → act (nudge replicas old→new by one allowed step) → repeat. A rollout is not a script that runs once; it is a control loop converging to "all replicas on the new template."

3 · Two strategies: Recreate vs RollingUpdate

The strategy answers one question: how much capacity may I lose, and how much extra may I spend, while moving from old to new?

StrategyWhat it doesCapacity during rollUse when
RecreateScale old ReplicaSet to 0, wait for all old Pods to terminate, then scale new to N.Drops to 0 — a deliberate outage window.The app cannot run two versions at once (e.g. an exclusive DB schema lock, a singleton). Simplicity over availability.
RollingUpdate (default)Incrementally add new Pods and remove old ones, bounded by maxSurge and maxUnavailable, until all replicas are new.Stays near N — never below N − maxUnavailable.Almost always — any stateless or version-compatible service that must stay up.

RollingUpdate is governed by two knobs, each an absolute count or a percentage of desired replicas (rounded — maxSurge rounds up, maxUnavailable rounds down):

maxSurge
How many Pods above the desired count may exist at once. It bounds the extra cost of the roll (transient pods you pay for). maxSurge = 25% of 10 = 3 → up to 13 Pods may exist mid-roll.
maxUnavailable
How many Pods below the desired count may be unavailable at once. It bounds the capacity loss. maxUnavailable = 25% of 10 = 2 → at least 8 Pods stay available throughout.

The two together set the width of the moving window. The controller never lets the live, available count fall below N − maxUnavailable, and never lets the total (old + new) Pod count exceed N + maxSurge. Inside that window it works as fast as new Pods become Ready.

4 · The rollout arithmetic (replicas = 10)

Take replicas = 10, maxSurge = 25% → 3 (rounded up), maxUnavailable = 25% → 2 (rounded down). The floor on available capacity is 10 − 2 = 8; the ceiling on total Pods is 10 + 3 = 13. Watch one representative step sequence (old = v1, new = v2, "available" = Ready Pods of either version):

startv1=10, v2=0 → available 10. The controller may surge to 13 total and may let availability dip to 8.
step 1surge up → create 3 new v2 Pods (booting). Total 13, available still 10 (v2 not Ready yet). When the 3 become Ready: available 13.
step 2scale old down → with 13 available and a floor of 8, the controller may delete up to 5 old Pods (13 − 8). It removes v1 Pods → v1=5, v2=3, available 8.
step 3surge again → top total back to 13: add 5 v2 Pods → v1=5, v2=8 (5 booting). Available 8 until they pass readiness, then 13.
step 4retire the rest → with 13 available and a floor of 8, delete the last 5 v1 → v1=0, v2=8 (all Ready), available 8. Still 2 short of 10.
step 5finish → surge the final 2 v2 Pods (total 10, 2 booting); once they pass readiness → v2=10, available 10. Old ReplicaSet kept at 0 as a revision.

The exact ordering depends on how fast new Pods pass readiness, but the invariants are hard: available never drops below 8, total never exceeds 13. Now break it: set maxUnavailable = 60% → 6. The floor becomes 10 − 6 = 4. If your service needs 7 Pods to hold peak traffic under its SLO, the roll is now allowed to spend a window at 4 available — below the line — and users see errors or latency even though the deploy "succeeded." That dip is invisible in the YAML; the widget below makes it visible.

Rolling-update simulator — watch the available-capacity dip
Set replicas, maxSurge, and maxUnavailable, then press deploy v2 to step the rollout. Grey = old v1 (Ready), amber = new v2 booting (not yet available), green = new v2 Ready, dashed = a slot being recreated. The min available KPI is the worst capacity reached during the whole roll. The dashed red line is an example SLO (60% of replicas): push maxUnavailable up and you will drive min-available below it.
idle — all v1, available = replicas
surge / unavail (pods)
total cap = N + surge
floor = N − unavail
min available reached

5 · History, rollback, readiness, and how traffic still finds Pods

Because the Deployment keeps each prior ReplicaSet scaled to 0, the cluster holds a revision history (default 10 revisions, set by revisionHistoryLimit). Rolling back is therefore not a redeploy — it is re-running the same scale-transfer in reverse:

kubectl rollout history deploy/web → lists revisions 1, 2, 3… each tied to a retained ReplicaSet.
kubectl rollout undo deploy/web → scales the previous revision's ReplicaSet back up and the current one down, under the same RollingUpdate strategy. No image rebuild, no redeploy — the v1 Pod template was never lost.
kubectl rollout status deploy/web → blocks until the new revision reports all replicas updated and available, so CI can gate on a finished, healthy roll.

What stops the controller from marching forward into a broken v2? Readiness. A new Pod counts as "available" only once it passes its readiness probe — a health check that says "this Pod can serve traffic" (covered fully in lesson 11). Until a fresh v2 Pod is Ready, it does not count toward the available total, so the floor (N − maxUnavailable) holds the line: the controller will not delete more old Pods until enough new ones are genuinely serving. If v2 never becomes Ready, the roll stalls at the floor instead of completing — your service stays up on v1 Pods while you investigate, and rollout undo reverses it. (Without good readiness probes this safety net is gone: Kubernetes counts a booting-but-broken Pod as available and rolls right over your last good replicas.)

One more thread from lesson 04 closes here. Throughout the roll, old and new Pods all carry the Deployment's selector labels (e.g. app=web) — the pod-template-hash only distinguishes ReplicaSets, not the app-level selector. So anything that finds Pods by that selector automatically tracks whichever Pods currently exist, v1 or v2, without being told. That label-query indirection is precisely how a stable front end (a Service) will keep routing to a moving set — which is the next lesson.

Failure modes & checklist

Failure modes

  • maxUnavailable too high. A roll allowed to drop well below the capacity your traffic needs. Signal: error rate / latency spikes during every deploy that vanish once it finishes; the Deployment still reports success.
  • maxSurge = 0 and maxUnavailable = 0. The controller can neither add nor remove a Pod — the roll deadlocks forever. Signal: rollout status hangs; both knobs cannot be zero at once.
  • No (or fake) readiness probe. Booting v2 Pods are counted as available, so the floor is meaningless and the roll steamrolls your last good v1 Pods. Signal: brief total outage mid-roll even though "min available" looked safe on paper.
  • Editing the ReplicaSet, not the Deployment. Changes get reverted by the Deployment controller (it owns the set) or create an orphan. Signal: your manual edits silently disappear within seconds.
  • Recreate on a multi-replica user-facing service. Full outage window because capacity hits 0. Signal: every deploy shows a hard gap in availability, not a dip.

Checklist

  • Change the Deployment template, never a ReplicaSet directly — let the higher controller own the roll.
  • Set maxUnavailable from your real capacity floor: N − maxUnavailable ≥ peak-load replica need.
  • Set maxSurge from headroom you can afford to pay for transiently (and that the cluster can schedule).
  • Ship a readiness probe that returns true only when the Pod can actually serve — it is what gates the roll (lesson 11).
  • Use kubectl rollout status in CI to gate on a finished, healthy roll; keep revisionHistoryLimit high enough to roll back.
  • Choose Recreate only when two versions genuinely cannot coexist, and accept the outage window.

Checkpoint exercise

Try it
In the widget set replicas = 12, maxSurge = 25%, maxUnavailable = 25%. By hand, compute the surge count (round up), the unavailable count (round down), the total ceiling, and the available floor. Press deploy v2 and step the roll — does the observed min available match your floor? Now raise maxUnavailable to 50% and redeploy: how far does min-available fall, and does it cross the SLO line? Finally, set both maxSurge and maxUnavailable to 0 and explain in one sentence why the roll can never make progress.

Where this points next

A Deployment now lets you change desired state safely: new ReplicaSet per version, bounded capacity loss, one-command rollback. But notice what just happened to every Pod. A rolling update deletes and recreates Pods — and from lesson 02 we know a fresh Pod gets a fresh cluster IP. So after a single roll, none of the IPs a client might have cached point at a live Pod; the entire backend set has moved. Worse, this happens on every scale event and every restart, not just deploys. A stable selector lets a controller find the current Pods, but a human or a remote client still has nothing fixed to dial. That is the problem lesson 06, Networking & Services, solves: a stable virtual IP and name that load-balances across whatever Pods the selector matches right now, no matter how often they churn.

Takeaway
A ReplicaSet pins one Pod template, so a version change is a destroy-and-replace with no undo. The Deployment is the fix: a controller of ReplicaSets that, on every template change, creates a new ReplicaSet (keyed by a pod-template-hash) and runs the lesson-00 loop to transfer replicas old→new under a strategyRecreate (drop to 0, then refill) or the default RollingUpdate, bounded by maxSurge (total ≤ N + maxSurge) and maxUnavailable (available ≥ N − maxUnavailable). For replicas=10 at 25%/25% the roll never exceeds 13 Pods nor drops below 8 available; push maxUnavailable too high and availability dips below your SLO even on a "successful" deploy. Old ReplicaSets are kept scaled to 0 as revision history, so rollout undo reverses the same transfer with no rebuild; readiness gates each forward step so a broken v2 stalls instead of steamrolling v1; and the shared selector labels let traffic-finding controllers track whichever Pods exist. The catch it leaves: rolled Pods get new IPs every time — which forces a stable Service.

Interview prompts