Part V — How the cluster decides and defends

Health & Autoscaling: Closing the Loop on Reality

Lesson 10 gave the scheduler a way to place a Pod well — filter the nodes it fits, score the survivors, bind it to the best one. But notice what the scheduler and the ReplicaSet actually know about a Pod once it lands: only that its process is running. Running is not the same as healthy. A process can be deadlocked, still loading a 40 GB model, or wedged because it lost its database connection — alive by the OS's measure, useless by yours. And the replica count the scheduler so carefully placed is itself a guess: three replicas is wasteful at 3 a.m. and far too few at the noon peak. This lesson closes both gaps the same way the whole track has closed every gap — with more control loops. Probes give the loop a truth signal beyond "process alive," and autoscaling makes the replica count, and even the node count, things that get reconciled too.

The previous step left this broken

The scheduler binds a Pod to a node and the kubelet starts its containers; the ReplicaSet (lesson 04) counts how many Pods match its selector. But both reason about liveness in the crudest sense — is the process still up? They cannot tell that a "running" Pod is deadlocked, mid-boot, or has lost its dependency, so they will happily route traffic to it and never restart it. And the replica count itself is a fixed number you typed once: it ignores that load is not constant. So we are simultaneously serving from sick Pods and paying for idle ones at night while dropping requests at peak. We need the loop to observe health, not just existence — and to drive the count, not just hold it.

Linear position

Forced by: the scheduler/ReplicaSet only know a process is running, not whether it is healthy or whether the fixed replica count matches current load.
New capability: probes (liveness, readiness, startup) give the loop a real health signal — restart the dead, pull the not-ready from Service rotation, grant grace to slow boots; and autoscaling (HPA on a metric vs target, VPA on requests, Cluster Autoscaler/Karpenter on nodes) makes the replica count and node count reconciled — three nested control loops.
Forces next: more replicas, more teams, more reach — yet any Pod can still read any Secret and call any API; there are no boundaries.

The plan

Six moves. (1) Show why "running" is the wrong truth signal. (2) Derive the three probes — liveness, readiness, startup — and the sharp liveness-vs-readiness distinction. (3) Add restartPolicy, CrashLoopBackOff, and the PodDisruptionBudget that caps voluntary disruption. (4) Derive autoscaling: the HPA reads a metric vs a target and computes a new replica count with a worked formula. (5) Add the stabilization window and tolerance that stop thrash, then VPA and the Cluster Autoscaler as the other two nested loops. (6) Drive the widget — watch replicas track load smoothly, then crank the aggressiveness to make them flap — then failure modes, checklist, and the hand-off to security.

1 · "Running" is the wrong truth signal

Recall the engine of the whole track: observe actual state, diff it against desired, act to close the gap, repeat. Every loop is only as good as what it can observe. The ReplicaSet observes "how many Pods match my selector exist and are running," and that is a far cruder signal than it looks. The kubelet starts a container; if the process is up, the Pod is Running. But a great many ways to be broken leave the process up:

Deadlocked

An event loop is wedged on a lock or an infinite loop. The process exists, accepts no work, and will never recover on its own. The OS sees a healthy process.

Still warming up

A server bound its port in 50 ms but is still loading weights, filling a cache, or running migrations for two minutes. "Running" since second one; unable to serve until minute two.

Lost a dependency

The app is fine but its database connection dropped, or a downstream is unreachable. It cannot serve correct responses right now, but the binary is perfectly alive.

Degraded, not dead

A memory leak has it thrashing, latency is 50× normal, but it has not crashed. By a process-up check it is fine; by any user's measure it is broken.

Two of these want different reactions, and conflating them is the single most common mistake in this whole lesson. A deadlocked Pod should be killed and restarted — a fresh process is the cure. A Pod that lost its database should not be killed — restarting it does nothing (the DB is still down), and worse, if the DB blip hit every replica at once you would restart your entire fleet into an outage. It should instead just stop receiving traffic until it recovers. So we need not one health signal but (at least) two, asking different questions: "is this process beyond saving — restart it?" and "is this Pod able to serve right now — send it traffic or not?"

2 · The three probes: liveness, readiness, startup

A probe is a periodic check the kubelet runs against a container — an HTTP GET on a path, a TCP connect, or a command exec that must exit 0. Each probe has a period (e.g. every 10 s), a timeout, a success threshold, and a failure threshold (e.g. 3 consecutive failures). Kubernetes defines three probes that ask three different questions, with three different consequences:

Probe	Question it asks	On failure	The crucial point
Liveness	Is this process beyond saving?	The kubelet restarts the container (per restartPolicy).	For unrecoverable states only — deadlock, wedged loop. Restart is the cure.
Readiness	Can this Pod serve traffic right now?	The Pod's IP is pulled from its Service's EndpointSlice — no restart, no kill.	For transient/dependency states. Traffic stops; the Pod lives and can rejoin.
Startup	Has the app finished booting yet?	Nothing destructive — it just holds off liveness/readiness until it passes.	A grace period so a slow boot is not mistaken for a deadlock.

Readiness is the direct payoff of lesson 06. There you saw the Service is a stable virtual IP over the live, label-selected set of Pod IPs recorded in the EndpointSlice, and that a Pod only joins that set once it is Ready. Readiness is exactly that gate. When a readiness probe fails, the EndpointSlice controller removes the Pod's IP, kube-proxy stops routing to it on every node, and traffic drains away — but the Pod is not killed. It keeps running, keeps being probed, and the instant readiness passes again its IP is added back. That is the right behavior for a lost dependency: quarantine, do not euthanize.

The startup probe exists to resolve a real conflict. Suppose a model server takes 90 s to load weights, and you also want a liveness probe that fails after 30 s of unresponsiveness to catch deadlocks. Without a startup probe, liveness fires at 30 s during the legitimate boot, the kubelet kills the container, it boots for 30 s again, dies again — a boot loop you created yourself. The startup probe says "until I pass, ignore liveness and readiness entirely," with its own generous budget (e.g. 30 failures × 10 s = 5 minutes of grace). Once it succeeds, the fast liveness/readiness probes take over. Slow to boot, then strict — exactly what you want.

→startup probe → from container start, gets a long grace budget; liveness & readiness are suppressed until it passes once.

→readiness probe → now runs every period; pass → IP in EndpointSlice (gets traffic); fail → IP removed (no traffic), Pod stays alive.

→liveness probe → also runs every period; fail past threshold → kubelet restarts the container in place.

The classic mistake

Putting a dependency check (e.g. "can I reach the database?") in the liveness probe. When the database has a hiccup, every replica's liveness fails at once, the kubelet restarts the entire fleet, and the restarts hammer the recovering database — you have converted a brief blip into a self-inflicted outage. Dependency checks belong in readiness (stop traffic, stay alive); liveness is only for "this specific process is wedged and a restart of it alone will fix it."

3 · restartPolicy, CrashLoopBackOff, and PodDisruptionBudget

When a liveness probe fails — or a container simply exits — what the kubelet does next is governed by the Pod's restartPolicy: Always (the default for long-running servers; restart on any exit), OnFailure (restart only on a non-zero exit — used by Jobs from lesson 09), or Never. Crucially the restart is in place on the same node — the kubelet re-runs the container; it does not reschedule the Pod or change its IP.

But a container that crashes immediately on start would, under Always, restart in a tight loop and burn the node. So the kubelet applies exponential backoff: after each crash it waits longer before the next restart — roughly 10 s, 20 s, 40 s, … doubling up to a 5 -minute cap. A Pod stuck in this cycle shows the status CrashLoopBackOff — one of the most common things you will ever see. It does not mean Kubernetes gave up; it means "this keeps dying, so I am restarting it more slowly." The backoff is the diagnostic: read the previous container's logs (kubectl logs --previous), because the loop is telling you the process cannot stay up.

There is a second kind of "going away" that is not a crash at all: voluntary disruption — you draining a node for maintenance, the Cluster Autoscaler removing an under-used node, a rolling update evicting Pods. These are deliberate, and Kubernetes will gladly evict Pods to do them — which can take down too many replicas at once if it is greedy. A PodDisruptionBudget (PDB) caps that: it declares, for a label-selected set, either minAvailable: 4 or maxUnavailable: 1. When something tries to voluntarily evict a Pod, the eviction API checks the PDB and refuses if it would breach the budget — so a node drain that would drop you below your floor simply blocks until enough replacement Pods are Ready elsewhere. (A PDB only governs voluntary disruptions; a node crashing is involuntary and no budget can stop physics.)

4 · Autoscaling: make the replica count reconciled

Probes fixed health. The other half of the gap is the count. A fixed replicas: 3 is the same anti-pattern the whole track exists to kill — a static number a human typed, pretending the world is constant. The fix is the same move once more: make the desired count itself the output of a control loop. That is the Horizontal Pod Autoscaler (HPA). It is a controller that, every ~15 s, reads a metric for the Pods, compares it to a target you declared, and adjusts the Deployment's replica count to bring the metric back to target. The arithmetic is one line:

  desiredReplicas = ceil( currentReplicas × ( currentMetric / targetMetric ) )

Work it. You run a Deployment at currentReplicas = 4, target CPU utilization targetMetric = 50% (utilization = actual usage ÷ the request from lesson 10 — so the request you set is the denominator the HPA scales against). A traffic spike pushes average CPU to currentMetric = 90%. Then:

  desiredReplicas = ceil( 4 × (90 / 50) ) = ceil( 4 × 1.8 ) = ceil(7.2) = 8

The HPA scales 4 → 8. With twice the Pods sharing the same load, per-Pod CPU should fall back toward 90 / 2 = 45% — just under target, and the loop settles. When the spike passes and CPU drops to 20%, the same formula gives ceil(8 × 20/50) = ceil(3.2) = 4, and it scales back down. The metric need not be CPU: the HPA also reads memory, custom metrics (requests-per-second, queue depth) via the custom-metrics API, or external metrics (a cloud queue's length). For the GenAI workloads in the sibling track, CPU is the wrong signal entirely — you scale on tokens-in-flight or KV-cache utilization, exactly the lesson the GenAI-on-Kubernetes track builds on top of this one.

5 · Stabilization, then the other two nested loops

Run that formula naively and you get a pathology: thrash (flapping). Load wobbles a little, the metric crosses target, the HPA scales up; the extra Pods drop the metric below target, it scales down; the load wobble pushes it over again — replicas oscillate every cycle, each change churning Pods, evicting work, and re-warming caches for no benefit. Two dampers prevent it. A tolerance (default 10%) means the HPA ignores a metric within 10% of target — no action for trivial deviations. And a stabilization window (default 300 s for scale-down, 0 for scale-up) makes the HPA, when deciding to shrink, look back over the window and pick the highest recommendation in it — so it scales up fast (responsive to spikes) but scales down slowly (immune to brief dips). Aggressive tuning — a short or zero stabilization window, a tiny tolerance — buys faster reaction at the cost of flapping. That trade-off is exactly what the widget below lets you feel.

The HPA reconciles the replica count, but it sits between two other loops, giving three nested control loops, each closing the gap the inner one cannot:

→Pod health (probes + kubelet) → keeps each individual Pod healthy: restart the dead, drain traffic from the not-ready. Inner loop, per Pod.

→Replica count (HPA) → keeps the number of Pods matched to load via the metric-vs-target formula. Middle loop, per workload.

→Node count (Cluster Autoscaler / Karpenter) → keeps the number of nodes matched to demand: when the HPA asks for Pods that cannot be scheduled, add nodes. Outer loop, per cluster.

That outer loop ties straight back to lesson 10. When the HPA raises replicas but the scheduler can find no node with room, those Pods sit Pending. The Cluster Autoscaler watches for exactly that — unschedulable Pods — and provisions new nodes from a node group; when nodes later sit under-used, it drains them (subject to your PDBs from §3) and removes them. Karpenter is a newer, faster variant that provisions right-sized nodes directly rather than scaling fixed groups. Note the HPA and the Cluster Autoscaler are complementary: the HPA changes Pods, the Cluster Autoscaler changes the nodes those Pods need.

One more autoscaler, on a different axis. The Vertical Pod Autoscaler (VPA) does not change the count — it right-sizes each Pod's resources.requests (the value the scheduler packs against and the HPA divides by), recommending, say, that a Pod requesting 2 CPU only ever uses 0.4 and should request less. The catch worth knowing for interviews: the VPA and HPA conflict if pointed at the same metric. If the HPA scales on CPU and the VPA also adjusts CPU requests, the VPA keeps moving the very denominator the HPA's utilization is computed against, and they fight. Run them on different signals (e.g. HPA on a custom RPS metric, VPA on memory) or not together on the same resource.

6 · Widget: track load smoothly, or make it flap

HPA control loop — smooth tracking vs. thrash

The blue line is incoming load (use the slider to push it, or leave "auto curve" on for a day/night wave with noise). The green line is the HPA's replica count, recomputed each tick from ceil(current × util / target). Drag the target utilization to move where it settles. Then flip the aggressiveness knob from Stable (long stabilization window, 10% tolerance) to Aggressive (zero window, no tolerance) and watch the green line stop tracking and start flapping — replicas churning every tick. Same load, same formula; only the dampers changed.

—

manual load: 50 target util %: 50

Current load

—

Per-Pod util %

—

Desired → current replicas

—

Scale changes (last 30 ticks)

—

Failure modes & checklist

Failure modes

Dependency check in the liveness probe. A DB blip fails liveness on every replica and the kubelet restarts the whole fleet, hammering the recovering dependency. Signal: a downstream hiccup turns into mass restarts and a far longer outage than the original blip.
No startup probe on a slow-booting app. A liveness probe tuned for deadlocks fires during the legitimate 90 s model load, killing the container mid-boot, forever. Signal: a Pod that never reaches Ready, restart count climbing, logs cut off mid-initialization.
No readiness probe behind a Service. Pods join the EndpointSlice the instant they are running, before they can serve. Signal: a burst of 5xx right after every scale-up or rollout (the lesson-06 failure mode, here is its cause).
HPA scaling on CPU for an I/O- or memory-bound app. CPU stays flat while the real bottleneck saturates, so the HPA never fires. Signal: latency climbing and queues growing while replica count sits stubbornly still.
Thrash from an aggressive HPA. Tiny tolerance / short stabilization window makes replicas oscillate on noise. Signal: replica count sawtoothing in the dashboard, constant Pod churn, caches never staying warm.
HPA and VPA on the same resource. The VPA moves the request the HPA computes utilization against; they fight. Signal: replica count and request size both oscillating, neither converging.
No PodDisruptionBudget. A node drain or autoscaler scale-down evicts too many replicas at once. Signal: availability dips during routine maintenance even though "nothing crashed."

Checklist

Liveness = "restart fixes it" only (deadlock, wedge). Never put a dependency check here.
Readiness = "can serve right now", including dependency checks — failure drains traffic without killing the Pod.
Add a startup probe to any app whose boot is slower than your liveness threshold; give it a generous budget.
Set requests honestly — they are both the scheduler's packing unit (lesson 10) and the HPA's utilization denominator.
Scale the HPA on the metric that actually binds (RPS, queue depth, KV-cache util), not reflexively on CPU.
Keep the scale-down stabilization window long (default 300 s) so brief dips do not trigger flapping.
Never run HPA and VPA on the same resource; split them across different signals.
Define a PodDisruptionBudget for every Service-backing workload so drains and node scale-downs respect a floor.

Checkpoint exercise

Try it

Open the widget in Stable mode with target 50% and watch the green replica line track the blue load wave with only a few step changes. Now switch to Aggressive and watch the same wave produce a flapping replica count — read the "scale changes (last 30 ticks)" KPI climb. Then, by hand: a Deployment runs at 6 replicas, target CPU 60%, and current average CPU is 80%. Compute desiredReplicas with the §4 formula and check it against the widget's behavior. Finally, name which probe you would add to a server that takes 2 minutes to load weights but should be restarted if it deadlocks after boot — and explain in one sentence why putting its "is the database reachable?" check in the liveness probe instead of readiness would be dangerous.

Where this points next

We have closed the loop on reality: probes give the kubelet a real health signal (restart the dead via liveness, drain the not-ready via readiness, grace the slow via startup), CrashLoopBackOff and the PDB tame restart storms and greedy evictions, and three nested control loops — Pod health, replica count, node count — keep health, capacity, and infrastructure all reconciled instead of hand-set. But every loop we just built makes the cluster bigger: more replicas, more teams onboarding workloads, more reach. And we have been quietly assuming a friendly world. Right now any Pod can mount and read any Secret (which, recall from lesson 08, is only base64-encoded, not encrypted), call any verb on the powerful API server, and open a connection to any other Pod on the flat network from lesson 06. There are no boundaries at all. Lesson 12, Security & Multi-Tenancy: Who Can Do What, What Can Reach What, draws those boundaries: authentication, RBAC and ServiceAccounts on the API plane, admission control and Pod Security Standards, and NetworkPolicy turning the flat any-to-any network into default-deny segmentation.

Takeaway

The scheduler and ReplicaSet only know a process is running — and running is not healthy, while a fixed replica count is wrong as load moves. Both gaps are closed with more control loops. Probes give the kubelet a real truth signal: liveness failure restarts the container (for unrecoverable wedges — restart is the cure), readiness failure pulls the Pod's IP from its Service's EndpointSlice so traffic drains but the Pod lives (for transient/dependency states — the direct payoff of lesson 06), and a startup probe grants slow boots grace so liveness does not kill them mid-load. restartPolicy governs restarts; repeated crashes back off exponentially into CrashLoopBackOff; a PodDisruptionBudget caps voluntary disruptions during drains and rollouts. Then autoscaling reconciles the count itself: the HPA computes desiredReplicas = ceil(current × currentMetric / targetMetric), with a tolerance and a (long, for scale-down) stabilization window to prevent thrash; the VPA right-sizes requests (and conflicts with the HPA on the same metric); the Cluster Autoscaler / Karpenter adds nodes when Pods can't be scheduled (lesson 10's Pending). Three nested loops — Pod health, replica count, node count — each closing the gap the inner one can't. The classic mistake to never make: a dependency check belongs in readiness, never liveness.

Interview prompts

Liveness vs. readiness — what is each for and what happens on failure? (§2 — liveness asks "is this process beyond saving?"; failure → kubelet restarts the container. Readiness asks "can it serve right now?"; failure → Pod's IP pulled from the Service EndpointSlice, traffic drains, but the Pod is not killed. Liveness for unrecoverable wedges, readiness for transient/dependency states.)
Why is a dependency check in the liveness probe dangerous? (§2 — a DB blip fails liveness on every replica at once, the kubelet restarts the whole fleet, and the restarts hammer the recovering dependency, converting a brief blip into a self-inflicted outage. Dependency checks belong in readiness.)
What problem does the startup probe solve? (§2 — a liveness probe tuned to catch deadlocks would fire during a legitimate slow boot (e.g. 90 s weight load) and kill the container in a boot loop; the startup probe suppresses liveness/readiness with a generous budget until the app finishes booting, then the strict probes take over.)
What is CrashLoopBackOff and what causes the backoff? (§3 — a container repeatedly crashing on start under restartPolicy: Always; the kubelet applies exponential backoff (~10s, 20s, 40s … capped ~5 min) between restarts. The status is a diagnostic that the process can't stay up — read the previous container's logs.)
Walk the HPA formula with numbers. (§4 — desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric). At 4 replicas, target 50%, current 90%: ceil(4 × 1.8) = 8; per-Pod load then halves toward 45% and settles. Utilization = usage ÷ the request you set, so the request is the denominator.)
What causes HPA thrash and what prevents it? (§5 — naive recompute on noisy metrics oscillates the replica count. A tolerance (default 10%) ignores small deviations and a stabilization window (default 300 s for scale-down, 0 for scale-up) makes scale-down pick the highest recent recommendation — fast up, slow down.)
Describe the three nested autoscaling loops and how HPA/VPA can conflict. (§5 — Pod health (probes), replica count (HPA), node count (Cluster Autoscaler/Karpenter, which adds nodes when Pods are Pending). HPA and VPA conflict on the same metric because the VPA moves the request the HPA computes utilization against — split them across different signals.)
What does a PodDisruptionBudget protect against, and what can't it stop? (§3 — it caps voluntary disruptions (drains, rollouts, autoscaler scale-down) via minAvailable/maxUnavailable, blocking evictions that would breach the floor. It cannot stop involuntary disruption like a node crashing.)