Part IV — State and config

Beyond the Stateless Server: StatefulSets, Jobs, DaemonSets

Lesson 08 gave a Pod two things it never had: configuration injected from outside the image (ConfigMaps and Secrets), and durable storage that outlives the container (a PersistentVolumeClaim binding to a real disk a StorageClass provisions). That made a single Pod stateful. But the controller we use to run Pods at scale — the Deployment — treats its Pods as interchangeable cattle: random names, one shared PVC if you wire one in, all replaced in parallel with no order. That is exactly the wrong shape for a clustered database, and it is the wrong shape for work that should finish or run once per node. This lesson derives the three workload controllers that fix each mismatch: the StatefulSet, the Job/CronJob, and the DaemonSet.

The previous step left this broken

A PersistentVolumeClaim hands a Pod a durable disk — but a Deployment manages a ReplicaSet of fungible Pods (lesson 04). They get hashed names like web-7d9f-x2k4, every replica mounts the same PVC if you attach one (and most disks are ReadWriteOnce — one writer — so the second Pod can't even mount it), and a rollout replaces all of them in any order, all at once. Try to run a 3-node database that way and it breaks three ways: replica db-1 can't find a stable db-0 to follow because names churn, two replicas fight over one disk, and an unordered restart can take down the primary and its only follower simultaneously. And a Deployment assumes its process never exits — so a one-shot migration or a per-node log agent has no controller that fits at all.

Linear position

Forced by: a Deployment's Pods are identity-less, share storage, and roll in parallel — fatal for a clustered database, and meaningless for work that should finish or run per-node.
New capability: three purpose-built controllers — the StatefulSet (stable ordinal identity, a per-Pod PVC via volumeClaimTemplate, ordered graceful rollout, headless-Service DNS), the Job/CronJob (run-to-completion and on-a-schedule), and the DaemonSet (exactly one Pod per node).
Forces next: every one of these controllers just declares "run this Pod" and assumes it will land on a node — but which node, and what stops a node from being overcommitted?

The plan

Five moves. (1) Pin down exactly why the Deployment's cattle model breaks for stateful, finite, and per-node work. (2) Derive the StatefulSet from those needs — identity, per-Pod disk, order, stable DNS — and work the db-0 → db-1 → db-2 bring-up. (3) Derive the Job and CronJob for run-to-completion and scheduled work. (4) Derive the DaemonSet for one-Pod-per-node agents, and say when a raw StatefulSet should become an operator. (5) Drive the widget — step the ordered rollout and contrast it with a Deployment's parallel roll — then failure modes, a checkpoint, and the hand-off to the scheduler.

1 · Three workloads the Deployment cannot serve

The Deployment is a magnificent fit for one shape of work: a stateless server — a process where any replica can answer any request, holds nothing important locally, and can be replaced by a fresh copy at any instant. Lesson 05's whole safety story (roll v2 in, surge spare capacity, kill old Pods in any order) depends on that fungibility. The trouble is that three common workloads violate it, each differently.

Clustered stateful app

A database, message broker, or consensus store where replicas are not interchangeable: db-0 is the primary, db-1 and db-2 are followers that must find db-0 by a stable name, each owns its own on-disk data, and they must come up and go down in a known order.

Work that should finish

A database migration, a batch ETL pass, a model-eval run. It executes, exits 0, and is done. A Deployment would treat that clean exit as a crash and restart it forever.

One agent per node

A log shipper, metrics agent, or CNI/storage plugin that must run on every node — exactly one copy, automatically following nodes as they join and leave. "Replicas: 50" doesn't track the node set.

Each mismatch is structural, not a tuning knob. A Deployment gives Pods names by hashing the template (db-7d9f-x2k4), so the name a follower would dial changes on every reschedule. It points all replicas at one Pod template — including one PVC reference — but a typical block disk is ReadWriteOnce (mountable read-write by exactly one node at a time), so a second replica trying to mount it stays stuck. And it has no concept of "this work completed" or "one per machine." We need three controllers, each the forced answer to one mismatch.

2 · The StatefulSet: identity, per-Pod disk, order

A StatefulSet manages a set of Pods that are not interchangeable. It changes four things relative to a Deployment, and each maps directly to a need from §1.

Property	Deployment (cattle)	StatefulSet (pets)
Pod names	Random hash suffix (web-7d9f-x2k4); a new name on every replace.	Stable ordinal: db-0, db-1, db-2. If db-1 dies it is recreated as db-1 — the name survives reschedule.
Storage	All replicas share the one PVC in the template (or none).	A volumeClaimTemplate mints a separate PVC per Pod — data-db-0, data-db-1 — and reattaches the same PVC to db-0 whenever it reschedules.
Rollout order	Parallel, any order, bounded only by surge/unavailable.	Ordered: create db-0, wait until it is Ready, then db-1, then db-2. Scale-down and rolling update go in reverse ordinal.
Network identity	Reached only via the Service VIP (lesson 06) — you can't address one Pod.	A headless Service gives each Pod a stable DNS name: db-0.dbsvc.ns.svc.cluster.local, resolving straight to that Pod's IP.

The volumeClaimTemplate is the heart of it. Instead of one PVC reference shared by all (the Deployment's broken model from §1), the StatefulSet stamps the template once per ordinal: db-0 gets PVC data-db-0, db-1 gets data-db-1, and so on, each binding to its own PersistentVolume (lesson 08's PVC→PV machinery, run N times). Crucially these PVCs are sticky: delete the db-1 Pod and the StatefulSet recreates a Pod named db-1 and reattaches PVC data-db-1 — same identity, same data, possibly on a different node. The PVC deliberately outlives the Pod, so scaling down does not delete the disk (your data is safe; you clean PVCs up by hand).

The headless Service (clusterIP: None, from lesson 06 §4) is what makes the names dialable. A normal Service hands out one VIP and load-balances — useless when a follower must reach the primary specifically. A headless Service publishes no VIP; instead DNS returns one A record per Pod, named by ordinal. So db-1's config can hardcode db-0.dbsvc as its primary and that name keeps resolving to db-0 across reschedules — the stable handle the cattle model never offered.

Ordered, graceful bring-up is the third piece, and the numbers matter. Suppose each replica takes ~20 s to become Ready (load data, join the quorum). A StatefulSet of 3 takes ~60 s to fully roll because it is strictly sequential — db-0 Ready before db-1 even starts. That is slower than a Deployment's parallel roll on purpose: it guarantees a follower never starts before the primary it depends on exists, and that scale-down (reverse order: remove db-2, then db-1) peels followers off before the primary, never the reverse.

→scale up to 3 → create db-0, bind PVC data-db-0, wait until db-0 reports Ready

→then → create db-1, bind data-db-1, wait for Ready (db-1 can already reach db-0 by DNS)

→then → create db-2, bind data-db-2, wait for Ready — set complete

→scale down to 1 → delete db-2 first, then db-1 (reverse ordinal); PVCs data-db-1/2 stay behind

3 · Jobs and CronJobs: work that finishes

A Deployment's reconcile loop has one fixed point: "N Pods Running, forever." Feed it a process that exits 0 on success and the loop sees a missing Pod and restarts it — turning a finished migration into an infinite loop. The forced answer is a controller whose definition of "done" is completion, not running: the Job.

A Job runs Pods until a target number succeed (exit code 0), then stops. Its three knobs:

completions

How many successful Pod runs the Job needs before it is Complete. completions: 1 for a single migration; completions: 100 for 100 work items.

parallelism

How many Pods may run at once. parallelism: 10 with completions: 100 means 10 in flight at a time until all 100 succeed — a worked example: ~10 waves.

backoffLimit

How many failed Pods (non-zero exit) to retry before the Job is marked Failed and gives up — so a genuinely broken task doesn't retry forever. Default 6.

The contrast with a Deployment is exact. A Deployment Pod that exits 0 is a problem to be fixed (restart it); a Job Pod that exits 0 is the goal (mark it Done, count it toward completions). This is why a Job's Pods default to restartPolicy: Never or OnFailure — never the Deployment's Always. Run-to-completion is a different control loop, not a flag on the old one.

A CronJob wraps a Job in a schedule: it holds a cron expression and creates a fresh Job each time it fires. "0 2 * * *" runs a backup Job at 02:00 daily; "*/15 * * * *" every 15 minutes. Concurrency policy decides what happens if a run is still going when the next is due (Allow, Forbid, or Replace), and history limits cap how many finished Jobs it keeps around for inspection. CronJob is to Job what a cron daemon is to a one-off command — scheduled, repeatable, finite work.

4 · DaemonSets, and when to reach for an operator

The last mismatch: some Pods are properties of the node, not of a desired replica count. A log shipper must read every node's logs; a metrics agent must scrape every node's kubelet; a CNI or CSI plugin must run wherever Pods or volumes live. Set "replicas: 50" and you've hardcoded today's node count — add a node and it has no agent; drain one and you have a stray. The DaemonSet changes the reconcile target from "N copies" to "exactly one Pod per node," and it tracks the node set automatically.

→node joins the cluster → the DaemonSet controller sees a node with no matching Pod → schedules one there

→node is drained / removed → its DaemonSet Pod goes with it → desired count auto-shrinks, no stragglers

→nodeSelector / tolerations → restrict to a subset (e.g. only GPU nodes) and still tolerate control-plane taints so even tainted nodes get the agent

DaemonSets often must run on nodes that ordinary workloads avoid — including tainted control-plane nodes — so they carry tolerations (the placement levers are lesson 10's subject). The point here is the controller's target: it reconciles against the live node inventory, not a number you typed.

That gives us four controllers — Deployment, StatefulSet, Job/CronJob, DaemonSet — covering stateless servers, clustered stateful apps, finite work, and per-node agents. The decision is a short table:

If the work is…	Use	Because
A stateless server, any replica answers anything	Deployment	Fungible Pods, fast parallel rollout, one-command rollback (lesson 05).
A clustered stateful app (DB, broker, quorum store)	StatefulSet	Stable identity, per-Pod disk, ordered bring-up, headless DNS.
A task that runs once / a batch / on a schedule	Job / CronJob	Run-to-completion semantics; exit 0 means Done, not crashed.
One agent on every node	DaemonSet	Reconciles against the node set, auto-adjusts as nodes come and go.

One honest limit. A StatefulSet gives a database stable identity, disk, and order — but it does not understand your database. It cannot promote a new primary when db-0 dies, take consistent backups, run a schema migration before a version bump, or rebuild a replica from a snapshot. Those are domain operations a human used to do from a runbook. When you need them, you reach past the raw StatefulSet for an operator — a custom controller that wraps a StatefulSet and adds failover, backups, and upgrades as automated reconcile logic. That is exactly the manual-ops problem lesson 00 set out to abolish, applied to your domain object; we build operators in lesson 13.

5 · Widget: ordered StatefulSet vs. parallel Deployment

StatefulSet ordered rollout — identity, per-Pod PVC, order

Pick a target size, then press Step to advance the rollout one action at a time. In StatefulSet mode, db-0 must reach Ready before db-1 is even created, and each Pod mints its own sticky PVC (pvc-db-0 …). Flip to Deployment mode to watch the same scale-up happen all at once: random names, one shared PVC, no order. Then scale down and note the StatefulSet removes Pods in reverse ordinal (db-2 first), keeping each disk.

target replicas: 3

Idle — target 3, current 0. Press Step to create the first Pod.

Mode

StatefulSet

Ready / target

0 / 3

Distinct PVCs

Next action

create db-0

Failure modes & checklist

Failure modes

Running a database as a Deployment. All replicas point at one ReadWriteOnce PVC. Signal: only one Pod ever runs; the rest sit in ContainerCreating with a "Multi-Attach error — volume already used by Pod X" event.
No headless Service for the StatefulSet. The Pods exist but db-1 can't resolve db-0.dbsvc. Signal: followers crash-loop on "host not found / connection refused" to the primary, even though all Pods show Running.
Expecting parallel speed from a StatefulSet. A 10-replica set rolls one Pod at a time and a stuck db-3 blocks everything after it. Signal: rollout "hangs" at one ordinal; the rest are Pending because the prior never went Ready.
A Job that never completes. The task exits 0 but you set restartPolicy: Always (or used a Deployment), so it restarts forever; or it fails silently up to backoffLimit and the failure is missed. Signal: a "batch job" Pod that has restarted hundreds of times, or a CronJob whose Jobs all show Failed.
Deleting a StatefulSet to "reset" it. The PVCs survive on purpose, so recreating it reattaches old data; conversely, assuming scale-down freed the disk. Signal: stale data after a "fresh" deploy, or orphaned PVCs quietly accruing storage cost.
DaemonSet missing tainted nodes. A node agent isn't running on control-plane or GPU nodes because it lacks the matching toleration. Signal: a coverage gap — logs/metrics absent from exactly the nodes that carry a taint.

Checklist

Match the controller to the work using §4's table before writing any YAML — stateless→Deployment, clustered-stateful→StatefulSet, finite→Job/CronJob, per-node→DaemonSet.
For a StatefulSet, always pair a headless Service (clusterIP: None) and use a volumeClaimTemplate, never a shared PVC.
Set a real readiness probe (lesson 11) — ordered rollout depends on "Ready" meaning the replica can actually serve, or the whole sequence stalls or races.
For a Job, set restartPolicy: Never/OnFailure, a sane backoffLimit, and (for big batches) parallelism < completions.
Treat StatefulSet PVCs as durable — clean them up explicitly; never assume scale-down deletes data.
For deep stateful ops (failover, backups), use an operator (lesson 13), not raw StatefulSet plus runbooks.

Checkpoint exercise

Try it

In the widget, set target to 3 in StatefulSet mode and press Step repeatedly. Count how many Steps it takes for all three to be Ready, and write down which PVC each Pod mounted. Now press Reset, switch to Deployment mode, and scale to 3 again — note that a single Step creates all three at once (no waiting for a prior to be Ready), the names are random, and the "Distinct PVCs" KPI stays at 1 (shared). Back in StatefulSet mode at 3 Ready, lower the target to 1 and Step: confirm db-2 is removed before db-1 (reverse order) and that the PVC count does not drop. Finally, in one sentence: explain why a 3-node database needs every difference you just watched — stable names, per-Pod disk, and order — and what specifically breaks if you run it as a Deployment instead.

Where this points next

We now have a controller for every shape of work, and each one ends its job at the same line: it declares "run this Pod" and hands it to the cluster. The StatefulSet says "create db-1"; the Job says "run a worker"; the DaemonSet says "put one agent on that new node." Every one of them assumes the Pod will simply land on a node and run. But a freshly created Pod is unbound — the API server holds it, no node has claimed it. Which of the cluster's machines should run it? Random ignores whether the Pod even fits the node's free CPU and memory, whether a node is reserved by a taint, and whether spreading replicas across failure domains matters. And what stops ten Pods from all landing on one node and overcommitting it into thrash? Lesson 10, The Scheduler: How a Pod Finds a Node, opens the loop that turns "run this Pod" into "run it here" — filtering nodes that fit, scoring the survivors, and binding the Pod, using requests, affinity, taints, and topology spread.

Takeaway

A Deployment's Pods are interchangeable cattle — hashed names, a shared ReadWriteOnce PVC, parallel unordered rollout — which is exactly wrong for a clustered database and meaningless for work that should finish or run per-node. Three controllers are the forced answers. The StatefulSet gives stable ordinal identity (db-0/1/2, names that survive reschedule), a volumeClaimTemplate minting a sticky per-Pod PVC (so each replica owns its disk, and the PVC outlives the Pod), ordered graceful rollout (db-0 Ready before db-1 starts; scale-down in reverse), and a headless Service for per-Pod DNS — the four things a DB, broker, or quorum store needs. The Job redefines "done" as completion (completions, parallelism, backoffLimit; exit 0 = Done, not crashed), and the CronJob schedules Jobs on a cron. The DaemonSet reconciles against the node set — exactly one Pod per node — for log/metrics/CNI/CSI agents. For domain operations a StatefulSet can't do (failover, backups, upgrades), wrap it in an operator (lesson 13). Every one of these still just declares "run this Pod" and assumes it lands somewhere — which is the scheduler's job, next.

Interview prompts

Why can't you run a 3-node database as a Deployment? (§1, §2 — Deployment Pods get random hashed names (no stable handle for a follower to dial), share one ReadWriteOnce PVC (a second replica can't mount it), and roll in parallel/any order (an unordered restart can take down primary and follower together). A clustered DB needs stable identity, a per-Pod disk, and ordered bring-up.)
What does a volumeClaimTemplate do that a Deployment's PVC reference can't? (§2 — it stamps a separate PVC per ordinal (data-db-0, data-db-1, …), each binding its own PV, and reattaches the same sticky PVC when a Pod reschedules; the PVC outlives the Pod, so each replica owns durable data and scale-down doesn't delete it.)
Why does a StatefulSet need a headless Service? (§2 — a normal Service is a load-balancing VIP, useless when a follower must reach the primary specifically; a headless Service (clusterIP: None) publishes one DNS A record per Pod (db-0.dbsvc…) that resolves straight to that Pod across reschedules — a stable, dialable per-Pod name.)
Describe StatefulSet rollout order and why it's deliberately slow. (§2 — scale-up is strictly sequential: db-0 must be Ready before db-1 is created; rolling update and scale-down go in reverse ordinal. 3 replicas at ~20 s each ≈ 60 s. It guarantees a follower never starts before its primary and peels followers off before the primary.)
How is a Job's control loop different from a Deployment's? (§3 — a Deployment keeps N Pods Running forever and treats exit 0 as a crash to restart; a Job runs Pods until completions succeed then stops, governed by parallelism and backoffLimit, with restartPolicy Never/OnFailure — exit 0 is the goal, not a failure.)
What is a DaemonSet's reconcile target, and how does it differ from "replicas: N"? (§4 — exactly one Pod per node, reconciled against the live node inventory; it schedules a Pod when a node joins and removes it when a node leaves, auto-tracking the node set instead of a fixed count — for log/metrics/CNI/CSI agents, often with tolerations for tainted nodes.)
When do you reach past a raw StatefulSet for an operator? (§4 — a StatefulSet gives identity, per-Pod disk, and order but doesn't understand your domain: it can't promote a new primary on failure, take consistent backups, or run pre-upgrade migrations. An operator is a custom controller that wraps the StatefulSet and automates those runbook operations — lesson 13.)