Part IV — State and config

Config & Storage: State That Outlives the Pod

Lesson 07 finished the reachability story: CoreDNS gives every Service a name so workloads discover each other by svc.namespace.svc.cluster.local instead of an opaque ClusterIP, and one Ingress / Gateway puts a single L7, TLS-terminating front door over many Services. Pods can now find and route to one another. But each Pod is still born two ways crippled. It is config-blind — the same image must run in dev, stage, and prod with different settings, yet anything baked into the image means rebuilding to flip a flag. And it is storage-less — a container's filesystem is ephemeral, so when the Pod dies its data dies with it. This lesson injects configuration without rebuilding, and gives a Pod a disk that survives it.

The previous step left this broken

A Pod is reachable now, but it is still a blank slate at birth. Two distinct failures. (a) Config is welded to the image. Bake the database URL or a feature flag into the container and you must rebuild and re-push a new image to change a single value — which breaks build-once-run-anywhere (the 12-factor rule that one immutable artifact runs in every environment, configured from outside). (b) The filesystem is ephemeral. A container's writable layer lives and dies with the container; restart the Pod and every byte it wrote is gone. That is merely annoying for a stateless web server and fatal for anything that must remember — a database, an upload store, a model cache. We need config injected from outside the image, and storage that outlives any single Pod.

Linear position

Forced by: one image must run with per-environment config without a rebuild, and a Pod's filesystem is ephemeral so stateful data cannot survive a restart.
New capability: ConfigMap and Secret inject configuration as env vars or mounted files (mounted files can hot-update; env vars cannot); and the PersistentVolumeClaim → PersistentVolume split lets a Pod claim durable storage by spec while a StorageClass + CSI driver provisions the real disk — decoupling the app author from the infrastructure.
Forces next: a PVC gives a Pod a disk, but a Deployment's Pods are interchangeable and share nothing stable — a clustered database needs its own stable disk and identity per replica, and not every workload is a long-running server at all.

The plan

Six moves. (1) State the two things a freshly born Pod lacks. (2) Derive ConfigMap and Secret, and the env-var-vs-mounted-file choice (only files hot-update). (3) The honest truth about Secrets — base64 is encoding, not encryption — and what real protection takes. (4) Volumes, then the PVC → PV split, and why the claim indirection exists. (5) Access modes, reclaim policy, dynamic provisioning via StorageClass + CSI. (6) Drive the widget — watch a claim bind, dynamically provision, or stay Pending — then failure modes, checklist, and the hand-off to stateful workloads.

1 · What a freshly born Pod lacks

Recall the engine of this whole track (lesson 00): you declare a Pod's desired state and a loop reconciles reality to it. The Pod spec so far names an image and a command — a frozen, identical artifact. That immutability is a feature: the same bytes you tested in stage run in prod. But it collides with two facts about real systems.

First, the same artifact must behave differently per environment. The dev Pod talks to a dev database; the prod Pod talks to the prod database; a feature flag is on in stage and off in prod. If those values live inside the image, you need a different image per environment — and to flip one flag you rebuild, re-push, and re-roll. That destroys build-once-run-anywhere: configuration is environment, and environment must come from outside the artifact at runtime.

Second, the container filesystem is ephemeral. A container image is a stack of read-only layers; at runtime the runtime adds one thin writable layer on top (lesson 01). Everything the process writes lands in that layer, and that layer is destroyed when the container is destroyed — which a Pod restart, reschedule, eviction, or roll does routinely. For a stateless web server this is fine; it holds nothing precious. For a database, an upload directory, or a multi-gigabyte model cache it is a data-loss bug waiting for the next reschedule. We need a place to write that is decoupled from the container's lifetime.

Config problem

One image, many environments. Baked-in settings force a rebuild per change — breaks 12-factor build-once-run-anywhere. Need: inject config from outside at runtime.

Secret problem

Some config is sensitive — DB passwords, API keys, TLS private keys. It cannot sit in the image, in git, or in a plain ConfigMap. Need: a config object handled with more care.

Durability problem

The writable layer dies with the container. Anything stateful loses its data on restart/reschedule. Need: storage whose lifetime is independent of any one Pod.

2 · ConfigMap and Secret: inject, don't bake

A ConfigMap is a Kubernetes object that holds non-confidential configuration as key-value pairs (or whole files). It is stored in etcd via the API server (lesson 03) like any other object, and it is referenced by — not embedded in — a Pod spec. The same image plus a dev ConfigMap is the dev app; the same image plus a prod ConfigMap is the prod app. Nothing rebuilds. A Pod consumes a ConfigMap in one of two ways, and the difference matters:

Injection	How the app reads it	Live update?	Fits
Environment variables	Keys become env entries the process reads at startup (DB_URL=…).	No. Env is fixed at process start; changing the ConfigMap does not reach a running process — it sees the new value only after a Pod restart.	Small scalars; classic 12-factor env config.
Mounted as files	The ConfigMap is mounted as a read-only volume; each key becomes a file (/etc/app/config.yaml).	Yes, eventually. The kubelet refreshes the mounted files (typically within ~1 minute); an app that re-reads the file (or watches it) picks up changes with no restart.	Config files, large blobs, apps that hot-reload.

The rule worth memorizing: env vars are a snapshot at start; mounted files can hot-update. If you need a running Pod to notice a config change without a restart, mount it as a file and have the app re-read it. If startup-time config is fine, env vars are simpler.

A Secret is the near-twin of a ConfigMap, meant for sensitive values (passwords, tokens, TLS keys, registry credentials). It is consumed exactly the same two ways — env var or mounted file — and structurally it is a ConfigMap with a different intent and slightly different handling. Which leads to the single most important and most misunderstood fact about it.

3 · The hard truth: a Secret is base64, not encrypted

Common misconception

A Kubernetes Secret is base64-encoded, not encrypted. base64 is a reversible text encoding — echo "czNjcjN0" | base64 -d reveals the value in one step, no key required. Storing a password in a Secret instead of a ConfigMap buys you almost nothing on its own. Treat a Secret as "marked sensitive," not "protected."

Why does the encoding exist at all, if it is not security? base64 lets a Secret safely carry arbitrary binary data (a TLS key, a certificate) and odd characters inside a text-based (YAML/JSON) object, and it lightly discourages shoulder-surfing. That is the entire benefit. Anyone who can kubectl get secret -o yaml can decode it instantly, and by default the value sits in etcd in plaintext-equivalent form. Real protection takes three layers, none of which the Secret object provides by itself:

Encryption at rest

Enable encryption-at-rest in the API server's config so Secrets are encrypted before they hit etcd (ideally with keys in a KMS). Otherwise an etcd backup or disk is a plaintext leak.

RBAC restriction

Lock down who and what can read Secrets with RBAC (lesson 12). By default many subjects can get Secrets — least-privilege is on you.

External secret store

For real secrets, source them from Vault or a cloud secret manager via tools like external-secrets or CSI secret drivers, so the source of truth is an audited vault, not etcd.

State this plainly in interviews: a Secret keeps a value out of the image and out of git, and marks it sensitive — but base64 is encoding, so durability and access still depend on encryption-at-rest, RBAC, and (for anything that truly matters) an external secret store.

4 · Volumes, then the PVC → PV split

Now the durability half. The first building block is a Volume: a directory, declared in the Pod spec, mounted into one or more of the Pod's containers. A Volume's lifetime is tied to the Pod, not the container — so an emptyDir volume survives a container crash-and-restart and is shared between containers in the Pod, which is great for scratch space and sidecar hand-off. But it still dies when the Pod dies. That is not durability; it is just a longer-lived scratchpad. For data that must outlive the Pod entirely, we need storage whose lifetime is independent of any Pod — and that is where Kubernetes introduces a deliberate split.

You might expect a Pod to just name a cloud disk directly. Kubernetes refuses, because that would weld the application author to one cloud's API — the very coupling this track keeps dissolving. Instead it splits durable storage into two objects that meet by matching, exactly like a Service meets its Pods by selector (lesson 06):

→PersistentVolume (PV) → a piece of real storage in the cluster — a cloud disk, an NFS export, a local SSD. It is a cluster resource with a capacity (20Gi), access modes, a StorageClass, and a reclaim policy. It represents the supply side, owned by whoever runs the infrastructure.

→PersistentVolumeClaim (PVC) → a request for storage by an application: "I want 10Gi, fast (a StorageClass), ReadWriteOnce." It names what the app needs, never which specific disk. This is the demand side, written by the app author in the Pod's namespace.

→binding → Kubernetes finds (or provisions) a PV that satisfies the PVC's size + access mode + class and binds them one-to-one. The Pod then mounts the PVC; it never names the underlying disk. Pod dies → PVC and its PV stay; a replacement Pod re-mounts the same data.

This claim indirection is the whole point. The app author writes "I need 10Gi of fast RWO storage" once, and that manifest runs unchanged on AWS, GCP, on-prem, or a laptop — because the cluster operator wires each environment's StorageClass to its real backend. The author is decoupled from the infrastructure: a PVC is portable; a raw cloud-disk ID is not. It is the same move as the Service VIP hiding moving Pod IPs — here the PVC hides the moving, environment-specific real disk.

  Pod                PVC (demand)              PV (supply)            real disk
  mounts ──► "10Gi, fast, RWO"  ──bind──►  pv-3: 20Gi, ssd, RWO  ──►  cloud volume vol-0a7…
             (app author)                   (infra / dynamic)

  Pod dies and is rescheduled; PVC + PV persist; the new Pod re-mounts the same PVC,
  same bytes. The app never learns the disk's real id.

5 · Access modes, reclaim policy, and dynamic provisioning

Three attributes on the claim and the volume decide whether a bind can happen and what survives a delete.

Access modes declare how many nodes may mount the volume and how:

Mode	Meaning	Typical use
RWO — ReadWriteOnce	Mounted read-write by a single node at a time (multiple Pods on that one node can share it).	The common case: a block device / cloud disk backing one database Pod.
ROX — ReadOnlyMany	Mounted read-only by many nodes at once.	A shared, immutable dataset or model weights fanned out to many readers.
RWX — ReadWriteMany	Mounted read-write by many nodes at once.	Shared scratch across replicas — requires a networked filesystem (NFS, CephFS); most block/cloud disks cannot do this.

A claim asking for RWX can only bind to a PV (and a backend) that actually supports many-node read-write. Ask a plain cloud block disk for RWX and the claim stays Pending forever — a classic trap the widget below reproduces.

Reclaim policy decides what happens to the PV (and its real disk) when its PVC is deleted:

Retain

The PV and the underlying disk are kept after the PVC is deleted; an operator must reclaim or clean it manually. Safe default for precious data — the bytes survive an accidental kubectl delete pvc.

Delete

Deleting the PVC deletes the PV and the real disk too. Convenient for ephemeral or dev storage — and a foot-gun for production data. Most dynamic StorageClasses default to this.

Finally, where do PVs come from? Two ways. Static provisioning: an operator pre-creates a pool of PVs by hand and claims bind to whatever fits — fine for a fixed, known set of disks. It does not scale: someone must keep stocking the shelf. Dynamic provisioning fixes that. A StorageClass names a kind of storage (fast-ssd, standard) and points at a CSI driver (Container Storage Interface — the pluggable contract, like the CNI for networking, by which Kubernetes asks a storage backend to create/attach/delete real disks). When a PVC names a StorageClass and no matching PV exists, the StorageClass's provisioner creates a brand-new PV and its real disk on the fly, then binds it. The app author just asks; the disk appears. A PVC with no class and no matching PV in the pool, by contrast, simply waits as Pending.

6 · Widget: bind, provision, or stay Pending

PVC → PV binding & dynamic provisioning

Write a claim: a size, an access mode, and a StorageClass (or none). Kubernetes tries, in order, to bind it to an existing PV from the static pool that satisfies all three (capacity ≥ requested, matching access mode, matching class). If none fits but the class supports dynamic provisioning, the StorageClass's CSI driver provisions a fresh PV sized to the claim and binds it. If nothing matches and dynamic provisioning can't help (no class, or the class can't satisfy the access mode), the claim stays Pending. Try asking for RWX on fast-ssd (a block class — can't do many-node write), or 200Gi with no class (no PV that big, nothing to provision it).

size: 10Gi access mode: StorageClass:

Static PV pool:

Claim status

—

Bound to PV

—

Capacity

—

How it bound

—

Failure modes & checklist

Failure modes

Treating a Secret as encrypted. Putting a password in a Secret and assuming it is protected. base64 decodes in one command, and without encryption-at-rest it sits readable in etcd. Signal: the secret appears decoded in an etcd dump or a -o yaml, and anyone with broad RBAC can read it.
Expecting env-var config to hot-reload. Editing a ConfigMap and waiting for a running Pod to notice — env vars are fixed at process start, so it never does. Signal: the ConfigMap shows the new value but the app still uses the old one until you restart the Pod.
Asking for an unsupported access mode. Requesting RWX from a block/cloud-disk class that only does RWO. Signal: the PVC sits Pending with an event like "no persistent volumes available for this claim and no storage class is set" — or a provisioning error.
Reclaim policy Delete on production data. A dev-default StorageClass with Delete backs a database; someone deletes the PVC and the real disk vanishes with it. Signal: deleting a PVC silently destroys data; no PV remains to reattach.
Storing data in the container layer or emptyDir. Writing important state to the writable layer or an emptyDir and assuming it survives a reschedule. Signal: data is fine across container restarts but gone after the Pod is evicted or rolled to a new node.

Checklist

Inject config, never bake it — same image, per-environment ConfigMap/Secret, so build-once-run-anywhere holds.
Mount config as files if you need hot-reload; use env vars only for startup-time settings.
Don't trust base64 — enable encryption-at-rest, restrict Secrets with RBAC (lesson 12), and source real secrets from Vault / external-secrets.
Claim by spec, not by disk — PVCs request size + access mode + StorageClass so the manifest is portable across clouds.
Match the access mode to the backend — RWX needs a networked filesystem; most block disks are RWO only.
Set reclaim policy deliberately — Retain for precious data, Delete only for genuinely disposable storage.

Checkpoint exercise

Try it

In the widget, start with a 10Gi, RWO, fast-ssd claim and confirm it binds to a matching PV from the static pool. Now raise the size to 200Gi with class (none) — observe it stay Pending and explain why (no static PV that big, and no class to provision one). Switch the class to fast-ssd at 200Gi — watch dynamic provisioning create a fresh PV and bind it. Finally set access mode to RWX on fast-ssd and explain, in one sentence, why a block-storage class cannot satisfy a many-node write claim while shared-nfs can. Then, away from the widget: state which ConfigMap injection method (env var vs mounted file) a running app needs to pick up a flag change with no restart.

Where this points next

Config now comes from outside the image, and a PVC gives a Pod a durable disk that survives reschedules — the bind indirection makes the claim portable across clouds. But notice what a PVC gives you: a disk for a Pod. A Deployment's Pods are interchangeable cattle (lesson 05) — random names, born and killed in any order, and if they reference one PVC they'd all fight over one RWO disk. That is exactly wrong for a clustered database, where db-0, db-1, and db-2 each need their own stable disk and their own stable network identity, brought up in order. And not every workload is even a long-running server — some work should run once and finish, or run per node. Lesson 09, Beyond the Stateless Server: StatefulSets, Jobs, DaemonSets, gives stable per-replica identity and per-replica storage (a StatefulSet's volumeClaimTemplate mints one PVC per Pod), plus the run-to-completion and one-per-node workload shapes.

Takeaway

A Pod is born config-blind and storage-less, so its image can't be welded to one environment and its writable layer can't hold anything precious. ConfigMap and Secret fix the first half by injecting configuration from outside the image — as env vars (a snapshot fixed at process start, no live update) or as mounted files (the kubelet refreshes them, so an app that re-reads can hot-reload). A Secret is base64-encoded, not encrypted: the encoding only carries binary safely and marks intent, so real protection needs encryption-at-rest in etcd, RBAC restriction, and ideally an external store like Vault / external-secrets. For durability, a Volume outlives a container but not the Pod; the PersistentVolume / PersistentVolumeClaim split outlives the Pod entirely. The app claims "10Gi, fast, RWO" (the PVC, demand side) and a matching PV is bound to it — found in a static pool, or minted on the fly by a StorageClass + CSI driver via dynamic provisioning. Access modes (RWO/ROX/RWX) gate which binds are even possible, reclaim policy (Retain/Delete) decides what survives a PVC deletion, and the claim indirection exists precisely to decouple the app author from the cloud — the same hide-the-moving-thing move as a Service VIP.

Interview prompts

Why inject config with a ConfigMap instead of baking it into the image? (§1, §2 — build-once-run-anywhere: one immutable artifact must run in every environment, so per-environment settings must come from outside at runtime; baking them in forces a rebuild and re-roll to flip a single flag.)
Env-var vs mounted-file config — what's the operational difference? (§2 — env vars are read once at process start and never update live; a mounted ConfigMap is refreshed by the kubelet (~1 min), so an app that re-reads the file hot-reloads with no restart. Need live updates → mount files.)
Is a Kubernetes Secret encrypted? What protects it? (§3 — no; it is base64, a reversible encoding that decodes with no key. Real protection is encryption-at-rest in etcd (KMS), tight RBAC on who can read Secrets, and/or an external store like Vault via external-secrets.)
Why the PVC → PV split instead of naming a disk directly? (§4 — the claim indirection decouples the app author (writes "10Gi, fast, RWO") from the infrastructure (the real cloud disk), so the manifest is portable across clouds/on-prem; the operator wires each environment's StorageClass to its backend.)
What do access modes mean and why might a claim stay Pending on one? (§5 — RWO = one node read-write, ROX = many nodes read-only, RWX = many nodes read-write; RWX needs a networked filesystem, so asking a block/cloud-disk class for RWX never binds and the PVC sits Pending.)
What is dynamic provisioning and what plays the CNI-equivalent role? (§5 — a PVC names a StorageClass; if no matching PV exists, the class's provisioner (a CSI driver — the storage analog of the CNI plugin) creates a brand-new PV and real disk on demand and binds it, so no one pre-stocks PVs.)
Reclaim policy Retain vs Delete — when does the choice bite? (§5 — Delete destroys the PV and underlying disk when the PVC is deleted (fine for dev, a foot-gun for prod); Retain keeps the data for manual reclaim. Most dynamic classes default to Delete, so production data needs Retain set deliberately.)