Part I — The engine and the unit

The Container: Boxing a Process

Lesson 00 built the whole engine: stop issuing commands, declare the desired state, and let a control loop drive actual toward desired forever. But that loop reconciles "a running app," and we left the word app dangerously vague — we said "a process" and moved on. A bare OS process is the wrong unit to declare. It borrows the host's shared libraries (so it runs here and breaks there — "works on my machine"), it shares the host's CPU and RAM with no ceiling (so one greedy process starves its neighbors), and its view of the filesystem is whatever the host happens to have. You cannot reconcile a thing that is not the same thing everywhere. This lesson derives the fix — the container — and shows it is not a tiny virtual machine but a perfectly ordinary Linux process wearing two pieces of armor.

The previous step left this broken

The reconcile loop in lesson 00 can recreate a process when it dies, but it has no portable, isolated, reproducible thing to recreate. A process inherits the host's library versions, its filesystem layout, and an unbounded slice of CPU and memory. Move it to another node and the libraries differ; co-locate it with a busy neighbor and they fight over RAM. "Restore the desired process" is meaningless until the process is identical everywhere and walled off from its neighbors.

Linear position

Forced by: to reconcile "a running app" you need a unit that is byte-for-byte identical on every node and cannot trample its neighbors; a bare process is neither.
New capability: the container — an image (a frozen, layered filesystem plus the start command) run as a normal process under namespaces (what it can see) and cgroups (what it can use). Ship the machine, not the recipe.
Forces next: a single container is rarely the whole app (it wants a log shipper, a proxy sidecar), and a lone container still is not a thing the cluster can schedule, place, and address as one unit.

The plan

Five moves. (1) Pin down exactly why a bare process is the wrong unit — the three things it shares with the host. (2) Derive the image: a frozen, layered filesystem plus a start command, so "the machine" itself becomes the artifact. (3) Derive the two pieces of armor — namespaces for isolation (what the process sees) and cgroups for limits (what it uses) — and insist a container is a process, not a VM. (4) Separate image (the template on disk) from container (one running instance). (5) Turn the cgroup memory ceiling yourself in the widget: watch a container get OOMKilled at its limit, and watch one with no limit starve its neighbor.

1 · Why a bare process is the wrong unit

Picture the lesson-00 loop trying to keep one web binary alive on a fleet of machines. The binary is just an executable on disk; to run it you fork a process. That process is not self-contained. It reaches out to the host for three things, and each one breaks the promise that the loop reconciles "the same app" everywhere.

Shared libraries → dependency hell

The process dynamically links against whatever libc, OpenSSL, Python, or CUDA version the host has. Node A has Python 3.11, node B has 3.9; the same binary behaves differently or refuses to start. This is the literal source of "works on my machine."

Shared CPU and RAM → noisy neighbor

A plain process can allocate memory until the kernel's out-of-memory killer fires, and grab as much CPU as the scheduler gives it. One runaway process degrades every other process on the box — the loop kept your app "running," but starved.

Shared filesystem and namespaces

It sees the host's whole filesystem, every other process (PIDs), the host's network interfaces, and the host's users. No boundary. Two copies of the app fight over the same config paths and ports.

So "declare the desired process and reconcile it" cannot work yet, for a sharp reason: the thing is not identical across nodes, and it is not walled off from what shares the node. We need to change the unit. Instead of shipping a recipe ("install these packages, then run this binary") and praying every host cooks it the same way, we will ship the finished machine.

2 · The image: freeze the machine, not the recipe

The first half of the fix attacks reproducibility. An image is a read-only, frozen snapshot of a complete userland filesystem — the binary, its libraries, the interpreter, the config, the certificates — plus metadata that records the start command (which process to launch and with what arguments). It carries everything the process links against, so the host's library versions stop mattering. The host kernel is still shared (that is the whole reason a container is cheap), but everything in user space comes from the image.

Images are not monolithic blobs; they are built as a stack of layers. Each build step (install base OS files, add Python, copy your app) produces one layer — a tarball of the filesystem changes that step made — identified by the cryptographic hash of its contents. The running container sees all layers merged top-down by a union/overlay filesystem, with one thin writable layer on top for runtime scratch. This is the OCI (Open Container Initiative) image format, the vendor-neutral standard, so an image built by Docker runs under any OCI-compliant runtime.

→base layer → a minimal OS userland (e.g. ~30 MB for a slim base), hash sha256:abc…

→runtime layer → add Python + your app's libraries, hash sha256:def…

→app layer → copy your code in, hash sha256:123…

→writable layer → a thin top layer created per container for scratch writes; discarded when it dies

Layering buys two concrete wins. First, deduplication and caching: ten images sharing the same base layer store and pull that 30 MB once, by hash; a registry and a node cache it once. Rebuilding after a one-line code change re-ships only the tiny top app layer, not the whole machine. Second, content-addressed reproducibility: an image referenced by its digest (sha256:…) is byte-for-byte identical wherever it lands — exactly the "same thing everywhere" the loop needed. The recipe is gone; the cooked machine is the artifact.

3 · The two pieces of armor: namespaces and cgroups

The image fixes reproducibility, but a process launched from an image still sees and uses the whole host. The second half of the fix is two independent Linux kernel features. Crucially, a container is just a normal Linux process the kernel started with these two features applied. There is no guest kernel, no hypervisor, no emulated hardware — that is what a virtual machine is, and it is not this. The container shares the host kernel; the isolation is the kernel pretending, per-process, that this process is alone.

Namespaces control what the process can see. The kernel can give a process a private view of a given resource so that, from inside, it looks like it owns the machine:

PID namespace

Its processes are renumbered from 1; the container's main process is PID 1 and cannot see the host's other processes. ps inside shows only its own tree.

Network namespace

Its own network stack — interfaces, routing table, ports. The container can bind port 8080 without colliding with the host or another container's 8080.

Mount namespace

Its own filesystem tree, rooted at the image's merged layers. It sees / as the image, not the host's /.

User & IPC namespaces

Its own user/group ID mapping (root inside can map to an unprivileged host user) and its own shared-memory/semaphore space. More boundaries the host process never had.

Cgroups (control groups) control what the process can use. Where a namespace is a view, a cgroup is a budget the kernel enforces. The two that matter most:

Cgroup control	What it caps	What happens at the edge
CPU shares / quota	A weight for contention plus an optional hard ceiling (e.g. "at most 0.5 of a core")	The process is throttled — it keeps running, just slower. CPU is compressible.
Memory limit	A hard ceiling on resident bytes (e.g. "at most 512 MB")	Cross it and the kernel OOMKills the process — exit code 137. Memory is incompressible: you cannot throttle a byte.

That asymmetry is worth burning in: a container over its CPU budget is merely slow, but a container over its memory limit is dead. The loop from lesson 00 will then notice the gap and restart it — which, if the limit is genuinely too low, becomes the endless restart loop you will meet later. The memory ceiling is the single most consequential knob on a container, which is exactly the knob the widget below hands you.

   HOST KERNEL  (one shared kernel — NOT a VM per container)
   ┌──────────────────────────────────────────────────────────┐
   │  container A (a process)          container B (a process)  │
   │  ┌───────────────────────┐        ┌──────────────────────┐│
   │  │ namespaces: own PIDs,  │        │ namespaces: own PIDs,││
   │  │  net, mounts, users    │        │  net, mounts, users  ││
   │  │ cgroup: ≤512MB, 0.5cpu │        │ cgroup: ≤256MB, 1cpu ││
   │  │ rootfs = image layers  │        │ rootfs = image layers││
   │  └───────────────────────┘        └──────────────────────┘│
   └──────────────────────────────────────────────────────────┘
        ↑ a VM would put a whole guest kernel inside each box ↑

4 · Image vs container: template vs instance

One distinction trips up everyone at first, and it maps cleanly onto something you already know. The image is the immutable artifact on disk (the layers + start command); the container is one running instance the runtime created from that image by giving the start command its namespaces, cgroups, and a fresh writable layer. It is exactly the class-versus-object relationship: one image, many containers.

Image

Frozen, read-only, content-addressed by digest. Lives in a registry and a node cache. Pulled once, shared by every container of that version. The class.

Container

A live process (or process tree) with namespaces + cgroups + a thin writable layer. Mortal: when it exits, the writable layer is discarded — its scratch state is gone. The instance.

Worked example: pull myapp:1.4 (one image, say 120 MB across four layers) once onto a node, then run three replicas. You get one cached image and three containers; the 120 MB of read-only layers is shared, and each container adds only its own thin writable layer plus its own namespaces and cgroup budget. That is why containers start in milliseconds and a VM takes tens of seconds: there is no kernel to boot, only a process to fork with armor attached. And because the writable layer dies with the container, anything you must keep cannot live there — a thread we pick up in the storage lesson later in the track.

5 · Turn the memory knob yourself

The widget puts two containers on one node (total node memory: 1,024 MB, illustrative). Each has a memory usage (what it is trying to hold) and a memory limit (its cgroup ceiling). Push a container's usage past its own limit and the kernel OOMKills it (exit 137). Turn a container's limit off and watch it behave like the bare process from §1: with nothing to stop it, a greedy container eats the node's RAM and starves its neighbor — the noisy-neighbor failure, reproduced.

cgroup memory ceiling — OOMKill and the noisy neighbor

Drag each container's usage and limit. When usage ≥ limit, that container is OOMKilled (a real event the loop would react to). Untick a container's limit to remove the cgroup ceiling: it can now consume node RAM unbounded, and when the node runs out the kernel kills whoever it must — often the innocent neighbor. The limit is what converts "noisy neighbor" into "the greedy one dies, alone."

container A (image myapp:1.4)Running

usage: 320 MB limit: 512 MB enforce limit (cgroup on)

container B (image logshipper:2.0)Running

usage: 200 MB limit: 256 MB enforce limit (cgroup on)

Container A

—

Container B

—

Node memory used

—

Last event

—

Two things to notice. With limits on, a container can only ever kill itself by exceeding its own ceiling — its blast radius is one container, and the node stays healthy for everyone else. With a limit off, the same overshoot becomes a node-level out-of-memory event whose victim the kernel chooses, and it may not be the offender. The cgroup did not make the app use less memory; it made the failure local and predictable instead of shared and random.

Failure modes & checklist

Failure modes

No memory limit set. The container behaves like the §1 bare process — unbounded, free to starve neighbors. Signal: sporadic node-level OOM events that kill seemingly random containers, not the greedy one.
Limit set too low. The app's real working set exceeds the ceiling, so the kernel OOMKills it (exit 137), the loop restarts it, and it dies again. Signal: a container in a restart loop with exit code 137 and rising restart counts.
Mutable latest tag. Referring to an image by a moving tag instead of a digest means two nodes can pull different bytes for the "same" version. Signal: "works on one replica, fails on another" despite identical spec.
Treating the writable layer as durable. Writing important data inside the container's top layer, which is discarded on exit. Signal: data vanishes on every restart, scale, or reschedule.
Confusing container with VM. Assuming a guest kernel / hardware isolation that is not there — a kernel bug or privileged escape can cross the boundary. Signal: security model assumes VM-grade isolation a shared kernel does not give.

Checklist

Pin images by digest (sha256:…) or an immutable tag, never bare latest, so every node runs identical bytes.
Set a memory limit on every container, sized from the real working set plus headroom — make failures local, not node-wide.
Set CPU requests/limits too, but remember CPU is throttled (slow), memory is killed (dead).
Keep images small and layered well — stable layers (base, deps) below volatile ones (your code) to maximize cache reuse.
Assume the writable layer is scratch. Anything that must survive a restart belongs in a volume, not the container.
Remember it is a process. Same kernel as the host; reason about isolation and security accordingly.

Checkpoint exercise

Try it

In the widget, give container A a limit of 512 MB and push its usage to 640 MB. Confirm A is OOMKilled while B keeps running — the blast radius is one container. Now turn A's limit off, set its usage to 900 MB while B sits at 200 MB, and watch the node total exceed 1,024 MB; note which container the node-level OOM event takes. In one sentence each, state: (a) what exit code an OOMKill reports and why memory (unlike CPU) cannot just be throttled, and (b) why pinning the image by digest rather than the latest tag is what makes "the same thing everywhere" actually true.

Where this points next

The container fixed the unit: an image makes the app identical on every node, and namespaces + cgroups wall it off so the loop can reconcile it safely. But two cracks are already showing. First, a real app is rarely one container — your web container wants a log shipper beside it, maybe a proxy sidecar that terminates TLS, and these must share a network and disk and live and die together. Scheduling them independently breaks that co-location. Second, a container description is still inert data, and a lone container is not something the cluster can place, address, and schedule as one thing. The next lesson, The Pod and the Node: the Atom and the Hands, derives the Pod (the smallest schedulable unit — a group of containers sharing one IP, localhost, and volumes) and the Node (the machine whose kubelet actually pulls images and runs containers, running the same reconcile loop locally).

Takeaway

A bare process is the wrong unit to reconcile because it shares the host's libraries (dependency hell, "works on my machine"), its CPU and RAM (noisy neighbor), and its filesystem. The container fixes both halves: an image — a frozen, layered, content-addressed filesystem plus a start command — makes the app byte-for-byte identical everywhere (ship the machine, not the recipe), and two kernel features wrap the process in armor: namespaces bound what it can see (its own PIDs, network, mounts, users) and cgroups bound what it can use (CPU shares it can be throttled on, a memory ceiling it is OOMKilled for crossing). A container is therefore a normal Linux process sharing the host kernel — emphatically not a VM — and the image (template, on disk) is distinct from the container (one mortal running instance with a throwaway writable layer). The memory limit is the load-bearing knob: with it, a runaway container kills only itself; without it, it is the §1 noisy neighbor again, taking down whoever the kernel picks.

Interview prompts

Why isn't a bare OS process a good unit to declare and reconcile? (§1 — it shares the host's libraries (dependency hell / "works on my machine"), its CPU and RAM with no ceiling (noisy neighbor), and its filesystem/PIDs/network; so it is neither identical across nodes nor isolated from neighbors.)
What is a container, precisely — and how is it not a VM? (§3 — a normal Linux process started from an image with namespaces and cgroups applied; it shares the host kernel. A VM runs a whole guest kernel on emulated hardware via a hypervisor; a container has neither, which is why it starts in milliseconds.)
Distinguish namespaces from cgroups. (§3 — namespaces control what a process can SEE (private PIDs, network, mounts, users); cgroups control what it can USE (CPU shares/quota, memory limit). View vs budget.)
What happens when a container exceeds its CPU limit vs its memory limit, and why the asymmetry? (§3, §5 — over CPU it is throttled (just slower) because CPU is compressible; over memory it is OOMKilled with exit 137 because memory is incompressible — you cannot throttle a byte.)
What is an image, and what do layers buy you? (§2 — a frozen, read-only, content-addressed (OCI) filesystem plus a start command, built as a hash-identified stack of layers; layers give dedup/caching (shared base pulled once, rebuild re-ships only the top layer) and digest-pinned reproducibility.)
Image vs container — what's the relationship? (§4 — class vs instance: one immutable image yields many mortal containers, each with its own namespaces, cgroup, and a thin throwaway writable layer; the read-only layers are shared.)
Why does a memory limit matter beyond protecting one container? (§5 — it localizes failure: with a limit a runaway container only OOMKills itself (blast radius one); without one it triggers a node-level OOM whose victim the kernel chooses, often the innocent neighbor.)