Part V — How the cluster decides and defends

The Scheduler: How a Pod Finds a Node

Lesson 09 finished the workload zoo: a Deployment runs interchangeable stateless replicas, a StatefulSet gives a clustered database stable identity and per-replica disk, a Job runs work to completion, a DaemonSet drops exactly one Pod on every node. Different shapes, one shared assumption we never examined — that the Pods they produce simply land on a machine. They do not, automatically. The moment any controller creates a Pod, that Pod is unbound: it exists in the cluster's store with no node assigned, and no kubelet will touch it. Something has to choose, out of N nodes, the one that runs it — and "pick a random node" ignores whether the Pod even fits, where its data lives, and whether you wanted replicas spread for safety. This lesson derives the kube-scheduler: the loop that turns an unbound Pod into a bound one.

The previous step left this broken

Every controller in lesson 09 — Deployment, StatefulSet, Job, DaemonSet — does its reconciliation by creating Pod objects. But a freshly created Pod has an empty spec.nodeName: it is stored in etcd and served by the API server, yet it points at no machine, so no kubelet claims it and nothing runs. The controllers know how many Pods and what they contain; not one of them decides where. Choosing a node naively — first one, random one — is a trap: a node already full has no room, a Pod whose data sits on node-3 wants node-3, and two replicas on the same node both die when that node does. Placement is its own problem, and it needs its own control loop.

Linear position

Forced by: controllers produce unbound Pods (no nodeName); a Pod that names no node never runs, and a naive choice ignores fit, locality, and spread.
New capability: the kube-scheduler — a control loop that watches for unbound Pods and, using each Pod's resources.requests as the placement contract, filters nodes that cannot host it, scores the survivors, and binds the winner by writing nodeName; with levers (selectors, affinity, taints/tolerations, topology spread) and QoS-based eviction when a node runs short.
Forces next: the scheduler places a Pod once, assuming it then stays healthy and rightly-sized — but a bound Pod can deadlock or lose a dependency, and a fixed replica count is the wrong count as load moves.

The plan

Six moves. (1) Show what "unbound" means and why placement is a distinct loop. (2) Pin the requests-vs-limits contract — requests gate placement, limits cap runtime. (3) Derive the filter → score → bind loop and work the bin-packing math. (4) The placement levers: nodeSelector / affinity / anti-affinity, taints & tolerations, topology spread. (5) QoS classes and the eviction order under node pressure. (6) Drive the bin-packing widget, then failure modes, checklist, and the hand-off to health & autoscaling.

1 · An unbound Pod is inert

Recall the engine of this whole track (lesson 00): nothing in Kubernetes acts on a command, it acts on stored desired state, and control loops drive reality toward it. A Pod object is desired state — "this set of containers should run." But desired state alone places nothing. A Pod has a field spec.nodeName, and until that field holds a node's name, the Pod is unbound (also called unscheduled): it sits in etcd, the API server serves it, but every kubelet ignores it, because a kubelet only runs Pods whose nodeName matches its own node. So an unbound Pod is inert — created, visible in kubectl get pods as Pending, and doing absolutely nothing.

You could imagine the API server itself picking a node at creation time. Kubernetes deliberately does not, and the reason is the same loose-coupling instinct from lesson 04: placement is a hard, evolving policy problem (fit, locality, spread, fairness, hardware) that should not be welded into the one component that stores truth. Instead, placement is just another controller — the kube-scheduler — that watches for Pods with an empty nodeName and assigns each one. Its single mutation is tiny: it writes nodeName back through the API server. That write is called the bind, and it is the entire visible output of the scheduler. Once bound, the Pod is a normal assignment and the destination kubelet (lesson 02) takes over: pulls images, runs containers, reports status. The scheduler never launches anything; it only decides.

→controller creates Pod → nodeName: "" — stored in etcd, served by API server, status Pending

→scheduler watches → sees an unbound Pod in its queue, runs filter → score → bind on it

→bind → scheduler writes nodeName: node-2 through the API server (its only mutation)

→kubelet on node-2 acts → "a Pod is now assigned to me" → pull images, start containers, report Running

2 · The contract: requests gate placement, limits cap runtime

Before the scheduler can decide if a Pod fits a node, the Pod must say how much it needs — and Kubernetes splits "needs" into two numbers per container that are constantly confused. Each container may declare resources.requests and resources.limits, for CPU (measured in cores; 1000m = 1 core) and memory (bytes; 512Mi, 2Gi). They do completely different jobs:

requests — the placement contract

The amount the scheduler reserves for this container. It is the only resource number the scheduler counts: a Pod fits a node only if the node's unreserved capacity covers the Pod's summed requests. Requests are about scheduling, not about what the container actually uses.

limits — the runtime cap

The ceiling the kubelet/cgroup (lesson 01) enforces while the container runs. Over the CPU limit, the container is throttled (slowed, not killed). Over the memory limit, it is OOMKilled (memory cannot be throttled, so the kernel kills it). Limits never influence placement.

allocatable, not total

The scheduler counts requests against a node's allocatable capacity — total machine resources minus what the kubelet, OS, and system daemons reserve. A 16-core box might expose ~15 allocatable cores. Capacity ≠ allocatable.

requests ≠ usage

The single most important fact: placement is driven by requests, never by live usage. A Pod requesting 2 cores but idling at 0.1 still reserves 2; a Pod requesting 0 can pile onto a node that is actually busy. The scheduler does bookkeeping on declared requests, not measurements.

This separation is what makes scheduling tractable and stable. If the scheduler chased real-time usage, every placement would be a moving target and bursts would cause thrashing. By reserving against declared requests, the node's free budget is a simple, durable number the scheduler can do arithmetic on. The cost of the simplicity: if you set requests far below true usage, you overcommit — the scheduler thinks the node is fine while it is actually saturated, and §5's eviction machinery has to clean up. Get requests honest and most scheduling problems disappear.

3 · The loop: filter → score → bind

For each unbound Pod the scheduler runs a three-stage cycle. It is the same observe-diff-act loop from lesson 00, specialized to "which node?":

1Filter (predicates) → walk every node and discard the ones that cannot host this Pod. Hard yes/no checks: does the Pod's summed CPU+memory request fit the node's remaining allocatable? Does the node match the Pod's nodeSelector? Does a taint on the node repel this Pod (§4)? Are required ports free? The output is the set of feasible nodes. If it is empty, the Pod stays Pending.

2Score (priorities) → rank the feasible nodes 0–100 on soft preferences and sum the weights: spread (prefer a node with fewer of this Pod's peers, for resilience), affinity preferences, balanced resource use, image locality (a node that already cached the image). The highest total wins; ties broken to spread load.

3Bind → write nodeName = the winner through the API server. Done — the kubelet there takes it from here.

Worked bin-packing. A node has 8 allocatable CPU cores. Each web Pod declares requests.cpu: 2000m (= 2 cores). How many fit? Filter keeps the node feasible while reserved + 2 ≤ 8:

  node-A allocatable: 8.0 CPU
  pod-1 (req 2) → reserved 2 / 8  ✓ fits   free 6
  pod-2 (req 2) → reserved 4 / 8  ✓ fits   free 4
  pod-3 (req 2) → reserved 6 / 8  ✓ fits   free 2
  pod-4 (req 2) → reserved 8 / 8  ✓ fits   free 0   ← node now full for this Pod
  pod-5 (req 2) → 8 + 2 = 10 > 8  ✗ filter rejects node-A
                  no other node feasible → pod-5 stays Pending

Exactly ⌊8 / 2⌋ = 4 fit; the fifth has nowhere to go and reports Pending with event "0/1 nodes available: Insufficient cpu". Note this is pure request arithmetic: even if those four Pods idle at 5% CPU and the node is 95% free in reality, the fifth is still rejected — the 8 cores are reserved, not measured. Memory works identically and in parallel; a Pod must fit on both axes (the binding axis is whichever runs out first). This is why honest requests matter: set them too high and you waste capacity (Pods pend on a near-empty cluster); set them too low and you overcommit into the eviction zone.

4 · Levers: steering and repelling placement

Filter+score gives sensible defaults, but real clusters need to direct placement — pin GPU work to GPU nodes, keep replicas apart, fence off dedicated hardware. Four levers, and the key distinction is pull (the Pod asks for a node) vs push (the node repels Pods):

Lever	Direction	What it does	Hard / soft
nodeSelector	Pod pulls	Pod runs only on nodes carrying matching labels (e.g. disktype=ssd). The simplest filter-stage constraint.	Hard — no match, Pod pends.
nodeAffinity	Pod pulls	Richer label matching with operators and either required… (filter) or preferred… (score) — "must be on a GPU node" vs "prefer zone-a."	Both — required is hard, preferred is soft.
pod (anti-)affinity	Pod pulls/repels peers	Place relative to other Pods: affinity co-locates (cache near its app); anti-affinity spreads (no two replicas on one node). Matched by label selector over a topology key.	Both — required or preferred.
taints & tolerations	Node pushes	A taint on a node repels every Pod that does not carry a matching toleration. The default-deny inverse of affinity: the node says "stay off unless invited."	Hard (NoSchedule) or eviction (NoExecute).

Taints and tolerations deserve a closer look because they invert the usual flow. A taint is a key=value:effect mark you put on a node; effects are NoSchedule (filter rejects Pods without a toleration), PreferNoSchedule (soft), and NoExecute (also evicts already-running, non-tolerating Pods). The canonical use: a node with expensive GPUs is tainted dedicated=gpu:NoSchedule so ordinary web Pods never land there and waste the card; only Pods that explicitly tolerate that taint — your inference workloads — are allowed on. Kubernetes itself uses this: control-plane nodes are tainted so user Pods stay off, and the node-pressure conditions in §5 apply taints like node.kubernetes.io/memory-pressure. Affinity is the Pod choosing a node; a taint is the node refusing the Pod — you often use both together.

The fourth spreading lever is topologySpreadConstraints: instead of the blunt "never co-locate" of anti-affinity, it bounds skew — "across zones, the number of these Pods on any one zone may differ from the least-loaded zone by at most maxSkew." With three zones and nine replicas and maxSkew: 1, the scheduler aims for 3/3/3 rather than 9/0/0, so losing one zone costs a third of capacity, not all of it. It is the precise tool for "spread evenly across failure domains," where anti-affinity only says "not the same one."

5 · QoS classes and the eviction order

Requests reserve capacity, but nodes can still run short — a Pod uses more memory than it requested, system daemons grow, several Burstable Pods burst at once. When a node hits memory pressure (free memory below the kubelet's eviction threshold), memory cannot be throttled, so the kubelet must evict (kill and reschedule elsewhere) some Pods to save the node. Which ones? The order follows each Pod's Quality of Service class, derived automatically from how its requests and limits relate:

QoS class	How you get it	Meaning	Eviction order under pressure
Guaranteed	Every container sets both requests and limits, and requests == limits for CPU and memory.	Fully reserved, cannot exceed its own request. The cluster promised it that memory.	Evicted last — only if it exceeds its own limit, or as a true last resort.
Burstable	At least one request set, but not the Guaranteed equality (requests < limits, or only some set).	Reserved a floor, allowed to burst above it toward the limit when the node has room.	Evicted second, worst offenders first — those using the most over their request.
BestEffort	No requests or limits at all.	Reserved nothing; runs on whatever scraps are free.	Evicted first — it promised nothing, so it is sacrificed first.

The order is the intuitive one: BestEffort → Burstable → Guaranteed. If you want a Pod to survive node pressure, give it requests == limits so it is Guaranteed; if you leave a Pod with no requests, you have implicitly volunteered it as the first casualty. This is also why "just don't set requests" is dangerous: a BestEffort Pod schedules anywhere (it reserves nothing, so it always "fits") but is the first thing killed the moment the node sweats — convenient placement, fragile life. The QoS class is the bridge from the placement contract (§2) to runtime survival.

6 · Widget: a bin-packing scheduler

Filter → score → bind — pack Pods onto nodes

Three nodes with CPU and memory allocatable. Add Pods of different sizes and watch the scheduler filter nodes that cannot fit the request, score the survivors (it prefers the least-loaded node, spreading), and bind the winner. node-C is tainted dedicated=gpu:NoSchedule, so ordinary Pods are repelled from it; only the "GPU Pod" carries the matching toleration — and, so it actually lands there rather than merely being allowed, it also carries node-affinity to node-C (push + pull used together, §4). When no node can fit a Pod's requests it goes Pending. "Overcommit a node" forces real usage above requests to trigger a memory-pressure eviction (BestEffort first). Placement is pure request arithmetic — usage only matters at eviction time.

Add a Pod to watch the scheduler place it.

Pods placed

Pending (no fit)

Cluster CPU reserved

0 / 24

Evicted

Failure modes & checklist

Failure modes

No requests set. Pods schedule anywhere (they reserve nothing, so they always "fit") and the node silently overcommits; the first memory pinch evicts them as BestEffort. Signal: Pods that ran fine suddenly get Evicted / OOMKilled under load, with no obvious cause.
Requests way above usage. Padding requests "to be safe" makes the scheduler reserve capacity nothing uses; Pods go Pending on a cluster that is nearly idle. Signal: FailedScheduling: Insufficient cpu while node dashboards show 20% utilization.
Confusing limits with requests. Setting a generous limit and a tiny request expecting it to reserve more — it does not; only the request gates placement. Signal: nodes pack far denser than expected, then throttle/OOM at runtime.
Forgotten taint. A node tainted dedicated=gpu:NoSchedule with no Pods tolerating it sits empty while other nodes overflow. Signal: one expensive node at 0% and Pods Pending elsewhere.
Anti-affinity that cannot be satisfied. Required pod-anti-affinity for 5 replicas across 3 nodes — the 4th and 5th can never be placed. Signal: replicas stuck Pending; a Deployment that never reaches its desired count.
Spread without spread constraints. All replicas land on one node by luck, then that node dies and takes the whole service. Signal: a single node failure causing a full outage despite "3 replicas."

Checklist

Set requests on every container, sized to real usage (measure, do not guess) — they are the only number the scheduler sees.
Set memory limits to bound blast radius (OOMKill one container, not the node); be cautious with CPU limits (they throttle).
Use requests == limits for anything that must survive node pressure (Guaranteed QoS).
Taint dedicated/special nodes (GPU, control plane) and add matching tolerations only on the workloads that belong there.
Add topologySpreadConstraints (or required anti-affinity) so replicas spread across nodes/zones — verify the math leaves a feasible placement.
Read the Pending event (kubectl describe pod) — it names exactly which predicate failed (Insufficient cpu, taint, no matching node).

Checkpoint exercise

Try it

Open the widget. (1) Add "batch Pod (4 CPU)" three times — confirm the third goes Pending if node-A and node-B (8 CPU each) cannot both hold two, and read the log to see which predicate rejected each node. (2) Add a "GPU Pod" and confirm it lands on tainted node-C while ordinary Pods never do — then try to fill node-C with web Pods and watch them refuse it. (3) Pack node-A near full with a BestEffort Pod and a Burstable web Pod, then hit "overcommit node-A" and predict, before clicking, which Pod gets evicted first. Finally, in one sentence: explain why a Pod requesting 2 CPU is rejected from a node whose real CPU usage is only 10% — what is the scheduler actually counting?

Where this points next

The scheduler closes the gap lesson 09 left: an unbound Pod now finds a node that genuinely fits it, with locality and spread respected and dedicated hardware fenced off. But notice the assumption baked into a successful bind — the scheduler places a Pod once and then walks away, trusting that the Pod will be healthy and that the replica count it was created for is the right count. Both are fragile. A bound, "Running" Pod can be deadlocked, still warming up, or have lost its database — running is not the same as serving — and nothing yet pulls a sick Pod out of rotation or restarts it. And a count fixed at deploy time (the 4 web Pods you packed) is simultaneously too many at 3 a.m. and too few at peak; placement does not adapt as load moves. Lesson 11, Health & Autoscaling: Closing the Loop on Reality, gives the loop a truth signal with liveness/readiness/startup probes and then makes the replica count itself a reconciled quantity with the Horizontal Pod Autoscaler, VPA, and the Cluster Autoscaler — three nested control loops on top of the one placement decision made here.

Takeaway

A Pod created by any controller is unbound — its spec.nodeName is empty, so it sits inert and Pending while every kubelet ignores it. The kube-scheduler is the control loop that fixes this: for each unbound Pod it filters nodes that cannot host it (predicates: do the Pod's summed requests fit the node's allocatable, does a nodeSelector match, does a taint repel it), scores the survivors (priorities: spread, affinity, least-loaded), and binds the winner by writing nodeName — its only mutation. The contract is sharp: requests gate placement (reserved, counted, never measured) while limits cap runtime (CPU throttle, memory OOMKill) — placement is request arithmetic, so ⌊8 CPU / 2 per Pod⌋ = 4 fit and a fifth pends even on an idle node. Levers steer it: nodeSelector/affinity pull a Pod toward nodes, pod anti-affinity and topologySpreadConstraints spread replicas across failure domains, and taints/tolerations let a node repel everything but invited Pods (GPU/dedicated). When a node still runs short of memory, QoS class sets the eviction order — BestEffort first, Burstable next, Guaranteed last — so requests == limits is how you buy survival. The scheduler decides where once; whether the Pod stays healthy and rightly-sized is lesson 11's problem.

Interview prompts

What does it mean for a Pod to be "unbound," and what makes it run? (§1 — its spec.nodeName is empty, so it is stored but no kubelet claims it (status Pending); the scheduler binds it by writing nodeName, after which the matching node's kubelet pulls images and runs containers.)
Requests vs limits — which one drives scheduling, and why? (§2 — requests drive scheduling: the scheduler reserves and counts only requests against a node's allocatable. Limits are runtime caps (CPU throttle, memory OOMKill) and never affect placement. Placement uses declared requests, not measured usage.)
Walk the scheduler's loop for one Pod. (§3 — filter nodes by hard predicates (fit, selector, taints, ports) to get feasible nodes; score the survivors on soft priorities (spread, affinity, least-loaded); bind the highest by writing nodeName. Empty feasible set → Pending.)
A node has 8 allocatable CPU; Pods request 2 each. How many fit, and what happens to the next one? (§3 — ⌊8/2⌋ = 4 fit; the fifth fails the filter (8+2 > 8) on every node and stays Pending with "Insufficient cpu" — even if the four are idle, because the 8 cores are reserved, not measured.)
Affinity vs taints/tolerations — how do they differ? (§4 — affinity/nodeSelector is the Pod pulling itself toward matching nodes; a taint is the node pushing Pods away unless they carry a matching toleration (e.g. GPU/control-plane nodes). One is opt-in attraction, the other default-deny repulsion; often used together.)
What are the three QoS classes and the eviction order under node memory pressure? (§5 — Guaranteed (requests==limits on all containers), Burstable (some requests, not equal), BestEffort (none). Under pressure the kubelet evicts BestEffort first, then Burstable (worst over-request first), Guaranteed last. requests==limits buys survival.)
How do anti-affinity and topologySpreadConstraints differ for spreading replicas? (§4 — required anti-affinity is binary "never co-locate" and can become unschedulable past one-per-node; topology spread bounds skew across a domain (maxSkew across zones/nodes), aiming for even distribution like 3/3/3 instead of just "not the same node.")