Part VI — Extension and synthesis

Extending Kubernetes, and the Whole Machine (Capstone)

Lesson 12 locked the cluster down: authentication, RBAC, admission, namespaces, quotas, and NetworkPolicy turned a wide-open multi-tenant box into one where every API call and every Pod-to-Pod connection is constrained. But all of that machinery still only governs the built-in nouns — Deployment, Service, ConfigMap, StatefulSet. Those nouns model a generic app. They have nothing to say about your domain: a database that needs ordered failover and nightly backups, a model rollout that must canary then promote. So today, to manage those, you are back to a human running runbooks by hand — which is precisely the manual ops we abolished in lesson 00. This final lesson closes that gap by teaching Kubernetes a new noun (Part A), and then steps back to trace the whole machine end to end and show that the thirteen primitives before it form a single derived chain (Part B).

1 · The forced fix: teach the API server a new noun

Return to the engine. Kubernetes works because intent is stored as a declarative object and a controller drives reality toward it. The whole reason a Deployment self-heals is that "I want 5 replicas of v2" is a first-class, queryable, watchable object the API server persists. Your domain intent — "I want a 3-node database, primary elected, replicas streaming" — has no such object. It cannot be stored, so it cannot be reconciled, so a human is the controller. To fix this the way Kubernetes fixes everything, you do not write a one-off script: you make your intent a declarative object the API server understands.

That is exactly what a Custom Resource Definition (CRD) is. A CRD is itself an object you kubectl apply; submitting it teaches the API server a brand-new kind — say Database — with a schema you define (an OpenAPI schema validating the fields). From that moment the API server treats your noun like any native one: kubectl get databases works, RBAC rules can grant verbs on it, admission webhooks can guard it, it gets stored in etcd, and clients can watch it. A single instance of the new kind is a custom resource (CR):

apiVersion: acme.io/v1          # your API group + version
kind: Database                  # the new noun the CRD registered
metadata: { name: orders-db }
spec:                           # desired state — what YOU want
  size: 3                       # number of database members
  version: "15"                 # engine version
status:                         # actual state — what the operator OBSERVES
  readyReplicas: 3
  primary: orders-db-0

Notice the shape is the universal Kubernetes shape from lesson 03: a spec (desired, written by the user) and a status (actual, written by the controller). The CRD gives you the noun and a place to store both halves of the loop. But a stored noun is inert — exactly like a declaration with no loop behind it (lesson 00, §3). The API server will happily persist a Database{size: 3} and do absolutely nothing about it. Something must watch this noun and close the gap. That something is the operator.

2 · The operator is just a controller over your noun

An operator is not a new mechanism. It is the controller from lessons 00 and 04 — the same observe → diff → act loop — pointed at a custom resource instead of a built-in one. It encodes the operational knowledge a human expert would otherwise apply by hand: how to bring a database up in order, elect a primary, fail over, take a backup. The pattern is so regular that frameworks (controller-runtime, scaffolded by kubebuilder or the Operator SDK) reduce writing one to filling in a single function. Its signature is the loop made literal:

// controller-runtime: called every time the watched object changes,
// AND periodically (re-sync) — because it is level-triggered (§00).
func (r *DatabaseReconciler) Reconcile(ctx, req) (Result, error) {
    var db Database
    r.Get(ctx, req.NamespacedName, &db)        // OBSERVE: read desired (spec)
    pods := r.listChildPods(db)                // OBSERVE: read actual (the world)
    diff := db.Spec.Size - len(pods)           // DIFF:    desired vs actual
    if diff > 0 { r.Create(... diff new Pods)  // ACT:     create the shortfall
    } else if diff < 0 { r.Delete(... -diff)   // ACT:     delete the surplus
    }
    return Result{RequeueAfter: 30 * time.Second}, nil   // REPEAT: forever
}

That is the entire idea. The reconciler returns, and controller-runtime calls it again on the next event or on the requeue timer. It is level-triggered (it re-reads desired and actual every call, so it never relies on catching a "Pod died" event) and idempotent (a call where diff == 0 does nothing) — the two properties from lesson 00 that make the loop self-heal, now yours to reuse. A real Database operator does more than count Pods, but every action is still a diff-and-act pass:

This is why the phrase "Kubernetes is a platform for building platforms" is precise, not marketing. The control-plane pattern — declarative noun in etcd, a watching loop that reconciles it — is generic and user-extensible. The built-in controllers (Deployment, ReplicaSet, StatefulSet, Job) are not privileged; they are simply the operators the Kubernetes authors shipped. You write your own with the same API, the same loop, the same guarantees. The manual runbook from the source-note becomes a Reconcile function, and the 3am page becomes a self-healing object.

3 · Drive the operator loop yourself

The widget below is the lesson-00 reconciliation loop, grown up. There it counted abstract Pods; here it is a real operator watching a custom Database resource. You edit one field — spec.size — and the operator reconciles by creating or deleting child StatefulSet Pods (db-0, db-1, …) to match. Watch that it is the same observe → diff → act pass: each tick it reads spec.size (desired) and counts live members (actual), computes the diff, and creates or deletes exactly enough to close it — newest ordinals removed first, just like a StatefulSet (lesson 09). Kill a member to simulate a node death and watch the operator recreate it on the next tick, exactly as the ReplicaSet did in lesson 04. The point of the widget is recognition: you have seen this loop before, in the very first lesson. Custom nouns did not add a new engine; they pointed the old engine at a new object.

That recognition is the hinge of the whole track. There was never a pile of unrelated features — there was one loop, applied to nouns the loop turned out to need. The next two sections make that claim literal by tracing the entire machine.

4 · Synthesis I — one kubectl apply, end to end

You type kubectl apply -f deployment.yaml. Follow the request through every component the track built, in order. Each hop is a lesson.

Twelve hops, and every one is a watch-driven, level-triggered reconcile against state in etcd. No component called another directly; each only watched the API server and acted on what it saw. That is the hub-and-spoke design from lesson 03 doing its job: the API server is the single source of truth, and a dozen independent loops converge the cluster on it. You declared a result; the machine computed the steps.

5 · Synthesis II — one external request, the map, and the chain

Now the other direction. An external user opens your app in a browser. Follow the HTTP request inward:

Put both directions together and the entire cluster is one picture — a control plane of stored truth plus watching loops, and worker nodes that run the actual Pods:

                         CONTROL PLANE (the brain)
   +-------------------------------------------------------------------+
   |   API SERVER  (only writer of etcd; authn -> authz -> admission)  |
   |        ^   ^        ^            ^               ^                 |
   |        |   |        |            |               |        +------+ |
   |   kubectl  |   scheduler   controllers      EndpointSlice |      | |
   |  (clients) |  (bind Pods)  (Deploy/RS/      controller -> | etcd | |
   |            |               StatefulSet/Job/             ^  |(truth| |
   |            |               YOUR OPERATOR)               |  +------+ |
   +------------|-------------------|--------------------------|--------+
                | watch/act         | bind                     | watch
                v                   v                          |
   ============ NODES (the hands) ===========================  |
   |  NODE A                         NODE B                 |  |
   |  kubelet  <-- reconciles Pods   kubelet                |  |
   |  containerd (CRI) runs:         containerd runs:       |  |
   |  [Pod: app  IP .11]  <--CNI-->  [Pod: app  IP .21]     |  |
   |  [Pod: db-0 IP .12]             [Pod: db-1 IP .22]     |  |
   |  kube-proxy (Service VIP rules) kube-proxy             |  |
   ==========================================================  |
            ^                                                  |
            |  external request: DNS -> Ingress -> Service VIP -> kube-proxy -> Pod
       [ user ] ------------------------------------------------+

And here is the payoff of the whole linear spine: every primitive exists because the previous one left a specific failure unanswered. Read this table top to bottom and you have re-derived Kubernetes — each row is forced by the row above.

The bridge to the sibling track. With this chain in hand, the entire Generative AI on Kubernetes track stops being a separate subject and becomes these exact primitives under one extra constraint: GPUs and their scarce HBM. A model server is a Pod (lesson 02) whose readiness probe (lesson 11) must wait for tens of gigabytes of weights to load; "schedule it" is the scheduler (lesson 10) filtering on a GPU resource and node taints (GPU nodes as a platform); autoscaling is the HPA (lesson 11) driven by KV-cache and TTFT signals instead of CPU; multi-tenant GPU fairness is RBAC + quota (lesson 12). The sibling track's opening lesson, Why GenAI Is Different on Kubernetes, even names the same failure modes — a CPU HPA that can't see the bottleneck, a readiness probe that lies — but now you can place each one on a row of the table above. Nothing there is new machinery; it is this machine, bound by HBM instead of CPU.

Failure modes & checklist

Checkpoint exercise

Where this points next

There is no lesson 15 — the chain closes here. The forced-derivation arc that began with a human typing ./server at 3am ends with that same human's runbook compiled into a Reconcile function the cluster runs forever. The honest next step is to re-read lesson 00 with new eyes: every noun it promised the loop would need — container, Pod, control plane, controllers, Services, scheduler — you have now built, and the operator you just wrote is literally the lesson-00 widget pointed at a custom object. From here, go wide instead of deeper: open the Generative AI on Kubernetes track and watch every one of these primitives reappear under GPU/HBM pressure — the scheduler placing model Pods on tainted GPU nodes, the HPA scaling on KV-cache instead of CPU, the readiness probe waiting on weights. You will not be learning a new system; you will be applying this one to the hardest workload it runs.

Interview prompts

• What problem does a CRD solve that a Deployment cannot? (§1 — it teaches the API server a new declarative noun (e.g. Database) with spec+status, so your domain intent becomes a stored, watchable, RBAC-able object that can be reconciled; otherwise domain knowledge lives only in a human runbook and nothing closes the gap.)
• How is an operator related to the controllers from earlier lessons? (§2 — it is the same observe → diff → act control loop (lessons 00, 04), level-triggered and idempotent, just pointed at a custom resource; Reconcile() reads spec, reads the world, diffs, and creates/deletes child objects. The built-in controllers are simply the operators Kubernetes shipped.)
• Why is "Kubernetes is a platform for building platforms" a precise claim? (§2 — the control-plane pattern (declarative object in etcd + a watching reconcile loop) is generic and user-extensible; Deployment/StatefulSet controllers aren't privileged, you write your own with the same API and the same guarantees.)
• Trace one kubectl apply -f deployment.yaml end to end. (§4 — kubectl → API server authn → authz/RBAC → admission → validate+write to etcd → Deployment controller creates a ReplicaSet → ReplicaSet controller creates Pods → scheduler binds Pods to nodes → kubelet pulls image and starts containers via CRI → CNI assigns Pod IP → readiness probe passes → EndpointSlice adds the Pod to its Service.)
• Trace one external HTTP request to a live Pod. (§5 — DNS resolves the hostname to the edge LB → Ingress/Gateway terminates TLS and routes by host/path to a Service → the Service ClusterIP (a stable VIP) → kube-proxy rewrites it to a real Pod IP from the EndpointSlice → a live Pod answers.)
• Pick any primitive and name the failure that forced it. (§5 recap table — e.g. the Service is forced because rolling Pods get new IPs so clients can't hold a moving target; the scheduler is forced because a freshly created Pod is unbound; the CRD/operator is forced because built-in nouns don't model domain objects. Each row is the answer to the row above.)
• How does the GenAI track map onto this one? (Bridge — it is the same primitives under scarce GPU/HBM: model server = Pod whose readiness waits on weight load; GPU placement = scheduler filtering on a GPU resource + node taints; serving autoscale = HPA on KV-cache/TTFT not CPU; tenant fairness = RBAC + quota. Nothing new, just this machine bound by HBM instead of CPU.)