Part V — How the cluster decides and defends

Security & Multi-Tenancy: Who Can Do What, What Can Reach What

Lesson 11 closed the last reality gap: probes tell the loop whether a process is truly healthy (liveness restarts the dead, readiness pulls the un-ready from a Service's endpoints), and three nested autoscalers — HPA on replicas, VPA on requests, Cluster Autoscaler on nodes — make even the replica count and the node pool reconciled quantities. The cluster now self-heals and self-sizes, so you put many teams' workloads on it. And there your luck runs out. Nothing constrains any of them. Every call to the all-powerful API server is accepted, and every Pod can open a socket to every other Pod. This lesson derives the two planes of defense — who may touch the API, and what may reach what on the network — that turn an open cluster into a multi-tenant one.

The previous step left this broken

Everything we built routes through one supremely powerful door: the API server is the only writer to etcd (lesson 03), so anyone who can talk to it can create a privileged Pod, read every Secret, or delete every Deployment in the cluster. Yet so far that door is wide open — there is no notion of who is calling or what they are allowed to do. And recall the two specific holes we noted and moved past: lesson 06's network model is flat and any-to-any (every Pod can dial every Pod by IP, with no NAT and no filter), and lesson 08's Secrets are merely base64-encoded in etcd — readable by anyone who can read the object or the disk. A single compromised Pod, or one careless intern's kubeconfig, can today own the whole cluster. With many tenants sharing one cluster, "no boundaries" is no longer tolerable.

Linear position

Forced by: one cluster now runs many teams, but nothing limits API access or Pod-to-Pod traffic — the powerful API server accepts every request and the flat network from lesson 06 lets any Pod reach any Pod.
New capability: two planes of control — the API request lifecycle (authentication → authorization via RBAC → admission) plus Namespaces, ResourceQuota and LimitRange for tenancy; and runtime/network defense via NetworkPolicy default-deny, Secret encryption-at-rest, and supply-chain checks.
Forces next: all of this still only governs Kubernetes' generic nouns; your domain objects (a database with failover, a model rollout) have no controller, so a human still runs the 3am runbook — the manual ops lesson 00 abolished.

The plan

Six moves. (1) Frame the two planes — API access vs runtime/network. (2) Walk the API request lifecycle: authentication (who are you). (3) Authorization with RBAC: Roles grant verbs on resources, RoleBindings attach them to subjects, ServiceAccounts are workload identities — and the default-SA trap. (4) Admission: validating/mutating webhooks, Pod Security Standards, policy engines. (5) Tenancy: Namespaces, ResourceQuota, LimitRange. (6) Network: turn the flat any-to-any model into default-deny segmentation with NetworkPolicy, plus Secrets-at-rest and supply chain. Then drive the RBAC widget, failure modes, and the hand-off.

1 · Two planes: the API door and the network fabric

Every attack surface in a cluster collapses into two questions, and they have two completely separate answers. The first is the control plane question: who is allowed to ask the API server to do what? Creating a Pod, listing Secrets, deleting a Deployment — all of these are HTTP requests to the one API server, and the defense is a gauntlet that request must pass before it touches etcd. The second is the data plane (runtime) question: once Pods are running, what may reach what? A compromised front-end Pod that can open a socket to the database Pod is a breach even if its ServiceAccount has zero API rights — the network is a separate door. Keep these planes distinct; conflating them is how people build clusters with airtight RBAC and a flat, wide-open network (or vice versa).

Plane A — API access control

Governs requests to the API server. A request runs a gauntlet: authentication (who are you), then authorization (are you allowed this verb on this resource — RBAC), then admission (does the object satisfy policy). Only then is it written to etcd. This is §2–§5.

Plane B — runtime / network

Governs running Pods: who can reach whom on the network (NetworkPolicy, §6), what a container may do on its host (Pod Security Standards, §4), whether Secrets are encrypted at rest, and whether the image was tampered with. RBAC never sees a Pod-to-Pod packet — this plane does. This is §4 and §6.

2 · The API request lifecycle, step 1: authentication

Picture the journey of one kubectl create pod. It is an HTTPS request to the API server, and before that server will even consider it, the request passes through three checkpoints in order. Any checkpoint can reject; only a request that clears all three is persisted to etcd.

1Authentication → who are you? The API server establishes an identity (a username and group memberships) from the request's credentials, or rejects it as anonymous. It does not yet decide what you may do.

2Authorization → are you allowed this? Given the identity, may this subject perform this verb (create) on this resource (pods) in this namespace? RBAC answers. Default answer: no.

3Admission → is the object acceptable? Even an authorized create can be mutated (defaults injected) or rejected (violates policy) by admission controllers before it lands in etcd.

Authentication answers only "who," never "what." Kubernetes has no built-in user database; it trusts external proofs of identity, of which there are two broad families:

Who is calling	Credential	How the API server verifies it
Humans (operators, CI)	A client TLS certificate (the CN/O fields carry username/groups), a static token, or — in production — an OIDC token (OpenID Connect: an external identity provider like Okta/Entra/Google issues a signed JWT the API server validates).	The cert is signed by the cluster CA; the OIDC token's signature is checked against the provider's public keys. The identity is derived from the token's claims.
Workloads (Pods)	A ServiceAccount token — a short-lived signed JWT automatically mounted into the Pod at /var/run/secrets/.../token. The Pod presents it on every API call.	The API server validates the token's signature (it issued it) and maps it to the identity system:serviceaccount:<ns>:<name>.

The key insight for the rest of the lesson: a ServiceAccount (SA) is a workload's identity. When a Pod talks to the API server (to watch its config, update its own status, etc.), it does so as its ServiceAccount. That identity is a first-class subject in authorization — which is exactly where the danger and the discipline live.

3 · Step 2: authorization with RBAC

Once the API server knows who, it must decide what. The mechanism is RBAC (Role-Based Access Control), and it is built from four nouns that compose with the same loose-coupling instinct as everything else in Kubernetes — you grant capabilities, then attach them to identities, separately.

Role

A namespaced set of permissions: a list of rules, each granting verbs (get, list, watch, create, update, delete) on resources (pods, secrets, deployments). A Role lives in one namespace and grants only there.

ClusterRole

The same, but cluster-scoped: usable in any namespace, and the only way to grant verbs on cluster-scoped resources (nodes, namespaces themselves) or across all namespaces at once.

RoleBinding

Attaches a Role (or ClusterRole) to subjects (users, groups, ServiceAccounts) within one namespace. This is what actually grants access. No binding = no access.

ClusterRoleBinding

Attaches a ClusterRole to subjects cluster-wide, in every namespace. Powerful and easy to over-grant — the usual source of accidental cluster-admin.

The model is purely additive and default-deny: a subject can do nothing until some binding grants it, and there are no "deny" rules — you simply never grant what you do not want. To evaluate a request, RBAC asks: is there any rule, reachable through a binding that applies to this subject in this namespace, whose verbs include the requested verb and whose resources include the requested resource? If yes → ALLOW. If no rule matches → FORBIDDEN.

The scoping rule is the one people trip on and the one the widget below makes vivid: a RoleBinding grants only inside its own namespace. Bind the pod-reader Role to ci-bot in namespace team-a, and ci-bot can list pods in team-a — and is still completely forbidden from listing pods in team-b. The same Role would have to be bound again, separately, in team-b. This namespace boundary is what makes RBAC a tenancy tool, not just a permissions list.

The default-ServiceAccount trap

Every namespace ships a ServiceAccount named default, and every Pod that does not name one is silently assigned it. On its own that SA has almost no rights — fine. The trap is the reflex of "my Pod can't reach the API, let me just bind it cluster-admin." Now a single compromised container in that Pod can read every Secret and delete every workload in the cluster. The discipline (least privilege): give each workload its own SA, grant it the minimum Role it needs, and disable token automounting on Pods that never call the API at all (automountServiceAccountToken: false).

4 · Step 3: admission — the policy gate

A request can be authenticated and authorized and still be dangerous: RBAC only checks "may you create a Pod," not "is this a privileged Pod that mounts the host filesystem." That second question belongs to admission controllers — plugins and webhooks the API server consults after authorization but before the object is persisted. They come in two flavors, run in this order:

→Mutating admission → may change the incoming object: inject a sidecar, set a default resource request, add a label. Runs first, so validation sees the final object.

→Validating admission → may only accept or reject: "this Pod runs as root — denied," "this image is from an untrusted registry — denied." Cannot modify.

The most important built-in policy is Pod Security Standards (PSS), enforced per-namespace by a label. It defines three tiers: privileged (no restrictions — for trusted infra workloads), baseline (blocks the obviously dangerous: host namespaces, privileged containers), and restricted (hardened: must run as non-root, drop all Linux capabilities, read-only root filesystem). Labeling a namespace pod-security.kubernetes.io/enforce: restricted means the API server rejects any Pod that would, say, run as root — at admission, before it ever schedules. For richer, custom rules ("every image must be from registry.corp/," "every Pod must carry a cost-center label"), teams add a general-purpose policy engine as a validating webhook — OPA/Gatekeeper (policies in the Rego language) or Kyverno (policies written as Kubernetes YAML). These are how an org expresses guardrails the built-in nouns cannot.

5 · Namespaces, ResourceQuota, and LimitRange — the tenancy boundary

RBAC scopes permissions per namespace; the Namespace is therefore the natural unit of tenancy — a soft partition of the one cluster into named slices (team-a, team-b, prod, staging) within which names are unique and most objects live. But a namespace boundary alone does not stop team-a from launching 10,000 Pods and starving everyone else's scheduling (recall lesson 10: requests gate placement, so one greedy tenant can exhaust the node pool). Two objects cap a tenant's footprint:

Object	What it caps	Worked example
ResourceQuota	A namespace's total consumption: summed CPU/memory requests and limits, and counts of objects (Pods, Services, PVCs, Secrets).	Quota requests.cpu: 50, requests.memory: 100Gi, pods: 200 on team-a: the 201st Pod, or the one that would push summed requests past 50 cores, is rejected at admission.
LimitRange	Per-Pod / per-container defaults and bounds: a default request/limit if the author set none, plus min/max ceilings.	Default cpu: 100m, max cpu: 2 per container: a Pod with no request gets 100m (so it counts against the quota); one asking for 4 cores is rejected.

Together they make a namespace a budgeted tenant: LimitRange ensures every Pod has a request (so quota accounting works and the scheduler can place it), and ResourceQuota caps the sum. Note this is soft multi-tenancy — tenants still share the same control plane, kernel, and nodes; for hostile tenants you reach for separate clusters or virtual-control-plane tooling. For internal teams, namespaces + RBAC + quotas are the standard, pragmatic boundary.

6 · The network plane: NetworkPolicy turns flat into default-deny

All of §2–§5 governs the API door. None of it touches a packet. By default, the lesson-06 network model is fully permissive: every Pod can open a connection to every other Pod, in any namespace, with no filtering. So a compromised marketing-site Pod can dial the payments database directly — RBAC never sees it, because no API call is made. The data-plane answer is the NetworkPolicy: a namespaced object that allows traffic to/from Pods selected by — the same tool yet again — a label selector (lesson 04).

The crucial mechanic is the switch from allow-all to deny-all. NetworkPolicies are additive allow rules over an implicit deny, but the implicit deny only activates for a Pod once at least one policy selects it. So the standard hardening move is to apply a catch-all default-deny policy to a namespace, then add narrow allow rules:

  BEFORE (lesson 06 default)          AFTER default-deny + allow rules
  ──────────────────────────          ────────────────────────────────
  any ──► any   (everything open)     [default-deny: all Pods]  ── drops everything
                                         + allow ingress: from app=web  ─► to app=api  :8080
                                         + allow egress:  from app=api  ─► to app=db   :5432
  web ─► db   ALLOWED  (bad!)         web ─► db    DENIED   (no rule permits it)
  web ─► api  ALLOWED                 web ─► api   ALLOWED  (matched ingress rule)
  api ─► db   ALLOWED                 api ─► db    ALLOWED  (matched egress rule)

A policy has ingress rules (who may connect to the selected Pods) and egress rules (where the selected Pods may connect out), each scoping the other end by podSelector, namespaceSelector, or IP block, plus ports. This is least privilege for the network: only the edges you explicitly draw exist. One hard caveat — a NetworkPolicy is inert unless the CNI plugin enforces it. The API server happily stores the object, but a CNI that only meets the basic flat-network contract (some Flannel setups) will not act on it; you need an enforcing CNI such as Calico or Cilium (lesson 06's choice now has teeth). Writing default-deny policies on a non-enforcing CNI gives a dangerous illusion of segmentation.

Two more runtime concerns round out plane B. Secrets at rest: lesson 08's Secrets are base64, not encrypted — anyone reading etcd or its disk reads them in clear. Turn on encryption-at-rest (the API server encrypts Secret values before writing etcd, ideally with keys in an external KMS) and lock down read access with RBAC; many teams go further with an external secret store (Vault, cloud secret managers). Supply chain: a Pod runs whatever its image contains, so verify what you run — pull by immutable digest, scan images for CVEs, and require signed images (an admission webhook checks the signature) so a tampered or unknown image never schedules.

7 · Widget: an RBAC allow/deny evaluator

RBAC evaluator — does this request clear authorization?

Pick a subject, a verb, a resource, and a namespace. The evaluator checks the request against the fixed Roles + RoleBindings shown below (default-deny, additive). It outputs ALLOW or FORBIDDEN and why — which rule matched, or that no rule grants it. Try ci-bot listing pods in team-a (allowed) vs. the same in team-b (forbidden — the binding is namespace-scoped). Try anyone get secrets (only deployer in team-a is granted). Note admin works everywhere — its binding is cluster-wide.

subject verb resource namespace

—

Pick a request above.

Request

—

Rules checked

—

Result

—

Failure modes & checklist

Failure modes

Cluster-admin to a workload. Binding a Pod's ServiceAccount to cluster-admin "to make it work." One RCE in that container now owns the cluster. Signal: a ClusterRoleBinding to a namespaced SA; kubectl auth can-i --list --as=system:serviceaccount:ns:sa shows * on *.
Relying on the default ServiceAccount. Pods inherit the namespace default SA and its mounted token even when they never call the API — needless attack surface. Signal: a leaked default-SA token can still authenticate; Pods carry a token they never use.
NetworkPolicy with no enforcing CNI. Authoring default-deny policies on Flannel/basic CNI; the object exists but nothing filters. Signal: a "blocked" Pod still curls the database; kubectl get networkpolicy lists rules that demonstrably do nothing.
Forgetting egress. A default-deny that only restricts ingress; a compromised Pod still exfiltrates data outbound. Signal: unexpected egress to the internet from a Pod that should only talk to one internal Service.
Treating Secrets as encrypted. Assuming base64 protects anything. Signal: etcdctl get or a node-disk read returns Secret values in clear; no encryption-at-rest configured.
No quota on shared namespaces. One team's runaway Job fills the node pool. Signal: other tenants' Pods stuck Pending for "Insufficient cpu" though they fit individually.

Checklist

One ServiceAccount per workload, granted the minimum Role; never bind cluster-admin to a workload.
Disable token automount (automountServiceAccountToken: false) on Pods that never call the API.
Prefer namespaced Role + RoleBinding over ClusterRole/ClusterRoleBinding; reach cluster-wide only when truly needed.
Apply a default-deny NetworkPolicy per namespace, then add narrow ingress and egress allows — and confirm the CNI (Calico/Cilium) enforces them.
Enforce Pod Security Standards (restricted where possible) and add OPA/Gatekeeper or Kyverno for org-specific guardrails.
Turn on Secret encryption-at-rest (KMS-backed) and gate Secret reads with RBAC; consider an external secret store.
Give each tenant a Namespace with ResourceQuota + LimitRange so no team starves the others.
Verify the supply chain: pin images by digest, scan for CVEs, require signatures via admission.

Checkpoint exercise

Try it

Open the widget. (1) Set ci-bot · list · pods · team-a and read the verdict — which rule matched? (2) Change only the namespace to team-b and observe it flips to FORBIDDEN; explain in one sentence why a RoleBinding in team-a grants nothing in team-b. (3) Try ci-bot · get · secrets · team-a (forbidden — pod-reader does not include secrets) then switch the subject to deployer (allowed). (4) Pick admin with any verb/resource/namespace and confirm it is always allowed; what kind of binding makes that true, and why is it the most dangerous object in the cluster? (5) Now write, in words, the second-plane defense you'd add so that even admin's pods cannot reach the db Pod from a web Pod.

Where this points next

The cluster is now multi-tenant: a request must clear authentication, RBAC authorization, and admission to touch the API; namespaces with quotas budget each team; and a default-deny NetworkPolicy plus encrypted Secrets harden the runtime. But notice what every one of these controls governs — Kubernetes' built-in, generic nouns: Pods, Deployments, Services, Secrets. None of them understands your domain. When a clustered database needs a failover, or a model rollout needs a canary-then-promote dance, there is no controller watching that — so a human reads a wiki runbook at 3am and runs the steps by hand. That is precisely the manual, command-issuing ops that lesson 00 set out to abolish, leaking back in at the top of the stack. The fix is to teach the API server a new noun and write a controller for it. Lesson 13, Extending Kubernetes, and the Whole Machine (Capstone), builds Custom Resource Definitions and operators — a custom controller running the exact reconcile loop from lessons 00 and 04 — then traces one kubectl apply end to end and maps every primitive to the failure it answered.

Takeaway

An open cluster shared by many teams has two doors, and each needs its own lock. Plane A (API access) is a gauntlet every request to the all-powerful API server must pass before reaching etcd: authentication (who are you — certs, OIDC tokens for humans, auto-mounted ServiceAccount tokens for workloads), then authorization via RBAC (Roles/ClusterRoles grant verbs like get/list/create on resources; RoleBindings/ClusterRoleBindings attach them to subjects; default-deny and additive; a RoleBinding grants only inside its own namespace — least privilege, and never bind cluster-admin to a workload's SA), then admission (mutating then validating webhooks, Pod Security Standards, OPA/Kyverno policy). Namespaces are the tenancy boundary; ResourceQuota caps a namespace's total CPU/mem/objects and LimitRange sets per-Pod defaults/bounds. Plane B (runtime/network) closes the holes RBAC can't see: a NetworkPolicy turns lesson 06's flat any-to-any fabric into default-deny segmentation by label selector (ingress and egress), but only if the CNI enforces it; Secrets need encryption-at-rest (base64 is not encryption); and the image supply chain must be verified by digest, scan, and signature. All of this still governs only generic nouns — your domain objects still need a controller, which is the operator, next.

Interview prompts

Walk the three checkpoints an API request passes. (§2 — authentication (establish identity from cert/token/OIDC), then authorization (RBAC: may this subject do this verb on this resource in this namespace), then admission (mutate/validate the object against policy); all three must pass before etcd is written.)
What is a ServiceAccount and what is the default-SA trap? (§2–§3 — an SA is a workload's identity; its auto-mounted token authenticates a Pod's API calls. The trap is binding the namespace default SA (which every un-specified Pod inherits) to broad/cluster-admin rights, so one compromised container owns the cluster; fix with one minimal-Role SA per workload and disabling token automount.)
Explain RBAC's four nouns and how a decision is made. (§3 — Role/ClusterRole list verbs on resources (namespaced vs cluster-scoped); RoleBinding/ClusterRoleBinding attach them to subjects (namespaced vs cluster-wide). Default-deny + additive: ALLOW iff some reachable rule's verbs and resources both match the request, else FORBIDDEN.)
Why does a Role bound in namespace A not grant access in namespace B? (§3, widget — a RoleBinding's grant is scoped to its own namespace; the same Role must be bound again in B. This namespace scoping is what makes RBAC a tenancy boundary, not a global ACL.)
What does admission add that authorization cannot? (§4 — RBAC only checks the verb/resource, not the object's contents. Mutating webhooks inject defaults/sidecars; validating webhooks + Pod Security Standards reject dangerous objects (privileged Pods, root, untrusted images); OPA/Gatekeeper and Kyverno express custom org policy.)
How does NetworkPolicy change lesson 06's network, and what's the caveat? (§6 — it turns the flat any-to-any fabric into default-deny: once a policy selects a Pod, only explicitly allowed ingress/egress (by label selector) is permitted. Caveat: it is inert unless the CNI (Calico/Cilium) enforces it — a basic CNI stores but ignores it.)
Why are Kubernetes Secrets not actually secret, and how do you fix it? (§6 — Secrets are base64-encoded, not encrypted, so anyone reading etcd or its disk reads them in clear. Enable encryption-at-rest (ideally KMS-backed), restrict reads via RBAC, and consider an external secret store.)
How do ResourceQuota and LimitRange differ? (§5 — ResourceQuota caps a namespace's total summed CPU/mem requests/limits and object counts; LimitRange sets per-Pod/container defaults and min/max bounds so every Pod has a request the quota can account for and the scheduler can place.)