Part III — Reaching the workload

Networking & Services: Flat IPs and a Stable Front

Lesson 05 made change safe: a Deployment ships v2 by spinning up a new ReplicaSet and shifting replicas across under a rolling strategy, so capacity never dips below your floor and a bad release rolls back in one command. But that safety has a cost we waved past — every rollout, scale event, and crash replaces Pods, and a fresh Pod gets a fresh IP. The IP a client connected to a minute ago now belongs to nothing. Worse, two Pods on two different machines have no agreed way to even find each other. This lesson builds the two layers that make a Pod reachable despite all that churn: a flat network where every Pod is routable, and a stable virtual front that hides the moving set behind one address.

The previous step left this broken

A Deployment's whole job is to keep replacing Pods — that is how a rolling update, a self-heal, or a scale-down works. So a Pod IP is ephemeral: it lives and dies with that one Pod and can never be the thing a caller holds. Point a client at 10.1.4.7 and the next roll leaves it talking to an address that points to no process. And before we even get to "which IP," there is a deeper gap: a Pod on node A and a Pod on node B share no network — the Docker-era reflex of mapping container port 8080 to host port 8080 does not compose (who gets host 8080 when ten Pods want it?), and NAT between nodes makes addresses lie. We need both a way for any Pod to reach any Pod, and a way for a client to reach a service without ever naming a Pod.

Linear position

Forced by: rolling Deployments make Pod IPs ephemeral, so no client can hold one; and Pods on different nodes have no shared, composable network.
New capability: a flat cluster network (every Pod gets a routable IP, any-to-any with no NAT, implemented by a CNI plugin) plus the Service — a stable virtual IP that load-balances across the live, label-selected set of Pod IPs, programmed into every node by kube-proxy.
Forces next: a ClusterIP is stable but still an opaque number you would hardcode, and external users need one named, L7, TLS-terminating door — not a cloud load balancer per Service.

The plan

Five moves. (1) State the two networking problems rolling Pods leave behind. (2) Derive the Kubernetes network model — one routable IP per Pod, no NAT — and how a CNI plugin makes it real (overlay vs routed). (3) Derive the Service: a stable ClusterIP backed by the label-selected EndpointSlice, with kube-proxy doing the packet rewriting on every node. (4) The four Service types — ClusterIP, NodePort, LoadBalancer, headless — and when each fits. (5) Drive the widget: watch the VIP stay constant while endpoints churn, then failure modes, checklist, and the hand-off to DNS and Ingress.

1 · The two problems rolling Pods leave behind

Lesson 05 gave us safe change, but change is replacement. Picture a 3-replica Deployment behind nothing. The Pods get IPs 10.1.4.7, 10.1.5.2, 10.1.6.9. A client caches one. You ship v2: the rolling update deletes those three Pods and creates three new ones, which land on whatever nodes have room and get whatever IPs the network hands out — say 10.1.4.11, 10.1.5.8, 10.1.7.3. The client's cached address now routes to a Pod that no longer exists. This is not an edge case; it is the normal operation of every primitive we have built. A Pod IP is a fact about one mortal Pod, never a contract.

The second problem is more primitive and easy to miss because Docker hid it. On a single Docker host you reach a container by mapping its port onto a host port: -p 8080:8080. That trick does not survive a cluster. There is exactly one host port 8080 per node, so ten Pods that all want it cannot coexist; and if Pod A on node-1 wants to call Pod B on node-2, host-port mapping plus NAT (network address translation — rewriting source/destination addresses at a boundary) means B sees the connection coming from node-1's address, not A's. Identities get smeared, return paths break, and policy can't tell who is really calling whom. Host port-mapping does not compose. We need a model where addresses are honest and abundant.

2 · The Kubernetes network model: a flat, routable fabric

Kubernetes refuses the host-port mess by stating three rules every conforming cluster must satisfy. They are deliberately strong:

One IP per Pod

Every Pod gets its own cluster-routable IP address — not a shared host IP with port juggling. All containers in the Pod share that one IP (recall lesson 02: a Pod is one network namespace), and talk to each other over localhost.

Any Pod reaches any Pod

A Pod on any node can open a connection to a Pod on any other node using that Pod's IP directly — across the whole cluster, flat, no gateways to hop.

No NAT between Pods

The destination sees the real source Pod IP, not a node's rewritten address. The address a Pod sends from is the address its peer sees. Identity is preserved end to end.

Pods see their own IP

A Pod's view of its own address matches what others use to reach it. No surprises from a hidden translation layer — what you bind is what peers dial.

This is the flat network: conceptually one giant L3 (layer-3, IP-routing) space where Pod IPs are first-class and globally meaningful inside the cluster. Kubernetes itself does not implement this — it specifies it, then delegates to a CNI plugin (Container Network Interface — the pluggable contract the kubelet calls when a Pod is created, to attach it to the network and hand it an IP). Picking a CNI plugin (Calico, Cilium, Flannel, the cloud's own) is how a cluster meets the model, and there are two broad strategies:

Strategy	How a packet crosses nodes	Trade-off
Overlay (e.g. VXLAN)	The source node wraps the Pod-to-Pod packet inside an outer packet addressed node-to-node (encapsulation), ships it over whatever underlying network exists, and the destination node unwraps it.	Works on any network with no router changes — easy, portable. Costs a little per-packet overhead and a few bytes of MTU for the wrapper.
Routed (e.g. BGP)	No wrapping. The network is taught real routes to each node's Pod IP range (often by advertising them with BGP), so Pod IPs are natively routable.	Native speed, real IPs on the wire, easier to debug. Needs control of the underlying routers/fabric — harder in arbitrary environments.

Either way, the result the rest of Kubernetes assumes is identical: a Pod IP you can dial from anywhere in the cluster with no NAT. That solves "Pods on different nodes can talk." It does not solve "the IP keeps changing." For that we need a stable thing in front of the moving set.

3 · The Service: a stable VIP over a live endpoint set

A Service is a named, stable address for a set of Pods, decoupled from any individual Pod's life. You define it with the exact tool from lesson 04 — a label selector. The Service says "I front every Pod matching app=web," and Kubernetes maintains the rest. Three objects cooperate:

→Service → you create it with a selector (app=web) and a port. Kubernetes assigns it a ClusterIP — a stable virtual IP from a reserved range (e.g. 10.96.0.0/12) that lasts the Service's whole life, immune to Pod churn.

→EndpointSlice → a controller continuously watches which Pods match the selector and are Ready, and writes their real Pod IPs into one or more EndpointSlice objects. This is the live membership list: scale up, an IP is added; a Pod dies or goes un-Ready, its IP is removed.

→kube-proxy → an agent on every node that watches Services and EndpointSlices and programs the node's kernel so that any packet sent to the ClusterIP is rewritten to one of the live endpoint Pod IPs, load-balanced across them.

The crucial, counter-intuitive fact: the ClusterIP is virtual — no process anywhere listens on it. There is no box at 10.96.0.10. When a Pod sends a packet to the ClusterIP, kube-proxy's rules (installed in the kernel's packet path) catch it and rewrite the destination to a randomly-or-round-robin-chosen live endpoint, e.g. 10.1.5.8:8080. The Service is a load-balancing rule baked into every node, not a server. That is why it has zero startup cost and no single point to crash: it is the same kernel doing the routing the packet was already going through.

  client Pod                Service (virtual)            live endpoints (EndpointSlice)
  10.1.9.4  ── dial ──►  ClusterIP 10.96.0.10  ──►   10.1.4.11:8080   (Ready)
                          (no process here;            10.1.5.8 :8080   (Ready)
                           kube-proxy rewrites          10.1.7.3 :8080   (Ready)
                           dest to a live Pod)          10.1.6.2 :8080   (NotReady — excluded)

  Pods come and go below; ClusterIP 10.96.0.10 never changes.

kube-proxy has two common backends. iptables mode installs Linux netfilter rules — simple and ubiquitous, but rule evaluation is roughly linear, so very large Services (thousands of endpoints) get slow to update. IPVS mode uses the kernel's in-built load balancer with hash-table lookups — near-constant time and real LB algorithms (round-robin, least-conn), better at large scale. Both achieve the same contract; IPVS just scales the data path further.

4 · The four Service types — internal to external

Every Service is, underneath, a ClusterIP. The types layer reachability outward from there:

Type	What it exposes	Reach	Use when
ClusterIP (default)	One stable virtual IP, in-cluster only.	Pod → Pod inside the cluster.	Service-to-service traffic. The other types build on it.
NodePort	The same ClusterIP, plus a fixed port (e.g. 30000–32767) opened on every node's IP.	Anything that can reach a node IP:port.	Quick external access, on-prem, or as the target a real load balancer points at.
LoadBalancer	A NodePort plus a cloud LB the cloud-controller provisions, with one external IP fronting the nodes.	The public internet, one external IP per Service.	Exposing a Service externally on a cloud. Costs one cloud LB each — the pain that motivates lesson 07.
Headless (clusterIP: None)	No VIP at all; DNS returns the Pod IPs directly, one record per Ready endpoint.	Client picks/addresses individual Pods.	Stateful sets and clients that must reach a specific Pod (e.g. a database replica) — covered in lesson 09.

Notice the headless case turns the load balancer off on purpose: when each Pod has a distinct identity (a primary database vs. a replica), you do not want a VIP scrambling you to a random one — you want to address db-0 by name. Everything else (ClusterIP, NodePort, LoadBalancer) shares the same VIP-over-EndpointSlice machinery from §3; only the outer edge differs. And note one tie-back to lesson 11: a Pod only joins the EndpointSlice once its readiness probe passes — readiness is precisely the signal that gates Service membership, which is why a still-warming Pod does not get traffic.

5 · Widget: the VIP stays put while endpoints churn

Service VIP → live EndpointSlice

The ClusterIP box is fixed — it is the address a client holds forever. Scale Pods up/down or kill one, and watch the EndpointSlice (the live membership) change underneath while the VIP never moves. Hit "Send 8 requests" to watch kube-proxy round-robin across only the live endpoints; a killed Pod drops out of rotation immediately. This is exactly why a client can hold the VIP through any rollout.

Service web selector app=web

ClusterIP 10.96.0.10 : 80 (virtual — no process listens here)

EndpointSlice (live, Ready Pods only):

ClusterIP (constant)

10.96.0.10

Live endpoints

—

Last request landed on

—

Dropped (dead Pod)

Failure modes & checklist

Failure modes

Hardcoding a Pod IP. Caching 10.1.4.7 in a client or config. The next rollout replaces that Pod and the address points to nothing. Signal: connections that worked yesterday now time out right after a deploy or scale event.
Selector / label mismatch. The Service selector is app=web but the Pods are labeled app=webserver. The EndpointSlice is empty, the VIP exists but routes to no one. Signal: the Service has a ClusterIP but every request gets "connection refused"; kubectl get endpointslices shows zero addresses.
No readiness probe (or a lying one). Pods join the EndpointSlice the instant they are "running," before they can actually serve. Signal: a burst of 5xx right after each scale-up, traffic hitting a Pod whose app is still booting.
LoadBalancer per Service. Giving every microservice type: LoadBalancer — each provisions its own cloud LB and external IP. Signal: a surprising cloud bill and a sprawl of public IPs; this is the exact pain lesson 07 fixes.
Expecting something to listen on the ClusterIP. Trying to ping or bind the VIP, debugging "why is nothing on 10.96.0.10." Signal: ping fails yet curl works — because the VIP is a rewrite rule, not a host.

Checklist

Talk to Services, never to Pod IPs — let the VIP absorb all Pod churn.
Verify the selector matches the Pod labels and that kubectl get endpointslices lists the IPs you expect.
Define a readiness probe on every Pod that backs a Service, so only serving Pods enter rotation (deep dive in lesson 11).
Default to ClusterIP; reach for NodePort/LoadBalancer only at the real edge, and prefer one shared edge (lesson 07) over a LoadBalancer per Service.
Use headless only when clients must address individual, identity-bearing Pods (StatefulSets, lesson 09).
Pick a CNI plugin that meets the flat-network model and matches your environment (overlay for portability, routed/BGP for native performance).

Checkpoint exercise

Try it

Open the widget with 4 live Pods. Send 8 requests and note the round-robin spread (each live Pod should take roughly 2). Now kill one Pod and immediately send 8 more — confirm (a) the ClusterIP in the KPI never changed, (b) the killed Pod takes zero requests, and (c) the other three now split the 8. Then scale down to 0 endpoints and send a request: what happens, and what does that tell you about a Service whose selector matches no Ready Pods? Finally, in words: explain why a client that held the ClusterIP through all of this never had to change anything, while a client holding a Pod IP would have broken at the first kill.

Where this points next

The Service fixed the churn problem: a stable VIP over a live, self-healing endpoint set, with no NAT thanks to the flat network beneath it. But it left two new gaps. First, a ClusterIP like 10.96.0.10 is stable yet still an opaque number — hardcoding it is only marginally better than hardcoding a Pod IP, and it does not survive recreating the Service. Workloads should discover each other by name. Second, the only built-in way to expose many Services to the outside world is a LoadBalancer per Service — one cloud LB, one external IP, one bill, each — with no shared hostname, path routing, or TLS termination. External users need a single, named, L7 front door. Lesson 07, DNS, Ingress & the Front Door, gives Services names with CoreDNS and puts one L7 reverse proxy at the edge (Ingress / the Gateway API) so many Services share one external entry point with host/path routing and TLS.

Takeaway

Rolling Deployments make Pod IPs ephemeral, so a client can never hold one; and Pods on different nodes have no composable shared network. Kubernetes answers in two layers. The network model demands every Pod get its own cluster-routable IP and that any Pod reach any Pod with no NAT — a flat fabric the CNI plugin implements via an overlay (VXLAN, encapsulate node-to-node) or a routed approach (BGP, native routes). On top of that, the Service gives a stable virtual IP — the ClusterIP — over the live, label-selected set of Pod IPs recorded in the EndpointSlice, and kube-proxy programs every node's kernel (iptables or IPVS) to rewrite a packet sent to the VIP onto a live endpoint. The VIP is virtual: no process listens on it; it is a load-balancing rule, which is why it is stable and free. Types layer outward — ClusterIP (internal), NodePort (per-node port), LoadBalancer (a cloud LB each, the cost that forces lesson 07), and headless (DNS straight to Pods). The selector is the same label query from lesson 04, and readiness (lesson 11) gates membership — so the moving set churns underneath while the one address out front never moves.

Interview prompts

Why can't a client just connect to a Pod IP? (§1 — Pod IPs are ephemeral; rolling updates, self-heal, and scaling all replace Pods and reassign IPs, so a held Pod IP routes to nothing after the next change.)
State the Kubernetes network model in one breath. (§2 — every Pod gets its own cluster-routable IP; any Pod can reach any Pod by that IP with no NAT; a Pod sees the same IP others use for it. Kubernetes specifies it; a CNI plugin implements it.)
Overlay vs. routed CNI — what's the trade-off? (§2 — overlay (VXLAN) encapsulates Pod packets node-to-node so it works on any network but adds per-packet/MTU overhead; routed (BGP) advertises real routes for native speed and real on-wire IPs but needs control of the underlying fabric.)
What is a ClusterIP and what actually listens on it? (§3 — a stable virtual IP for a Service; nothing listens on it. kube-proxy installs kernel rules on every node that rewrite a packet destined for the VIP onto a live endpoint Pod IP — it is a load-balancing rule, not a server.)
How does a Service know which Pods to send to, and what keeps it current? (§3 — a label selector (same model as lesson 04) plus the EndpointSlice controller, which continuously writes the IPs of matching, Ready Pods; un-Ready or deleted Pods drop out so the VIP only fans out to live endpoints.)
Name the four Service types and when each fits. (§4 — ClusterIP for internal traffic; NodePort opens a fixed port on every node; LoadBalancer provisions a cloud LB + external IP (one per Service — costly); headless (clusterIP: None) returns Pod IPs via DNS for addressing individual identity-bearing Pods.)
iptables vs. IPVS kube-proxy mode — why does it matter at scale? (§3 — iptables evaluates netfilter rules roughly linearly, so large Services get slow to update; IPVS uses the kernel LB with hash-table lookups (near-constant time, real LB algorithms), scaling the data path to thousands of endpoints.)