all_lessons/kubernetes/07lesson 8 / 14

Part III — Reaching the workload

DNS, Ingress & the Front Door

Lesson 06 gave each Pod a routable IP and then planted a stable ClusterIP in front of the churning, label-selected Pod set — so a caller dials one virtual address and kube-proxy fans it across the live endpoints, no matter how the Pods roll. That fixed the moving target. It left two things ugly. First, that stable address is still a number10.96.0.42 — that something would have to hardcode, and numbers change between clusters and namespaces. Second, that number is only reachable from inside the cluster: nothing here gives an outside user one hostname, one path-routing rule, or TLS. This lesson builds the front door: CoreDNS so workloads find each other by name, and Ingress / the Gateway API so the outside world reaches many Services through one L7 edge.

The previous step left this broken
A ClusterIP is stable, but it is an opaque integer with two flaws. (1) You cannot hardcode it. The address is assigned at Service-create time and differs per cluster, per namespace, per recreation — bake 10.96.0.42 into config and the next environment breaks. A caller wants to say "the checkout service," not memorize an IP. (2) It is internal only. ClusterIP lives on the cluster's private virtual-IP range; a browser on the public internet cannot route to it. And the lesson-06 escape hatch — give every app its own LoadBalancer Service — is wasteful (one cloud load balancer, and often one public IP, per app) and blind (an L4 load balancer sees TCP/ports, not HTTP hosts or paths, so it can neither route /api vs / nor terminate TLS centrally).
Linear position
Forced by: a ClusterIP is a stable but un-hardcodable, internal-only number; external users need one named, L7, TLS-terminating door, and a LoadBalancer-per-app is wasteful and L4-only.
New capability: CoreDNS turns every Service into a name (service.namespace.svc.cluster.local) so workloads discover each other by name; Ingress (and its successor, the Gateway API) put one L7 reverse proxy at the cluster edge — host/path routing plus TLS termination — fronting many Services through a single external load balancer.
Forces next: workloads now reach each other and the outside world, but a Pod is still born config-blind and storage-less — settings baked into images, no durable disk that survives a restart.
The plan
Five moves. (1) Replace the opaque ClusterIP with a name: how CoreDNS gives every Service a DNS record, the search-domain shortcuts, and what headless Services resolve to. (2) State the external-edge problem and why a LoadBalancer-per-app does not scale. (3) Derive Ingress — one L7 reverse proxy doing host/path routing and TLS for many Services behind one LB — and why it is inert without an ingress controller. (4) The Gateway API as the modern successor, and why its role split (infra owns the Gateway, app teams own the routes) exists. (5) Drive the routing-table widget, then failure modes, a checklist, and the hand-off to config and storage.

1 · CoreDNS: every Service gets a name

The fix for "you can't hardcode a number" is the oldest fix in networking: give the number a name and resolve the name at call time. Kubernetes runs a cluster DNS server — almost always CoreDNS, itself a Deployment of Pods sitting behind its own ClusterIP — and wires every Pod to use it. The API server's Service controller (recall lesson 06: Services and their EndpointSlices are just more objects in etcd) feeds CoreDNS, so the instant you create a Service named checkout in namespace shop, a DNS name exists for it.

The fully-qualified name follows a fixed grammar. For a normal (ClusterIP) Service it is:

  <service>.<namespace>.svc.cluster.local
       │          │       │        └─ the cluster DNS suffix (configurable; this is the default)
       │          │       └────────── "svc" = this is a Service (vs "pod" for Pod records)
       │          └────────────────── the namespace the Service lives in
       └───────────────────────────── the Service's metadata.name

  checkout.shop.svc.cluster.local   →  resolves to  →  10.96.0.42   (the ClusterIP)

The name resolves to the Service's ClusterIP, and from there kube-proxy does the lesson-06 fan-out to a live Pod. So the caller holds a name that never changes, while the IP behind it and the Pods behind that are free to churn. That is the whole point: one more layer of indirection buys stability.

Typing the full suffix everywhere would be miserable, so every Pod gets a DNS search list baked into its /etc/resolv.conf. Inside namespace shop, a Pod's resolver appends, in order, shop.svc.cluster.local, svc.cluster.local, cluster.local. So the short forms below all reach the same place — the resolver just tries each suffix until one resolves:

Same namespace
Just checkout. The search list appends shop.svc.cluster.local and it resolves. This is why intra-namespace code reads like http://checkout — clean and portable.
Cross namespace
checkout.shop from another namespace. The svc.cluster.local suffix completes it. You name the namespace only when you cross one.
Fully qualified
checkout.shop.svc.cluster.local. (trailing dot optional). Unambiguous — skips the search list. Use it in config that must not depend on the caller's namespace.
Named ports / SRV
A Service can also publish SRV records for its named ports, so a client can discover which port a service exposes, not just its address.

One special case matters for stateful systems (and returns in lesson 09). A headless Service — one declared with clusterIP: None — has no virtual IP at all. Instead of resolving to a single VIP, its DNS name resolves to the list of individual Pod IPs behind it (an "A record per ready Pod"). And each Pod gets its own stable per-Pod name, e.g. db-0.db.shop.svc.cluster.local. That is exactly what a clustered database needs: not "any replica behind a VIP," but "address replica 0 specifically." Headless = DNS without load-balancing — name discovery, you do the rest.

2 · The external-edge problem

DNS solved naming inside the cluster. Now point a browser at it from the public internet. The ClusterIP range is private and unroutable from outside, so we need something with a public address that forwards inward. Lesson 06 already gave us one tool: a LoadBalancer-type Service, which asks the cloud to provision an external load balancer and a public IP that funnels to that one Service. Use it once and it is fine. Use it per app and the costs stack up fast.

cost → each LoadBalancer Service provisions its own cloud LB (and often a billable public IP). Forty microservices that need exposure = forty load balancers and forty IPs.
L4 blindness → a cloud LB is layer-4: it sees TCP and ports, not HTTP. It cannot route by hostname, cannot route /api to one Service and / to another, and cannot terminate TLS centrally.
no shared policy → host-based virtual hosting, a single TLS certificate store, redirects, and rate limits all have to be re-implemented per LB instead of declared once at the edge.

The pattern any web operator reaches for here is a single reverse proxy at the edge: one public entry point that reads the HTTP request — its Host header and URL path — and forwards it to the right internal backend. That is a layer-7 (application-layer) device, and it is what Kubernetes models with Ingress.

3 · Ingress: one L7 door for many Services

An Ingress is a Kubernetes object that declares L7 routing rules: "for host shop.example.com, send path /api to Service api and everything else (/) to Service web; for host admin.example.com, send everything to Service admin; terminate TLS for these hosts using this Secret." One object, many backends, behind one external load balancer and one public IP.

                       ┌──────────────────────── one external LB + one public IP
   internet ──TLS──►  ┌─┴───────────────┐
                      │ ingress         │   reads Host + path, terminates TLS,
                      │ controller (L7) │   then routes to the right ClusterIP
                      └──┬───────┬───────┘
        Host: shop.example.com   Host: admin.example.com
           /api │   / │              │ /
                ▼      ▼              ▼
          Svc api   Svc web      Svc admin      ← ClusterIPs from lesson 06
             │         │            │
          [pods]    [pods]       [pods]         ← live EndpointSlice, kube-proxy fans out

Here is the catch that trips up newcomers: the Ingress object does nothing by itself. It is just a declaration sitting in etcd — desired state with no controller. To make it real you must run an ingress controller: a Pod (commonly NGINX, but also HAProxy, Traefik, Envoy-based, or a cloud-native one) that watches all Ingress objects and continuously programs itself to be the proxy they describe — the lesson-00 reconcile loop again, applied to routing rules. No controller, and every Ingress you create is silently inert; traffic goes nowhere. Two things Ingress centralizes are worth naming explicitly:

Host virtual-hosting
Many hostnames share one IP. The controller inspects the request's Host header and picks the rule set for that host — shop.example.com and admin.example.com resolve to the same public IP yet route to different Services.
Path routing
Within a host, route by URL prefix: /api → the API Service, /static → a CDN-origin Service, / → the web Service. One front, many backends, no client awareness.
TLS termination
The HTTPS connection ends at the edge: the controller holds the certificate (from a referenced Secret), decrypts, then forwards plaintext over the trusted cluster network. One cert store, not one per app.
Default backend
A request matching no host/path rule falls through to a configured default backend (often a tidy 404 page) instead of erroring — the catch-all at the bottom of the routing table.

So the L4 LoadBalancer does not disappear — it shrinks to one, in front of the ingress controller, and all the L7 intelligence (hosts, paths, TLS, redirects) moves into Ingress rules the controller enforces. Forty apps, one LB, one IP, one cert store.

4 · The Gateway API: the same job, split by role

Ingress works, but it aged badly. Its spec is thin, so every controller bolted its extra features (rewrites, canary weights, auth) onto annotations — free-text key/value strings on the object that are vendor-specific and unvalidated. An Ingress for NGINX does not port to Traefik. And one Ingress object mixes concerns that belong to different people: the public hostname and TLS cert (a platform/security concern) sit in the same YAML as the path-to-Service map (an app-team concern).

The Gateway API is the modern successor (graduated to stable, the direction Kubernetes is steering toward) that fixes both by splitting one object into a small family of typed resources — and the split is deliberately drawn along who owns what:

ResourceWhat it declaresWho owns it
GatewayClassWhich implementation runs the data plane (e.g. an Envoy-based controller). Cluster-scoped — the "kind of gateway available."Infrastructure / vendor
GatewayAn actual edge: which ports/protocols listen, which hostnames and TLS certs are allowed, what the public address is.Cluster operators / platform
HTTPRouteThe L7 rules: for these paths/headers on an allowed host, route to these Services (with weights, rewrites, mirroring).App teams

Why split it at all? Because one cluster serves many teams, and they have different authority. The platform team must control the public surface — which hostnames are claimable, which certificates are trusted, how the edge is provisioned — for security and cost. The app teams must be free to change their own routing — add /v2, shift 10% of traffic to a canary — without a ticket to platform and without the power to hijack another team's hostname or mint certs. Ingress jammed both into one object, so either app teams got edit rights to TLS and hosts (unsafe) or every route change waited on the platform team (slow). The Gateway API encodes that boundary in the type system: an HTTPRoute may only attach to a Gateway that explicitly permits it, so an app team owns its routes while the operator still owns the door. Separation of concerns, made structural.

5 · Trace a request through the front door

Tie naming and the edge together with one external request, end to end. This is the path the widget below lets you poke at — change a rule, change the URL, watch where it lands.

public DNS → the user's browser resolves shop.example.com to the edge load balancer's public IP (this is ordinary internet DNS, not CoreDNS — CoreDNS is cluster-internal only).
edge LB → the one cloud L4 load balancer forwards the TCP connection to the ingress controller Pod.
TLS + L7 match → the controller terminates TLS, reads Host: shop.example.com and path /api, and matches the most specific Ingress/HTTPRoute rule (or the default backend → 404).
to the Service → it forwards to the matched Service's ClusterIP — typically by its CoreDNS name api.shop.svc.cluster.local.
kube-proxy fan-out → kube-proxy (lesson 06) rewrites that to one live Pod IP from the EndpointSlice, and the Pod answers.
L7 routing-table playground
This is the ingress controller's decision. Each rule is host + path → Service, with a TLS flag. Toggle rules on/off, then type a request as a host + path and hit Route it. The controller matches by host first, then the longest matching path prefix (most specific wins); if nothing matches it falls through to the default backend (404). Watch host virtual-hosting (same edge, different host → different Service) and the TLS indicator (does the matched rule terminate TLS for that host?).
request:
Toggle rules, set a request, and press Route it.
Matched rule
Backend Service
TLS at edge

Failure modes & checklist

Failure modes

  • Ingress with no controller. You apply Ingress objects expecting traffic to flow; nothing routes because no controller is installed to enact them. Signal: the Ingress shows no ADDRESS, external requests time out or hit the LB's default, and controller logs are empty (there is no controller).
  • Hardcoding the ClusterIP. Config bakes 10.96.0.42 instead of the CoreDNS name; it works in one cluster and breaks on redeploy when the VIP is reassigned. Signal: connection-refused only in a new namespace/cluster, fixed by switching to checkout / checkout.shop.
  • Wrong-namespace short name. Using bare checkout from a different namespace; the search list never appends the right suffix, so it resolves to nothing (or the wrong same-named Service). Signal: NXDOMAIN or surprising target; qualify it as checkout.shop.
  • TLS terminated but plaintext assumed. Forgetting the edge decrypts: backends receive HTTP, or a backend redirects to HTTPS and loops. Signal: redirect loops, or mixed-content / wrong X-Forwarded-Proto handling.
  • Headless treated as load-balanced. Pointing a normal client at a headless Service expecting one VIP; instead it gets a list of Pod IPs and picks the first. Signal: traffic skewed to one Pod, or breakage when that Pod rolls.

Checklist

  • Refer to Services by CoreDNS name, never by ClusterIP — bare name in-namespace, service.namespace across.
  • Install and verify an ingress controller (or Gateway controller) before creating any Ingress/HTTPRoute.
  • Front many Services with one Ingress/Gateway behind a single LB; reserve LoadBalancer Services for genuinely non-HTTP (raw TCP/UDP) needs.
  • Terminate TLS at the edge with a managed cert (cert-manager / a referenced Secret); decide explicitly whether backend hops are plaintext or re-encrypted.
  • For stateful per-replica addressing use a headless Service and the stable per-Pod DNS names; don't expect VIP load-balancing.
  • On a multi-team cluster, prefer the Gateway API split so platform owns the Gateway/TLS and app teams own HTTPRoutes.

Checkpoint exercise

Try it
In the widget, turn off the shop.example.com + /api rule, then route a request for shop.example.com /api/orders. Predict before you press: which rule wins now, and is TLS still terminated? Then explain in one sentence why /api/orders matched /api rather than / when both were on. Finally, write the CoreDNS name a Pod in namespace web would use to reach a Service named api in namespace shop — and the shorter form a Pod inside shop could use for the same Service.

Where this points next

Workloads now find each other by name and the outside world reaches them through one L7 door with TLS. But notice what every Pod we have built still lacks. Its configuration — database URLs, feature flags, the very hostnames and certs this lesson routed — is baked into the container image, so flipping a setting between dev and prod means rebuilding the image, which breaks the build-once-run-anywhere promise from lesson 01. And its filesystem is ephemeral: when the Pod dies (and lesson 05 told us Pods die constantly), everything it wrote dies with it. A routable, named, TLS-fronted Pod that loses all its data on restart and can't be reconfigured without a rebuild is still only half a workload. Lesson 08, Config & Storage: State That Outlives the Pod, separates configuration from the image (ConfigMaps and Secrets) and gives a Pod durable disk that survives it (Volumes, then the PersistentVolumeClaim → PersistentVolume binding).

Takeaway
A ClusterIP is stable but un-hardcodable and internal-only, so we add two layers of front door. CoreDNS gives every Service a name — service.namespace.svc.cluster.local, with search-domain shortcuts so a Pod can say just checkout in its own namespace or checkout.shop across one — turning a churning IP into a stable name (a headless Service skips the VIP and resolves to per-Pod IPs and stable per-Pod names, for stateful systems). For external traffic, an Ingress is one L7 reverse proxy at the edge doing host virtual-hosting, path routing, and TLS termination, fronting many Services behind a single load balancer instead of one wasteful, L4-only LoadBalancer per app — but the Ingress object is inert until an ingress controller (NGINX, etc.) watches it and becomes the proxy. The Gateway API is the modern successor, splitting the job into GatewayClass/Gateway (owned by infra/platform) and HTTPRoute (owned by app teams) so a multi-team cluster gets separation of concerns by construction: platform owns the public surface and certs, app teams own their routes.

Interview prompts