all_lessons/kubernetes_genai18 lessons - production platform track

Generative AI on Kubernetes, From First Principles

An operational platform track for running LLMs and AI applications on Kubernetes: deployment, model data, GPUs, routing, autoscaling, observability, fine-tuning jobs, RAG, agents, security, FinOps, and release management.

Prerequisite
This track assumes you already know how Kubernetes works (pods, controllers, Services, scheduling, RBAC). If you don't — or want the substrate derived from first principles — read Kubernetes from First Principles first. Every lesson here is then just those same primitives under GPU and memory pressure.
Recommendation
Add this as a new Generative AI on Kubernetes series under all_lessons/kubernetes_genai/lessons/. It belongs in the GPU, kernels & serving area because it starts from inference and GPU scheduling, but it should also be searchable by Kubernetes, systems, data, agents, and security.
Shape of the syllabus
This is intentionally 18 lessons, not a duplicate of the existing vLLM, System ML, or Agentic Systems tracks. It follows the book linearly, then adds the production gaps a platform owner needs: release management, security governance, FinOps, capacity planning, and incident response.

The Platform Loop

model artifact -> GPU runtime -> router/gateway -> users/apps
      ^                |               |             |
      |                v               v             v
 registry        observability     policy       eval feedback
      ^                |               |             |
      +-------- release gates <- incidents <- cost/security

Coverage Map

IntroLesson 01: why GenAI workloads change the platform problem.
Chapter 1Lessons 02-03: manual deployments, model servers, KServe, Ray Serve, KubeRay.
Chapter 2Lessons 04-05: model formats, registries, PVs, modelcars, OCI image volumes.
Chapter 3Lessons 06-07: GPU discovery, device plugins, DRA, GPU Operator, topology.
Chapter 4Lessons 08-10: tuning, autoscaling, AI routing, disaggregated serving.
Chapter 5Lessons 11-12: observability, quality, responsible AI, guardrails.
Chapters 6-7Lessons 13-15: customization, fine-tuning jobs, gang scheduling, quotas.
Chapters 8-9Lessons 16-17: RAG, agents, MCP, A2A, state and identity.
Added ops layerLesson 18: GitOps, canary, rollback, security, FinOps, and incident drills.

Lessons

01
Why GenAI Is Different on Kubernetes
Model size, cold starts, GPU scarcity, token latency, privacy, and why ordinary web-service instincts break.
PDF Introduction: challenges of running GenAI at scale, Kubernetes for AI workloads, and LLM fundamentals.
02
First Manual LLM Deployment
Deploy an LLM server by hand with a Deployment, Service, PVC or model mount, GPU request, probes, and a first request.
PDF Chapter 1: deploying models manually to Kubernetes before adding controllers.
03
Model Servers and Controllers
vLLM, TGI, SGLang, KServe, Ray Serve, and KubeRay: runtime ownership versus lifecycle ownership.
PDF Chapter 1: model servers, KServe, Ray Serve, and KubeRay.
04
Model Artifacts and Registries
Safetensors, GGUF/GGML, ONNX, Hugging Face, MLflow, Kubeflow Model Registry, and OCI registries.
PDF Chapter 2: model data formats and model registries.
05
Getting Model Weights Into Pods
Init copies, PersistentVolumes, modelcars, OCI image volumes, node cache behavior, and cold-start economics.
PDF Chapter 2: accessing model data in Kubernetes; refreshed with current Kubernetes image-volume status.
06
GPU Nodes as a Platform Layer
Node Feature Discovery, GPU Feature Discovery, device plugins, NVIDIA GPU Operator, MIG, time slicing, and DRA.
PDF Chapter 3: Kubernetes and GPUs; refreshed with current DRA status.
07
Multi-GPU Inference and Topology
Data parallel, tensor parallel, pipeline parallel, single-node versus multinode inference, and topology-aware placement.
PDF Chapter 3 and Chapter 4: multi-GPU inference, topology, and production placement.
08
Benchmarking and Runtime Tuning
Task evaluation, quantization, pruning, vLLM parameters, startup-time optimization, and repeatable load tests.
PDF Chapter 4: model and runtime tuning, language-model evaluation, benchmark, vLLM tuning, and startup time.
09
Autoscaling and AI-Aware Routing
Token-based autoscaling, queue depth, cache-aware routing, LoRA-aware routing, AI gateways, and Gateway API Inference Extension.
PDF Chapter 4: autoscaling, LLM-aware routing, AI gateways, and Gateway API Inference Extension.
10
Disaggregated Serving
Prefill/decode split, specialized hardware pools, KV transfer, LeaderWorkerSet-style patterns, and when complexity pays.
PDF Chapter 4: disaggregated serving.
11
Observability for LLM Serving
Logs, metrics, traces, TTFT, TPOT, throughput, queue metrics, GPU/DCGM, quality metrics, and OpenTelemetry GenAI signals.
PDF Chapter 5: model observability.
12
Safety, Quality, and Guardrails
Hallucination checks, runtime guardrails, explainability, fairness, OWASP LLM risks, policy boundaries, and audit trails.
PDF Chapter 5: quality metrics, responsible AI, hallucination, and runtime guardrails; expanded with production security risks.
13
Model Customization Choices
Prompt/context engineering, RAG, PEFT, LoRA, full fine-tuning, and when customization is worth the ops cost.
PDF Chapter 6: model customization, prompt engineering, fine-tuning, PEFT, and LoRA.
14
Fine-Tuning Jobs on Kubernetes
Kubeflow Trainer, distributed jobs, checkpoints, storage, restart policy, training observability, and job security.
PDF Chapter 6 and Chapter 7: running tuning jobs, Kubeflow Trainer, storage, security, and observability.
15
Batch Scheduling and GPU Fairness
Bin packing, descheduling, gang scheduling, Kueue/Volcano-style queues, topology awareness, quotas, chargeback, and GPU-as-a-service.
PDF Chapter 7: job scheduling optimization, gang scheduling, topology, quota, multitenancy, networking, and storage.
16
RAG on Kubernetes
Ingestion jobs, embedding consistency, vector database StatefulSets, provenance, reranking, restore plans, and query-time orchestration.
PDF Chapter 8: retrieval-augmented generation components, ingestion, query processing, and RAG on Kubernetes.
17
Agents on Kubernetes
MCP, A2A, service identity, token exchange, SPIFFE/SPIRE, state stores, checkpoints, human review, and long-running workflows.
PDF Chapter 8 and Chapter 9: agentic workflows, MCP, MCP security, A2A, and agent state management.
18
Capstone: Production GenAI Platform
GitOps, release promotion, eval gates, canary, rollback, security, FinOps, incident drills, and the full production platform map.
Synthesis of the full PDF plus operational gaps: release management, governance, capacity planning, and incident response.

External Anchors To Keep Current