Part IV - Customization and training jobs

Fine-Tuning Jobs on Kubernetes

Lesson 13 helped you decide whether to customize and how — prompt engineering, RAG, full fine-tune, or a parameter-efficient adapter like LoRA. Once you have chosen to train, you have to run that training somewhere, and a GPU training run is a fundamentally different beast from the serving deployments the rest of this track has built. A server is always-on and stateless; a training job runs to completion, holds many expensive GPUs at once, must survive node death without losing days of compute, and reads terabytes of data at line rate. This lesson takes the customization decision and turns it into a reproducible batch workload with the right scheduling, storage, checkpoint, and security boundaries — so that the artifact you hand back to the registry is one you can trust and re-create.

1 · A training job is not a service

Everything earlier in this track ran as a Deployment: a controller that keeps N identical pods alive forever, replacing any that die, scaling on load. That is exactly right for an inference server — you want continuous availability and fast, transparent replacement. A training run is the opposite shape. It starts, consumes resources for hours or days, produces an artifact, and terminates successfully. The Kubernetes primitive for that is a Job: a controller that runs one or more pods to completion and then stops, tracking success/failure rather than uptime.

This is not a cosmetic difference. Because a Job terminates, its success is an event you can gate a pipeline on. Because it holds GPUs for the whole run rather than serving short requests, idle time is pure waste — a GPU that sits unused while the job waits for something costs real money every minute (an H100 rents for roughly $2-$4/GPU-hour on cloud, so a single idle 8-GPU node burns on the order of $20-$30/hour doing nothing). And because the run is long, the probability that some piece of hardware fails or gets preempted — evicted by the scheduler to free its GPUs for a higher-priority job (the mechanics of which are lesson 15's subject) — before the run finishes is no longer negligible, which is why checkpoints (§3) exist.

2 · Distributed training needs gang scheduling — and an operator that knows it

A small LoRA fine-tune of a 7B model might fit on one GPU and run as a single-pod Job — nothing exotic. But a full fine-tune of a large model, or any data-parallel run that wants throughput, spreads across many GPUs on many nodes. Those workers are not independent: they perform a synchronized all-reduce of gradients every step (the collective-communication machinery from the system_ml track). If worker 3 of 8 is not running, the other seven cannot make progress — they block on the collective, holding their GPUs idle while they wait.

This is the defining hazard of distributed training on a shared cluster. The default Kubernetes scheduler places pods one at a time, greedily. Hand it an 8-worker job and it may schedule 6 workers, then run out of free GPUs because some other job grabbed the last two — leaving you with 6 GPUs held hostage, idle, waiting for two that will never come, while the other job is in the same deadlock. The fix is gang scheduling: place all workers of a job together or place none of them (all-or-nothing), so a job only starts when its full quota is available. We preview the term here because it motivates everything in lesson 15, which builds the queueing and fairness machinery (Kueue, scheduler plugins) properly.

Kubeflow Trainer is the Kubernetes-native operator that orchestrates distributed training jobs — the successor to the older Kubeflow Training Operator and its per-framework CRDs (PyTorchJob, TFJob), consolidating them behind one TrainJob API. You hand it a TrainJob describing the runtime (PyTorch, etc.), the number of nodes, and the per-node resources; it creates the worker pods, wires up the rendezvous endpoints so the workers find each other, injects the per-worker environment each one needs — its rank (the worker's integer index, 0…N−1, that identifies it in the collective) and the world size (the total worker count) — and tracks the run to completion. It is the training-specific layer that a plain Job lacks.

The platform choice is about repeatability and ownership, not raw training speed. All three can train a model; what differs is who operates the rendezvous, how the job integrates with the cluster's queueing, and how reproducible the run is six months later.

Gang scheduling fixes the worst case — a job that never fully starts. The subtler, everyday hazard is the straggler: all workers did start, but one is slower than the rest (a degraded NIC, a thermally throttled GPU, a noisy node), so every synchronized all-reduce stalls at the speed of the slowest rank, and the whole job's throughput drops to that worker's pace while the other GPUs idle on the barrier. A straggler is invisible in aggregate metrics — average GPU utilization still looks healthy — which is why a training platform needs rank-aware observability: logs and metrics tagged with each worker's rank, so you can see that rank 5 is taking 1.4× as long per step as ranks 0–4 and 6–7. Without per-rank visibility, a distributed job that has quietly halved its throughput looks identical to a healthy one. (Lesson 15 returns to the straggler as a scheduling and topology problem; here it is the reason rank-aware logging is a first-class platform requirement, not an afterthought.)

3 · How often to checkpoint — the quantitative knob

A checkpoint is periodically saved training state — model weights, optimizer state, the step counter, the data-loader position — written to durable storage so that if a worker dies or gets preempted, the run resumes from the last save instead of from step zero. On a long run this is not optional: over hours or days, hardware will fail or the scheduler will preempt your job for a higher-priority one. Without a checkpoint, every such event throws away all the GPU time since the run began.

So you checkpoint often. But checkpoints are not free: writing the full optimizer + weight state for a large model can be tens to hundreds of GB, and that write blocks (or at least taxes) training while it happens. Checkpoint too often and the writes dominate wall time; checkpoint too rarely and a single preemption costs hours. There is a sweet spot, and it has a number.

// per hour: write overhead + expected recompute on failure
write = (60 / T) * W;                 // T smaller -> more writes -> more overhead
loss  = (60 / MTBF) * (T / 2);        // T larger  -> more lost work per failure
total = write + loss;
Tstar = Math.sqrt(2 * W * MTBF);      // analytic minimum of write+loss

Two refinements production platforms add. First, asynchronous / sharded checkpointing (each rank writes its shard in parallel, overlapping with compute) drives W down so you can afford to checkpoint more often. Second, the math above assumes checkpoint storage is fast and reliable — if it is not, W balloons and the whole curve shifts up, which is the first failure mode below.

4 · A training Job, concretely

Here is the shape of a single-pod fine-tuning Job — enough to see the four things a serving Deployment does not have: a GPU request, a fast shared data volume, a durable checkpoint path, and a restart policy.

apiVersion: batch/v1
kind: Job
metadata:
  name: lora-tune-llama-7b
spec:
  backoffLimit: 3            # retry the failed pod up to 3x, then mark Job failed
  template:
    spec:
      restartPolicy: OnFailure       # Jobs cannot use "Always"; OnFailure or Never
      containers:
      - name: trainer
        image: registry.internal/trainer@sha256:...   # pinned digest (lesson 04)
        command: ["python", "tune.py", "--config", "/cfg/run.yaml"]
        resources:
          limits:
            nvidia.com/gpu: 1        # the GPU request
        env:
        - name: HF_TOKEN             # data/model credential, per-job
          valueFrom: { secretKeyRef: { name: tenant-a-data, key: hf-token } }
        volumeMounts:
        - { name: data, mountPath: /data, readOnly: true }   # fast shared input
        - { name: ckpt, mountPath: /ckpt }                   # durable checkpoints
      volumes:
      - name: data                   # high-throughput shared dataset
        persistentVolumeClaim: { claimName: training-data-fast }
      - name: ckpt                   # durable, resumable checkpoint store
        persistentVolumeClaim: { claimName: ckpt-tenant-a }

For a multi-node distributed run you would not hand-write the rank wiring; you would hand Kubeflow Trainer a TrainJob that declares numNodes and the per-node resources, and it generates the gang-scheduled worker pods, the rendezvous, and the rank/world-size env for you. The single-Job sketch above is the mental anchor — the distributed version is the same four concerns multiplied across workers, plus the all-or-nothing placement from §2.

Note the credential handling. The data token is a per-job secretKeyRef scoped to this tenant's dataset — not a broad service-account permission the container inherits, and never baked into the image. That separation is what keeps one tenant's training run from reading another's data, and it is the third checklist item below.

5 · Where training jobs go wrong

Checkpoint exercise

Where this points next

This lesson treated a single job in isolation — its workers, its storage, its checkpoints. But a real cluster runs many training jobs from many teams against a finite pool of GPUs, and the partial-placement deadlock in §5 showed that the default scheduler cannot handle that safely. The next lesson, Batch Scheduling and GPU Fairness, builds the machinery that was previewed here as "gang scheduling": all-or-nothing placement, job queues, quotas, and fair-share policies (Kueue and scheduler plugins) that decide which job runs when, so big runs wait their turn instead of thrashing the cluster and stranding GPUs.

Interview prompts

• Why model a training run as a Job instead of a Deployment? (§1 — a Job runs to completion with a terminal success/failure state and restarts only on failure, matching training's shape; a Deployment is always-on and restarts forever, matching serving. Idle GPU time in a Job is pure waste.)
• What is gang scheduling and why does distributed training need it? (§2 — all-or-nothing placement of a job's workers; without it the scheduler may place some workers and strand the rest, leaving the running ones blocked on the gradient all-reduce while holding GPUs idle. Full treatment in lesson 15.)
• What does Kubeflow Trainer give you over a plain Kubernetes Job? (§2 — distributed-training orchestration: worker creation, rendezvous wiring, rank/world-size injection, and run-to-completion tracking that a plain Job does not provide.)
• How do you choose a checkpoint interval? (§3 — balance write overhead (∝ 1/T) against expected lost work on preemption (∝ T); the optimum is near √(2·W·MTBF). Too frequent and writes dominate; too rare and a single failure costs hours.)
• What four things does a training Job's YAML have that a serving Deployment does not? (§4 — a GPU resource request, a fast shared data volume, a durable checkpoint path, and a non-Always restart policy (OnFailure/Never with a backoffLimit).)
• How do you keep one tenant's training run from reading another tenant's data? (§4, §5 — scope data credentials per job/tenant via Secrets referenced as env, not baked into images or granted as broad service-account permissions; isolate namespaces and authenticate framework endpoints.)
• A distributed job has GPUs allocated but near-zero utilization — what do you suspect and how would you confirm? (§5 — partial-placement deadlock: some workers scheduled, others not, the rest blocked on the collective. Confirm with per-rank logs/metrics showing missing ranks; fix with gang scheduling.)