Part IV - Customization and training jobs

Batch Scheduling and GPU Fairness

Lesson 14 got a single fine-tuning job running on Kubernetes — a distributed training workload that asks for several GPUs at once and runs for hours. But the moment a second team submits a job, you stop running a job and start running a shared GPU platform, and the questions change shape entirely. Who gets the scarce accelerators? Can a distributed job wait until all its workers are ready, or does it half-start and hold GPUs idle? Should a research experiment be allowed to block a production incident? And when finance asks what the cluster cost, can anyone answer? This lesson turns Kubernetes from a pod placer into a fair, accountable GPU platform — a scheduling contract welded to a financial one.

1 · Scheduling is now a product decision

When one team owns a GPU node, the scheduler's job is trivial: place the pod, done. The default Kubernetes scheduler does exactly this — it finds the first node that fits the pod's resource request and binds it. That policy, "first pod that fits," is perfectly fine for stateless web pods. It is a disaster for expensive accelerators shared by many teams, for three reasons that this lesson unpacks: it places pods one at a time (bad for jobs that need all their workers together), it is greedy about packing (bad for topology), and it knows nothing about who owns what budget (bad for fairness and cost).

The mental model to carry through the lesson: GPU fairness is a scheduling contract plus a financial contract. The scheduling contract decides, mechanically, who waits in a queue, who is allowed to preempt whom, and which hardware topology a job is guaranteed. The financial contract decides who pays for the GPU-hours consumed and whether that spend bought anything useful. The platform fails the moment either contract is implicit — and, as the chapter stresses, GPU fairness usually fails socially before it fails technically: a team that does not know the preemption rules will be furious the first time their job is killed, even if the kill was correct.

This is why projects like Kueue (a Kubernetes-native job queueing and quota controller) and Volcano (a batch scheduler with gang and fair-share semantics) exist. They are not competing "better schedulers"; they each move a piece of policy out of the implicit default and make it explicit and auditable. The real design question the chapter poses is not "which project is best?" but where should each policy live — in the batch scheduler, in a queue/admission controller, or in a platform API your teams call.

2 · The four scheduling primitives, defined

Four mechanisms do the structural work. Each one exists to prevent a specific failure you would otherwise hit on shared GPUs.

Why gang scheduling is non-negotiable for training. A PyTorch distributed job needs rendezvous: every worker discovers the others, agrees on ranks, and only then begins collective operations (the all-reduce that averages gradients, covered in the System ML track). If only half the pods are scheduled, the job is not half-useful — it is stuck, holding scarce GPUs while blocked at the rendezvous barrier waiting for ranks that the default scheduler may never place because the GPUs are gone. Gang scheduling admits the group only when enough resources are free for all of it, so a job either runs or politely waits in the queue. Volcano-style schedulers center on this; Kueue expresses it through all-or-nothing admission of a job's pods.

3 · A multitenancy scenario: when best-effort blocks production

Make it concrete. A cluster has 64 GPUs. Three teams share it, each with a quota:

One night the cluster is quiet, so Team Batch's quotas let them borrow idle capacity (a cohort feature) and a best-effort eval sweep spreads across 40 GPUs — far above their 8-GPU quota, which is allowed precisely because nothing else wanted them. Then a production incident hits: Team Serving needs to scale inference and submits a job for 16 more GPUs. But the 64 are already accounted for — Serving's own steady-state is using 16, Batch's borrowed sweep holds 40, and only 8 are truly free. The job needs 16 and can see only 8. With no preemption policy, the production job queues behind a best-effort eval sweep for the other 8 it needs. The incident drags on while a nightly job no one is watching holds the accelerators.

The fix is two pieces of explicit policy. Priority classes rank workloads (production > research > best-effort). Preemption lets a higher-priority job that cannot otherwise be admitted evict lower-priority running pods. With both configured, the production job's admission triggers preemption of enough best-effort eval pods to free the missing 8 GPUs (the 16-GPU job already sees 8 free, so 8 more must be reclaimed); Batch's borrowed-over-quota work is the first to be reclaimed because it was running on capacity it had no guarantee to.

# Kueue: a ClusterQueue gives each team a guaranteed quota and a borrowing cohort,
# and admits jobs by priority. Higher-priority jobs preempt lower ones when needed.
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata: { name: serving-cq }
spec:
  cohort: shared-64gpu              # lets teams borrow each other's idle quota
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: a100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 32            # Serving's guaranteed share
  preemption:
    reclaimWithinCohort: Any        # reclaim borrowed GPUs from lower priority
    withinClusterQueue: LowerPriority

The lesson the chapter draws: best-effort work must be able to use idle GPUs (utilization) but must be the first to lose them (fairness). Borrowing without reclaim is how a nightly eval blocks an incident. The contract — Serving is guaranteed 32, can borrow more, and reclaims its 32 by preempting borrowers — must be published so Team Batch is not surprised when their job is killed mid-sweep.

4 · Utilization versus topology: the central tension

Now the harder trade-off, the one that makes GPU scheduling genuinely different from packing web pods. Bin packing maximizes a scalar — utilization — but large training and tensor-parallel jobs care about a shape: they need several GPUs that are physically close, connected by NVLink (NVIDIA's high-bandwidth intra-node GPU interconnect) within a node, or rack-local across nodes, so their collectives are fast. Bin packing and topology pull in opposite directions.

Worked example. Take 8 nodes, each with 8 NVLink-connected GPUs — 64 GPUs total. A flurry of small 2-GPU jobs arrives. A greedy bin-packer, optimizing utilization, scatters them to fill gaps wherever they fit:

Cluster utilization is 48 / 64 ≈ 75% running, and if a few more small jobs land it climbs toward 95%. By the utilization dashboard, the cluster looks healthy. But now the next big job arrives: an 8-GPU tensor-parallel fine-tune that needs 8 NVLink-connected GPUs on one node. There are 16 GPUs free in total — more than enough by count — yet no single node has 8 free. The largest contiguous NVLink island is 2 GPUs. The big job cannot be placed despite 25% of the fleet being idle. Utilization optimized the average and starved the shape.

  greedy bin packing                 topology-aware (islands reserved)
  node-1 [#][#][#][#][#][#][.][.]    node-1 [#][#][#][#][#][#][.][.]
  node-2 [#][#][#][#][#][#][.][.]    node-2 [#][#][#][#][#][#][.][.]
  node-3 [#][#][#][#][#][#][.][.]    node-3 [#][#][#][#][#][#][#][#]  <- still
  ...                                ...                                whole
  16 free, largest island = 2        16 free, largest island = 8
  8-GPU job: CANNOT place            8-GPU job: places on node-3
  (# = used GPU, . = free GPU)

Same 16 free GPUs, two different shapes. The greedy layout maximized a number; the topology-aware layout deliberately kept node-3 whole so the next big job has a home. The difference is invisible on a utilization gauge and decisive for the job that matters.

Topology-aware scheduling makes the scheduler reason about the interconnect, not just GPU counts. It keeps a distributed job inside a topology domain — same NVLink node, then same rack, then same secondary-network fabric — so its collectives stay on the fast path, and it reserves islands so large jobs are placeable. The cost: the scheduler has fewer feasible placements, so it is slower to admit and can leave GPUs idle waiting for the right-shaped hole. That is the deliberate trade — a worse average for a better worst case on the jobs that matter.

5 · Closing the financial loop: chargeback and showback

The scheduling contract decides who runs. The financial contract decides who pays, and it is the half most platforms skip — which is why "no one can explain GPU spend" is a near-universal failure. The chapter's prescription: expose cost at three granularities.

The distinction between chargeback (teams are actually billed their share) and showback (teams are shown their share without a real invoice) is a maturity choice: showback first, to build the accounting and trust; chargeback once the numbers are trusted enough to drive a budget. Either way, the critical design point is that platform metrics must not stop at allocation. Allocation says a job held 8 GPUs; usefulness says whether those GPUs did anything. A team whose chargeback is in $/useful-token has a direct incentive to fix the batching and KV-cache settings from lessons 08–10; a team billed only in $/GPU-hour has no signal that their job ran at 30% utilization.

Checkpoint exercise

Where this points next

You can now run GPUs as a fair, accountable, multi-tenant platform — training and serving workloads share hardware under a published scheduling and financial contract. Part IV's customization arc is done. Lesson 16, RAG on Kubernetes, shifts from training a model on your data to retrieving your data at inference time: it adds the machinery — a vector database, an embedding service, and a retrieval step in front of the model server — that lets a model answer from a knowledge base it was never trained on. The scheduling discipline you just built still applies (the embedding and retrieval services are pods competing for the same nodes), but the new question becomes a data and latency one: how to keep the retrieval hop fast and the index fresh.

Interview prompts

• Why isn't the default "first pod that fits" scheduler enough for shared GPUs? (§1 — it places pods one at a time (bad for jobs needing all workers), packs greedily (bad for topology), and ignores ownership/budget (bad for fairness and cost).)
• What problem does gang scheduling solve, and what breaks without it? (§2 — all-or-nothing admission; without it a distributed job half-starts and stalls at the rendezvous barrier holding scarce GPUs idle while waiting for ranks the scheduler may never place.)
• A best-effort eval job is blocking a production incident. What's the fix? (§3 — priority classes plus preemption: borrowing-over-quota work can use idle GPUs but is reclaimed first; the production job preempts the best-effort pods to free its GPUs.)
• The cluster is at 95% utilization but an 8-GPU job won't schedule with 16 GPUs free. Why, and how do you fix it? (§4 — fragmentation: GPU count is fine but no contiguous NVLink island remains. Fix with topology-aware placement that reserves islands and a descheduler that defragments.)
• Why isn't $/GPU-hour a sufficient cost metric? (§5 — it measures allocation, not usefulness; a job at 95% allocation but 30% busy looks fine. $/useful-token and $/training-run tie spend to value and expose idle-but-allocated GPUs.)
• Where should scheduling policy live — Kubernetes or an HPC scheduler? (§1, §5 — an ownership decision: Kueue/Volcano make Kubernetes the batch scheduler, or Kubernetes hosts services around a Slurm-style scheduler via bridges; pick one so the contract lives in a single place.)
• Why is "publish the rules" listed as an operational requirement, not a nicety? (§1, §3 — GPU fairness fails socially before it fails technically: a team that doesn't know the preemption/quota/wait rules treats a correct preemption as betrayal, generating escalations and side-channel requests.)