Part III - Cluster and platform layer

Ray Clusters, KubeRay, and Autoscaling

Lessons 06–11 built real applications — RL rollouts, tuning sweeps, data pipelines, distributed training, online serving — but every one of them ran on a cluster that someone, somewhere, had already stood up. ray.init() quietly connected to "a cluster" and the GPUs were simply there. This lesson opens that box. A Ray cluster is not an undifferentiated pool of machines: it is a head node running the control plane plus worker groups, each a node type, that the autoscaler grows and shrinks. The central idea you must own by the end is that Ray's autoscaler scales on pending logical resource demands, not CPU utilization — a distinction that decides whether your bursty job gets its GPUs in time or times out waiting.

1 · The anatomy of a cluster

A Ray cluster is a set of nodes — physical or virtual machines, or Kubernetes pods — running Ray processes that together present one logical runtime to your code. It has exactly two kinds of node, and the asymmetry between them is the first thing to internalize.

The reason worker groups exist rather than one flat pool is heterogeneity. Recall from Lesson 07 that an RLlib job wants many cheap CPU rollout workers and a few expensive GPU learners; from Lesson 10 that training wants GPU boxes; from Lesson 09 that data preprocessing wants high-core CPU boxes. A single homogeneous pool cannot serve all three without massive waste. A worker group lets you say "this group is A100 boxes, scale it 0→8; that group is CPU boxes, scale it 2→64," and the autoscaler picks the right group to satisfy each pending demand.

2 · Three ways to stand one up

Ray gives you a ladder of cluster-creation methods, from a manual two-command setup good for a quick benchmark, to a fully operator-managed Kubernetes deployment good for a platform team.

The manual path makes the model concrete: the head process opens a port, and every ray start --address is a worker registering its resources with the GCS. The cluster launcher automates exactly that over cloud VMs. KubeRay does the same but expresses the cluster as Kubernetes objects, which is where most production Ray now lives — so it is worth seeing the CRD in full.

The three KubeRay CRDs differ only in lifecycle. A RayCluster is a long-lived cluster you submit many jobs to. A RayJob creates a cluster, runs one job, and tears the cluster down — clean isolation and cost attribution per job (Lesson 08's tuning sweep is a natural RayJob). A RayService wraps a Ray Serve app (Lesson 11) with zero-downtime upgrades. Here is a minimal RayCluster with one CPU group and one autoscaling GPU group:

# raycluster.yaml — applied with: kubectl apply -f raycluster.yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: ml-cluster
spec:
  enableInTreeAutoscaling: true        # the Ray autoscaler runs as a sidecar on the head
  headGroupSpec:
    rayStartParams:
      dashboard-host: "0.0.0.0"
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray-ml:2.9.0-gpu
          resources:
            limits:   { cpu: "8",  memory: "32Gi" }
            requests: { cpu: "8",  memory: "32Gi" }   # size the head generously: GCS lives here
  workerGroupSpecs:
  - groupName: cpu-group
    minReplicas: 2                      # warm pool: always 2 CPU workers
    maxReplicas: 64
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray-ml:2.9.0-gpu
          resources:
            limits:   { cpu: "16", memory: "64Gi" }
            requests: { cpu: "16", memory: "64Gi" }
  - groupName: a100-group
    minReplicas: 0                      # scale to zero when no GPU work is pending
    maxReplicas: 8
    template:
      spec:
        containers:
        - name: ray-worker
          image: rayproject/ray-ml:2.9.0-gpu
          resources:
            limits:   { nvidia.com/gpu: 1, cpu: "12", memory: "80Gi" }
            requests: { nvidia.com/gpu: 1, cpu: "12", memory: "80Gi" }

Read the two worker groups as policy, not plumbing. cpu-group keeps a warm floor of 2 (fast scheduling for steady work); a100-group floors at 0 so you pay nothing for idle GPUs — at the cost of a cold start when GPU work arrives, which §4 quantifies. The nvidia.com/gpu: 1 limit is what makes Kubernetes place each of these pods on a GPU node and what Ray reads back as num_gpus=1 of schedulable resource.

3 · The autoscaler scales on pending demand, not utilization

This is the load-bearing idea of the lesson, and it is the one most engineers get wrong because they reason by analogy to web autoscaling. The Ray autoscaler is a process (on the head) that, in a loop, asks the GCS one question: are there tasks or actors that cannot be scheduled because no node has the logical resources they requested? If a @ray.remote(num_gpus=1) task is sitting in the pending queue and no node has a free GPU, that is a pending resource demand. The autoscaler sums those demands, computes the smallest set of worker-group nodes that would satisfy them, and requests exactly those nodes.

import ray

# These declared resources are the ONLY thing the autoscaler sees.
@ray.remote(num_gpus=1)            # -> creates GPU demand when no GPU is free
def train_shard(shard):
    return fit(shard)

@ray.remote(num_cpus=4)            # -> creates CPU demand the cpu-group can satisfy
def preprocess(block):
    return transform(block)

# Submitting 8 GPU tasks at once on a cluster with 0 GPU workers:
#   pending demand = 8 x {GPU:1}  ->  autoscaler asks for up to 8 a100-group nodes
refs = [train_shard.remote(s) for s in shards[:8]]
ray.get(refs)

Two corollaries fall straight out of this design. First, the autoscaler is blind to what you do not declare. A task that secretly forks a process onto a GPU it never requested with num_gpus produces zero demand — the autoscaler will not add a node, and the task will fight for a GPU that, on paper, is fully free. Declared logical resources are the entire contract. Second, scale-down is the mirror: a worker node that has been idle (no running tasks/actors, no objects in use) for longer than idleTimeoutSeconds is removed.

On Kubernetes the picture becomes two-level autoscaling, and seeing the full chain prevents a class of confusing stalls:

The trap: if your Kubernetes cluster has no GPU node group with headroom and no cluster-autoscaler, step 3 never completes — the Ray pods sit Pending forever and your job hangs with no error. Ray asked correctly; Kubernetes simply had nowhere to put the pods. Both levels must be configured to scale, or you cap out at whatever fixed nodes exist.

4 · The two trade-offs that decide your config

Autoscaling looks like free money — pay only for what you use — until you price the latency it injects and the idle it leaves on the floor. Two worked numbers govern every real autoscaler config.

4.1 · Scale-up latency budget

"Add a GPU worker" is not instant. It is a sum of stages, and for a fresh GPU node from zero it runs roughly:

Sum the worst case: 60 + 240 + 10 + 90 ≈ 400 s, about 6½ minutes, before the first request can be served on that new GPU. Now compare against an SLO. If a serving deployment (Lesson 11) has a traffic spike and its p99 budget is 2 seconds, a purely reactive autoscaler is hopeless: every request that arrives during those 6½ minutes either queues past the SLO or gets dropped. The conclusion is not "autoscaling is bad" — it is that bursty latency-sensitive workloads need warm headroom, not just reactive scale-up. You keep minReplicas above zero (a warm pool already past all four stages) sized to absorb the burst while reactive scale-up catches up. A batch training job, by contrast, happily eats the 6½-minute cold start because its "SLO" is hours — there, minReplicas: 0 is correct and saves real money.

4.2 · Idle cost versus cold start — setting the idle timeout

Scale-down has the symmetric trade-off, controlled by idleTimeoutSeconds: how long a worker sits idle before you release it. Too short and you thrash — release a GPU, pay the 6½-minute cold start to get it back 90 seconds later. Too long and you pay for idle GPUs.

This is the same shape as Serve's batching trade-off in Lesson 11 (wait time vs. utilization) and the predictive-vs-reactive choice everywhere in capacity planning: a warm pool (minReplicas > 0) is a bet that future demand justifies present idle cost. Predictive scaling — pre-scaling on a known daily traffic curve — beats reactive scaling exactly when your cold start is long relative to how fast load ramps, which for GPU workloads it almost always is.

5 · Failure modes & checklist

Checkpoint exercise

Where this points next

You can now stand up a cluster and reason about how it grows. But a cluster is just capacity — it does not yet say how Data, Train, Tune, and Serve hand work to one another across it. Each of those libraries produced and consumed checkpoints, datasets, and models in Lessons 09–11, yet we never named the contract that lets a checkpoint flow train → tune → serve without train/serve skew. Lesson 13 reads the ML platform (the "AIR" pattern) not as a product but as exactly that handoff contract layered on the cluster you just built — and confronts the trade-off that one unified runtime simplifies handoffs while concentrating the blast radius.

Interview prompts

• What does the Ray autoscaler scale on, and how is that different from a Kubernetes HPA? (§3 — pending unschedulable logical resource demands (num_cpus/num_gpus no node can satisfy), not CPU/memory utilization; an HPA scales pods on observed metrics. A 100%-CPU job with nothing queued triggers no Ray scale-up.)
• A GPU task is running but the autoscaler never adds GPU nodes. What is the likely cause? (§3 — the task didn't declare num_gpus=1, so it created no pending GPU demand; the autoscaler is blind to undeclared resources.)
• Walk the two-level autoscaling chain on Kubernetes and name where it can stall. (§3 — Ray autoscaler asks KubeRay for pods → KubeRay creates them → Kubernetes cluster-autoscaler provisions VMs → workers register. It stalls if there's no GPU headroom and no cluster-autoscaler: pods sit Pending and the job hangs silently.)
• Budget the scale-up latency for a fresh GPU serving node and say why it matters. (§4.1 — VM provision ~30–60 s + image pull ~120–240 s + Ray start ~5–10 s + model load ~90 s ≈ 6½ min; a reactive scale-up can't meet a 2 s p99 spike, so latency-sensitive bursts need warm minReplicas.)
• How do you set idleTimeoutSeconds? (§4.2 — against the measured inter-burst gap versus the cold-start penalty: hold the node if a missed SLO costs more than the idle $/min over the gap; release it fast for latency-insensitive batch.)
• When is minReplicas: 0 right, and when is it wrong? (§2, §4 — right for latency-insensitive batch/GPU groups (save idle cost, eat the cold start); wrong for bursty latency-SLO serving, which needs a warm pool already past VM+image+model-load.)
• Why is the head node both a SPOF and a bottleneck, and how do you mitigate each? (§1 — all control-plane traffic (actor creation, task metadata, resources) flows through the GCS on the head; mitigate the bottleneck by sizing it and keeping compute/coordination off it (tree-reduce), and the SPOF by enabling GCS fault tolerance with external Redis.)
• RayCluster vs RayJob vs RayService — when each? (§2 — RayCluster: long-lived, many jobs; RayJob: spins up a cluster for one job then tears down (clean isolation/cost attribution, good for a Tune sweep); RayService: wraps a Serve app with zero-downtime upgrades.)

See also: Kubernetes GenAI — GPU nodes and the platform for how the GPU node group and cluster-autoscaler are provisioned underneath KubeRay, and System Design — scaling and load balancing for the reactive-vs-predictive autoscaling trade-off in the general case.