Part III - Cluster and platform layer

Capstone: Design a Ray ML Platform

Lesson 15 drew the boundary of where Ray belongs in a stack; the fifteen lessons before it built the pieces — tasks and actors, the object store, Data, Tune, Train, Serve, KubeRay, the failure model. This lesson spends none of its budget on new API. Instead it takes one concrete product requirement and designs the whole platform end to end, walking every decision with the trade-off it leaves on the table and the lesson it draws from. The deliverable is a senior-engineer design-interview answer: requirements, an architecture you can defend, a decisions table, a worked cost-and-latency budget, and what changes at 10× scale.

Book source

Book synthesis: Chapters 1-11 as one end-to-end design exercise. No new mechanism is introduced; every claim cites a prior lesson. Calibrated to current Ray (Data / Tune / Train / Serve as the library names, KubeRay RayCluster/RayService CRDs, AIR treated as the platform pattern of lesson 13 rather than a separate API).

Linear position

Prerequisite: all of Parts I-III — in particular Data (09), Tune (08), Train (10), the checkpoint handoff (13), Serve (11), KubeRay autoscaling (12), the failure model (14), and the ecosystem boundary (15).
New capability: the ability to take a workload requirement and produce a defensible Ray platform design — naming, for each stage, its unit of parallelism, where the large bytes live, who owns retries, and what it costs per month.

The plan

Six moves. (1) State the product and pin SLOs, because the SLOs decide everything downstream. (2) Lay out the architecture as a stage flow — Ingest → Tune+Train → Checkpoint → Serve — on one KubeRay cluster, each stage with its resources. (3) The decisions table: choice, trade-off, and the lesson behind each. (4) Make the cross-cutting concerns real design decisions: failure model, autoscaling+cost, train/serve skew, observability/SLOs, build-vs-integrate, and the blast radius of one unified cluster. (5) A worked end-to-end resource / cost / latency budget. (6) What you would do differently at 10× scale, then a design-review grid.

1 · The product, and the SLOs that decide the rest

Concrete brief: a session-ranking service for a mid-size e-commerce site. Every page view sends ~40 candidate items; the model scores them and returns a ranked list. Three workloads share one model lineage:

online serve~3,000 ranking requests/sec at peak, each scoring 40 candidates. The product SLO: p99 end-to-end ≤ 80 ms, availability 99.9%. This is the workload that pays the bills, so it is the constraint everything else bends around.

nightly trainRe-train the ranker on the last 30 days of click logs — ~2 TB of Parquet that already lands in S3 from an existing Spark ETL job. Must finish inside a 6-hour window so the new model is live by morning.

weekly tuneA hyperparameter sweep (learning rate, embedding dim, dropout) over ~64 configs to pick next week's architecture. Has a generous 24-hour budget but should not cost more than the value of the improvement it buys.

The SLOs are not decoration — they pick the architecture. The 80 ms p99 forces GPU-backed Serve replicas with bounded dynamic batching (lesson 11). The 6-hour train window sets the worker count for data-parallel Train (lesson 10). The "2 TB already in S3 from Spark" line is the single most important constraint in the brief: it tells us, before any design, that we integrate with an existing Spark/Parquet lake rather than rebuild ingestion in Ray (lesson 15). Write the SLOs down first; a platform tuned for a workload whose SLO you never stated is tuned for nothing.

2 · The architecture: four stages on one KubeRay cluster

The pipeline is the same fan-out / object-store / fan-in graph the whole course has been about (lesson 00), specialized four times. Each stage names its unit of parallelism and where its large bytes live.

1INGEST — Ray Data (lesson 09). Read the 2 TB Parquet as a stream of blocks (Arrow tables, one unit of parallelism each), apply a fitted Preprocessor (normalize numerics, hash categoricals), and feed Train. Streaming execution means the dataset never fully materializes — it flows block-by-block, so 2 TB moves through a cluster with far less than 2 TB of RAM. Large bytes: the Parquet in S3 and the in-flight blocks in the object store.

2TUNE + TRAIN — Tune (08) over Train (10). Tune is the control plane: 64 trials, an ASHA scheduler that kills weak configs at each rung so compute reallocates to promising ones. Each surviving trial is a data-parallel TorchTrainer — N worker actors, one per GPU, exchanging gradients by allreduce. Unit of parallelism: the trial (Tune) and the DDP rank (Train). Large bytes: model replicas in GPU memory, gradients on the wire.

3CHECKPOINT — the handoff (lesson 13). The winning trial writes a Checkpoint to durable storage (S3): model weights plus the fitted Preprocessor. This object is the contract between train-time and serve-time. It is the only thing that crosses the train→serve boundary; nothing else should.

4SERVE — Ray Serve (11). A deployment graph: an ingress deployment fans the 40 candidates into a dynamically batched model deployment running on GPU replicas, each replica loading the same Checkpoint (weights + Preprocessor). Independent autoscaling per deployment. Unit of parallelism: the replica; the batch within a replica. Large bytes: model weights in GPU memory, request/response payloads.

All four run on one KubeRay cluster (lesson 12) with worker groups split by accelerator, because the stages do not scale alike — the whole point of Ray's heterogeneous scheduling (lesson 00):

Worker group	Resource / role	Scaling signal
head	1 pod: GCS, dashboard, the Ray autoscaler. No workload.	fixed (the SPOF — §4)
cpu-ingest	CPU pods for Ray Data blocks + Preprocessor.	pending Data tasks
gpu-train	A100/H100 pods, used only nightly/weekly.	pending Train/Tune demand → scale to 0 when idle
gpu-serve	L4/A10 pods, always warm, behind the SLO.	Serve replica autoscaling (request load)

Splitting gpu-train from gpu-serve is deliberate. Training wants a few big expensive GPUs for hours; serving wants many cheaper GPUs always on. One pooled group would either starve serving during a training run or pay A100 prices to serve 40-item rankings. The Ray autoscaler scales each group on pending logical resource demand, not CPU utilization (lesson 12) — when no Train job has requested GPUs, gpu-train drains to zero and the k8s cluster-autoscaler reclaims the nodes.

3 · The decisions table

The core of any design doc. Each row is a real fork in the road with what it costs and where it was argued.

Decision	Choice	Trade-off you accept	Lesson
Ingest engine	Read existing S3 Parquet with Ray Data; do not rebuild the Spark ETL	Two systems to operate (Spark + Ray), but you avoid re-implementing mature SQL ETL in a tool that is not built for it	15, 09
HPO strategy	ASHA early-stopping over 64 configs	Aggressive halving can kill a late-blooming config; you trade a small accuracy risk for ~3-4× less GPU-hours than training all 64 fully	08
Train parallelism	Data-parallel `TorchTrainer`, 4 GPUs	Works only because the ranker fits on one GPU; a larger model would need tensor/pipeline parallel (system_ml)	10
Train→serve handoff	One Checkpoint carrying weights + Preprocessor	Couples the two stages to a serialization format; in exchange, train/serve skew is structurally impossible (§4)	13
Serve batching	Dynamic batching, `max_batch_size=64`, `batch_wait_timeout_s=0.005`	Up to 5 ms added latency under light load, in exchange for filling the GPU and hitting throughput	11
Cluster topology	One KubeRay cluster, worker groups per accelerator	Shared object store and simple handoffs vs a single blast radius (§4)	12, 13
LLM-style models	If a future feature needs an LLM, serve it via a dedicated engine (vLLM) behind a Serve deployment	One more dependency, but you do not reimplement a paged-attention server in Ray	15

4 · Cross-cutting concerns — each a real decision, not a checkbox

The failure model across stages (lesson 14)

Ray's defaults differ by stage, and the design must match retry semantics to idempotency. Ingest: Ray Data tasks are deterministic and side-effect-free, so a lost block is reconstructed from lineage by re-reading its Parquet input — cheap, automatic, idempotent. Train: a worker dying mid-epoch is not idempotent — the model state is in GPU memory and would be lost. So Train checkpoints to S3 every epoch; on failure it resumes from the last checkpoint. With a 6-hour run over, say, 12 epochs (30 min each), a worker failure costs at most one epoch of redo — ~30 min of expected lost work, not 6 hours. Serve: a replica crash is handled by the deployment respawning it; in-flight requests on that replica fail and the client retries (idempotent — scoring is a pure read). The head node is the exception: it owns the GCS, so its loss takes the control plane down. Mitigate with GCS fault tolerance (external Redis) so a head restart does not drop the running cluster (lesson 12).

The contract: at-least-once means design for idempotency

Ray retries give at-least-once execution (lesson 14). That is safe for the pure-function stages (ingest, scoring) and unsafe for anything with a side effect. Concretely: never let a Train step append to an external metrics table without a dedup key, or a retried task double-counts. The serving path writes nothing, so it is safe by construction — which is itself a design choice worth making on purpose.

Autoscaling and cost (lesson 12)

The cost lever is the gpu-train group scaling to zero when idle, against a warm pool for serve. Back-of-envelope monthly bill for the GPU groups, at ~$2/hr per serve GPU (L4-class) and ~$3.5/hr per train GPU (A100-class):

Worked monthly GPU cost

Serve: to hold 3,000 req/s at p99 (see §5) we keep 6 L4 replicas warm 24/7. 6 × $2/hr × 730 hr = $8,760/mo. We do not scale serve to zero — a cold start (pod schedule + image pull + 4 GB model load ≈ 90 s) would blow the 80 ms SLO on the first request, so the floor stays warm and only the headroom autoscales.
Train: nightly run uses 4 A100 for ~4 hr = 16 GPU-hr/night × 30 = 480 GPU-hr. Weekly tune (after ASHA) ≈ 200 GPU-hr/run × 4 = 800 GPU-hr/mo. Total ≈ 1,280 GPU-hr × $3.5 = ≈ $4,480/mo — but only because the group sits at zero the other ~95% of the time. If it stayed pinned at 4 A100 all month it would be 4 × $3.5 × 730 = $10,220/mo, so scale-to-zero saves ~$5,700/mo. Combined ≈ $13,240/mo for GPUs.

The residual trade-off: scale-to-zero on train means the first nightly job each day eats cold-start latency (node provision + image + weights ≈ 3-5 min). For a 4-hour batch job that is noise; for serving it would be fatal — which is exactly why the two groups have opposite policies.

Train/serve skew prevention (lesson 13)

The classic production bug: the model trained on features normalized one way, but the serving path normalizes differently, so live accuracy quietly degrades. The design makes this impossible by construction: the Preprocessor is fitted once during ingest, then travels inside the Checkpoint. Serve loads the same object, so the exact same normalization runs in both places. There is no second code path to drift. This is the whole reason the handoff in §2 carries the Preprocessor and not just the weights.

Observability and SLOs (lesson 14)

Two SLO families, watched separately (lesson 14's "jobs vs serving" split). For serving: p99 latency (the 80 ms budget), replica queue depth, and object-store pressure — if the object store fills, Ray spills to disk and the spill latency cliff (lesson 04) shows up as p99 spikes; alert on object-store memory before it hits the spill threshold. For jobs: train completion time against the 6-hour window, and GPU utilization (if Train workers sit < 80% busy, the Ray Data pipeline is starving them — rebalance CPU ingest workers, lesson 09). The dashboard, timeline, and ray memory are the instruments (lesson 14).

Build on Ray vs integrate vs replace (lesson 15)

The boundary is drawn three times in this design. Integrate: read from the existing Spark/Parquet lake — Ray Data is not a Spark replacement for big SQL ETL, so we consume its output instead. Build on Ray: the train→tune→serve loop, where Ray's dynamic graph and shared object store are exactly the fit. Replace nothing rashly: if an LLM feature appears, serve it through a dedicated engine (vLLM) behind a Serve deployment — Serve does the autoscaling and composition it is good at, the engine does the paged-attention decoding Ray should not reimplement.

The generality trade-off and blast radius (lessons 01, 13)

One unified cluster is the choice that buys the most and risks the most. The win (lesson 13): one runtime, one object store, trivial handoffs — a Checkpoint written by Train is one ref away from Serve. The cost (lesson 01's "generality isn't free"): a single blast radius. A bad model rollout, an OOM in a Train job, or a head-node fault can touch the serving path that owns the 80 ms SLO. The mitigation is the worker-group split (training cannot evict serving pods) plus GCS fault tolerance — but the honest design note is that a fully separate serve cluster would isolate blast radius at the cost of a harder handoff. We accept shared-cluster risk here because the handoff simplicity and utilization are worth more at this scale; §6 revisits that at 10×.

5 · The worked end-to-end budget

Three numbers a reviewer will ask for: does serving meet 80 ms, does training fit 6 hours, does tuning fit its budget.

Serving latency budget (the 80 ms p99)

One request scores 40 candidates. On an L4, a forward pass over a batch of 40 takes ~6 ms. Dynamic batching coalesces concurrent requests up to max_batch_size=64; at 3,000 req/s spread over 6 replicas that is 500 req/s/replica, so batches fill in well under the 5 ms batch_wait_timeout. Budget: network in/out ≈ 8 ms, Preprocessor ≈ 3 ms, queue wait ≤ 5 ms, GPU forward ≈ 6 ms, ingress fan-out/serialize ≈ 10 ms → p50 ≈ 32 ms. p99 adds queueing + a GC pause + tail batch wait ≈ +35 ms → p99 ≈ 67 ms, inside the 80 ms SLO with ~13 ms of headroom. If load doubles, the batch-wait knob and replica autoscaler absorb it before the budget breaks.

Training-window budget (the 6-hour limit)

2 TB Parquet, 4 A100 data-parallel. Suppose ~1.2 B training rows, 12 epochs. At 4 GPUs each doing ~40k rows/s (ranker is small), aggregate ≈ 160k rows/s → 1.2 B / 160k ≈ 7,500 s ≈ 2.1 hr/epoch is far too slow — so we sub-sample to the most recent + hard negatives (~150 M rows): 150 M / 160k ≈ 940 s ≈ 16 min/epoch × 12 ≈ 3.1 hr, plus ingest warmup and checkpointing ≈ 3.5 hr. Fits the 6-hour window with margin, and the margin is the failure budget — one epoch of redo (16 min) after a worker crash still lands before morning. Scaling efficiency caveat (lesson 10): going 4→8 GPUs is not 2× because allreduce comm grows; measure before assuming it.

Tuning-budget math (ASHA vs brute force)

64 configs. Brute force = train all fully: each config is 3.1 wall-hours on 4 GPUs = 12.4 GPU-hr, so 64 × 12.4 ≈ 800 GPU-hr. ASHA with halving rungs (lesson 08) trains all 64 for a short rung, keeps the top 32, then 16, 8, 4, 1 — most configs die after a fraction of training, so total ≈ 200-250 GPU-hr, roughly 3-4× cheaper for nearly the same winner. The residual risk: a config that only shines late gets cut early; if the team sees that pattern, raise ASHA's grace period — trading some savings back for safety.

6 · What changes at 10× scale

Push the brief to 30,000 req/s and 20 TB nightly. The design does not break uniformly — specific decisions flip:

split the clusterThe shared-cluster blast-radius bet (§4) stops paying. At 10× the serving fleet is large enough that a training OOM taking it down is unacceptable. Move gpu-serve to its own KubeRay cluster (or KServe), keep the Checkpoint as the handoff over S3. You trade handoff simplicity for isolation — the opposite call from §4, made for a reason (lesson 13).

ingest stops being free20 TB streamed nightly may need more than re-reading Parquet — materialize a feature store or partition by recency so each train reads only its slice (lesson 09 block-size and partition tuning).

train hits the memory wall maybeIf the 10× traffic justifies a much larger model that no longer fits one GPU, data-parallel is not enough — you cross into tensor/pipeline parallelism (system_ml). The Train abstraction stays, the ScalingConfig changes.

serve autoscaling gets a warm-pool tierAt 30k req/s a traffic spike that needs +20 replicas can't wait 90 s of cold start per pod. Keep a warm spare pool (lesson 12) sized to the spike rate, paying idle cost to protect p99 — the cost/SLO knob, turned toward SLO.

7 · Design review — failure modes & checklist

Failure modes

Reaching for Ray before the SLO is stated. Without the 80 ms / 6-hour numbers, every later choice (batch size, worker count, warm pool) is arbitrary. SLOs first (§1).
Preprocessor left behind at the handoff. Shipping only weights and re-implementing normalization in the serving code is the canonical train/serve skew bug. The Preprocessor must travel in the Checkpoint (§4).
Cost as a dashboard afterthought. If gpu-train never scales to zero you pay $10k/mo for ~5% utilization. Cost is a scheduling/topology decision, not a post-hoc chart (§4).
Scaling serve to zero. A cold start blows the latency SLO on the first request. Floor stays warm; only headroom autoscales (§4).
Assuming retries are free. At-least-once execution double-counts any non-idempotent side effect. The serving path writes nothing on purpose (§4).
One cluster forever. The shared-cluster blast radius is fine at 1× and a liability at 10×; revisit the boundary as scale grows (§6).

Implementation checklist

Are the serving and job SLOs written down as numbers before any component is chosen?
For each stage: what is the unit of parallelism, and where do the large bytes live?
Does the Checkpoint carry the Preprocessor, so train and serve share one feature path?
Which stages are idempotent, and what is the expected redo work after a failure of each?
Does each worker group scale on the right signal, and which group must stay warm?
Is the head node's GCS made fault-tolerant, given it is the shared blast radius?
What is integrated (Spark lake), built on Ray (train→serve loop), and left to a dedicated engine (LLM)?
Have you run a failure drill: kill a Train worker, crash a Serve replica, restart the head?

Checkpoint exercise

Try it

The product adds a second model: a "similar items" embedding model that the ranking model now consumes as a feature, and a new EU region with data-residency rules. Walk the decisions that change. Which become a multi-deployment Serve graph (the embedding model feeding the ranker) versus a separate service? Does the single-cluster bet (§4) survive a region that legally cannot share storage with the US? What happens to the Checkpoint handoff when there are now two model lineages with different train cadences? The right answer should flip several §3 rows — and the region constraint should force at least one shared-cluster decision the other way, exactly as the 10× case did.

Where this points next

This is the last lesson, so "next" is the rest of the stack. The Ray track gave you one substrate; the boundary lesson (15) named the systems on either side of it. From here the natural continuations are the ones this design kept gesturing at: the model-parallelism that takes over when data-parallel hits the memory wall, the Kubernetes-native serving platforms that compete with a Ray-only serve cluster, and the broader ML-systems design that treats Ray as one layer among many. Follow those cross-track links, or return to the index to revisit any stage you want to redesign.

Takeaway

A Ray ML platform is not a pile of Ray APIs — it is a set of explicit contracts. State the SLOs first; they pick the architecture. Build the pipeline as one fan-out/object-store/fan-in graph specialized four times — Ingest (Data) → Tune+Train → Checkpoint → Serve — on one KubeRay cluster with worker groups per accelerator, because the stages do not scale alike. Make the cross-cutting concerns real decisions: match retry semantics to idempotency (at-least-once means the serving path writes nothing), scale gpu-train to zero but keep gpu-serve warm (cold start would blow the SLO), and carry the Preprocessor inside the Checkpoint so train/serve skew is structurally impossible. The single unified cluster buys handoff simplicity and utilization at the price of one blast radius — a bet that is correct at this scale and the first thing you reverse at 10×. The whole skill is naming, for every stage, its unit of parallelism, where the bytes live, who owns retries, and what it costs.

Interview prompts

You're asked to design a Ray platform. What do you nail down before drawing any architecture? (§1 — the SLOs as numbers: serving p99 and availability, job completion windows. They decide GPU choice, batch size, worker count, and warm-pool policy; a design without stated SLOs is tuned for nothing.)
Why split gpu-train and gpu-serve into separate worker groups? (§2, §4 — opposite needs: training wants a few expensive GPUs for hours and can scale to zero; serving wants many cheaper GPUs always warm. The Ray autoscaler scales each on pending logical demand, so a pooled group would starve serving or overpay.)
How does this design prevent train/serve skew? (§4, lesson 13 — the Preprocessor is fitted once at ingest and travels inside the Checkpoint, so serve runs the identical feature path; there is no second normalization code path to drift.)
A Train worker dies 3 hours into the nightly run. What's the expected lost work, and why? (§4 — at most one epoch (~16-30 min), because Train checkpoints to S3 each epoch and resumes; GPU model state is not idempotent so it must be checkpointed, unlike ingest blocks which reconstruct from lineage for free.)
Why not scale the serving group to zero off-peak to save money? (§4 — a cold start (pod schedule + image pull + ~4 GB weight load ≈ 90 s) would violate the 80 ms p99 on the first request. The floor stays warm; only headroom autoscales. Training, a batch job, can absorb the cold start, so it scales to zero.)
Walk the 80 ms p99 latency budget. (§5 — network ≈ 8 ms, Preprocessor ≈ 3 ms, queue/batch wait ≤ 5 ms, GPU forward over 40 candidates ≈ 6 ms, ingress fan-out ≈ 10 ms → p50 ≈ 32 ms; tail effects push p99 ≈ 67 ms, inside the SLO with headroom.)
What's the cost of the single unified cluster, and when do you reverse it? (§4, §6 — one blast radius: a training OOM or head-node fault can touch the serving SLO. Mitigated by worker-group isolation + GCS fault tolerance at 1×; at 10× you split serve onto its own cluster, trading handoff simplicity for isolation.)
Where do you integrate rather than build on Ray, and why? (§3, §4, lesson 15 — read the existing Spark/Parquet lake instead of rebuilding ETL, and serve any LLM via a dedicated engine behind a Serve deployment. Ray is a general substrate, not a replacement for mature SQL ETL or a paged-attention server.)