Part III - Cluster and platform layer

Ecosystem, Integrations, and Alternatives

Lesson 14 closed the runtime story: you now understand Ray's failure model, observability, and the at-least-once semantics that make a Ray cluster operable. But no Ray cluster runs in a vacuum — it lands in an organization that already has a data warehouse, a Spark ETL pipeline, a Kubernetes platform, and maybe an HPC allocation. The hardest decisions left are not about Ray's API; they are about its boundary. This lesson makes the honest positioning argument: Ray is strongest when you know precisely what it should replace, what it should integrate with, and what it should leave alone. The through-line is the generality trade-off from lesson 01 — a general substrate buys flexibility by giving up the whole-program optimization a narrow engine performs.

1 · The generality trade-off, made concrete

Lesson 01 asserted that Ray's generality has a price: a runtime that runs arbitrary Python — stateless tasks and stateful actors, with a dependency graph discovered at runtime — cannot perform the whole-program optimization a narrow engine performs over a restricted, declarative program. This lesson cashes that claim out, because it is the single fact that decides every boundary question below.

Consider a SQL aggregation: SELECT region, SUM(revenue) FROM sales WHERE year = 2025 GROUP BY region. Spark's Catalyst optimizer (its query planner) sees the entire query as a declarative plan before a single byte moves. It can push the year = 2025 filter down into the Parquet reader so most rows are never read, prune every column except region and revenue, and pick a hash-aggregate strategy — all because the program is small, closed, and relational.

Now write the same job in Ray as a graph of Python tasks: a task per file that filters and sums. Ray's scheduler sees opaque Python closures. It cannot know that year = 2025 means it should skip Parquet row-groups, nor that only two columns matter. It will faithfully run exactly the tasks you wrote.

Flip the workload and the trade-off flips with it. The moment your job is "filter a bit of data, then run a Python UDF that loads a PyTorch model and scores each row on a GPU, then feed the survivors into distributed training," the declarative engine is the one out of its element: it has no first-class notion of a long-lived GPU actor, heterogeneous CPU+GPU placement, or a dynamic graph where tasks spawn tasks. That is exactly where Ray's generality pays for itself. The rule that falls out: match the engine to the shape of the program, not to a feature checklist.

2 · Three verbs: replace, integrate, leave alone

Every ecosystem decision reduces to assigning one of three verbs to each incumbent system. There is no fourth option, and "rewrite the whole platform in Ray" is the mistake of choosing replace for systems that should be integrate or leave alone.

The cost of the verb is asymmetric. Choosing integrate when you could have replaced leaves a little glue around; choosing replace when you should have integrated means re-implementing (and re-debugging) someone's mature SQL planner, governance layer, or scheduler — a multi-quarter project that usually ends back at integrate. When unsure, default to integrate and earn the right to replace later.

3 · The alternatives, honestly compared

Each row defines the system in one line, then states what it genuinely wins and where Ray wins. The verb column is the default recommendation — the boundary to draw on the whiteboard.

Two rows deserve emphasis because they are the most commonly mis-decided.

Spark is the canonical "integrate, do not replace." Ray Data is not a Spark-SQL replacement. It has no Catalyst, no decade of join optimization, no lakehouse governance. What Ray Data is good at is the last mile: streaming blocks through a Python/GPU transform that feeds Ray Train without materializing the whole dataset (lesson 09). So the pattern is: Spark does the heavy relational ETL and writes Parquet/Delta; Ray reads those files and does the GPU-bound preprocessing + training. You replace neither — you hand off at the file boundary.

vLLM is the canonical "specialized engine inside a general one." A dedicated LLM serving engine like vLLM (see the vllm track) wins on raw token throughput because it does whole-program optimization for one shape: a paged KV cache and continuous batching that a general @serve.batch deployment cannot match. The right move is not to compete with it — it is to let Ray Serve compose it: Serve owns autoscaling, request routing, multi-model ensembles, and the pre/post-processing graph, while each replica wraps a vLLM engine. Same generality trade-off, resolved by nesting the narrow engine inside the general substrate.

# Ray Serve as the orchestrator; vLLM as the specialized inner engine.
from ray import serve
from vllm import LLM, SamplingParams

@serve.deployment(num_replicas=4, ray_actor_options={"num_gpus": 1})
class LLMReplica:
    def __init__(self):
        # the narrow engine does the hard part: paged KV cache, continuous batching
        self.engine = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

    async def __call__(self, request):
        prompts = (await request.json())["prompts"]
        outs = self.engine.generate(prompts, SamplingParams(max_tokens=256))
        return [o.outputs[0].text for o in outs]

# Serve owns what it is good at: HTTP ingress, autoscaling 4 replicas,
# routing, and (next line) composing this with a retriever / re-ranker.
app = LLMReplica.bind()        # serve.run(app) — Ray supplies the *platform* around vLLM

4 · A worked positioning example

Make it concrete. A team has a mature Spark pipeline producing a 5 TB feature table in Delta on S3, refreshed nightly. They now need to train a recommendation model on GPUs and serve it online. The reflex is "migrate everything to Ray." The right answer is one replace, one integrate, one leave alone.

The boundary that makes this work is a boring interface: Parquet/Delta files on S3. Each side owns its strength; neither reaches into the other's internals. That is the shape of almost every good Ray integration — files, checkpoints, metrics, HTTP/gRPC, explicit configs — never a hidden in-process dependency.

5 · When Ray Serve fronts a workflow engine vs is fronted by one

Workflow engines (Temporal, Airflow) sit at a different layer than Ray, and conflating the two is a common boundary error. A durable workflow engine guarantees that a multi-step, possibly multi-day process survives crashes: it persists every step's state, replays on failure, fires timers, and waits on human approval. Ray's own task graph is a lighter, in-cluster dependency graph (lesson 02) — fast and fine-grained, but its durability is lineage-based reconstruction within a live cluster, not a persisted multi-day journal.

The decision is "which guarantee do I need at this layer?" Need durable, day-spanning, human-in-the-loop orchestration? That is the workflow engine's job — leave it alone and make Ray a step. Need fast in-job fan-out/fan-in? That is Ray's task graph — do not pull in a workflow engine for it.

6 · Failure modes & checklist

Checkpoint exercise

Where this points next

You can now defend Ray's boundary against every major alternative and assign each incumbent a verb. The final lesson stops arguing boundaries and builds the platform: lesson 16 is the capstone — one end-to-end pipeline (ingest with Ray Data → tune+train with Tune+Train → checkpoint → serve with Serve, on KubeRay) with the cross-cutting concerns (failure model from lesson 14, autoscaling from lesson 12, cost, and train/serve skew from lesson 13) made explicit. Everything you decided here — what Ray owns and what it hands off — becomes a design constraint there.

Interview prompts

• Why isn't Ray Data a drop-in replacement for Spark? (§1, §3 — Spark's Catalyst does whole-program relational optimization (predicate pushdown, column pruning) that a general Python task graph cannot; for heavy SQL ETL Ray reads Spark's output rather than replacing it.)
• State the generality trade-off and give a number. (§1 — a general substrate gives up the narrow engine's plan optimization; on a 2 TB Parquet table, pushdown+pruning reads 16 GB vs 2 TB for naive whole-file Ray tasks, ~125×.)
• A team has Spark ETL and needs GPU training. What do you do? (§4 — leave the Spark ETL alone, integrate via Ray Data reading its Delta/Parquet output, replace only the bespoke training/inference glue with Ray Train/Serve. Hand off at the file boundary.)
• How does Ray relate to vLLM for LLM serving? (§3 — vLLM out-serves a generic deployment on raw token throughput (paged KV cache, continuous batching); Ray Serve composes it — autoscaling, routing, ensembles — wrapping a vLLM engine per replica.)
• Ray vs Slurm/MPI — when each? (§3 — HPC wins on tightly-coupled bulk-synchronous SPMD jobs at extreme scale with low-overhead collectives; Ray wins on dynamic graphs, heterogeneous tasks, and Python ergonomics. Ray can run inside a Slurm allocation.)
• When is a workflow engine the right layer, not Ray's task graph? (§5 — when you need durable, multi-day, human-in-the-loop orchestration with a persisted journal; Ray's graph is fast in-cluster lineage, not durable orchestration. Make Ray a step inside, or run a workflow layer on top of Ray.)
• How do you avoid over-subscribing nodes when integrating an external library? (§6 — wrap its unit of work as a task/actor with explicit num_cpus/num_gpus so the scheduler sees the resources; never let it spawn hidden threads/GPU contexts.)