Part I - Core mental model

Why Ray Exists

Lesson 00 drew the map: ordinary Python tasks and actors, an object store between them, and a runtime that builds the dependency graph. It also named the gap Ray fills, but only in a sentence. This lesson is the rigorous version of that sentence. We start from why a single Python process runs out of room — the GIL, then the three independent scaling walls — and then put Ray on trial against the tools that already exist: multiprocessing, Dask, Spark, MPI. The point is not to declare Ray the winner; it is to see exactly which shape of problem each tool fits, where the seam between them is, and what generality costs you. By the end you should be able to look at a workload and say "this is a Spark job" or "this needs Ray" with a reason, not a vibe.

1 · The constraint: one process, one GIL

A CPython process executes Python bytecode under a single Global Interpreter Lock (GIL) — a mutex that lets exactly one thread interpret bytecode at any instant. Threads still help when the work is I/O-bound (a thread holding the GIL releases it while it waits on a socket or disk), but for CPU-bound Python — a scoring loop, a feature transform, a simulation step — adding threads buys nothing: they take turns under the lock. The only way to actually run Python bytecode on more than one core is to run more than one process. That single fact is the root of everything that follows; distribution in Python is fundamentally about coordinating processes, not threads.

So "scale Python" decomposes into two sub-problems: run many processes, and move data between them (because separate processes do not share memory — values must be serialized, shipped, and deserialized). Every tool below is some answer to those two sub-problems. They differ in how much they automate, and in what they assume about the shape of your computation.

2 · Three walls, not one

"I ran out of one machine" is actually three different complaints, and the right tool depends on which one you are making. Lesson 00 introduced these; here they are the axes we will score every tool against.

These are independent. A batch-scoring job hits the compute wall but not the model wall. Training a 70B model hits the model wall and usually the data wall too. A real ML platform hits all three at different stages — which is the deep reason a single specialized engine rarely covers an end-to-end pipeline, and the first hint at what Ray is trying to be.

3 · The tools that already exist — and where each stops

Before adopting anything new, walk the existing options. Each is excellent inside its assumed shape and awkward outside it. The seam is always the same: how rigid is the computation graph it expects, and does it model state.

multiprocessing — the standard-library floor

Python's multiprocessing.Pool forks worker processes on one machine and maps a function over an iterable. It directly solves the GIL problem for embarrassingly parallel CPU work on a single box. Two hard limits: it does not cross machine boundaries (a 4-core laptop stays 4-core), and it makes you hand-marshal data — every argument is pickled to the worker and every result pickled back, with no shared store, so passing a 500 MB array to 100 tasks copies it 100 times. It has no notion of a task that depends on another task's output except through your own orchestration, and no first-class long-lived stateful worker. It is the right tool when the job fits one machine and the data per task is small.

Dask — multiprocessing's distributed, Python-native cousin

Dask scales Python — particularly NumPy/pandas-shaped work — across a cluster, with a distributed scheduler and a real task graph. For "my pandas dataframe got too big" or "parallelize this array computation," Dask is often the lightest answer and stays idiomatically Python. Its sweet spot is collection-style parallelism (Dask DataFrame / Array) and its scheduler does handle arbitrary task graphs. Where it gets thinner than Ray: stateful long-lived services (actor support exists but is not the center of gravity), heterogeneous GPU+CPU pipelines, and the dense ML-serving / RL / distributed-training stack. Dask and Ray overlap most on data; they diverge as the workload becomes stateful, online, or accelerator-heavy.

Spark — the bulk-synchronous dataflow engine

Spark is the mature answer to the data wall for ETL and SQL. Its model is bulk-synchronous partitioned dataflow: data lives as partitions, you express a DAG of transformations (map, filter, join, groupBy), and Spark executes it in synchronized stages separated by shuffles. Crucially the graph is declared up front and largely static — which is exactly what lets the Catalyst optimizer rewrite it (push down filters, reorder joins, pick join strategies) before a single byte moves. That whole-program optimization is Spark's superpower and its constraint: the model assumes coarse, stage-structured batch transforms over data, and is awkward for fine-grained dynamic graphs, mutable stateful workers, tight iterative loops with per-step control flow, or GPU-bound model serving. You would not build an online inference service or an RL training loop in Spark.

MPI — the HPC substrate

MPI (Message Passing Interface) is the HPC standard for the model wall and tightly-coupled numerical computing. It is SPMD — single program, multiple data: every one of N ranks runs the same program, and you, the programmer, explicitly orchestrate communication (send, recv, allreduce) between them. It is the fastest path to a gradient allreduce across GPUs because nothing is hidden. Its cost is also that nothing is hidden: the world size is fixed (no rank joins or leaves mid-run), there is no built-in fault tolerance (one dead rank typically kills the job), no dynamic task spawning, and no scheduler — you hand-manage placement and communication. MPI is superb when the computation is a fixed, regular, tightly-synchronized numerical kernel and brutal when it is dynamic or heterogeneous.

4 · The gap Ray fills, and the principles that fill it

Read the table down the "shape" axis and a hole appears. Spark assumes a static, coarse, stateless dataflow. MPI assumes a static, regular, hand-communicated SPMD kernel. multiprocessing and Dask are Python-native but thin on first-class state and (for multiprocessing) on cluster scale. Nothing in the list is a general-purpose way to distribute arbitrary Python that is simultaneously: stateless and stateful, dynamic (a task can decide at runtime to spawn more tasks), heterogeneous (CPU and GPU work in the same graph), and fault-tolerant — without making you a distributed-systems engineer first.

That hole is Ray's reason to exist. Its design is five principles, and each one is the direct answer to a wall or a limitation above — this is the part to actually understand, not memorize.

The smallest possible demonstration: the local loop and its Ray form are nearly the same code, but the Ray form returns futures immediately and lets a runtime fan the work across a cluster of processes that the GIL would otherwise have prevented.

import ray
ray.init()                                    # start or connect to a cluster

# --- local: one process, GIL-bound, 13.9 hours for 1M @ 50 ms ---
results = [score(r) for r in records]

# --- Ray: stateless tasks fanned across the cluster ---
@ray.remote                                   # a function becomes a task
def score(record):
    return expensive_model(record)            # ~50 ms of CPU work

refs = [score.remote(r) for r in records]     # returns instantly: N ObjectRefs (futures)
results = ray.get(refs)                        # block once, collect all results

# --- state when you need it: a class becomes an actor (lesson 03) ---
@ray.remote
class Model:                                   # loaded ONCE, reused across calls
    def __init__(self, path): self.net = load(path)
    def score(self, r): return self.net(r)
m = Model.remote("ckpt.pt")
out = ray.get(m.score.remote(record))          # method call dispatched to the actor

5 · The residual trade-off: generality forfeits whole-program optimization

Generality is never free, and it is intellectually dishonest to pretend Ray is strictly better than the engines it competes with. The price is precise: because Ray discovers the task graph dynamically and runs arbitrary Python, it cannot perform the whole-program optimization a narrow engine can. Spark's Catalyst optimizer sees your entire query as a static relational plan and rewrites it — pushing filters before joins, choosing a broadcast vs shuffle join, pruning unread columns — before execution. A bespoke serving engine can fuse a fixed model graph and pre-compile kernels. Ray sees a stream of opaque Python tasks arriving at runtime; it schedules them well, but it will not fuse two of your maps into one pass, will not reorder your computation, and will not auto-pick a join strategy. You own those decisions.

6 · Failure modes & checklist

Checkpoint exercise

Where this points next

We have argued why Ray exists and named its five principles, but we have not yet written a real program against the smallest one. The whole edifice rests on a single mechanism: f.remote() returns an ObjectRef — a future — immediately, and passing one ref as another task's argument is what builds the dynamic graph. Lesson 02 makes that concrete: how futures defer work, why ray.get in a loop quietly re-serializes your program back into a serial bottleneck, how ray.put shares a big value once, and the granularity math that decides whether a task is worth its scheduling overhead.

Interview prompts

• Why can't threads parallelize CPU-bound Python, and what does that imply for any scaling tool? (§1 — the GIL serializes bytecode; only multiple processes run Python on multiple cores, so every tool here is really process orchestration + data movement.)
• Name the three scaling walls and give an example of each. (§2 — data: 2 TB table > 64 GB RAM; compute: 1M × 50 ms = 13.9 h serial; model: 70B fp16 ≈ 140 GB > 80 GB GPU.)
• How is Ray's execution model different from Spark's? (§3, §4 — Spark is a static, coarse, bulk-synchronous dataflow DAG (which is what lets Catalyst plan it); Ray builds a fine-grained dynamic task graph at runtime and models first-class state via actors.)
• Why is MPI a poor fit for an RL training loop? (§3 — MPI is fixed-size SPMD with hand-coded messages, no dynamic task spawning, no fault tolerance; RL needs heterogeneous resources, asynchronous rollouts, and worker restarts — Ray's model.)
• State Ray's five design principles and tie each to a problem it solves. (§4 — functions→tasks & classes→actors (keep writing Python); dynamic graphs (vs static DAG/SPMD); actor state (vs stateless engines); heterogeneous resources (CPU+GPU in one graph); object store (zero-copy share, avoids re-serializing big values).)
• What does Ray give up by being general-purpose? (§5 — whole-program optimization: it will not fuse maps, reorder, or pick join strategies the way Spark's Catalyst does; it optimizes scheduling and data movement, not your algorithm.)
• A nightly job is a big filter-then-join over Parquet. Ray or Spark, and why? (§5 — Spark: it is a static plannable relational transform; Catalyst pushes the filter into the scan and broadcasts the small side, saving I/O Ray would not avoid automatically.)