Part I - Core mental model

Orientation - Ray as distributed Python

Before the first line of API, you need a map — not of Ray's features, but of the one idea those features repeat at every scale. Ray lets you write ordinary Python and have a runtime turn it into a distributed program. This lesson lays out the whole series as a single line of reasoning: start from why a laptop runs out of room, build the two primitives (stateless task, stateful actor) and the object store that connects them, then watch five ML libraries appear as recurring shapes on top of that one runtime. The goal is that nothing later feels like a new system — only the same graph, specialized.

Book source

Map of the whole book: Preface and Chapters 1-11 of Learning Ray. Current-docs calibration: modern Ray frames its ML stack as Ray Data, Train, Tune, Serve, RLlib, and KubeRay, treating the older "AIR" umbrella as a platform pattern rather than a separate API — so this course follows the library names you will actually import today.

Linear position

Prerequisite: comfortable Python (functions, classes, list comprehensions) and the basic ML loop (data → train → evaluate → serve). No distributed-systems background assumed — every term is defined at first use.
New capability: a mental model that lets you look at any workload and name its unit of parallelism and where its large bytes live, before deciding whether Ray is even the right tool.

The plan

Five moves. (1) Establish why single-machine Python hits a wall, and which of three walls you are hitting. (2) Introduce the two primitives — task and actor — and the object store that lets them compose. (3) Make "Ray is not magic" quantitative: serialization, scheduling overhead, and Amdahl's law put hard limits on what wrapping a function in @ray.remote can buy you. (4) Show the five ML libraries as the same task/actor graph in five recurring shapes. (5) Give the reading order and the two questions to ask of every workload.

1 · The wall a laptop hits

A Python process is a single operating-system process with, effectively, one thread doing CPU-bound work — the Global Interpreter Lock (GIL) means two Python bytecode instructions never run truly in parallel inside one interpreter. That is fine until a workload pushes on one of three walls:

data wallThe dataset no longer fits in one machine's RAM, or reading it serially takes longer than the rest of the job. A 2 TB feature table will not load into a 64 GB box.

compute wallThe work is embarrassingly parallel but serial Python is too slow: 100,000 model-scoring calls at 50 ms each is 83 minutes on one core, and the GIL means more threads do not help CPU-bound Python.

model wallThe model itself no longer fits in one accelerator's memory, so training or inference must be split across devices. A 70B-parameter model in fp16 is ~140 GB of weights — larger than a single 80 GB GPU.

The standard escapes each cover part of the problem. multiprocessing forks processes on one machine and makes you marshal data by hand. Spark scales data processing beautifully but assumes a bulk-synchronous dataflow — map a partition, shuffle, reduce — and is awkward for stateful services or fine-grained dynamic task graphs. MPI (the HPC standard) gives you raw collective communication but a static, single-program-multiple-data world where you manage everything. None of them is a general way to take arbitrary Python — stateless functions and long-lived stateful objects — and spread it across a cluster while the dependency graph is still being discovered at runtime.

That gap is Ray's reason to exist, and lesson 01 makes the argument in full. For now, hold the framing: Ray is a runtime that turns ordinary Python units into a distributed, dynamically-built graph of work.

2 · Two primitives and the thing between them

Almost everything in Ray reduces to two ways of shipping Python to a worker, plus one shared place to keep the values that flow between them. A worker here means an ordinary Python process that Ray manages on some node of the cluster.

Task (stateless)

A plain function marked @ray.remote. Calling f.remote(x) ships the function to some worker and returns immediately with a handle. No state survives between calls. This is your unit of compute parallelism — lesson 02.

Actor (stateful)

A class marked @ray.remote. Its instance lives on one worker and remembers state between method calls — for a loaded model, a counter, a connection pool, a simulator. This is your unit of stateful parallelism — lesson 03.

Object store

A shared, in-memory store of immutable values on each node. Task results and ray.put values live here, addressed by an ObjectRef (a future). Workers on the same node read them zero-copy. This is what lets tasks compose without routing every byte through your driver — lesson 04.

Driver

The process running your main(). It submits tasks, creates actors, and calls ray.get to pull results back. The driver owns control flow — and is the first thing you accidentally turn into a bottleneck.

The single most important fact is that f.remote(x) does not wait for the answer. It returns an ObjectRef — a promise of a future value — so your driver can fire thousands of tasks and only later block on the results. The dependency graph builds itself: if you pass one task's ObjectRef as another task's argument, Ray knows the second depends on the first and schedules accordingly. That asynchrony is the entire source of Ray's parallelism, and also the entire source of its sharp edges.

import ray
ray.init()                                   # start/connect to a local cluster

@ray.remote                                  # a stateless task
def score(record):
    return expensive_model(record)           # ~50 ms of CPU work

refs = [score.remote(r) for r in records]    # returns instantly: 100k ObjectRefs
results = ray.get(refs)                       # blocks once, here, for all of them

3 · Ray is not magic — three numbers that bound it

It is tempting to read "wrap it in @ray.remote and it gets faster." Three costs say otherwise, and internalizing them now prevents most beginner disappointment.

Serialization. A value that crosses a worker boundary must be serialized (Ray uses a zero-copy Arrow/Plasma path for many types, and cloudpickle otherwise), shipped, and deserialized. If score does 50 ms of work but each record is a 20 MB array, you may spend more time moving the 20 MB than computing. The fix — put the big value in the object store once and pass its ref — is a recurring theme (lesson 04).

Scheduling overhead. Submitting a task is not free: the system spends on the order of a few hundred microseconds to a millisecond routing it, reserving resources, and tracking its ref. If your tasks each do 100 µs of real work, overhead dominates and the parallel version is slower than the serial loop. Worked number: 1,000,000 tasks at ~0.5 ms overhead each is ~500 s of pure bookkeeping spread across the cluster — so tasks should be coarse enough (rule of thumb: tens of milliseconds or more) that overhead is a few percent, not the whole bill. Lesson 02 makes granularity concrete.

Amdahl's law is the real ceiling

Speedup is capped by the part you did not parallelize. If a job is 95% parallelizable and 5% inherently serial (reading config, a final reduce on the driver), then even with infinite workers the speedup is 1 / 0.05 = 20× — never more. With 100 workers it is 1 / (0.05 + 0.95/100) ≈ 16.8×. So the lever that matters is shrinking the serial 5% (e.g. tree-reduce instead of reducing on the driver, lesson 05), not adding the 101st worker. Most "Ray didn't speed up my code" reports are an unnoticed serial section — often a ray.get inside a loop that quietly re-serializes the program.

4 · The five libraries are five graph shapes

Once tasks, actors, and the object store are clear, the ML libraries stop looking like five separate products. Each is a hardened version of a graph you could build by hand — which is exactly why lesson 06 builds one by hand before lesson 07 hands it to a library.

Library	The recurring shape it packages	Lesson
Ray Data	Map/transform over a stream of data blocks, with shuffle and repartition — the map-shuffle-reduce shape, pipelined so data larger than cluster RAM still flows.	09
Ray Train	N worker actors, one per device, each holding a model replica and exchanging gradients (allreduce) — data-parallel training with checkpoints.	10
Ray Tune	Many independent trials (tasks/actors) racing under a scheduler that kills losers early to reallocate compute — a control plane over experiments.	08
Ray Serve	Autoscaling groups of replica actors behind HTTP, batching requests to fill the GPU and composing models into a graph.	11
RLlib	Many cheap rollout-worker actors generating experience, a few expensive learner actors consuming it — fan-out/fan-in with heterogeneous resources.	07

Notice the through-line: every one is some arrangement of fan out work → move data through the object store → fan results back in, differing only in what is stateful, where the GPUs are, and who owns retries. That is why the core lessons (00–05) come first: learn the shape once, and the libraries become specializations rather than novelties.

5 · Where the bytes live, and where this series goes

Two questions answer most "how should I structure this in Ray" decisions, and they recur in every later lesson:

1What is the unit of parallelism? A task, an actor, a dataset block, a training worker, a tuning trial, a serving replica, or an RL rollout worker. Name it before writing code.

2Where do the large bytes live? Input files, the object store, GPU memory, checkpoints on durable storage, model weights, or response payloads. The answer tells you what to ray.put, what to shard, and where the next bottleneck will appear.

Part I (00–05) builds the runtime: tasks and refs, actors, the scheduler and object store, and the map-shuffle-reduce pattern. Part II (06–11) lifts it into the ML libraries. Part III (12–16) is where toy examples become operational systems: clusters and autoscaling, the platform handoff, the failure model, ecosystem boundaries, and a capstone design.

Failure modes & checklist

Failure modes (this whole series)

Treating Ray as a speed button. A serial bottleneck wrapped in @ray.remote is still serial — now plus scheduling overhead. Amdahl, not worker count, sets the ceiling.
Many tiny tasks. Sub-millisecond tasks pay more in scheduling than they do in compute. Batch the work.
Big objects through small tasks. Passing a 100 MB array into 1,000 trivial tasks serializes 100 GB. Put it once, pass the ref.
Reaching for every library at once. Data + Train + Tune + Serve before the Core graph is clear hides the shape you most need to see.

Orientation checklist

Which of the three walls (data / compute / model) is this workload hitting?
What is the unit of parallelism, named explicitly?
Where do the large bytes live, and which of them should be a shared object-store value?
What fraction of the job is inherently serial — and can it shrink?
Which layer owns retries and recovery: my code, Ray Core, a library, or the cluster manager?

Checkpoint exercise

Try it

Take one Python loop you wrote recently. Annotate each value as input, independent, shared state, or too large to copy casually. Now estimate: how long is one iteration, and how many iterations? If an iteration is 100 µs, Ray's per-task overhead alone may exceed it — the right move is to batch many iterations per task. If an iteration is 2 s and there are 5,000 of them, it is a textbook task-graph. The annotation usually tells you whether the shape is a task graph, an actor service, a dataset pipeline, or not a Ray problem yet.

Where this points next

The map is drawn; now we earn it. Lesson 01 makes the case for Ray's existence rigorously — why the GIL, multiprocessing, Spark, and MPI each leave a gap, and what design principles a runtime needs to fill it. Then lesson 02 writes the first real Ray program and meets the ObjectRef that everything else is built on.

Takeaway

Ray is one idea at several scales: write ordinary Python tasks (stateless) and actors (stateful), let an object store carry values between them, and a runtime builds and schedules the dependency graph for you. It is not magic — serialization, scheduling overhead, and Amdahl's law bound what parallelism can buy, so coarse tasks and a small serial fraction matter more than raw worker count. The five ML libraries (Data, Train, Tune, Serve, RLlib) are not five systems; they are five recurring shapes of the same fan-out / object-store / fan-in graph. Learn the shape in Part I and the rest of the course is specialization.

Interview prompts

Why can't you just use threads to parallelize CPU-bound Python? (§1 — the GIL serializes Python bytecode; you need separate processes/workers, which is what Ray manages.)
What does f.remote(x) return, and why does that matter? (§2 — an ObjectRef (future), immediately; the asynchrony lets the driver build a large dependency graph instead of blocking call-by-call.)
Name the three walls a single machine hits and give an example of each. (§1 — data (dataset > RAM), compute (too many serial calls), model (weights > one accelerator).)
You wrapped your function in @ray.remote and it got slower. Name three causes. (§3 — tasks too fine (scheduling overhead dominates), large objects serialized repeatedly, or an unnoticed serial section / ray.get in a loop capped by Amdahl.)
With a 95%-parallel job, what is the maximum speedup, and what should you optimize? (§3 — 1/0.05 = 20× ceiling; shrink the serial 5%, don't just add workers.)
How is Ray different from Spark for distributed work? (§1 — Spark is bulk-synchronous dataflow over partitions and excels at ETL/SQL; Ray runs arbitrary stateless tasks AND stateful actors with a dynamic task graph, suiting mixed/online/RL workloads.)
Why does this course build an RL app by hand (06) before teaching RLlib (07)? (§4 — every library is a hardened version of a task/actor graph; seeing the shape first makes the library a specialization, not a black box.)