Part II - Applications and libraries

RLlib: Scalable Reinforcement Learning

In lesson 06 you built a reinforcement-learning loop by hand and watched four roles emerge: an environment to simulate, a policy to choose actions, rollout workers to collect experience, and a learner to update the policy from that experience. You also watched the bottleneck migrate — from compute, to coordination, to the single learner — as you added simulators. RLlib is that exact graph, hardened. This lesson maps each hand-built role onto an RLlib component, and then spends its weight on the one engineering fact that makes RL a Ray problem rather than a single-box problem: the two halves of the loop want completely different hardware, and reconciling that is the whole game.

1 · The four roles, now named by RLlib

In lesson 06 you wrote the roles yourself. RLlib gives each one a component and a contract, so the code you keep writing is the part that is actually specific to your problem — the environment and the reward — while the distributed plumbing is configured, not rebuilt.

The data path is exactly the fan-out / object-store / fan-in shape lesson 00 promised: EnvRunners fan out across cheap CPU nodes, each producing sample batches that travel through Ray's object store to the learner, which fans them in, computes one gradient step, and then broadcasts the updated weights back out to every EnvRunner so the next rollouts use the fresh policy. That last broadcast — weights flowing back the other way — is the detail lesson 06 had you write by hand and is the source of the staleness question in §4.

2 · The minimal loop, and what one train() does

RLlib's modern API is a builder: you chain configuration methods onto an AlgorithmConfig subclass (here PPOConfig), then .build_algo() it into a running Algorithm and call .train() in a loop. Each call to train() is one full iteration of the loop you built in lesson 06: collect a train batch from the EnvRunners, do one (or several) gradient updates on the learner, broadcast new weights, and return metrics.

from ray.rllib.algorithms.ppo import PPOConfig

config = (
    PPOConfig()
    .environment("CartPole-v1")          # the Gym/Gymnasium env contract
    .env_runners(num_env_runners=4)       # 4 CPU rollout-worker actors
    .training(train_batch_size=4000)      # transitions per learner update
    .resources(num_gpus=0)                # CartPole is tiny; no GPU needed
)

algo = config.build_algo()                # spins up the EnvRunner actors + learner
for i in range(20):
    result = algo.train()                 # ONE iteration: collect → update → broadcast
    print(i, result["env_runners"]["episode_return_mean"])
algo.save("checkpoints/cartpole")         # checkpoint = resumable policy + optimizer

Read the contract in that loop. train_batch_size=4000 means each train() waits until the four EnvRunners have collectively produced 4,000 environment transitions before the learner updates. So iteration time is dominated by whichever side is slower: if the four runners take 0.8 s to make 4,000 steps and the learner takes 0.2 s to consume them, you are sampling-bound and the GPU (if any) sits idle 80% of the time. That ratio is the entire subject of §3. The older API spelled the runner line .rollouts(num_rollout_workers=4); same actors, renamed — if you read old examples, translate rollouts→env_runners and num_rollout_workers→num_env_runners.

3 · Resource heterogeneity is the whole point

Here is why RL maps onto Ray specifically and not onto a simpler tool. The two halves of the loop want opposite hardware. Sampling is stepping an environment — usually CPU-bound Python/physics with little arithmetic, and embarrassingly parallel across env copies. Learning is a dense neural-net gradient step — the thing GPUs exist for. A single homogeneous pool of machines is wrong for both: a box of GPUs wastes them on env stepping; a box of CPUs starves the gradient step. The right cluster is heterogeneous — many cheap CPU EnvRunners feeding a few expensive GPU learners — and configuring exactly that asymmetric placement is what Ray's resource model (lesson 04) is for.

config = (
    PPOConfig()
    .environment("MyHeavyEnv-v0")
    .env_runners(
        num_env_runners=16,               # 16 cheap CPU rollout actors
        num_envs_per_env_runner=4,         # 4 env copies each → 64 envs total, vectorized
    )
    .learners(num_learners=1, num_gpus_per_learner=1)   # one expensive GPU learner
    .training(train_batch_size=32000)
)

That config is the heterogeneous topology in code: 64 environment copies stepping on CPU actors, one GPU doing the gradient. num_envs_per_env_runner lets a single actor vectorize several envs in one process, amortizing the per-actor overhead (lesson 02's granularity point — don't pay actor scheduling cost for one tiny env).

const LEARNER_CAP = 30000;                 // transitions/s the GPU can consume
const throughput  = nRunners * perRunnerRate;
// if sampling < capacity, learner waits on data → utilization = throughput/capacity
// if sampling >= capacity, learner is the bottleneck → utilization = 100%, surplus wasted
const util = Math.min(1, throughput / LEARNER_CAP);
const bottleneck = throughput < LEARNER_CAP ? "sampling (GPU idle)" : "learner (good)";

4 · On-policy vs off-policy: how stale may a sample be?

The §3 surplus question — what if you sample faster than the learner can consume? — is answered differently by two families of algorithm, and the difference is the most important conceptual knob in RL systems.

A policy update changes the policy. The samples an EnvRunner collected a moment ago were generated by the old policy. On-policy means a sample is only valid for updating the exact policy that produced it — the moment you take one gradient step, every un-consumed sample is stale and must be discarded. PPO (the default above) is on-policy. Off-policy means samples from older policies remain usable: you store transitions in a replay buffer and the learner trains on a mix of fresh and old. DQN and SAC are off-policy.

5 · Four modes that change the loop's shape

Everything above is single-agent online RL. RLlib packages four variants; you do not need their full API here, only what each is for and how it bends the loop.

6 · Failure modes & checklist

Checkpoint exercise

Where this points next

You can now configure a single RL experiment and reason about its CPU/GPU ratio. But one config is one point in a space — learning rate, batch size, runner count, network size all interact, and RL is notoriously sensitive to them. Running configs one at a time by hand is exactly the waste lesson 00 warned about. Lesson 08 introduces Ray Tune: it treats each config as a trial (a Ray task/actor), runs many in parallel, and — crucially — uses a scheduler to kill losing trials early and reallocate their compute. The algo.train() loop you just wrote becomes the inner body that Tune drives, and the question shifts from "how do I run this config" to "how do I search the space without burning the whole GPU budget."

Interview prompts

• Map the four hand-built RL roles onto RLlib components. (§1 — environment → Gym env contract; rollout workers → EnvRunner actors; learner → the GPU gradient component; experience → sample/train batch; plus the weight broadcast back to runners.)
• Why is distributed RL specifically a Ray problem? (§3 — the two halves want opposite hardware (CPU sampling vs GPU learning); Ray's per-actor resource model expresses the heterogeneous many-CPU-few-GPU placement that a homogeneous pool cannot.)
• How many CPU EnvRunners do you need for one GPU learner? (§3 — divide learner consumption rate by per-runner sampling rate; e.g. 30,000 ÷ 2,000 = 15. Fewer idles the GPU; more is wasted on-policy.)
• What does on-policy vs off-policy change about your distributed design? (§4 — on-policy samples are valid only for the policy that made them, forcing tight sync and discarding surplus; off-policy reuses old samples from a replay buffer, tolerating staleness and surplus runners.)
• Explain the throughput-vs-staleness trade-off and what async collection buys. (§4 — async keeps both CPU and GPU busy by overlapping collection and update (e.g. 1.0 s → 0.8 s/iter) at the price of training on slightly-stale samples, which on-policy must correct with importance sampling.)
• Reward is climbing but GPU utilization is 20%. Diagnose. (§3, §6 — sampling-bound: not enough/too-slow EnvRunners to feed the learner; add CPU runners or vectorize envs, don't touch the algorithm.)
• What is the central risk of offline RL versus online? (§5 — distribution shift: the logged behavior policy differs from the learned policy, so the data never covers what the new policy would do; throughput stops being the concern.)