Part II - Applications and libraries

Ray Tune: Experiments and Hyperparameter Search

Lesson 07 gave you RLlib: many cheap rollout actors feeding a few expensive learners, one training run at a time. But which learning rate? Which network width, which batch size, which discount factor? A single training run answers none of that — you need to run dozens of configurations and let the good ones survive. Ray Tune is that layer. The trap is to read it as "random search, but on more machines." It is something sharper: a control plane over a population of experiments that decides, while they run, which configs deserve more compute and which to kill. This lesson builds that control plane from the problem up, and makes the central engineering claim — that early stopping, not parallelism, is where the compute savings live — quantitative.

1 · Why HPO is hard (and why machines alone don't fix it)

Stochastic gradient descent tunes a model's weights because the loss is differentiable in the weights — you have a gradient pointing downhill. Hyperparameters are different. Learning rate, batch size, network depth, dropout, weight decay — these sit outside the differentiable graph. You cannot take the gradient of "final validation accuracy" with respect to "learning rate," because evaluating that relationship requires training a whole model. HPO is therefore the optimization of a function with four hostile properties:

Now the key insight that separates Tune from a job scheduler. Because evaluation is expensive and the budget is fixed, the question is never "can I run all the configs in parallel?" — you usually cannot afford to. The question is "given a fixed pile of GPU-hours, how do I spend them to find the best config?" That reframes HPO as a resource-allocation problem under uncertainty: partial results arrive at different times, and something must continuously decide which running experiments are worth feeding. That "something" is the control plane Tune provides. Parallelism is table stakes; allocation is the value.

2 · Three separated concerns: trial, search, scheduler

Tune's design cleanly splits three jobs that beginners tend to conflate. Define each at first use:

The separation matters because the two decisions answer different questions and can be mixed and matched. The search algorithm answers "where in the space should I look next?" The scheduler answers "of the trials I am already running, which are wasting my compute right now?" You can pair random search (cheap, embarrassingly parallel) with an aggressive scheduler, or Bayesian search (sample-efficient but sequential — it must see results before proposing) with a gentle one. The book's chapter walks random search first precisely because it is the robust baseline against which any clever search must justify itself.

Exploration vs exploitation, concretely. Pure random search is all exploration: it never uses what it learned, so it wastes samples re-checking obviously-bad regions, but it parallelizes perfectly and never gets stuck. Bayesian search exploits: after 20 trials it has a surrogate model and concentrates samples near the best region — far fewer trials to a good answer, but each proposal waits on prior results (a serial dependency, recall the Amdahl ceiling from lesson 00), and it can tunnel into a local optimum. The practical answer is usually random/grid for breadth plus a scheduler for the savings, escalating to Bayesian only when each trial is so expensive that sample-efficiency dominates.

3 · Successive halving: where the compute is actually saved

This is the heart of the lesson. ASHA — Asynchronous Successive Halving Algorithm — is the default scheduler worth understanding in full, because its arithmetic is the entire argument for using Tune over a loop.

The idea: do not train every config to completion to find out it was bad. A config that is in the bottom third after 1 epoch is very unlikely to be the winner after 27. So organize training into rungs — checkpoints at increasing budgets — and at each rung promote only the top fraction, killing the rest. A rung is a budget milestone (e.g. "1 epoch trained"); successive halving is the repeated cull at each rung. The grace period is the minimum budget every trial gets before it is eligible to be killed — your protection against murdering a config before it has had a fair chance.

The "A" in ASHA — asynchronous — matters on a real cluster. Classic Hyperband waits for an entire rung to finish before promoting, so one straggler stalls the whole population (a synchronization barrier; recall the shuffle bottleneck of lesson 05). ASHA promotes a trial the moment enough peers have reported at its rung, so GPUs never sit idle waiting for the slowest trial. That is why it is the cluster default.

4 · Resource-aware concurrency and checkpointing

A scheduler that kills trials is only useful if the freed compute is immediately reused. Tune is resource-aware: each trial declares what it needs (CPUs, GPUs, fractional GPUs), and Tune packs trials onto the cluster honoring those reservations — the same placement logic from lesson 04, applied to a trial population.

Checkpointing is what makes pause/resume possible. When ASHA stops a trial at a rung, it does not always discard it — with pause-and-resume schedulers it saves the trial's state to durable storage so it can be reloaded later. More importantly, checkpoints are how Tune survives node failure: a preempted spot instance or a dead worker loses a trial's process, but the trial resumes from its last checkpoint instead of restarting from epoch 0. The rule from the old skeleton holds — checkpoint often enough to lose little work, but cheaply enough not to dominate runtime, and write to shared/durable storage (S3, NFS) not local disk, because trials migrate between nodes.

One scheduler deserves a special note because it blurs the search/scheduler line: Population Based Training (PBT). PBT runs a population, periodically copies the weights of top performers onto poor ones, and then mutates their hyperparameters live — so the config is not fixed for a trial's lifetime, it evolves. This is uniquely powerful for things like learning-rate schedules (it discovers a schedule, not just a constant), at the cost of more bookkeeping and the requirement that trials checkpoint frequently so weights can be cloned mid-flight.

5 · The real code

Putting the three concerns together. The train function is an ordinary loop that calls tune.report each epoch; the Tuner wires up the search space, the search algorithm, and the scheduler:

from ray import tune, train
from ray.tune.schedulers import ASHAScheduler
from ray.tune.search.optuna import OptunaSearch

def train_fn(config):                         # ONE trial
    model = build_model(width=config["width"])
    opt = make_optimizer(lr=config["lr"])
    for epoch in range(27):                    # up to the ASHA max
        loss, acc = train_one_epoch(model, opt)
        # report the metric AND a checkpoint so Tune can rank + resume
        tune.report({"accuracy": acc, "loss": loss})

tuner = tune.Tuner(
    tune.with_resources(train_fn, {"gpu": 0.5}),   # resource-aware: 2 trials / GPU
    param_space={                                  # the search space
        "lr":    tune.loguniform(1e-5, 1e-1),      # continuous, log scale
        "width": tune.choice([64, 128, 256]),      # categorical
    },
    tune_config=tune.TuneConfig(
        metric="accuracy", mode="max",
        search_alg=OptunaSearch(),                 # Bayesian exploration
        scheduler=ASHAScheduler(                    # successive halving
            max_t=27, grace_period=1, reduction_factor=3),
        num_samples=81,                            # 81 configs to draw
    ),
)
results = tuner.fit()
best = results.get_best_result(metric="accuracy", mode="max")
print(best.config, best.metrics["accuracy"])

Read the separation in the code: param_space is the space, search_alg proposes within it (exploration), scheduler reallocates compute across the running trials (the savings), and tune.with_resources governs packing. Swap OptunaSearch() for the default random search, or ASHAScheduler for None, and you can isolate exactly how much each contributes — which is precisely the baseline discipline the book insists on.

6 · Make ASHA's budget tangible

Drag the knobs to see how the number of configs and the reduction factor trade off against compute. The bars compare full training (every config to the max) against ASHA's successive halving over the same population.

// rungs at budgets 1, eta, eta^2, ... up to maxEpochs; keep top 1/eta survivors each.
let survivors = N, prevBudget = 0, asha = 0, budget = 1;
while (true) {
  asha += survivors * (budget - prevBudget);   // incremental epochs this rung
  if (budget >= maxEpochs || survivors <= 1) break;
  survivors = Math.max(1, Math.floor(survivors / eta));
  prevBudget = budget;
  budget = Math.min(maxEpochs, budget * eta);
}
const full = N * maxEpochs;                      // train everything to the end
const speedup = full / asha;                     // the early-stopping win

7 · Failure modes & checklist

Checkpoint exercise

Where this points next

Tune governs a population of trials, but it has been treating each trial as a black-box train function. What if one trial is itself too big for a single GPU and must be data-parallel across many workers? That is the recurring distributed-training shape — and Tune composes directly with it: a trial can be a Ray Train run, and the data feeding it comes from Ray Data (lesson 09). Lesson 10 builds Ray Train's N-worker, gradient-exchanging machinery, after which a single Tune trial can itself span a multi-GPU cluster. The control plane you just built sits one layer above the training engine you are about to meet.

Interview prompts

• Why can't you use gradient descent to tune hyperparameters? (§1 — hyperparameters sit outside the differentiable graph; evaluating one requires a full training run, so HPO is a black-box, gradient-free optimization.)
• What distinguishes Tune from "random search on more machines"? (§1, §3 — parallelism gives a linear, hardware-bounded speedup; Tune's scheduler spends less compute per config by early-stopping losers, a 5–10× effective win from arithmetic, not nodes.)
• Separate the roles of the search algorithm and the scheduler. (§2 — search proposes the next config (exploration of the space); scheduler decides which running trials to kill/keep (allocation of compute over time). They are independent and composable.)
• Walk through ASHA's budget for 81 configs, η=3, max 27 epochs. (§3 — rungs at 1/3/9/27 keeping top 1/3 each: 81+54+54+54 ≈ 243 vs 81×27 = 2,187 epoch-units, a 9× saving, winner still trains full length.)
• What is the grace period and what does it trade off? (§3, §4 — the minimum budget before a trial can be killed; raising it protects late-blooming configs at the cost of more compute spent on eventual losers.)
• Why is ASHA asynchronous rather than synchronous Hyperband? (§3 — synchronous promotion waits for a whole rung (a barrier a straggler stalls); async promotes as soon as enough peers report, keeping GPUs busy on a real cluster.)
• How does Tune pack trials onto a cluster, and what does a fractional GPU buy/cost? (§4 — each trial declares resources via tune.with_resources; 0.5-GPU trials double exploration throughput but share memory/compute, risking thrash and noisy timings.)
• Why must checkpoints go to durable shared storage? (§4 — trials migrate between nodes and nodes die; a local-disk checkpoint is lost on migration or failure, breaking pause/resume and fault tolerance.)