Part II - Applications and libraries

Ray Serve: Online Inference

Lesson 10 finished a model: N worker actors trained it, checkpointed it, and handed you a set of weights on durable storage. That checkpoint is worthless until someone can call it. But serving is not training run backwards — training is a throughput job that may take hours and only owes you a final number, while serving is a stream of individual requests, each owed an answer inside a latency budget, arriving in unpredictable bursts. This lesson takes the actor primitive from lesson 03 and the autoscaling instinct from the whole track and turns it into an online service: a group of model-holding replicas behind an HTTP door, with the one knob that decides whether your GPU is busy or your users are waiting — dynamic request batching.

1 · Online inference is a latency contract

A batch job (lesson 09's Ray Data, lesson 10's training) is judged on total time: process 50 million rows by 6 a.m., and nobody cares whether row 12,000,000 took 3 ms or 30 ms. Online inference inverts every one of those properties.

The consequence is that you cannot serve a model by wrapping model(x) in a Flask route and calling it done. You need: a process that loads the weights once and stays warm (or every request pays the load cost), several such processes so one slow request does not block the next, a front door that spreads requests across them, and a policy that grows and shrinks that pool as traffic moves. Ray Serve is that machinery, built on the actor you already know.

2 · The Serve object model

Serve adds three nouns on top of Ray Core. Learn them as the runtime objects they are, not as config keys.

The minimal deployment is one class with a request handler. Note that __init__ runs once per replica — this is the lesson-03 "load the expensive thing once" pattern, now managed by Serve:

from ray import serve
from starlette.requests import Request

@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class Classifier:
    def __init__(self):
        self.model = load_model_from_checkpoint("s3://ckpt/best")  # once per replica

    async def __call__(self, request: Request):
        payload = await request.json()
        return {"label": self.model(payload["text"])}

app = Classifier.bind()    # bind() builds the deployment graph node
serve.run(app)             # start ingress + 2 GPU replicas

Two replicas, each pinned to one GPU, means two requests run truly in parallel and a third queues behind whichever frees first. That is already better than a single Flask process — but each replica still runs one request at a time through its model, and on a GPU that is almost always the wrong way to use the hardware. Fixing that is the next section.

3 · Dynamic request batching — the central knob

A GPU is a throughput device: it is built to apply the same matrix multiply to a wide batch in roughly the time it takes to do one row, because the cost is dominated by launching the kernel and moving weights, not by the per-row arithmetic. Feed it one request at a time and you might use 5% of it. Dynamic batching is the mechanism that fixes this for online traffic: when a request arrives, instead of running it immediately, the replica holds it briefly to see if more arrive, then runs them together as one batched forward pass.

In Serve this is a decorator on a coroutine that takes a list of inputs:

@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class Classifier:
    def __init__(self):
        self.model = load_model_from_checkpoint("s3://ckpt/best")

    async def __call__(self, request: Request):
        payload = await request.json()
        return await self.classify(payload["text"])   # one item in, one out

    # Serve gathers concurrent calls into a list, runs ONE forward pass.
    @serve.batch(max_batch_size=16, batch_wait_timeout_s=0.01)
    async def classify(self, texts: list[str]) -> list[str]:
        return self.model(texts)                       # vectorized over the batch

Two knobs govern it. max_batch_size caps how many requests go in one pass. batch_wait_timeout_s caps how long the first waiting request will tolerate the replica holding the door open for friends. The batch fires when either the batch is full or the timeout expires — whichever comes first.

The math, worked

Say a single forward pass on this GPU takes T(b) ≈ 8 + 1.5 × b ms for a batch of b (8 ms fixed kernel/launch overhead, 1.5 ms of marginal work per extra item — the classic "wide is cheap" curve). Requests arrive at a steady λ = 800 requests/second.

• No batching (b = 1). Each pass is 8 + 1.5 = 9.5 ms, so one replica does at most 1000 / 9.5 ≈ 105 req/s. To serve 800 req/s you need ⌈800 / 105⌉ = 8 GPU replicas, and the GPU spends 84% of its cycles on fixed overhead. Expensive and wasteful.
• Batch of 16. One pass is 8 + 1.5 × 16 = 32 ms and clears 16 requests, so throughput is 16 / 0.032 = 500 req/s per replica — almost 5× the per-replica rate. Now ⌈800 / 500⌉ = 2 replicas suffice. You went from 8 GPUs to 2.
• What the latency cost was. At 800 req/s a batch of 16 fills in 16 / 800 = 20 ms on average, so a typical request waits ~10 ms (it lands mid-fill) plus the 32 ms compute → p50 ≈ 42 ms. The unlucky first arrival in a batch waits the full 20 ms fill + 32 ms compute = 52 ms, and under burst the queue stacks, pushing p99 toward 80–120 ms. Compare the unbatched p50 of ~10 ms: batching quadrupled median latency to cut the GPU bill by 4×.

That is the trade in one breath: bigger batch → higher throughput and utilization, fewer replicas, lower cost — but higher p50 and a fatter p99 tail. The timeout is the safety valve: set batch_wait_timeout_s so that even when traffic is light (batches never fill), no request waits longer than your tail budget. The widget lets you feel both knobs at once.

// time to fill a batch from empty at rate lambda (s^-1):
fill_ms = 1000 * max_batch_size / lambda;          // could exceed the timeout
wait_ms = Math.min(fill_ms, timeout_ms);           // batch fires at whichever is first
eff_b   = Math.min(max_batch_size, lambda * wait_ms / 1000); // items actually gathered
compute = 8 + 1.5 * eff_b;                          // one forward pass, ms
throughput = 1000 * eff_b / compute;                // req/s ONE replica sustains
replicas = ceil(lambda / throughput);               // right-size the deployment
rho = lambda / (replicas * throughput);             // per-replica load, in [0,1)
// a request waits a uniform share of the fill window, then computes:
p50 = 0.5 * wait_ms + compute;
p99 = wait_ms + compute + compute * rho;            // + ~one burst pass; grows with batch

4 · Composing models — and right-sizing each stage

A real API is rarely one model. A moderation endpoint tokenizes text (cheap, CPU), runs a small fast classifier (cheap, GPU or CPU), and only on a flagged result escalates to a large expensive model (heavy, GPU). Serve composes these as a deployment graph: separate deployments wired together with handles, where each deployment scales independently.

@serve.deployment(num_replicas=2)                       # cheap CPU stage
class Tokenizer:
    def __call__(self, text): return tokenize(text)

@serve.deployment(autoscaling_config={"min_replicas": 2, "max_replicas": 8,
                                      "target_ongoing_requests": 5},
                  ray_actor_options={"num_gpus": 1})    # the bottleneck stage
class Classifier:
    @serve.batch(max_batch_size=16, batch_wait_timeout_s=0.01)
    async def __call__(self, batch): return self.model(batch)

@serve.deployment                                        # ingress: wires the graph
class Pipeline:
    def __init__(self, tok, clf):
        self.tok, self.clf = tok, clf                    # DeploymentHandles
    async def __call__(self, request):
        text = (await request.json())["text"]
        toks = await self.tok.remote(text)               # call stage 1
        return await self.clf.remote(toks)               # call stage 2

app = Pipeline.bind(Tokenizer.bind(), Classifier.bind())  # graph via .bind()
serve.run(app)

The reason each stage is its own deployment is that they have different resource profiles and different throughput. Collapsing them into one deployment forces every stage to scale together — you would buy a GPU for every CPU tokenizer you add. Separating them lets you give replicas only to the stage that needs them.

Autoscaling per deployment is driven by target_ongoing_requests — the number of concurrent requests Serve aims to keep queued+running per replica — bounded by min_replicas and max_replicas. If each replica handles ~5 in-flight requests comfortably and 40 are in flight, Serve wants ⌈40 / 5⌉ = 8 replicas; the min keeps a warm floor so a burst does not hit a cold start, the max caps cost. The deep autoscaling story — how this rides on the cluster autoscaler and pod cold-start latency — is lesson 12.

5 · The three trade-offs you are actually choosing

6 · Failure modes & checklist

Checkpoint exercise

Where this points next

This lesson assumed replicas could appear when target_ongoing_requests demanded them — but where do the GPUs come from, and how fast? A burst that needs four new replicas may wait on a new pod, a pulled image, and a model load before it serves a single request, and that scale-up latency is itself part of your tail. Lesson 12 takes the autoscaling instinct down a layer: the Ray autoscaler scaling on pending resource demand, KubeRay's RayService running this graph on Kubernetes, and the two-level autoscaling dance between Ray and the cluster — where online inference's "just add replicas" meets the physics of cold starts and idle cost.

Interview prompts

• How is online inference a different problem from a batch job? (§1 — request-level, SLO-bound, tail (p99) sensitive, and bursty; you owe each request an answer inside a budget, not an aggregate by a deadline.)
• What is a deployment vs a replica in Ray Serve? (§2 — a deployment is an autoscaling group serving one logical thing; a replica is one Ray actor running an instance of the deployment class, loading the model once in its __init__.)
• Why does batching help on a GPU, and what does it cost? (§3 — a GPU's cost is dominated by kernel launch + weight movement, so a wide batch is nearly as cheap as one row, multiplying throughput; the cost is the queueing tax each batched request pays, raising p50 and p99.)
• How do max_batch_size and batch_wait_timeout_s interact? (§3 — the batch fires when it fills OR the timeout expires; at low arrival rates the timeout binds (small batches), at high rates the size binds. Tune the timeout against the live λ so light traffic never over-waits.)
• You must sustain 1000 req/s through tokenize → classify → fallback. How do you size replicas? (§4 — capacity per replica per stage = 1000/latency; ceil(load/capacity) per stage. Replicas go to the real bottleneck — here the 50 ms fallback dominates the GPU bill even at 30% of traffic.)
• Why split a pipeline into separate deployments instead of one? (§4 — stages have different resource profiles and throughput; independent deployments scale (and fail) independently, so you don't buy a GPU per CPU tokenizer, and the bottleneck is visible.)
• When is Serve the wrong tool, or only half of it? (§5 — for a single hot LLM a specialized engine (vLLM, TensorRT-LLM, Triton) has better kernels and KV-cache batching; run it inside a Serve deployment and keep Serve for the Python composition around it.)