Serving non-LLM models — features, cascades, and decisions

Lessons 04-06 served autoregressive LLMs, where KV memory, prefill/decode, and GPU bandwidth dominate. Most production ML does not look like that. A fraud model, ranking model, or forecast scorer is often cheap to execute; the hard part is fetching correct features, staying under a p99 budget, preserving train-serve parity, and turning scores into actions. Same loop, different wall.

The replacement wall

In LLM serving, the synchronous path is usually prompt -> GPU -> tokens. In general ML serving, it is often request -> feature hydration -> candidate retrieval -> model(s) -> decision. The bottleneck moves from HBM/KV to online lookups, fan-out, ANN recall, serialization, calibration, and downstream action capacity.

Plain English dictionary

Feature hydration means gathering the facts needed for this request. Candidate generation means shrinking a huge universe, like 100M items, to a small set worth scoring. A cascade means cheap broad filtering first, expensive precise scoring later. Calibration means a score like 0.8 really behaves like an 80% probability or risk when the product uses it.

0 · The serving thought process

For a non-LLM model, the model call may be the easiest part. Think through the request in this order:

What must happen before the user gets a decision? Only this belongs on the synchronous path.
Where does each required fact come from? Request payload, cache, online store, ANN index, database, or another service.
How much p99 latency does each stage spend? Budgets force tradeoffs before the system is built.
What can be precomputed, cached, or skipped on failure? This is where reliability and cost come from.
How does the score become an action? Thresholds, ranking policy, review queues, business rules, and safety constraints are part of serving.

1 · Draw the synchronous path

Start with the request path and assign a p99 budget to each stage. If a stage is not required before the decision, move it async.

Budget recipe

Start with the user-visible SLO, then reserve 10-20% for network, serialization, queues, and variance. Split the remaining budget across only the sequential stages. If two calls run in parallel, the slower tail dominates; if three calls fan out, the request tail is worse than each component tail. Keep slack for retries and fallbacks, then load-test the whole path, not just each service alone.

Stage	What binds first	Design lever
Feature fetch	Online store p99, fan-out, missing values	Denormalize hot features, cache, precompute, fallback defaults
Candidate generation	Recall vs latency/RAM	Two-tower embeddings, ANN, sharded index, filtering
Model score	CPU/GPU latency, batchability, model size	Fast tree or linear model first; distill a large model into a smaller one; export to optimized runtimes; batch requests
Decision	Calibration, threshold, downstream capacity	Cost-sensitive threshold, review queue, business rules

2 · Cascades are the default for ranking

A ranking service rarely scores every item with the most expensive model. It uses a cascade: cheap high-recall retrieval, medium-cost ranking, expensive reranking only on a small set. This is the recsys/search analogue of LLM batching: spend the expensive computation only where it changes the decision.

latency_total = latency_retrieve + latency_rank(K) + latency_rerank(k)

Worked: why K matters more than model glamour

Suppose retrieval returns K=1000 candidates in 20 ms, ranker scores them at 20 microseconds each (20 ms), and reranker scores top 50 at 0.4 ms each (20 ms). Total is ~60 ms before serialization. Doubling K for a tiny recall gain adds 20 ms to ranking and may break a 100 ms SLO. The binding wall is candidate set size under latency, not whether the ranker is fashionable.

For the mechanism details, the search/recsys track covers the funnel, candidate generation, ANN, and ranking models: funnel, candidate generation, ANN, and ranking models.

3 · Feature fetch is part of latency

Non-LLM models are often fast enough that feature hydration dominates p99. A gradient-boosted tree model scoring 500 features may take sub-millisecond CPU time; fetching those features from three stores across the network can take tens of milliseconds and produce tail spikes.

Pattern	Use when	Cost
Request carries features	Features are already known at the edge	Large payloads, client trust issues
Online feature store lookup	Fresh user/entity state needed	Network p99 and missing keys
Prejoin / denormalize	Hot features change slowly	Staleness and larger storage
Nearline cache	Session-level features update every seconds-minutes	Cache invalidation and warmup
Fallback defaults	Availability beats perfect quality	Lower accuracy, must monitor fallback rate

The p99 multiplication

Three independent feature calls at p99 20 ms do not make a 20 ms feature stage. Fan-out turns component tails into request tails. Co-locate features, batch lookups, or denormalize until the feature stage has a single predictable p99 budget. Classic backend fan-out rules apply; see latency and queueing.

4 · Score means action only after calibration

A model score is not automatically a decision. Classification and ranking systems need calibration and thresholds because downstream actions have costs.

System	Score	Action conversion
Fraud	P(fraud)	Block if expected fraud loss exceeds false-positive/user-friction cost
Ads	pCTR / pCVR	Bid and rank by expected value; calibration directly affects money
Recsys	Utility score	Blend engagement, diversity, safety, freshness, exploration
Moderation	Risk probability	Allow, downrank, blur, review, remove

This is why lesson 10a exists: the serving design and eval design are coupled. A ranker can be accurate but uncalibrated; a fraud model can have high AUC but a bad operating threshold; a recommender can lift clicks while hurting retention.

What carries forward

General serving is often feature-bound, not model-bound. Put feature p99 and missing-rate beside model latency.
Cascades spend compute where it matters. Retrieve many cheaply, rank fewer, rerank very few.
Scores become actions through calibration and thresholds. The action surface is part of the system, not product garnish.
Use LLM serving tools only when the LLM wall binds. For ranking/fraud/scoring, the walls are usually freshness, fan-out, recall, and calibration.