Batch, streaming, and trust
The Part 0 capstone. Three fundamentals that decide whether a pipeline is correct, trustworthy, and affordable: when work runs (batch vs streaming), whether you can rely on the result (idempotency, determinism, reproducibility), and what it costs to run (throughput, unit cost, bottlenecks).
Theme 1 — Batch vs streaming: the time dichotomy
Every pipeline answers one question before any other: when does the work run? Either you collect input and process it in chunks on a schedule, or you process each event the instant it arrives. This single choice cascades into how you reason about correctness, ordering, and failure — so it is worth getting the picture right first.
Think about doing laundry. The normal way is batch: you wait until you have a full hamper, then wash it all in one load. It's efficient per item — one cycle of water, soap, and electricity cleans thirty shirts at once — but your favorite shirt isn't available until the whole load finishes. You trade freshness for efficiency, and you always know exactly what went in: a fixed pile of clothes.
Now imagine a magic machine that washes each sock the instant it gets dirty. That's streaming. Everything is always fresh — no waiting for a load to fill — but the machine never stops running, and keeping it correct is much harder. What if two socks arrive at once? What if a sock from yesterday's mud shows up late, after you already folded today's? The always-on machine has to answer questions the hamper never raised.
Batch processes a bounded dataset to completion at intervals (hourly, nightly). Because the input is fixed and finite, batch optimizes for throughput and cost — it can sort, group, and shuffle the whole thing — and accepts latency as the price. It is easy to make reproducible: re-run the same job over the same fixed input and you get the same output.
Streaming processes an unbounded sequence of events with low latency, and that single difference introduces hard problems batch never faces:
- Event-time vs processing-time — when something happened vs when your system saw it. They differ, sometimes by minutes.
- Windowing — to compute "events in the last hour" over an infinite stream you must cut it into finite windows. Where do the boundaries go?
- Watermarks — a heuristic for "I've probably seen all events up to time T," so a window can finally close even though late data might still arrive.
- Exactly-once processing — making sure a crash-and-retry doesn't count the same event twice. (Delivery itself can't be exactly-once over an unreliable network; you get the effect by combining at-least-once delivery with the idempotent writes of Theme 2 — which is where Theme 2 becomes survival, not luxury.)
The dichotomy is fundamental because correctness, ordering, and failure-recovery semantics differ completely between the two. A mental model built for one will quietly mislead you in the other.
SAME EVENTS (1..7), TWO PROCESSING MODELS
───────────────────────────────────────────────────────────────────────
BATCH — accumulate into a bin over an interval, process the whole bin at once
┌──── bin 1 ────┐ ┌──── bin 2 ────┐ ┌──── bin 3 ────┐
arrivals ─────1──2────3───────┼─4─────5─────────┼─6──────7────────┼──▶ time
results ─────────────────────█─────────────────█─────────────────█──▶
▲ results appear in chunks
└ whole bin 1 emitted here (latency ≈ bin-fill time)
STREAM — process each event the instant it arrives
arrivals ─────1──2────3─────────4─────5───────────6──────7──────────▶ time
results ─────▌──▌────▌─────────▌─────▌───────────▌──────▌──────────▶
└──┴────┴── one result per event, immediately ──┴──────┘
(low latency, machine always on)
The trade is exactly latency vs throughput: batch waits a full bin before any result appears but amortizes fixed costs across the whole bin (cheap per record); streaming emits each result the instant its event lands (low latency) at the cost of being always-on and paying per-event overhead.
| Batch | Streaming | |
|---|---|---|
| Input | Bounded — a fixed, finite dataset | Unbounded — an endless sequence of events |
| Optimizes for | Throughput & cost per record | Latency — freshness of each result |
| Runs | On a schedule, to completion | Continuously, never finishes |
| Hard problems | Mostly scale (the shuffle) | Event-time, windows, watermarks, exactly-once |
| Reproducible? | Naturally — input is fixed | Hard — input is a moving target |
Picture a sink filling faster than it drains: the water rises and eventually overflows. The only fixes are to slow the tap or widen the drain. A streaming pipeline has the same physics. If a downstream stage consumes slower than an upstream stage produces, the queue between them grows without bound — and backpressure is the signal that flows back up to throttle the producer until it matches the consumer.
This is load-bearing later: it is exactly how the producer (rollout generation) and consumer (the trainer) stay rate-matched in lesson 10's RL online dataplane, where a mismatch either starves the GPUs or lets data go stale.
Theme 2 — The trust properties
Distributed pipelines (from F6) will fail mid-run and retry. A machine dies, a network blips, a job is restarted from a checkpoint. The question is not whether retries happen — they do — but whether a retry leaves your data correct or corrupts it. Three properties decide that, and they stack on top of each other.
Idempotent — an elevator call button. Press it once or press it five times in frustration: the result is identical, one elevator arrives. The repeated presses don't stack up into five elevators. An operation is idempotent when doing it again changes nothing.
Deterministic — a recipe so precise that two different cooks, following it exactly, produce identical cakes. No "season to taste," no "bake until it looks done." Same instructions, same result, every time.
Reproducible — keeping not just the precise recipe but the exact batch of flour and the exact oven settings filed away, so you can bake that same cake next year and it comes out identical. Determinism plus a frozen record of everything that went in.
- Idempotent — applying an operation N times yields the same state as applying it once. In pipelines this usually means overwrite-by-key rather than append: writing row
id=7twice leaves one row 7, not two. This is what makes retries safe — re-running a half-finished job can't double-count. - Deterministic — the same input always produces the same output. No hidden randomness (seed it), no order-dependence (a set, not a list, unless order is defined), no wall-clock or "now()" dependence baked into the result.
- Reproducible — deterministic plus versioned inputs, code, and config, so an old result can be regenerated byte-for-byte. Determinism alone isn't enough: if the input file silently changed, the deterministic code still produces a different (correct-for-the-new-input) answer.
Here is the bug these properties prevent. A job appends its output, then crashes after writing but before recording success. The orchestrator retries the whole job — and the rows land again:
NON-IDEMPOTENT (append on rerun) IDEMPOTENT (overwrite by key)
───────────────────────────────── ──────────────────────────────
run 1: append [A,B,C] → A B C run 1: upsert {A,B,C} → A B C
crash + retry the job crash + retry the job
run 2: append [A,B,C] → A B C run 2: upsert {A,B,C} → A B C
A B C (no change)
─────────────────────────────────── ──────────────────────────────
result: 6 rows ✗ silently doubled result: 3 rows ✓ retry was safe
The non-idempotent pipeline doesn't error — it succeeds, and quietly returns wrong numbers. That's the worst kind of failure, because nothing alerts you. Idempotency turns a retry from a corruption risk into a no-op.
| Property | Precise statement | Buys you |
|---|---|---|
| Idempotent | N applications = 1 application | Safe retries & re-runs |
| Deterministic | Same input → same output | Predictable, testable behavior |
| Reproducible | Deterministic + versioned inputs/code/config | Regenerate any past result exactly |
Theme 3 — Cost & throughput thinking
The final fundamental is economic. A pipeline that is correct but costs ten times what it should, or runs ten times slower than it could, is a failed pipeline. Reasoning about cost and speed is its own skill — and the trap is optimizing the wrong thing.
Plan a road trip on a budget and you naturally think in rates and bottlenecks: miles per gallon, dollars per mile, hours per leg. And you ask the sharp question: which leg is the slow one? Is it traffic through the city? The time lost refueling? The mountain pass that drops you to 30 mph?
The crucial insight: a faster engine is pointless if you're stuck in traffic. Doubling your top speed saves nothing on the leg where you're bumper-to-bumper. To go faster you must find and fix the actual bottleneck — and optimizing anything else is wasted effort, money, and time.
Characterize a pipeline with two numbers:
- Throughput — work per unit time: records/sec, GB/sec. How fast it moves.
- Unit cost — money per unit work: $/TB, $/M records. How much each unit costs to move.
Then find the bottleneck resource — the one thing the pipeline is waiting on — because optimizing anything else changes nothing:
- I/O-bound — waiting on disk or object-store reads/writes. More CPU won't help; faster/parallel I/O or less data (columnar pruning, from the lakehouse lesson) will.
- Compute-bound — waiting on the CPU/GPU doing the work. More cores or a cheaper algorithm helps; faster disks don't.
- Shuffle/network-bound — waiting on data crossing the network between machines. This is the F6 villain, and it's where adding workers can backfire.
That last point connects directly to F6. For narrow (embarrassingly parallel) work, adding workers W scales nearly linearly — more hands, more done. But for wide work that requires a shuffle, every worker may talk to every other worker, so coordination grows like O(W²). Past a point, another worker adds more shuffle overhead than useful work, and the job gets slower. A faster engine in traffic.
WHERE IS THE TIME GOING? (profile before you optimize) ──────────────────────────────────────────────────────── stage bar (share of wall-clock) verdict read ████████████████████████ 62% ◀ BOTTLENECK (I/O-bound) transform ██████ 16% shuffle ████████ 20% write █ 2% ──────────────────────────────────────────────────────── buying faster CPUs here speeds up the 16% transform → barely moves total. reading less data (column pruning, partition filter) attacks the 62% → wins.
How these three set up the post-training series
These three themes are fundamental, general data engineering — they apply to any pipeline, anywhere. But they also set up the post-training series you're about to read, directly:
- The batch→streaming pivot is the whole story of Part III's RL online dataplane. In reinforcement learning the model generates its own training data during training — so the comfortable batch assumption of a fixed, bounded input breaks, and you're suddenly reasoning about an unbounded stream of fresh experience.
- Idempotency and determinism are formalized in lesson 02, and they power the retries and backfills that orchestration depends on in lesson 09 — a backfill is just a deliberate re-run, and only idempotency makes it safe.
- Cost and throughput become the explicit subject of the capstone, lesson 11, where you put real numbers on a real post-training pipeline.