The pipeline abstraction
You know what data is, how it's stored, and why layout decides cost. Now the central organizing idea of the whole field: why we think in pipelines — a chain of small, well-defined steps — instead of one big script that does everything at once.
The first-principles question
Suppose you have raw event logs and you need a clean, deduplicated, summarized table out the other end. You could write one script: read the logs, parse them, drop duplicates, join in some reference data, aggregate, write the result. One file, one run. So why does every serious data team instead break this into a chain of separate steps that hand off to each other? It costs more code and more moving parts up front. What do you get back?
Think of an assembly line, or a series of water filters. Raw material enters one end. Each station does one well-defined job — strip the casing, drill the hole, attach the part — and hands the result to the next station. A clean product exits the far end.
Two things make this powerful, and they are exactly the things a single do-everything worker can't give you:
- You can inspect quality between any two stations. Bad product coming out the end? Walk the line backward and look at what's sitting between each pair of stations until you find where it went wrong.
- If one station breaks, you fix that station — and you only re-run from there, because the work of the earlier stations is still sitting on the belt, finished.
Now picture the alternative: one worker at one bench doing every job in sequence, holding everything in their head. If they fumble the second-to-last step, the half-built product in their hands is worthless and they start over from raw stock. And while they work, you can't see anything — you just wait and hope. That's the do-everything script: opaque, all-or-nothing, and painful to fix.
A pipeline is a sequence of stages, where each stage is — conceptually — a pure function over datasets:
output = f(input)
"Pure" means a stage's output depends only on its declared inputs, with no hidden state, so the same input always yields the same output. (The honest exception is the very first Extract stage that reads from a live external source — its input is a moving target, not a fixed dataset; but everything downstream of the landed raw data can be pure, which is the whole reason we land raw first.) Each stage has a defined input contract (what columns/types/grain it expects) and an output contract (what it promises to produce). Stages compose by contract: stage B's input contract must be satisfied by stage A's output contract.
The decisive engineering move is that intermediate results are materialized — written down to storage between stages, not just kept in memory. That single choice buys you the two properties from the intuition, now stated precisely:
- Independently re-runnable. Because stage N's input is persisted, you can re-run stage N alone without re-running 1…N−1. A failure at the last stage costs you the last stage, not the whole job.
- Independently debuggable. Every intermediate is a real, inspectable dataset. "Where did the bad row come from?" becomes a question you can answer by reading the table between two stages, instead of a guess about what a long script was doing in memory.
The one-big-script approach forfeits both: nothing is materialized between steps, so any failure restarts from raw input, and the intermediate states never exist as data you can look at.
A pipeline, drawn
Here is the shape. Read each arrow as a pure function with a defined input and output; read each box-between-arrows as a materialized dataset you could query right now.
SOURCES STAGE STAGE CONSUMER
─────── ───── ───── ────────
logs/ ──┐
│ f_ingest f_clean f_aggregate
api/ ──┼──▶ [ raw ] ──────▶ [ clean ] ──────▶ [ daily ] ──────▶ dashboard
│ table table table / model
files/ ──┘
▲ ▲ ▲
│ │ │
in: raw bytes in: raw table in: clean table
out: raw table out: clean table out: daily table
(materialized) (materialized) (materialized)
Each arrow = a pure function output = f(input).
Each [box] = a persisted intermediate you can inspect and re-run from.
Notice you can point at any box and ask "is this correct?" — and if the answer is no, you know the broken stage is the arrow that produced it. That is the whole payoff in one sentence.
From a line to a DAG
The assembly-line picture is a straight line, but real pipelines branch and merge: a "clean" table might feed three different downstream stages; an "aggregate" stage might need two upstream tables joined together. The general shape isn't a line — it's a DAG.
Picture a flowchart of stations where arrows mean "this must finish before that can start." Some stations feed several others; some pull from several. The only rule is: no loops — you can never follow the arrows and end up back where you started. If you could, a stage would depend on its own output, and there'd be no valid order to run things in.
A DAG is a Directed Acyclic Graph:
- Nodes are stages.
- Edges are dependencies — a directed arrow A → B means "A must run before B," because B reads what A produced.
- Acyclic means no cycles: follow the arrows and you can never return to a node you started from.
Acyclicity is what guarantees a valid execution order exists (a topological order): stages with no unfinished dependencies can run, possibly in parallel, until everything is done. A straight pipeline is just the simplest DAG — a single path. We are only naming the structure here. The runtime that actually walks the DAG, decides what runs when, retries failures, and backfills history is orchestration, and it gets its own treatment in lesson 09.
Fan-in and fan-out
We said a clean table "might feed three downstream stages." Here is what that branching actually looks like — and it cuts both ways. Fan-in and fan-out are the two basic dataflow topologies a DAG is built from: many inputs funneling into one node, and one node feeding many outputs.
FAN-IN (many sources → one table) FAN-OUT (one table → many consumers)
────────────────────────────────── ─────────────────────────────────────
logs/ ──┐ ┌──▶ SFT trainer
api/ ──┤ │
files/ ──┼──▶ [ bronze ] ──▶ … ──▶ [ silver / gold ] ──────┼──▶ preference / DPO trainer
vendor/ ──┤ (one curated table) │
scrapes/──┘ ├──▶ eval / decontamination
│
└──▶ ad-hoc analytics
Many sources collapse INTO bronze (fan-in);
one curated table fans OUT to many independent readers (fan-out).
Note that this is dataflow fan-out — one materialized dataset, many readers, each pulling the same bytes for a different purpose. That is distinct from control-flow fan-out — one task dynamically spawning many downstream tasks at runtime — which is an orchestration concern covered in lesson 09. Same word, two planes again.
Two flows over one graph: data vs. control
That forward-reference hides the single most common point of confusion for newcomers. The DAG you just drew is used by two different things at once, and people conflate them. One is the data moving through the stages. The other is the control that decides when each stage runs. They travel over the same boxes and arrows, but they are not the same thing — and they often move in opposite directions.
Walk back onto the factory floor. A conveyor belt carries parts forward, station to station: a part leaves the casing station, rolls to the drilling station, rolls on to assembly. That forward motion of physical parts down the belt is the data flowing — it only ever goes forward.
Standing off to the side is a floor manager. The manager doesn't ride the belt. They decide which station fires up first, watch each station for a jam, and if one stalls they walk over and restart it. That's control — instructions and status-checks, on their own separate loop.
Same factory, same stations — but two different things are moving. The parts move forward along the belt. The manager's attention moves on its own loop, and it often reasons backward: "the final crate is empty — which upstream station didn't deliver?" Forward flow of stuff; a separate, often-backward flow of decisions about that stuff.
Lay both flows over the same DAG:
- Data flow (forward, materialized). Data moves downstream and is written down between stages — bronze → silver → gold. These are bytes at rest you can open and inspect at any point. The data only ever flows in the direction of the edges.
- Control flow (separate, often backward). The orchestrator's signal is a different thing entirely. It senses input freshness, triggers a stage, monitors it while it runs, and retries on failure. And it frequently reasons backward from the goal: "gold is stale because silver changed → re-run silver, then re-run gold." The control signal walks up the dependency edges to decide what to do, then dispatches work back down.
So data arrows ≠ control arrows, even though they overlay the same graph. The graph tells you the dependencies; data rides those edges forward and lands as materialized tables; control rides the same edges — often in reverse — to decide what to run and when. The machinery of that control plane (schedules, triggers, retries, backfills) is exactly lesson 09.
DATA (forward, materialized between stages)
─────────────────────────────────────────────
f_clean f_aggregate
[ bronze ] ─────────▶ [ silver ] ─────────▶ [ gold ]
(raw, at rest) (clean, at rest) (curated, at rest)
CONTROL (separate plane — trigger down, status up)
─────────────────────────────────────────────────────
┌───────────────────┐
│ scheduler │ reasons backward:
│ (orchestrator) │ "gold stale <- silver changed"
└───────────────────┘
trigger ┊ trigger ┊ trigger ┊
status ▲ status ▲ status ▲
▼ ▼ ▼
[ bronze ] [ silver ] [ gold ]
── solid = data moves forward and is materialized.
┊ dashed = control: scheduler triggers each stage (▼ down)
and reads ▲ status back (up). Same DAG, two flows.
If you remember one thing from this lesson beyond the pipeline itself, make it this: the data flowing forward and the control deciding what runs are two distinct flows over one graph. Keeping them separate in your head is what makes lesson 09 click.
ETL vs ELT: where the transform happens
Every pipeline does three broad kinds of work: Extract (pull data from sources), Transform (clean, reshape, aggregate), and Load (write it where consumers read it). The classic question is the order of the last two — and the answer flipped over the last decade.
| ETL — Extract → Transform → Load | ELT — Extract → Load → Transform | |
|---|---|---|
| Order | Transform before storing | Land raw first, transform in place |
| What lands | Only the cleaned, modeled result | The raw source, then derived tables beside it |
| Assumes | Storage is scarce/expensive; transform on a separate box | Storage is cheap (the lake, F3); the engine is powerful enough to transform at query time |
| Re-transform? | Hard — raw is gone; you must re-extract from the source | Easy — raw is still sitting there; just re-run the transform |
| Era | Classic warehouse default | The modern default |
ELT won because the assumptions changed. Cheap object-store lakes (F3) made "keep the raw forever" affordable, and powerful query engines made "transform it later, in place" fast. Landing raw first means that when you discover a bug in your cleaning logic, or someone wants a column you threw away, you re-run the transform — you don't have to go back and re-pull from a source that may have changed or disappeared. The transform stages become just more nodes in the DAG, reading raw and writing derived tables next to it.
This shape is the spine of the whole series
The reason we spend a whole foundations lesson on this: every pipeline you'll meet in this series is this same shape with different stages named in. It is fundamental, general data engineering — it runs every analytics warehouse, every BI dashboard, every product feature, and every ML system. The "medallion" layout you'll see later (bronze → silver → gold) is literally just this pipeline with the intermediate layers given names and persisted: bronze is raw, silver is cleaned, gold is the consumer-ready table.
GENERAL (analytics / any ML) POST-TRAINING DATA (this series)
───────────────────────────── ────────────────────────────────
source ─▶ ingest ─▶ clean ingest ─▶ clean ─▶ dedup
─▶ dedup ─▶ aggregate ─▶ tokenize ─▶ pack ─▶ serve
Same spine. Post-training just adds the tokenize + pack stages
(and, much later, an online loop that feeds rollouts back in).
bronze ─▶ silver ─▶ gold is this exact pipeline with the
intermediate layers named and persisted.
So the post-training pipeline that is the worked example of Parts I–III is not a new idea — it's this abstraction with two extra domain-specific stages. Learn the shape once here and the rest of the series is "which stages, in which order, with which contracts."
f(input) fits and runs on a single machine. What happens when one stage's input is a terabyte and won't fit, or its work is too slow for one box? That's exactly where F6 picks up — why one machine isn't enough, and what a stage looks like when it's split across many.