What data engineering is — records, schemas, storage, pipelines, distributed processing — built from zero, then narrowed into one of the most demanding pipelines in the field today: the data plane for post-training large language models.
Almost everything useful you can do with data — answer a business question, power a product, train a model — needs that data clean, organized, and reachable. It is almost never born that way. Turning the mess into something usable, reliably and at scale, is a distinct engineering discipline with its own tools (Spark, Ray, Parquet, orchestrators) and its own failure modes. This series teaches that discipline twice over: Part 0 (F0–F7) builds the foundations with no machine-learning background assumed — what data is, the relational model, where data lives, why storage layout decides cost, the pipeline abstraction, the shuffle, batch vs streaming, and the properties that make a pipeline trustworthy. Then Parts I–III apply every one of those ideas to a single worked example: the pipeline that turns raw text, human annotations, and model rollouts into training-ready batches for supervised fine-tuning, preference learning, and reinforcement learning.
Who this is for
You can read Python and do basic data wrangling. You do not need to have trained a model or know what SFT, DPO, or RL are — Part 0 assumes none of it, and the post-training concepts are introduced as we reach them. If you are already fluent in ML and only want the post-training pipeline, you can skip Part 0 and start at orientation. Either way, by the end you'll be able to design a real data pipeline — for an analytics warehouse or an SFT/preference/RL dataset — and reason about its correctness, throughput, and cost.
New to data engineering? Start with Part 0
Begin at F0 · What data engineering is. The eight foundations lessons each follow the same shape — an intuitive on-ramp (a concrete analogy) followed by the mechanical, precise version it graduates to — so each idea lands twice, two ways. Already fluent? Jump to 00 · Orientation.
Choose your path
The lessons form one ladder, but you don't have to climb all of it. Three routes, depending on where you're starting:
Two information flows in one picture. Top: data flows left to right through the pipeline (sources → bronze → silver → gold → trainer), with the main lessons that teach each stage. Bottom: the Part 0 foundation that gives you the intuition for each stage sits directly beneath it. The dashed arc is the RL loop, where the trainer's own output re-enters as data.
Interlude · see the whole flow once, concretely
Between orientation and Part I, 00b · The journey of one record follows a single annotation — a request to write a median function — all the way from a raw source through bronze, silver, gold, and into the trainer (plus its RL cousin around the online loop). Read it to feel the flow as one continuous story before the later lessons dissect each hop.
The pipeline you're learning
Every post-training dataset flows through the same shape: raw sources land in a bronze layer, get cleaned and curated into silver, and are tokenized and packed into gold batches the trainer consumes. RL adds the dashed loop — the trainer generates its own data every step.
Part 0 · Foundations (F0–F7 · data engineering from zero, no ML assumed)
Start here if data engineering is new to you. These eight lessons build the whole vocabulary the post-training series leans on — and each one teaches the idea twice: an intuitive analogy first, then the precise mechanics. Nothing here is post-training-specific; it's the same data engineering behind any analytics warehouse or ML system.
Linearly. Each lesson assumes the previous one. New to the field? Start at F0 — Part 0 builds the vocabulary the rest leans on. By lesson 07 we packed the tokens lesson 04 told us how to store; by lesson 10 the batch pipeline becomes an online one.
Two ways, every idea. Part 0 teaches each concept intuition-first (a concrete analogy) then mechanics-second (the precise definition). When a later lesson moves fast, the intuition is already in place.
Touch every knob. Each interactive widget has a setting that blows up your cost or starves the trainer. Find it — that's the lesson.
Map it to a tool. Every stage names the real tools (Spark, Ray, Daft, Parquet, Dagster) so you can go from the concept to the library you'd actually reach for.