data_engineering / lessons / index 8 foundations + 13 lessons · ~6.5h read

Data Engineering, From First Principles

What data engineering is — records, schemas, storage, pipelines, distributed processing — built from zero, then narrowed into one of the most demanding pipelines in the field today: the data plane for post-training large language models.

Almost everything useful you can do with data — answer a business question, power a product, train a model — needs that data clean, organized, and reachable. It is almost never born that way. Turning the mess into something usable, reliably and at scale, is a distinct engineering discipline with its own tools (Spark, Ray, Parquet, orchestrators) and its own failure modes. This series teaches that discipline twice over: Part 0 (F0–F7) builds the foundations with no machine-learning background assumed — what data is, the relational model, where data lives, why storage layout decides cost, the pipeline abstraction, the shuffle, batch vs streaming, and the properties that make a pipeline trustworthy. Then Parts I–III apply every one of those ideas to a single worked example: the pipeline that turns raw text, human annotations, and model rollouts into training-ready batches for supervised fine-tuning, preference learning, and reinforcement learning.

Who this is for
You can read Python and do basic data wrangling. You do not need to have trained a model or know what SFT, DPO, or RL are — Part 0 assumes none of it, and the post-training concepts are introduced as we reach them. If you are already fluent in ML and only want the post-training pipeline, you can skip Part 0 and start at orientation. Either way, by the end you'll be able to design a real data pipeline — for an analytics warehouse or an SFT/preference/RL dataset — and reason about its correctness, throughput, and cost.
New to data engineering? Start with Part 0
Begin at F0 · What data engineering is. The eight foundations lessons each follow the same shape — an intuitive on-ramp (a concrete analogy) followed by the mechanical, precise version it graduates to — so each idea lands twice, two ways. Already fluent? Jump to 00 · Orientation.

Choose your path

The lessons form one ladder, but you don't have to climb all of it. Three routes, depending on where you're starting:

🌱
New to data engineering — the full ladder
F0 → F7 (foundations) → 00 orientation → 00b one record's journey → 01 → 11. Every concept intuition-first; nothing assumed. This is the intended path.
ML-fluent — straight to the pipeline
00 orientation → 00b journey → 01 → 11. Dip back into F4 (layout) or F6 (the shuffle) only if lesson 04 or 05 feels fast.
🎯
"I just need to design one pipeline" — fast path
02 skeleton → 04 formats → 05 compute → 09 orchestration → 11 cost. The load-bearing five; backfill the rest as needed.

The map — where each lesson sits in the flow

Two information flows in one picture. Top: data flows left to right through the pipeline (sources → bronze → silver → gold → trainer), with the main lessons that teach each stage. Bottom: the Part 0 foundation that gives you the intuition for each stage sits directly beneath it. The dashed arc is the RL loop, where the trainer's own output re-enters as data.

10 · RL online loop — the trainer's rollouts re-enter as data every step DATA FLOWS → SOURCES annotate·log·scrape BRONZE raw · immutable SILVER clean·dedup·gated GOLD tokenized·packed TRAINER SFT·DPO·RL 01 · 03 02 · 04 05 · 06 · 08 07 consumes 09 · orchestration + 11 · scaling, cost & monitoring — span the whole pipeline PART 0 FOUNDATIONS — the intuition under each stage F1 shape F2 relational F3 where it lives F5 pipeline F6 the shuffle F7 trust F4 layout & cost F0 what DE is F7 batch/stream See one record travel this entire path in the interlude: 00b · the journey of one record
Interlude · see the whole flow once, concretely
Between orientation and Part I, 00b · The journey of one record follows a single annotation — a request to write a median function — all the way from a raw source through bronze, silver, gold, and into the trainer (plus its RL cousin around the online loop). Read it to feel the flow as one continuous story before the later lessons dissect each hop.

The pipeline you're learning

Every post-training dataset flows through the same shape: raw sources land in a bronze layer, get cleaned and curated into silver, and are tokenized and packed into gold batches the trainer consumes. RL adds the dashed loop — the trainer generates its own data every step.

SOURCES annotate · synth logs · scrape BRONZE raw · immutable provenance SILVER clean · dedup decon · quality GOLD tokenized packed batches TRAINER SFT · DPO · RL REGIMES pairs · prefs · prompts ETL pack batches schema RL: rollouts re-enter as data every step

Part 0 · Foundations (F0–F7 · data engineering from zero, no ML assumed)

Start here if data engineering is new to you. These eight lessons build the whole vocabulary the post-training series leans on — and each one teaches the idea twice: an intuitive analogy first, then the precise mechanics. Nothing here is post-training-specific; it's the same data engineering behind any analytics warehouse or ML system.

F0
What data engineering is
The discipline of getting data from where it's born to where it's used — clean, on time, reproducible, affordable. Why it's a role of its own, and why the pipeline, not the dataset, is the product. Post-training as one destination among many.
F1
The shape of data
Records, fields, types, schemas. The structured → semi-structured → unstructured spectrum. Schema-on-write vs schema-on-read. Why we keep talking about data's "shape." (Three filing cabinets.)
F2
The relational model
Keys, foreign keys, joins, normalization — and the normalize↔denormalize read/write trade. Why operational data lives in many small tables but training data is one flat wide one. (A wedding guest list.)
F3
Where data lives
OLTP vs OLAP. Data warehouse vs lake vs lakehouse, and object storage underneath. Why the app's database can't also be the analytics store — and why the medallion layout is a lakehouse. (Cash register vs accountant.)
F4
Why layout decides cost
Rows vs columns; projection, predicate pushdown, compression. How the same bytes laid out differently change a query's cost by tens of ×. The intuition that lesson 04's Parquet section builds on. (Library shelving.)
F5
The pipeline abstraction
Dataflow as a DAG of pure-function stages with defined contracts. ETL vs ELT. Why a pipeline beats one do-everything script. The shape the whole series runs on. (An assembly line.)
F6
Why one machine isn't enough
Horizontal scaling, partitions, narrow vs wide operations, the shuffle, skew and stragglers. Why the shuffle — not the compute — is the cost center. (Counting votes in an election.)
F7
Batch, streaming, and trust
Batch vs streaming as a fundamental fork. Idempotency, determinism, reproducibility — the properties that survive retries. Throughput, unit cost, and finding the bottleneck. The bridge into the post-training series. (Laundry, an elevator button, a budget road trip.)

Part I · The substrate (01–03 · post-training data, and how it gets in)

01
The data, by regime
The unit of data for each post-training regime: SFT = (prompt, response); preference = (prompt, chosen, rejected); RLVR = (prompt, verifier/answer). The shared schema, and why "the dataset is the product."
02
The ETL / ELT skeleton
Extract → transform → load. The medallion layout (bronze/silver/gold = raw/curated/training-ready). Batch vs streaming. Idempotency and determinism — the two properties that make a pipeline reproducible.
03
Ingestion & provenance
Where data comes from (human annotation, synthetic generation, logs, public datasets, scrape) and how it lands. Incremental / CDC ingestion, schema-on-read, and capturing license + provenance at the door — before it's too late.

Part II · The pipeline, one stage per lesson (04–09 · bronze → gold)

04
Storage & file formats
JSONL vs Parquet vs Arrow; row vs columnar; compression, partitioning, sharding; predicate & column pushdown. Why the format you pick decides your scan cost. Live format / scan-cost comparator.
05
Transformation at scale — Spark, Ray, Daft
The distributed compute model: map / filter / shuffle over partitions. Why the shuffle is the cost center. Spark vs Ray Data vs Daft vs single-node, and how to choose. Live partition → worker throughput simulator.
06
Dedup & decontamination
Normalization, PII handling, exact + near-duplicate detection via MinHash / LSH, and n-gram decontamination against eval sets. Why duplicates and test leakage quietly wreck a run. Live LSH bucketing demo.
07
Tokenization & packing
Tokenize as a pipeline stage. Sequence packing to fill the context window; loss masking for prompt and tool tokens; the padding waste you pay without packing. The throughput math. Live packing-efficiency simulator.
08
Quality & validation
Schema contracts, expectation checks, and gates that fail the build. Quality scoring and filtering (length, language, toxicity, reward-model score). The quality-vs-quantity trade, and where to put each filter. Live quality funnel.
09
Orchestration & versioning
DAG orchestrators (Airflow, Dagster, Prefect, Flyte): dependencies, retries, backfills, idempotency. Data versioning and lineage so a dataset is reproducible. Materialization and caching. Live DAG task-state walk-through.

Part III · From batch to online (10–11 · the RL dataplane and the cost model)

10
The RL online dataplane
In RL the pipeline runs inside the training loop: rollout → verify → buffer → train, every step. Replay buffers, on-policy freshness, streaming trajectory data. Why batch ETL assumptions break and what replaces them. Live freshness / staleness simulator.
11
Scaling, cost & monitoring (capstone)
The throughput and cost model of the whole pipeline: records/sec, $/M tokens, where the bottleneck sits (I/O vs compute vs shuffle). Monitoring volume, freshness, and quality drift. The end-to-end reference architecture. Live pipeline sizer.

How to use this

  1. Linearly. Each lesson assumes the previous one. New to the field? Start at F0 — Part 0 builds the vocabulary the rest leans on. By lesson 07 we packed the tokens lesson 04 told us how to store; by lesson 10 the batch pipeline becomes an online one.
  2. Two ways, every idea. Part 0 teaches each concept intuition-first (a concrete analogy) then mechanics-second (the precise definition). When a later lesson moves fast, the intuition is already in place.
  3. Touch every knob. Each interactive widget has a setting that blows up your cost or starves the trainer. Find it — that's the lesson.
  4. Map it to a tool. Every stage names the real tools (Spark, Ray, Daft, Parquet, Dagster) so you can go from the concept to the library you'd actually reach for.