The shape of data

Isn't data just text in a file? Why do engineers keep talking about its "shape"? Because the shape is exactly what decides whether a machine can find what it needs without reading everything — and that decision is the first one every pipeline makes.

Where we are

Lesson 2 of Part 0 · Foundations. In F0 we said data engineering moves data from where it's born to where it's used. But to move it well you first need to know what you're holding. This lesson builds the core vocabulary — record, field, type, schema — and the spectrum from structured to unstructured. Same shape as every foundations lesson: an intuition on-ramp, then the mechanics it graduates to.

"It's just text in a file" — and why that's not the whole story

You can open almost any data with a text editor and see characters. So in one literal sense, yes, it's just bytes. But that view hides the question that actually matters in practice: how much does the layout already tell a machine, before any program reads a single value? A file where every line has the same labeled columns lets a query jump straight to "the third column of row 9,000." A folder of voice memos does not — to answer anything you must process every file end to end. That difference — how much structure is baked in — is what "shape" means.

Intuition — three filing cabinets

Picture three cabinets, each holding "data" about your customers, but stored very differently.

Cabinet 1 — the spreadsheet (structured). Every row is a customer; every row has the exact same labeled columns: name, email, signup date, plan. You want "all customers on the Pro plan who signed up in May"? You glance down two columns. You never read a customer's whole row to know where their email is — column 2 is always the email. A machine loves this: the location of every fact is known in advance.
Cabinet 2 — the stack of filled-in forms (semi-structured). Mostly the same fields on every form, but people scribbled extras in the margins: one added a second phone number, another wrote a note that doesn't fit any box. To find someone's email you still know roughly where to look, but you have to glance at each form because they aren't perfectly uniform.
Cabinet 3 — the shoebox (unstructured). Loose photos, voice memos, napkin sketches, a printed email. The information is in there, but there is no layout at all. To answer "which customers mentioned refunds?" you must open and interpret every single item.

The rule to carry forward: the more structure is baked in, the less a machine has to "read" to find things. Structure is pre-paid work — someone organized it once so every later query is cheap.

Mechanics — record, field, type

Strip the analogy and three precise objects appear, the atoms of nearly all data:

A record is one observation — one row, one form, one item. "Customer #4471" or "the event that happened at 09:14:02." It is the unit you count, store, and process one of.
A field is one named attribute of a record: email, signup_date, plan. A record is a bundle of fields.
A type is the kind of value a field holds — string, integer, boolean, timestamp, or a nested structure. The type is what lets a machine compare, sort, sum, or validate without guessing: it can only add amount values if it knows they're numbers, not text.

So a record is a set of (field name, typed value) pairs. The spreadsheet cabinet works because every record agrees on the same field names and types; the shoebox is hard precisely because its items have no agreed fields at all.

The schema: the contract that makes a cabinet a cabinet

What turns a loose pile of records into the orderly spreadsheet is an agreement that they all share the same fields with the same types. Written down, that agreement is a schema.

Intuition — the blank form's template

The schema is the blank form before anyone fills it in: it declares the boxes, names them, says what goes in each ("date here, in DD/MM/YYYY"), and marks which boxes are mandatory. Hand someone the blank form and every filled-in form comes back compatible — that's why the form cabinet stays tidy and the shoebox never does. The shoebox has no blank form; everyone contributed whatever they liked.

Mechanics — schema as contract

A schema is the declared contract for a set of records: the field names, the type of each field, and which fields are required versus optional (and often constraints — "email must be unique," "age >= 0"). It is data about the data — metadata — that lets every reader and writer agree in advance on what a valid record looks like.

Why a contract is worth so much: it lets a machine reject bad data at the door, store records compactly (it need not repeat field names or guess types per row), and plan a query without inspecting the data first. The schema is what converts "find things by reading everything" into "find things by jumping to a known location."

The spectrum: structured, semi-structured, unstructured

The three cabinets are not three separate worlds; they're points on one axis — how much schema is present and enforced.

  MORE STRUCTURE  ◀───────────────────────────────────────────▶  LESS STRUCTURE
  (cheaper to query)                                          (cheaper to store anything)

  STRUCTURED                 SEMI-STRUCTURED                UNSTRUCTURED
  ──────────                 ───────────────                ────────────
  spreadsheet / SQL table    stack of forms w/ margins      shoebox of stuff
  fixed labeled columns      mostly-shared fields, some     raw text, images,
  every row identical        records add or omit fields     audio, PDFs
  e.g. CSV, Parquet,         e.g. JSON, JSONL, XML,         e.g. .txt, .jpg,
       a SQL table                log lines                      .wav, scanned docs
       │                            │                              │
       └─ schema fixed & enforced   └─ schema loose / self-        └─ no schema; meaning
          up front                     describing per record          must be extracted

Shape	Every record same fields?	Typical formats	Find a value by…
Structured	Yes, enforced	SQL tables, CSV, Parquet	Jumping to a known column
Semi-structured	Mostly; records may vary	JSON, JSONL, XML, log lines	Walking a record's self-described fields
Unstructured	No fields at all	Text, images, audio, PDFs	Reading / interpreting the whole item

Semi-structured is the sweet spot for a lot of real data because it carries its own labels. A JSONL file (one JSON record per line) is the workhorse format you'll meet constantly. Here are two records — note the second one adds a field the first lacks, which is perfectly legal:

{"id": 4471, "name": "Ada", "plan": "pro", "signup_date": "2026-05-03"}
{"id": 4472, "name": "Linus", "plan": "free", "signup_date": "2026-05-04", "referral": "newsletter"}

Each line is a record; each key (id, name, …) is a field; the values carry implied types (4471 is a number, "Ada" a string). There's a schema here, but it lives inside each record rather than being declared once and enforced — which is exactly what makes it semi-structured.

Two philosophies: schema-on-write vs schema-on-read

If a schema is so valuable, when do you enforce it? There are two answers, and the choice shapes whole systems.

Intuition — bouncer at the door vs. sort it at the table

Schema-on-write is a bouncer at the door: nothing gets into the club unless it matches the dress code. Bad records are turned away at ingestion, so everything inside is guaranteed clean and uniform — but you can't store anything that doesn't fit, and changing the dress code is disruptive.

Schema-on-read is "let everyone in, sort it out when you actually need to seat them." You dump the raw stuff in a big room (a data lake) exactly as it arrived, and only impose structure at the moment a query asks for it. Flexible and cheap to ingest — but every reader now shares the burden of making sense of the mess, and a malformed record only blows up later, at read time.

Mechanics — when validation happens

The two strategies differ only in when the schema is applied:

Schema-on-write — validate and conform records at ingestion, before they're stored. The classic example is a SQL table: you CREATE TABLE with column names and types first; any INSERT that violates the schema is rejected. Storage is then compact and queries are fast, because every row is already known-good and uniform. Cost: ingestion is rigid, and evolving the schema means migrating existing data.
Schema-on-read — store records raw (e.g. JSON files in a data lake) and apply a schema only when something reads them. The same files can be read under different schemas by different consumers. Cost: validation is deferred, so errors surface at query time, and every reader pays the parsing/interpretation cost again.

Neither is "right." Tight, well-understood data headed for many fast queries leans schema-on-write; large, varied, exploratory or raw data leans schema-on-read. Real pipelines often do both: land raw (on-read), then promote validated, conformed data into structured tables (on-write).

A note on the lake

Schema-on-read is the defining idea behind the data lake, and it returns in force in lesson 03 · ingestion, where we land raw post-training data first and structure it later. Hold the term loosely for now — just remember it means "store now, structure later."

This is general — and post-training is one user of it

Everything above is plain data engineering: the same record / field / type / schema idea underlies every database table, every log file, and every dataset on earth. Post-training large language models is simply one demanding user of these atoms — worth a moment because it's the worked example this series keeps returning to.

Every "unit of data" you feed a post-training run is a record with a schema. The fields just happen to be text:

Regime	A record's fields (its schema)
Supervised fine-tuning (SFT)	`(prompt, response)`
Preference / RLHF	`(prompt, chosen, rejected)`

An SFT dataset is most often a JSONL file — semi-structured, one record per line, exactly like the example above but with prompt and response fields. So the abstraction transfers directly: a training set is a pile of records, each conforming (we hope) to a schema, and "is this dataset valid?" becomes "does every record have the required fields, with the right types?" The full menu of which regime needs which fields is the subject of lesson 01 · data by regime.

Takeaway

Data is never "just text" — its shape is how much structure is baked in, and that decides how cheaply a machine can find things. The atoms are the record (one observation), the field (one named, typed attribute), and the schema (the contract: which fields, which types, which required). Data sits on a spectrum from structured (enforced uniform tables) through semi-structured (self-describing JSON/JSONL) to unstructured (raw text, images). You either enforce the schema at ingestion (schema-on-write, like a SQL table) or impose it at query time (schema-on-read, like a data lake). This vocabulary is universal; post-training datasets are just records with text-valued fields. Next, F2 takes the most structured end of the spectrum and makes it rigorous — the relational model.