Where data lives

The same database that runs an application is the wrong place to do analytics or model-training prep — and it's not a tuning problem, it's a built-for-a-different-job problem. This lesson is why, and the map of the stores you copy data to.

Where we are

We have rows and columns and a relational model (F2). Now: where does that data physically sit, and why is there more than one kind of store? The first-principles question driving this lesson: why can't the same database that runs the app also be the one we run analytics and model-training data prep on? The answer splits the whole field in two — and names the stores (warehouse, lake, lakehouse) that the rest of this series, including the post-training pipeline, actually lives in.

Two completely different jobs

It feels wasteful to keep two copies of your data in two different systems. Surely one good database can serve the app and answer business questions? In practice it can't — not because the database is bad, but because those two jobs pull a storage system in opposite directions. The clearest way to see it is a shop.

Intuition — the cash register vs the accountant

A shop has a cash register at the front. It handles thousands of tiny transactions a day: ring up one customer's three items, take payment, update stock by three, print a receipt — each in a fraction of a second, each touching almost no data. It must never be slow and must never be wrong: a frozen register or a double-charge is a disaster happening to a real person standing right there.

At the end of the quarter, an accountant sits in the back office and asks a different kind of question: "what sold best in Q3?", "which suppliers are we over-paying?", "what's the margin trend by region?" To answer, they read everything at once — every receipt, every line item, all quarter — and crunch it into a few summary numbers. One question, enormous read.

Now imagine making the accountant do their giant quarter-wide scan on the live register, while customers are checking out. The register would crawl; checkout would freeze. So you don't. You take a copy of the day's receipts to the back office — a place built for big, slow, thorough reads — and let the accountant work there without ever touching the front-of-shop machine. That copy-to-the-back-office move is the entire reason two kinds of data store exist.

Mechanics — OLTP vs OLAP

The two jobs have names. The register is OLTP (Online Transaction Processing); the accountant is OLAP (Online Analytical Processing). Every design choice flows from their opposite workloads:

	OLTP — runs the app	OLAP — runs analytics / ML
Workload	Many small concurrent reads and writes; each touches a few rows	Few huge queries; each scans/aggregates millions of rows, mostly read-only
Storage layout	Row-oriented (a whole record together — grab one order fast)	Column-oriented (a whole column together — scan one field cheaply)
Schema	Normalized (no duplication, safe to update — see F2)	Denormalized (pre-joined, wide tables — fewer joins per query)
Guarantees	Strong ACID: a transaction is all-or-nothing, never half-applied	Relaxed; correctness of a batch matters more than per-write atomicity
Optimized for	Low latency per transaction, high concurrency	High throughput per scan, cheap storage of huge volumes
Examples	PostgreSQL, MySQL behind a web app	Snowflake, BigQuery, a Spark job over Parquet

A single store cannot be excellent at both at once: row-vs-column is a physical layout decision (the subject of F4), and a giant analytical scan would lock up or starve the latency-sensitive transactional traffic. So you separate them, and data flows from one to the other:

  OLTP (app database)                          OLAP (analytical store)
  ───────────────────                          ───────────────────────
  orders, users, inventory   ──extract──▶      copy, reshaped for big scans
  row-oriented, normalized     (E of ETL)       columnar, denormalized
  never blocked by analytics                    analytics · BI · ML data prep

The arrow is an extract — the E in the ETL skeleton from lesson 02. The app keeps running on OLTP, undisturbed; everything analytical happens on a copy, downstream, in a store built for it.

The three analytical destinations

So data flows out of the app into an analytical store. But "analytical store" turns out to be three different things, and the difference is one of the most-asked-about distinctions in the field. Again, start with a picture.

Intuition — warehouse vs lake vs lakehouse

A data warehouse is a tidy, governed physical warehouse. Nothing gets in until it's been inspected, labeled, and shelved by strict rules. Everything inside is in a known place and a known form, so finding and trusting it is fast — but the inspect-and-shelve step means it's more work and more cost to get things in, and you can only store what fits the shelving system.

A data lake is the opposite: a vast, cheap, open expanse where you dump anything — JSON, images, logs, CSVs, half-broken exports — exactly as it arrived, and sort it out later. Wonderfully flexible and cheap. The danger is famous: with no labeling discipline, a lake silently rots into a "data swamp" — terabytes nobody can find, trust, or explain.

A lakehouse is the move that won: keep the cheap, sprawling lake for actually storing the bytes, but lay a smart catalog and ledger over the top that records what every file is, enforces shape, and tracks every change. You get the lake's cost and flexibility and warehouse-like order and trust — without paying to shovel everything onto rigid shelves up front.

Mechanics — the three stores, precisely

First, the substrate underneath two of them. Object storage (Amazon S3, Google Cloud Storage) stores immutable blobs addressed by a key — you PUT a file under a name and later GET it by that name. It is effectively infinite and very cheap per byte, but it is high-latency and has no in-place edit: you cannot change byte 500 of an existing object. To "edit," you read the whole object, change it, and write a whole new object. That single constraint is why lakes are immutable / append-only and why "updating" data means rewriting files, not patching them.

Now the three stores:

Data warehouse — structured tables only, schema-on-write (you must declare the shape before loading; bad-shaped data is rejected at the door), tightly governed. Storage and query are a managed bundle. Snowflake, BigQuery, Redshift.
Data lake — raw files of any type sitting directly in object storage, schema-on-read (no shape enforced on write; you interpret the bytes when you read them). Cheapest and most flexible, but ungoverned — no enforced schema, no transactions, no built-in catalog — which is the swamp risk.
Lakehouse — lake storage (files in object storage) plus a transactional table layer — Delta Lake or Apache Iceberg — that adds a metadata ledger on top of the raw files. That ledger buys back the warehouse properties: ACID transactions at the table/commit level (a whole write batch commits atomically — not the per-row transaction concurrency OLTP needs, which is why the OLAP side stayed "relaxed" above), schema enforcement, and time-travel (query the table exactly as it looked at an earlier version), all over cheap object storage.

	Warehouse	Lake	Lakehouse
Data shape	Structured tables only	Any type — files, blobs, raw	Tables defined over raw files
Schema timing	Schema-on-write	Schema-on-read	Schema-on-read + enforcement layer
Cost	Higher (managed, governed)	Lowest (raw object storage)	Low (object storage) + small metadata
Governance	Strong, built-in	None — swamp risk	Strong, via the table layer
Edit model	In-place updates supported	Immutable; rewrite files to "edit"	Immutable files + transactional rewrite (ACID)

The lakehouse insight in one line

A lake is just files; a lakehouse is the same files plus a transaction log that tells the truth about them. The bytes are cheap object storage either way — what you add is a metadata ledger that makes a pile of immutable files behave like a real, governed, versioned table.

This series lives in a lakehouse too

Here's the payoff, and it's deliberately deflating: the post-training data pipeline this series builds toward is not special infrastructure. It lives in a lakehouse, exactly like an analytics team's data does. The medallion layout you'll see throughout — bronze, silver, gold — is the lakehouse pattern wearing a different name.

Medallion layer	What it is	Lakehouse property
Bronze	Immutable raw capture — exactly as ingested, append-only	Lake storage: immutable files in object storage
Silver	Cleaned, deduplicated, validated, conformed records	A curated, schema-enforced table
Gold	Training-ready datasets shaped for the consumer	A denormalized table built for big reads

The defining rule of the medallion layout — each layer is reproducible from the one before it — is precisely lakehouse semantics: immutable inputs plus a recorded, re-runnable transformation, so any layer can be rebuilt byte-for-byte and any version recovered (that's time-travel). The raw bronze sitting as Parquet on object storage (the columnar file format that's the subject of the next lesson, and lesson 04) is lake storage in the literal sense. The medallion structure introduced in lesson 02 and the map in orientation are this same picture, applied to text and model data.

So the emphasis to carry forward: OLTP/OLAP and warehouse/lake/lakehouse are fundamental, general data engineering. They are how analytics, BI, and every ML system organize storage. Post-training data isn't exotic — it just lives in a lakehouse like everyone else's data, and benefits from the same governance, reproducibility, and cheap columnar storage.

What F4 answers next

We kept asserting that analytical stores are "columnar" and that this makes huge scans cheap, without saying why. The next lesson, F4 · Why layout decides cost, opens up the physical layout — row-vs-column on disk — and shows how the choice of layout, more than any clever query, is what decides whether a quarter-wide scan costs cents or dollars.

Takeaway

One database can't run the app and the analytics because those are opposite jobs: OLTP (many tiny, fast, ACID transactions — row-oriented, normalized) versus OLAP (few enormous scans — columnar, denormalized). So you extract a copy from OLTP into an analytical store, which comes in three shapes: a governed warehouse (schema-on-write), a cheap raw lake (schema-on-read on immutable object storage, with swamp risk), or a lakehouse (lake files plus a transactional table layer giving ACID, schema enforcement, and time-travel). The post-training pipeline's bronze/silver/gold medallion is just a lakehouse — the same fundamental storage every data team uses. Next: why the columnar layout these stores use is so cheap to scan.