Data-Intensive Systems, From First Principles
A linear, mechanism-first track on the data systems beneath reliable software and ML products. It follows one spine: a single truthful copy, pressured one step at a time — meaning over time, demand, redundancy, distribution, concurrency, failure, derived history, derived views, cloud/AI, and finally correctness and society — then an applied part that locates real systems in that design space and works six full interview cases.
Source note
An original educational synthesis inspired by Martin Kleppmann's Designing Data-Intensive Applications (and the expanded 2nd edition with Chris Riccomini). It does not reproduce the book's prose or figures; it uses the book's conceptual arc as a launchpad for site-native, first-principles lessons with ML-infrastructure examples.
The linearized idea
Start with one truthful copy on one machine. Then apply one pressure at a time: meaning drifts as code changes; demand wants it fast; redundancy keeps copies; distribution splits it; concurrency and messages threaten its invariants; machines fail; you derive history and then views from it; the cloud turns the disk into a service; and law and ethics shape what you may keep at all. Every mechanism is a contract bought at a cost charged somewhere else.
The recurring question: "Where is truth?"
Every lesson ends with the same artifact. For its mechanism, name the system of record, the copies / derived views, the freshness budget, the owner, the deletion path, the reconciliation/repair path, and the evidence it is correct. That single question ties the whole spine together.
The spine
a single truthful copy (1) model, storage, invariants
-> meaning over time (2) schemas, compatibility, migrations
-> demand (3) indexes, caches, SLOs, latency
-> redundancy (4) replication, lag, local-first sync
-> distribution (5) sharding, tenants, routing
-> invariants under concurrency (6) transactions, messages, workflows
-> failure (7) clocks, leases, partitions, consensus, verification
-> derived history (8) logs, CDC, batch, stream
-> derived views (9) warehouse, search, vector, features, models
-> cloud / AI systems (10) managed services, object stores, RAG, ML pipelines
-> correctness and society (11) audits, privacy, fairness, accountability
-> [applied] (12) production atlas, failure drills, interview cases
Syllabus
Orientation
Part 1 · A single truthful copy
01
Data Models: Relational, Document, and Graph
The shape of thought a system makes cheap: tables for joins, documents for aggregate reads, graphs for relationships that keep moving.
02
Query Languages and Access-Pattern Design
Declarative languages say what you want and let the engine plan; at scale the access pattern becomes part of the storage contract.
03
Storage Engines I: Logs, Hash Indexes, and B-Trees
A write becomes durable by entering a log; a read becomes fast by using an index. The first fork: overwrite pages or append history.
04
Storage Engines II: LSM-Trees and Columnar Storage
LSM trees make writes sequential and defer cleanup to compaction; column stores arrange bytes by question, not by object.
Part 2 · Meaning over time
Part 3 · Demand
06
Reliability, Scalability, SLOs, and Tail Latency
What the system must keep true as load grows and parts fail; percentiles, tail-latency amplification, and the social timeline as the canonical NFR case.
07
Indexes, Caches, and Serving Under Demand
Making the read path fast: index trade-offs, caching patterns, stampede control, and SLO-driven latency budgets — with a freshness bill.
Part 4 · Redundancy
08
Replication I: Leaders, Followers, Lag, and Failover
Copies buy availability, read scale, and local latency. The price is disagreement, and the gap between copies is where anomalies live.
09
Replication II: Quorums, Leaderless, and Conflicts
Once more than one node accepts writes, availability rises and conflict resolution becomes part of the application contract.
10
Local-First and Offline Sync
Multi-leader at its limit: every device is a usually-offline writer. CRDTs converge without a central authority.
Part 5 · Distribution
11
Partitioning, Hot Keys, and Rebalancing
Split data across machines. The shard key is a performance contract: it fixes locality, balance, and which queries become scatter-gather.
12
Multitenancy, Request Routing, and Sharding Operations
Running a real multi-tenant cluster: tenant isolation, noisy neighbors, request routing, and rebalancing without an outage.
Part 6 · Invariants under concurrency
13
Transactions and Isolation
A transaction lets the app pretend concurrency and crash faults did not happen. Isolation level decides how convincing the illusion is.
14
Messaging, Idempotency, and the Transaction Boundary
When an operation spans a database and a broker, ACID stops. The outbox pattern, idempotency keys, and effectively-once as a scoped guarantee.
Part 7 · Failure
15
Partial Failure, Unreliable Clocks, and Fencing
Distributed systems fail partially and nondeterministically. You cannot tell crashed from slow; clocks lie; fencing tokens stop zombies.
16
Consistency, Causality, and Linearizability
What reads are allowed to see. The strongest models make distributed state look single-copy — and charge latency and availability.
17
Consensus, Coordination, and Verification
Make unreliable machines agree irrevocably — total-order broadcast, Raft, 2PC limits — and verify them with formal methods and fault injection.
Part 8 · Derived history
18
Batch Processing: Dataflow Engines, Joins, and Recompute
Bounded input to derived output. MapReduce as history; dataflow engines, sort-merge vs broadcast joins, and deterministic recompute.
19
Stream Processing: Logs, CDC, Windows, and Stream Joins
Batch without an end-of-file. Logs, change data capture, event time vs processing time, windows, watermarks, and stream joins.
Part 9 · Derived views
20
Derived Data: Keeping Views Correct
Compose specialized stores around one system of record. Drive every derived view from a single ordered log, never dual writes.
21
Analytics and Query-Execution Internals
Why warehouses feel fast: star/snowflake schemas, vectorized and compiled execution, columnar pipelines, and compression-aware operators.
22
Search, Vector Indexes, and RAG
Full-text inverted indexes and approximate-nearest-neighbor vector search as derived views, and the RAG pipeline kept fresh by CDC.
23
Feature Stores and ML Data Pipelines
DataFrames, point-in-time joins without label leakage, offline vs online stores, train-serve skew, and predictions as derived data.
Part 10 · Cloud / AI systems
Part 11 · Correctness and society
25
Law, Regulation, and Ethics as Architecture Input
Data minimization, consent, the deletion path against an immutable log, auditability, GDPR-style rights, surveillance, and data as power.
26
Current DDIA: What the 2nd Edition Adds
A signpost to DDIA 2e (Kleppmann & Riccomini, ~672 pp): cloud, GraphQL, event sourcing/CQRS, vector search, durable execution, local-first, and ethics.
Part 12 · Applied & interviews
27
Production Systems Atlas
Locate Postgres, Cassandra/Dynamo, Bigtable/HBase, Kafka/Pulsar, Redis, Elasticsearch, and Spark/Flink at their coordinates in the design space.
28
Failure Timelines and Quantitative Drills
How systems break — failover, stale reads, CDC backlog, rebalance storm, cache stampede, dual-write divergence — and the drills to size them.
29
Case: Social Home Timeline
Fan-out-on-write vs on-read, the celebrity hot key, and the hybrid real timelines use — the canonical NFR case, with an answer rubric.
30
Case: Global Profile Store
Multi-region low-latency reads, read-your-writes after an edit, concurrent conflicts, and the CAP/PACELC trade made explicit.
31
Case: ML Feature Store
Online lookups vs offline training-set builds, point-in-time correctness without leakage, train-serve skew, and CDC freshness.
32
Case: Search Index Without Dual Writes
The flagship "don't dual-write" case: why writing Postgres and Elasticsearch separately diverges, and the CDC/outbox fix.
33
Case: Model Registry
Immutable versioned artifacts plus metadata, the atomic linearizable "promote to prod" pointer, audit/lineage, and safe rollback.
34
Case: Metrics / Observability Dashboard
High-cardinality time-series ingest, columnar rollups, hot/cold tiers, approximate aggregates, and timeliness over integrity.
How this differs from the neighboring tracks
| Track | Focus | This track's role |
|---|---|---|
| Distributed Systems Design | Interview patterns and backend architecture moves | Goes deeper on the data-system mechanisms behind those moves |
| Data Engineering for Post-Training | ML training-data pipelines and lakehouse workflows | Explains the storage, schema, batch, stream, and correctness substrate |
| ML Systems Design | Designing model-serving, training, evaluation, and product systems | Provides the database/log/index/derived-state vocabulary those systems depend on |