Part 11 · Correctness & society

Current DDIA: What the 2nd Edition Adds (and the Ethics Chapter)

The previous lesson (25) took the field's grown conscience — privacy, surveillance, bias, data as power — and treated law, regulation, and ethics as a hard architecture input, the same way an SLO or a partition key is. That lesson's mechanisms, and every mechanism this track drew, were mapped from the first edition of Designing Data-Intensive Applications, published in 2017, and the field did not stand still. This lesson is a signpost, not a syllabus: the first-edition arc you learned still holds — B-trees, LSM-trees, replication, quorums, partitioning, isolation, consensus, the log, derived data — and the second edition adds and expands on top of it. We walk the new and grown topics briefly, say where each one extends a lesson you already own, and hand off to the applied part (lesson 27, the production systems atlas) where these mechanisms get scored against real databases.

DDIA source

Signpost to Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd edition (O'Reilly Media) — 14 chapters, 672 pages. Note a publication-date discrepancy in the public listings: O'Reilly's catalog lists the release as February 2026, while Martin Kleppmann's own site lists March 2026; treat the exact ship date as not-yet-settled. This lesson does not reproduce the book; it maps its expanded scope back onto the first-edition arc this track taught (lessons 00–26). Treat topic boundaries as approximate — the final table of contents may shift — and read this as "where to look next," not a substitute for the book.

Linear position

Prerequisite: the whole track, especially the derived-data capstone (20) and the law/regulation/ethics lesson (25). You should already be able to name a system of record, derive every other store from one ordered log, give each view a lag budget and a rebuild path, and reason about replication, partitioning, isolation, and consensus by their contracts and bills.
New capability: a current map. You can place each topic the second edition foregrounds — cloud/serverless storage–compute split, GraphQL, event sourcing and CQRS, vector search, durable execution, local-first sync, multitenancy, formal methods, and data ethics — onto a mechanism you already understand, so the new vocabulary lands as extension rather than novelty, and you know which lesson to revisit.

The plan

Four moves. (1) Frame what changed between 2017 and now and why a "second edition" is mostly re-weighting, not rewriting — the mechanisms are stable, the defaults and the deployment substrate moved. (2) Walk the expanded architecture and modeling topics — cloud/serverless storage–compute separation, GraphQL, event sourcing and CQRS, and multitenancy — each as "what it is / why it matters now / where it extends our track." (3) Walk the expanded correctness, scale, and ML-adjacent topics — vector search, durable execution, local-first sync, and formal methods and testing — the same way, with a single extends-Lxx grid tying it together. (4) The chapter this track has been building toward: the dedicated ethics chapter — privacy, surveillance, bias and accountability, data as power — then a real closing takeaway and a hand-off to the neighboring site tracks.

1 · What changed, and what did not

The first edition's thesis was that you should understand data systems by the guarantees they make and the costs those guarantees charge, not by brand names. That thesis is timeless, and nothing in the second edition retracts it. A B-tree still does point lookups and an LSM-tree (lesson 04) still trades read amplification for write throughput; quorum arithmetic w + r > n (lesson 09) is still the same inequality; linearizability (lesson 16) still costs a coordination round trip. So a "second edition" of a mechanisms book is not a rewrite — it is a re-weighting. What moved is the substrate and the defaults:

The deployment substrate moved to the cloud. In 2017 you sized disks and racks; now the default is a managed object store plus elastic compute, and "separation of storage and compute" is an architectural primitive, not an exotic choice.
New workloads became first-class. Similarity search for embeddings, AI/ML retrieval (RAG), and high-cardinality observability were niche in 2017 and are now mainstream — so the topics that serve them get their own treatment.
Some patterns graduated from "advanced" to "expected." Event sourcing, CQRS, change data capture, and durable workflow execution were specialist techniques; they are now common enough to deserve dedicated coverage.
The field grew a conscience. The first edition closed with a short reflection on ethics; the second elevates it to a dedicated chapter, because the consequences of getting data systems wrong are now societal, not just operational.

So the right way to read the rest of this lesson is: each "new" topic is a mechanism you already hold, viewed under a pressure that grew louder. We name the pressure, the topic, and the home lesson.

2 · Expanded: architecture and data modeling

Cloud and serverless data architecture — the separation of storage and compute. What it is: instead of a database where one machine owns both the disk holding the data and the CPU querying it, the data lives in a shared, durable, cheap object store (think S3-style buckets) and stateless compute nodes are spun up on demand to read and process it. Why it matters now: the two resources scale independently — pay for petabytes of cold storage without paying for idle CPUs, and burst to a thousand query workers for one minute without keeping them around. This is the architecture of every modern cloud warehouse and lakehouse. Where it extends our track: it is the columnar/analytics store of lesson 04 pulled apart along its seams, and it makes the rebuild path of a derived view (lesson 20) cheap — recompute is just "rent compute, scan the object store, write the view back."

GraphQL — a query language at the API edge. What it is: a query language and runtime where the client specifies exactly which fields it wants in one request, traversing related objects in a single round trip, instead of calling a fixed REST endpoint per resource. Why it matters now: it directly attacks two pains of REST APIs — over-fetching (the endpoint returns a fat object when the client wanted three fields) and under-fetching (the client must make N follow-up calls to assemble one screen, the chatty round-trip problem). Where it extends our track: it is an access-pattern design choice (lesson 02) made at the network boundary, and an encoding/schema-evolution concern (lesson 05) — a GraphQL schema is a contract that must evolve with backward and forward compatibility just like any wire format, and its resolver-per-field model has its own fan-out and N+1 hazards to watch.

Event sourcing and CQRS. What it is: event sourcing stores the append-only log of what happened (an immutable sequence of events) as the system of record, rather than only the current mutable state. CQRS (Command Query Responsibility Segregation) splits the write side (commands appended to the log) from the read side (one or more derived views shaped per query). Why it matters now: together they give a full audit trail, time-travel ("what did this look like last Tuesday?"), and the freedom to build a new read model at any time by replaying the events. Where it extends our track: this is the log-as-integration-backbone of lessons 19 and 20 stated as a deliberate architecture — the event log is the system of record, and every read model is a derived view with a source, a lag budget, and a rebuild path (replay from offset 0).

Multitenancy and sharding for SaaS. What it is: running many customers (tenants) on shared infrastructure while keeping their data isolated — from "one shared schema with a tenant_id column" through "schema-per-tenant" to "database-per-tenant," with sharding to spread the load. Why it matters now: nearly every B2B product is multi-tenant SaaS, and the noisy-neighbor problem (one big tenant starving the rest) is a real production hazard. Where it extends our track: this is partitioning (lesson 11) with the partition key chosen as the tenant boundary — the same hot-key and rebalancing problems, now wearing a business label, where a single whale tenant is the celebrity hot key of lesson 29.

3 · Expanded: correctness, scale, and the ML tie

Vector embeddings and vector search (ANN indexes). What it is: represent items (text, images, users) as high-dimensional vectors so that "similar" means "nearby," then index those vectors with an approximate nearest-neighbor (ANN) structure — HNSW graphs, IVF lists, product quantization — so "find the 20 most similar" is sub-linear rather than a full scan. Why it matters now: this is the retrieval engine of semantic search and RAG, the backbone of LLM applications. Where it extends our track: it is a new index shape answering a new access pattern (lessons 01 and 04 — it sits beside the B-tree and the inverted index), and in a composed system it is exactly a derived view of the source documents (lesson 20) — rebuildable by re-embedding and re-indexing, with a lag budget when the embedding model changes. ANN trades exactness for speed: you accept a recall below 100% in exchange for orders-of-magnitude faster lookup, which is the read/space/time amplification trade-off of lesson 04 in a new costume.

Durable execution — workflow engines. What it is: a programming model (Temporal-style) where a long-running workflow's progress is persisted at every step, so if the process crashes mid-way it resumes from the last completed step instead of restarting or leaving a half-done mess. The engine records each step's result to a durable log and replays it to reconstruct state. Why it matters now: it makes reliable multi-step orchestration — payment then shipment then notification, or a multi-call agent pipeline — tractable without hand-rolling state machines and retry logic. Where it extends our track: it is the durable ordered log (lesson 03) and the end-to-end idempotency-key discipline (lesson 20) packaged as an execution runtime, and it leans on coordination/consensus (lesson 17) to make "this step ran exactly once" hold across failures.

Local-first and offline sync (CRDTs). What it is: applications that keep a full working copy of the data on the user's device, let them read and write offline, and sync changes peer-to-peer or through a server when connectivity returns — merging concurrent edits automatically with CRDTs (Conflict-free Replicated Data Types), data structures whose merge function is commutative, associative, and idempotent so any two replicas converge to the same value regardless of message order. Why it matters now: collaborative editors and offline-capable apps need conflict resolution that does not lose user work or require a central lock. Where it extends our track: it is multi-leader and leaderless replication with concurrent-write conflicts (lesson 09), but where the conflict-resolution strategy is baked into the data type so convergence is automatic rather than last-write-wins or a manual merge.

Formal methods and testing. What it is: techniques to gain confidence that a distributed design is actually correct — TLA+ (a specification language and model checker that exhaustively explores a protocol's state space to find ordering bugs no test would hit), deterministic simulation testing (run the whole system on a simulated clock and network so a failing run can be replayed exactly), and property-based and fault-injection testing (assert invariants over randomized inputs and inject crashes, partitions, and delays). Why it matters now: the subtle concurrency and partial-failure bugs of lessons 15 and 17 are precisely the ones that survive ordinary unit tests and surface in production at 3am. Where it extends our track: it is how you verify the partial-failure, clock, and consensus reasoning of lessons 15 and 17 — turning "I argued this is correct" into "a model checker explored a million interleavings and found no violation."

The whole expansion, on one map:

2nd-edition topic	One-line what	Extends
Storage–compute separation	Cheap object store + elastic on-demand compute	L04, L20
GraphQL	Client picks exact fields; fixes over/under-fetch	L02, L05
Event sourcing & CQRS	Event log is the truth; read models are derived	L19, L20
Vector search (ANN)	Similarity index for embeddings / RAG	L01, L04, L20
Durable execution	Persist each workflow step; resume after crash	L17, L20
Local-first / offline (CRDTs)	Offline edits that auto-merge on sync	L09
Multitenancy & SaaS sharding	Many isolated tenants on shared infra	L11
Formal methods & testing	TLA+, deterministic sim, fault injection	L15, L17
Data ethics	Privacy, surveillance, bias, data as power	L20

How to read each row

None of these rows introduces a primitive you have not met. "Extends" means: go back to that lesson, recall the contract and the bill, and read the new topic as that same trade-off pushed by a 2026 pressure. Vector search is still an index choosing read speed over exactness; CRDTs are still concurrent-write conflict resolution; multitenancy is still partitioning by a key that happens to be a customer. The vocabulary is new; the physics is the one this track taught.

4 · The ethics chapter — the obligation the whole track was circling

The first edition ended on a short ethical reflection. The second edition gives ethics its own chapter, and that promotion is the most important signal in the new book: a data engineer's decisions now have consequences that are societal, not merely operational. Lesson 25 already treated this chapter's concerns as a hard architecture input — law, regulation, and ethics shaping the design the way an SLO does — so here we only note where each of the chapter's four threads lands on a mechanism this track taught, without re-deriving the ethics treatment itself.

Privacy & consent

Data collected for one purpose is reused for another the user never agreed to. Because every byte flows along the lineage of lesson 20, the same log that lets you rebuild a view is what lets you honor deletion and prove it propagated — privacy is enforceable only if you know where data went.

Surveillance & tracking

Logging "every event for analytics" is also building a surveillance record. The high-cardinality event capture of the metrics case (L34) and the CDC event log (L19) are powerful precisely because they remember everything — which is exactly why retention and minimization are ethical choices, not just storage-cost choices.

Bias & accountability

A model trained on a derived view inherits whatever skew the pipeline introduced. The online/offline skew and label-leakage hazards of L20 and the feature-store case (L31) are not only accuracy bugs — when the output decides a loan or a sentence, they are fairness failures someone must be accountable for.

Data as power

Whoever holds the system of record holds leverage over the people in it. The "system of record vs derived view" distinction from L20 is also a question of who controls the truth — an architectural decision with a political weight the engineer should name out loud.

Lesson 20 already drew the line between integrity (no corruption or loss — usually non-negotiable) and timeliness (freshness — often relaxable). The ethics chapter adds a third axis the engineer cannot offload to the architecture: responsibility. The mechanisms are neutral; the choice to collect, retain, join, and act on data is not. A system that can prove it deleted a user's data, that minimizes what it tracks, that audits its derived views for skew, and that names who owns the truth is not just a well-engineered system — it is a defensible one.

What stays the same

The mechanisms: storage engines, replication, quorums, partitioning, isolation, consensus, the log, derived data — all first-edition, all still load-bearing.
The method: judge a system by its guarantee and its bill, not its brand.
The capstone discipline: one system of record, derive the rest from one ordered log, every view gets a source, lag budget, and rebuild path.

What grew louder

The cloud substrate: storage and compute separated by default.
ML-adjacent workloads: vector search and RAG as first-class retrieval.
Graduated patterns: event sourcing, CQRS, durable execution, local-first sync.
Verification: formal methods and deterministic simulation, not just tests.
Ethics: from an afterword to a dedicated chapter and an engineering obligation.

Checkpoint exercise

Try it

Take the RAG assistant you designed in the lesson-20 checkpoint and re-examine it through the second-edition lens. (a) Which boxes are now best described with a 2nd-edition term — the vector index as an ANN derived view, the ingestion pipeline as durable execution, the read API as GraphQL, the chunk-and-embedding store as event-sourced? Map each to its extends-Lxx home. (b) If you make it multi-tenant for several customers, what is your partition key and what is your noisy-neighbor mitigation (L11/L29)? (c) Pick one invariant ("a deleted document never appears in a retrieval result") and say how you would gain confidence in it — an audit, a property-based test, or a TLA+ spec — and why. (d) Write the three-sentence ethics note for this system: what you collect, how long you keep it, and how you prove a deletion propagated to the vector index.

Where this points next

This signpost closes Part 11. The applied part opens next: lesson 27, the production systems atlas, takes the current map you just built and scores real databases — Postgres, Cassandra, DynamoDB, Bigtable, Kafka, Redis, Elasticsearch, Spark, Flink — against the design-space axes, so a 2nd-edition term lands on a product you can place. Beyond this track, three neighboring tracks are built on the same substrate, and you can now read all three as consequences of what you learned here. The data-engineering track builds pipelines on a log it treats as a primitive — and you have opened that primitive: CDC, offsets, replay, event sourcing, durable execution, the real engines (Postgres as a system of record, Kafka as the integration backbone) seen by their guarantees rather than their brands. The ML-systems design track trains on a feature store and serves from a vector index as if they were given — and you now know they are derived views with lag budgets, online/offline skew, label-leakage hazards, and an ethics obligation, kept honest only by audit and recompute. The distributed training & inference track shards a single model across many GPUs — the same partitioning, replication, and collective-communication ideas you met here (lessons 08, 09, and 11), now applied to weights and activations instead of rows. Every product you meet from here — first edition or second — is a point in the design space the orientation lesson drew, and you can name its guarantees, its bill, and who owns its truth.

Where is truth?

Every 2nd-edition topic still answers the track's one question. System of record: the event log, in the patterns the new edition foregrounds — event sourcing makes the append-only log the truth, durable execution persists each step to a log, the relational/columnar store stays the source for vector search and lakehouse compute. Copies / derived views: CQRS read models, ANN vector indexes, materialized lakehouse tables, CRDT replicas on each device. Freshness budget: per view — a read model's replay lag, a vector index's re-embed lag when the model changes, a local-first replica's sync window. Owner: the team that owns the log and each derived consumer; for the ethics threads, also whoever is accountable for the decision the data drives (L25). Deletion path: tombstone in the log, replayed into every derived view, provable because lineage is explicit (L20). Reconciliation / repair: replay from offset 0 to rebuild any view; CRDT merge to converge replicas. Evidence it is correct: formal methods and deterministic simulation (TLA+, fault injection) verify the protocol; an audit of derived views catches skew before it becomes a fairness failure.

Takeaway

The first-edition arc this track taught — B-trees and LSM-trees, replication and quorums, partitioning, isolation, consensus, the durable log, and derived data — is the stable physics of data systems, and the Designing Data-Intensive Applications second edition (O'Reilly; 14 chapters, 672 pages, by Kleppmann and Riccomini; listed as February 2026 by O'Reilly and March 2026 on Kleppmann's site) does not retract it; it re-weights it for a cloud-native, ML-adjacent, ethically-aware world. The new topics are old mechanisms under louder pressure: storage–compute separation is columnar storage pulled apart (L04); GraphQL is access-pattern and schema-evolution design at the API edge (L02/L05); event sourcing and CQRS make the log the system of record with derived read models (L19/L20); vector search is a new ANN index and another derived view (L01/L04/L20); durable execution packages the ordered log and idempotency keys as a runtime (L17/L20); CRDTs bake conflict resolution into the data type (L09); multitenancy is partitioning by tenant (L11); and formal methods verify the partial-failure and consensus reasoning you could previously only argue (L15/L17). Above all, the second edition promotes ethics from an afterword to a chapter: privacy, surveillance, bias and accountability, and data as power are now engineering obligations — treated as a hard architecture input back in L25 — and the same lineage that lets you rebuild a derived view is what lets you honor deletion, minimize tracking, audit for skew, and name who controls the truth. You can now place any system — in either edition — by its guarantee and its bill, and now also by its responsibility, ready for the production atlas that opens the applied part.

Interview prompts

Is a "second edition" of a mechanisms book a rewrite? What actually changed? (§1 — no; the mechanisms are stable. What moved is the substrate (cloud, storage–compute split), new first-class workloads (vector/RAG, high-cardinality observability), graduated patterns (event sourcing, CQRS, durable execution), and ethics promoted to a chapter.)
What problem does GraphQL solve, and how is it an access-pattern and schema-evolution concern? (§2 — it lets the client request exactly the fields it needs, attacking REST's over-fetching and under-fetching/N+1 round trips; its schema is a wire contract that must evolve with back/forward compatibility, so it extends L02 and L05.)
How do event sourcing and CQRS relate to the derived-data capstone? (§2 — event sourcing makes the append-only event log the system of record; CQRS splits writes (commands to the log) from reads (derived views). It is L19/L20's log-as-backbone stated as a deliberate architecture, with read models rebuilt by replay.)
Where does vector search fit in the design space, and what does ANN trade? (§3 — it is a new index shape (beside the B-tree and inverted index, L01/L04) answering similarity queries, and a derived view of the source documents (L20). ANN trades exact recall for sub-linear lookup — the read/space/time amplification trade-off of L04.)
What is durable execution and which track mechanisms underpin it? (§3 — a runtime that persists each workflow step so it resumes after a crash instead of restarting; it is the durable ordered log (L03) plus end-to-end idempotency keys (L20), leaning on coordination/consensus (L17) for exactly-once steps.)
How are CRDTs different from ordinary multi-leader conflict handling? (§3 — CRDTs bake a commutative, associative, idempotent merge into the data type so any two replicas converge regardless of message order — conflict resolution by construction, versus last-write-wins or manual merge in L09.)
Why did the second edition promote ethics to its own chapter, and how does it connect to lineage? (§4 — data decisions now have societal consequences (privacy, surveillance, bias, data as power). The same log/lineage from L20 that rebuilds a derived view is what lets you honor deletion provably, minimize tracking, and audit derived views for skew — ethics is enforceable only if you know where data flowed.)