Part 9 · Derived views

Derived Data: Keeping Views Correct

Lesson 19 gave us the change log as a live wire: write to the system of record, capture every change with CDC, and feed it forward. Lesson 18 gave us the other half — recompute a whole view from scratch when the live wire is not enough. This lesson opens Part 9 · Derived views by joining them around one question: a real product is never one database, it is a dozen specialized stores that must agree. So which one is the truth, how do the rest stay in sync with it, and — when something inevitably drifts — how do you prove they agree and repair them when they don't? This is keeping derived views correct: every mechanism you learned was a contract bought at a cost, and here we assemble them into one composed system whose correctness holds end to end.

DDIA source

Arc drawn from Martin Kleppmann, Designing Data-Intensive Applications, Chapter 12 (The Future of Data Systems) — unbundling the database, derived data and the log as integration backbone, the lambda/kappa debate, and the end-to-end argument applied to correctness. Original synthesis; the worked ML-product example and ASCII derivation map are ours.

Linear position

Prerequisite: the whole spine so far. You need the durable ordered log (03), replication lag and read-your-writes (08), quorums (09), the shard key (11), transactions and isolation (13), unreliable clocks and fencing (15), linearizability (16), total-order broadcast and consensus (17), batch recompute (18), and stream/CDC processing (19). This lesson does not introduce a new primitive; it composes them.
New capability: design a composed data system — pick the system of record, derive every other store from a single ordered log, give each derived view a lag budget and a rebuild path, and make the whole thing auditable and repairable so end-to-end correctness is a property you can demonstrate, not a hope.

The plan

Six moves. (1) The thesis: no single database is right, so real systems compose a system of record plus derived views — and the defining property of a derived view is that it is recomputable. (2) Keeping them in sync: the log as the integration backbone, and why dual writes diverge while a single ordered log does not. (3) Lambda vs Kappa — batch layer plus speed layer, versus one replayable log reprocessed — unified by the idea that batch and stream are the same computation over bounded vs unbounded input. (4) End-to-end correctness: per-hop exactly-once is not enough; you need end-to-end idempotency keys, audit, and repair-by-recompute. (5) A worked ML-product example that wires the spine so far together and shows what breaks when a feature definition changes. (6) Failure modes, a checklist, and the hand-off into the rest of Part 9.

1 · The thesis: compose specialized stores, derive all but one

The whole track has resisted the question "which database should I use?" because it is the wrong question. No single store is best at every access pattern: a relational database gives clean transactions (lesson 13) but cannot serve full-text search; a B-tree (lesson 03) is great for point lookups but a columnar store (lesson 04) crushes it on analytic scans; a vector index answers "what is similar?" but cannot enforce a uniqueness constraint. So real systems do not choose — they compose. One store is the system of record (the authoritative copy of the facts — the one you would rebuild everything else from after a disaster), and every other store is a derived view: a copy reshaped for one read pattern.

System of record

The source of truth. A transactional row store, or an append-only event log treated as primary (event sourcing, lesson 19). Writes land here first; its commit defines "what happened." Everything else is downstream.

Cache

A derived view holding hot keys in memory for cheap reads (lesson 03 idea, promoted). Recomputed trivially: drop it and refill on miss.

Search / vector index

A derived view shaped for full-text or similarity queries. Rebuildable by re-indexing every record from the source — the RAG embedding index is exactly this.

Materialized view

A query result stored rather than computed on read — a precomputed aggregate or join (lesson 18). The canonical derived view: pure function of the source, refreshed on change.

Feature store

Precomputed model inputs, online (low-latency serving) and offline (training tables). A derived view whose freshness and event-time correctness are model-critical.

Analytics / lakehouse

Columnar tables for aggregation and training-data extraction (lesson 04). A derived view recomputed by batch jobs (lesson 18).

The word doing all the work is derived: a derived view is, by definition, a deterministic function of the system of record. That single property is what makes the architecture safe — because you can recompute it, you can throw it away and rebuild it. A search index that drifts is not a crisis; you re-index. A feature value computed under a buggy definition is not lost data; you recompute it from the events. The orientation lesson stated the iron rule and it is the spine of this one: any view you cannot recompute from the log is not a derived view — it is a second source of truth you forgot you had, and when it disagrees with the real one there is no clean answer to "what is true?"

The one question to ask of every store in your diagram

For each box, answer: is this the system of record, or is it derived? If derived, name (a) its source, (b) its update path, (c) its lag budget (how stale is allowed), and (d) its rebuild path. A store that cannot answer (d) has secretly become authoritative. Most production incidents in composed systems trace to a box whose owner could not answer (d).

2 · The log is the integration backbone — dual writes diverge, one log does not

Granted you have one source of truth and several derived views, how do changes get from the source to the views? There are two designs, and only one of them survives contact with failure.

The wrong way: dual writes. A dual write is application code that, on each change, writes to several stores itself — "update the database, then update the search index, then update the cache." It is the obvious thing and it is broken in two distinct ways:

DUAL WRITES (application fans out the write itself) app ──write A──▶ database ✓ committed ──write B──▶ search index ✗ times out / process crashes here ──write C──▶ cache never runs result: database says X, search index says (old) Y, cache empty. no ordering between A,B,C across concurrent requests either: req1: db ← "Sam" index ← "Samantha" (writes interleaved) req2: db ← "Samantha" index ← "Sam" db ends "Samantha", index ends "Sam" → silently divergent forever

The two failures: (1) partial failure (lesson 15) — the process can crash, or a write can time out, between the two writes, leaving the stores permanently inconsistent with no record of the gap; and (2) race conditions — two concurrent requests can apply their writes to the two stores in different orders, so the stores converge to disagreeing values even though no write was lost. There is no single agreed order, which is exactly the concurrency problem from lessons 09 and 17, now spread across different systems where you cannot run a transaction.

The right way: write once, capture the change, fan it out from a single ordered log. Write only to the system of record. Capture its committed changes — via change data capture (CDC), tailing the database's own write-ahead log (the durable ordered log from lesson 03, exposed as a stream, lesson 19) — into one totally ordered log. Every derived view is a consumer of that one log, applying changes in the same order.

SINGLE ORDERED LOG (the integration backbone) WRITE (one place only) │ ▼ ┌───────────────────────┐ │ SYSTEM OF RECORD │ the commit defines truth & order └───────────────────────┘ │ CDC: tail the WAL → one ordered change log ▼ ╔═══════════════════════╗ ║ ORDERED CHANGE LOG ║ offset 0,1,2,3,… (total order) ╚═══════════════════════╝ ┌───────────┬───────────┬───────────┬──────────────┐ ▼ ▼ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌────────────┐ │ cache │ │ search/ │ │ feature │ │analytics │ │ training │ │ │ │ vector │ │ store │ │ columnar │ │ tables │ └─────────┘ └─────────┘ └──────────┘ └──────────┘ └────────────┘ each consumer tracks its own OFFSET → it has a measurable LAG, and can be REBUILT by resetting its offset to 0 and replaying.

This single move dissolves both dual-write failures. There is now one order (the log's offsets — the total-order broadcast of lesson 17, made durable), so no two views can disagree about sequence. And a consumer that crashes has not lost anything: it resumes from its last committed offset and catches up. The log decouples the source from the views in time — a view can be down for an hour and recover — which is the same async-derived-view bargain the orientation lesson named: you buy decoupling and per-view freshness, and the bill lands on freshness (views are eventually consistent with the source) and on the operational care of running the log.

Worked number — measuring the lag you bought

A profile update commits to the source at t = 0. CDC reads the WAL and appends to the log within 20 ms. The feature-store consumer polls the log every 100 ms and takes 30 ms to apply a batch. Worst-case staleness for the online feature = 20 + 100 + 30 = 150 ms. That is your lag budget made concrete: a re-rank that fires within 150 ms of a profile edit may score on the old feature value. If the ranker's read-your-writes requirement is tighter than 150 ms, you either route that one read to the source (lesson 08) or shrink the poll interval — and either way you have a number to defend, not a vibe. Compare a dual write, whose failure mode has no number at all: it is simply "diverged, unknown when, unknown by how much."

3 · Lambda vs Kappa: two ways to reconcile batch and stream

The log gives you fresh, incremental updates. But streams have a weakness: a bug in the stream code, or a brand-new derived view, needs the full history reprocessed, not just new events. The two famous architectures differ in how they supply that full-history recompute.

Lambda architecture runs two layers in parallel. A batch layer (lesson 18) periodically recomputes each derived view from scratch over all historical data — slow, high-latency, but complete and correct, and trivially re-runnable after a bug. A speed layer (lesson 19, stream processing) computes incremental updates over just-arrived events for freshness. Reads merge the two: "the batch view as of last night, plus today's streamed deltas." It works, but you maintain the same logic twice in two engines, and merging the two views at read time is fiddly and error-prone.

Kappa architecture collapses the two into one. Keep a single replayable log with long enough retention to hold the history you'd ever reprocess. To get a fresh view, stream over the tail; to fix a bug or add a view, reset the consumer's offset to 0 and reprocess the whole log through the same code. One codebase, one engine, batch-as-replay. The cost is log storage (you retain history you might replay) and a reprocessing job that can chew through the full log fast enough.

The unifying idea

Batch and stream are the same computation over different input bounds. A batch job is a stream computation over a bounded input (the file ends); a stream job is a batch computation over an unbounded input (the file never ends — lesson 19's "batch without an end-of-file"). Kappa takes this literally: there is only the log, and "batch" is just "replay from offset 0," "stream" is just "keep reading past the tail." Once you see batch and stream as one operator over bounded vs unbounded input, Lambda's two layers look like an accident of immature tooling rather than a law.

Choice	Buys (contract)	Costs (the bill)
Lambda (batch + speed)	Complete correct batch view + fresh streamed deltas; recompute path is the batch layer you already trust	Logic maintained twice in two engines; read-time merge of the two views is complex and a bug surface
Kappa (one replayable log)	One codebase and engine; recompute = replay from offset 0; "batch" and "stream" are the same job	Log retention storage for full history; reprocessing must be fast enough to backfill from the start

4 · End-to-end correctness: per-hop exactly-once is not enough

Now the deepest point, and the one interviews probe. Each hop in the pipeline can be made reliable on its own: lesson 19's stream processor offers exactly-once effects via idempotent writes and offset-with-output transactions. So if every hop is exactly-once, is the whole pipeline correct end to end? No — and the reason is the end-to-end argument: a guarantee that must hold for the whole system generally cannot be assembled purely from guarantees at the lower layers; it must be enforced (or at least checked) at the endpoints that actually care.

Concretely: a client submits "charge $20." The request times out (lesson 15 — you cannot tell "didn't happen" from "happened, ack lost"). The client retries. Now two identical events enter the log. Every downstream hop processes each one exactly once — and the user is charged twice. Per-hop exactly-once was honored perfectly; the duplicate was created above the pipeline, at the client. The fix lives at the endpoint: the client attaches an idempotency key — a unique ID for the intent, not the attempt — and the system of record dedups on it, so the second arrival is recognized as the same operation and collapses to one effect. The log's total order (lesson 17) then carries that single deduped event to every derived view identically.

Three layers of dedup, and you need all three

(1) End-to-end idempotency keys at the source kill duplicate intents (double-submit, client retry). (2) Per-hop exactly-once (idempotent applies keyed by log offset) kills reprocessing duplicates when a consumer replays. (3) Audit + repair catches what slips through (1) and (2): drift from bugs, schema changes, and operator error. Skipping the end-to-end key is the classic mistake — teams harden every hop, feel safe, and still double-charge because the duplicate was born before the first hop.

The second pillar of end-to-end correctness is that asynchrony makes derived views eventually consistent with the source — and the right response is not to hide that, but to make it measurable and recoverable. Two practices:

Audit. Periodically recompute a check from the source and compare to the derived view: counts, checksums, or spot recomputation of a sampled key. "The source has 4,001,233 active users; the search index reports 4,001,180 — 53 missing." Now drift is a detected number, not a customer complaint. Because each view tracks its offset, you can also alert on lag directly: "feature store is 9,000 offsets / 4 minutes behind."
Repair by recompute. Because the view is a function of the log, the fix is mechanical: reset the consumer's offset and replay (Kappa), or re-run the batch job (Lambda). No surgical manual patching of the broken view — you regenerate it. This is why "recomputable" in §1 was not a nicety; it is the entire recovery story.

Notice the through-line back to lesson 16: a composed async system does not offer linearizability across stores — a read of the search index right after a write to the source can miss it. You do not pretend otherwise. You publish the lag budget (§2's 150 ms), route the few reads that truly need freshness to the source (read-your-writes, lesson 08), and make everything else auditable and repairable. Make the eventual consistency explicit and recoverable, not hidden.

5 · Worked example: an ML product, end to end

Let us wire the spine so far into one system. The product ranks items for users; it needs online features for serving, a vector index for retrieval, and training tables for the model.

user/item EVENTS (clicks, views, purchases, profile edits) │ write to ONE place ▼ ┌───────────────────────┐ │ SYSTEM OF RECORD │ event store / OLTP DB (lessons 03,13) │ • dedup on idempotency-key (§4) └───────────────────────┘ │ CDC tail WAL → ordered log (lessons 17,19) ▼ ╔═══════════════════════╗ ║ ORDERED EVENT LOG ║ partitioned by user-id (shard key, lesson 11) ╚═══════════════════════╝ ┌───────────────┬──────────────────┬─────────────────────┐ ▼ ▼ ▼ ▼ ┌───────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ ONLINE │ │ VECTOR / │ │ ANALYTICS / │ │ OFFLINE FEATURE │ │ FEATURE │ │ SEARCH │ │ COLUMNAR │ │ / TRAINING │ │ STORE │ │ INDEX │ │ (lesson 04) │ │ TABLES │ └───────────┘ └──────────────┘ └──────────────┘ └──────────────────┘ serves ranker RAG retrieval aggregates model training set (lag ≤ 150ms) (re-index on (batch, lesson18) (event-time correct!) embed change) each view: source ✓ update-path ✓ lag-budget ✓ rebuild-path ✓ (§1)

Every piece is a lesson: the source of truth is a transactional store (13) whose WAL is the durable log (03); CDC turns its commits into a totally ordered stream (17, 19) partitioned by user-id, the shard key (11); replicas behind the source serve read-your-writes for the rare fresh read (08); the analytics/training tables are columnar (04) and batch-recomputed (18); the online and offline feature stores are derived views with explicit lag budgets; cross-store reads are eventually consistent, never linearizable (16).

The scenario that tests the design: a feature definition changes. Someone redefines ctr_7d from "clicks/impressions over 7 calendar days" to "over the last 7 days with active sessions." Every stored value of that feature is now computed under the old definition — in the online store, in the offline training tables, everywhere. Because each is a derived view of the event log, the repair is not a frantic manual backfill:

Recompute from the log. Deploy the new feature code, reset its consumer to offset 0, and replay the entire event history (Kappa). The view rebuilds itself under the new definition. The events never changed — only the function over them did.
Detect drift you didn't intend. Audit catches the dangerous failure mode unique to ML: online/offline skew. If the online store and the offline training tables compute the feature with even slightly different code, the model trains on values it will never see at serving time. Audit recomputes a sample of online feature values from the same log the offline job used and compares; a mismatch is a detected number before it becomes a silent accuracy regression.
Respect event time. When rebuilding training tables, the feature for a labeled example must be computed from events as of the label's timestamp — never from later events (lesson 19's event time vs processing time). Replaying the log in offset order makes this enforceable: you can reconstruct exactly what was known at any point. This is label leakage prevention, and it is only possible because you kept the ordered history.

The same lineage that lets you recompute also satisfies obligations the orientation lesson hinted at: deletion (a user's delete event propagates to every derived view, and you can prove it did because each view is replayable and auditable), and provenance ("why did the user see this item? which document version and embedding model produced this vector?"). You cannot honor deletion, retention, or audit if you do not know where data flowed — so the log that debugs your drift is the same log that makes the system governable.

Timeliness vs integrity

End-to-end correctness has two halves, and conflating them is a common engineering mistake. Integrity means the data is not corrupted, duplicated, or lost — every committed fact is present exactly once and unmangled. Timeliness means the data is recent — a read reflects writes from a moment ago. They are not the same guarantee and they do not carry the same weight. Integrity is usually non-negotiable: a double-charged payment or a dropped order is a correctness violation no amount of freshness excuses, and it is exactly what the end-to-end idempotency keys, total-order log, and audit of §4 enforce. Timeliness is usually relaxable: a search index that is 150 ms — or even a few seconds — behind the source (§2) is almost always fine, which is precisely why the eventually-consistent derived-view architecture works at all. The discipline is to decide, per view, which writes need integrity (always) and how much staleness is acceptable (a budget), and never to silently trade integrity away to buy freshness.

A system that derives, retains, and replays personal data is not only a correctness problem — the same ordered log that lets you recompute also makes the system a long-lived memory of people's behavior, where "recomputable forever" sits in direct tension with deletion and retention limits. That is a real obligation, but it is no longer this lesson's job to develop it: lesson 25 (Law, Regulation, and Ethics as Architecture Input) is the dedicated treatment of privacy, tracking and surveillance, bias and accountability, and data as power. Note here only that the lag budgets, idempotency keys, and lineage you built for correctness are the same levers you will use there to honor deletion, limit retention, and stay accountable.

6 · Where this breaks, and a checklist

Failure modes

Dual writes. App fans the write out to several stores itself; a crash between writes or a concurrent reorder leaves stores permanently divergent with no record of the gap (§2). Symptom: search index and DB disagree, nobody knows since when.
The forgotten second source of truth. A view nobody can rebuild from the log becomes authoritative by accident; when it drifts there is no clean answer to "what is true?" (§1).
Hardened hops, no end-to-end key. Every hop is exactly-once but the client double-submitted; the duplicate was born above the pipeline and ships as two effects (§4). Symptom: double charges despite "exactly-once everywhere."
Hidden staleness. Async lag is real but unpublished, so a read-your-writes path silently serves stale data and no one has a number to reason about it (§2, §4).
Online/offline skew. Online and offline feature code diverge; the model trains on values it never sees at serving time. Silent accuracy loss, invisible without audit (§5).
Label leakage. Training features computed from events after the label's time; offline metrics look great, production collapses (§5).
Lambda merge bugs. The read-time reconciliation of batch + speed layers is subtly wrong, or the two layers' logic drifts apart (§3).

Design checklist

Which single store is the system of record? Every other store: name its source, update path, lag budget, and rebuild path (§1).
Are changes fanned out from one ordered log (CDC), never via dual writes (§2)?
What is each derived view's lag budget, and which reads (if any) need the source for read-your-writes (§2, lesson 08)?
Lambda or Kappa? If Lambda, who owns the read-time merge? If Kappa, is retention long enough and reprocessing fast enough to backfill from offset 0 (§3)?
Is there an end-to-end idempotency key at the source, not just per-hop exactly-once (§4)?
Is there an audit (counts/checksums/sampled recompute) that turns drift into a detected number, and a one-command repair-by-recompute (§4)?
For ML: is online/offline skew audited, and is event-time enforced to prevent label leakage (§5)?
Does deletion propagate to every derived view, provably (lineage) (§5)?

Checkpoint exercise

Try it

Design the data backbone for a RAG assistant over a corporate document store. (a) Name the system of record and list every derived view (chunk store, embedding/vector index, full-text index, usage analytics), and for each give source, update path, lag budget, and rebuild path. (b) A document is deleted — trace exactly how the deletion reaches the vector index, and state the worst-case window during which retrieval could still return the deleted chunk; defend the number. (c) You change the embedding model — which views must be rebuilt, and do you use Lambda or Kappa to do it without taking the assistant offline? (d) A user double-clicks "ingest this file" — where does the duplicate die, and what would happen if you relied only on per-hop exactly-once? (e) Propose one audit that would catch the vector index silently missing 0.1% of chunks.

Where this points next

This opens Part 9 · Derived views: we have established the discipline — one system of record, every other store a recomputable view fed from a single ordered log, made fresh within a budget and provably repairable. The rest of Part 9 builds the specific views that discipline governs. Next, lesson 21 · Analytics and Query-Execution Internals opens the columnar/OLAP derived view and shows how a query engine actually executes the aggregations and joins that feed dashboards and training-data extraction. Part 9 then continues with the other two big derived views: lesson 22 · Search, Vector Indexes, and RAG (the search/similarity view, kept correct without dual writes) and lesson 23 · Feature Stores and ML Data Pipelines (the online/offline feature view, where event-time correctness and online/offline skew are model-critical). Every one of them is a point in the design space this lesson framed: name its source, its bill, and how to rebuild it.

Where is truth?

System of record: the one authoritative store whose commit defines "what happened" — a transactional row store or an event-sourced log; everything else is downstream. Copies / derived views: cache, search/vector index, materialized view, feature store, analytics/training tables — each a recomputable function of the source. Freshness budget: a published per-view lag (e.g. the worked 150 ms: 20 ms CDC + 100 ms poll + 30 ms apply); reads that need it tighter are routed to the source for read-your-writes. Owner: for each box, an owner who can name its source, update path, lag budget, and rebuild path — the box that cannot answer "rebuild path" has silently become a second source of truth. Deletion path: a delete event flows through the one ordered log to every view, provably, via lineage. Reconciliation / repair: repair-by-recompute — reset the consumer's offset and replay (Kappa) or re-run the batch job (Lambda); never patch a view by hand. Evidence it is correct: end-to-end idempotency keys at the source, per-hop exactly-once, and an audit (counts / checksums / sampled recompute) that turns drift into a detected number rather than a customer complaint.

Takeaway

No single database is right, so real systems compose: one system of record holds the truth and every other store — cache, search/vector index, materialized view, feature store, analytics tables — is a derived view, defined by the property that it is recomputable from the source. Keep them in sync not by dual writes (which diverge under partial failure and concurrent reordering) but by writing once and fanning every change out from a single ordered log (CDC + total order, lessons 17/19), giving each view a measurable lag and a rebuild path. Lambda (batch + speed layers) and Kappa (one replayable log) are two ways to supply the full-history recompute, unified by the fact that batch and stream are the same computation over bounded vs unbounded input. End-to-end correctness is not the sum of per-hop exactly-once: duplicates born above the pipeline need an end-to-end idempotency key at the source, and because asynchrony makes views eventually consistent, you make that explicit and recoverable with audit (drift becomes a detected number) and repair-by-recompute (replay the log). This is the discipline that opens Part 9 — every mechanism was a contract bought at a cost, and a correct composed system is one where, for every store, you can name its source, its bill, and how to rebuild it. Part 9 now builds the specific derived views (analytics, search/vector, feature stores) on top of it.

Interview prompts

Why don't real systems just pick one database, and what is the defining property of a derived view? (§1 — no store is best at every access pattern, so you compose a system of record plus derived views; a derived view is a deterministic, recomputable function of the source, which is what makes it safe to rebuild.)
What is wrong with dual writes, and what replaces them? (§2 — partial failure between writes and concurrent reordering leave stores permanently divergent with no record; replace with writing once to the source and fanning changes out from one ordered log via CDC, so there is a single order and crashed consumers resume by offset.)
Contrast Lambda and Kappa, and state the idea that unifies them. (§3 — Lambda runs a batch layer and a speed layer and merges them (logic twice, fiddly merge); Kappa keeps one replayable log and reprocesses from offset 0 (one engine, storage cost). Unified: batch and stream are the same computation over bounded vs unbounded input.)
If every hop is exactly-once, is the pipeline correct end to end? (§4 — no; a duplicate created above the pipeline (client retry on an ambiguous timeout) ships as two effects despite per-hop exactly-once. The end-to-end argument says the dedup must live at the endpoint: an idempotency key on the intent, deduped at the source.)
How do you handle that derived views are eventually consistent with the source? (§4 — don't hide it: publish a lag budget, route the few reads needing freshness to the source (read-your-writes, lesson 08), and make drift measurable with audit and recoverable with repair-by-recompute.)
A feature definition changes after a year of data. How do you fix every stored value? (§5 — deploy the new code and replay the event log from offset 0 (Kappa) or re-run the batch job (Lambda); the events are unchanged, only the function over them, so the view rebuilds itself. Audit online vs offline to catch skew.)
Why is keeping the ordered event log essential for ML training correctness? (§5 — replaying in offset order lets you compute each example's features as of the label's timestamp (event time), preventing label leakage, and lets you audit online/offline skew against the same history; lineage also makes deletion and provenance provable.)