Part 11 · Correctness & society

Law, Regulation, and Ethics as Architecture Input

Lesson 24 pushed storage into the cloud and treated durability, replication, and tiering as a subsystem you rent rather than build. That was the last technical constraint. This lesson names the one that is not technical at all: law and ethics. They are not a closing reflection you bolt on after the architecture is done — they are an input, the same as throughput or latency, and they shape the schema, the retention policy, and the deletion path from day one. Lesson 20 built a system designed to remember everything forever — an immutable log you recompute every view from. This lesson is where "forever" collides with the right to be forgotten, and where we turn privacy, consent, and accountability into concrete mechanisms and a design-review checklist.

1 · Law and ethics are an architecture input, not a footnote

Every prior lesson treated the design space as a set of technical trade-offs: a quorum buys consistency at the cost of latency, a derived view buys a read pattern at the cost of freshness. Law and ethics add a different kind of constraint — one you cannot buy out of with more hardware. A regulator does not care that crypto-shredding is operationally inconvenient; "delete this person's data" is a requirement with the same standing as "p99 under 100 ms," and like that SLO it is far cheaper to design for than to retrofit.

The failure pattern is always the same: a team builds the system for correctness and scale, ships it, and only then asks "how do we delete a user?" — and discovers the answer is "we can't, cleanly," because the architecture was built to remember. The personal fact has, by then, been replicated to five followers, snapshotted into nightly backups, indexed into a search cluster, aggregated into analytics tables, and embedded into a vector index, and nobody wrote down where it all went. The fix at that point is archaeology. The fix at design time is a single column in the "Where is truth?" artifact: who owns this data, and what is its deletion path?

2 · Data minimization, consent, and lawful basis

The first principle is data minimization: collect and retain only the data a stated purpose actually needs. It is the privacy analogue of "don't add an index you have no query for." Each field you store is not free — it is a future deletion obligation, a future breach blast-radius, and a future audit liability. The discipline is to start from the purpose ("rank items this user is likely to click") and admit only the fields that purpose requires, rather than logging everything "because the log makes it cheap" (lesson 20's own warning) and mining it later.

This directly shapes the schema. A recommender needs interaction events; it probably does not need the user's precise home address, and if it does not, that column should not exist. Purpose limitation is the sibling rule: data collected for one purpose may not be silently repurposed. The training pipeline that quietly starts joining a support-ticket table into the ad-targeting features has changed the purpose, and under most regimes that is a new decision requiring a new basis — not a free optimization.

Consent and lawful basis govern what you may store and process at all. Under GDPR every processing activity needs a lawful basis — consent, contract, legitimate interest, legal obligation, and others — and consent specifically must be freely given, specific, and revocable. The engineering consequence is concrete: consent is itself data you must store, version, and join against at processing time. A feature pipeline that computes on a user's events must be able to ask, per user and per purpose, "do we have a basis to process this?" — and when consent is withdrawn, that withdrawal must propagate to the same derived views a deletion would (§3). Consent you cannot query is consent you cannot honor.

3 · The deletion path: where "remember forever" meets the right to be forgotten

This is the central collision of the lesson, and it falls squarely on lesson 20's architecture. There, the design was: write to the system of record, capture every change into an immutable ordered log, and derive every view by replaying that log — so any view is recomputable forever. That immutability is exactly what made the system correct, auditable, and repairable. It is also what makes GDPR Article 17, the right to erasure, hard: a user asks you to delete their data, and your architecture's defining feature is that it never forgets.

Deleting the system-of-record row is the easy 5%. The hard 95% is everything downstream, and the immutable log is the worst of it — you cannot just delete an entry from an append-only, offset-addressed log without breaking the replay guarantee every derived view depends on. There are four engineering answers, used together:

• Tombstones. Instead of physically removing a record, append a deletion marker — a tombstone keyed to the deleted entity. Replay and compaction (lesson 4's LSM idea) honor the tombstone: the entity is treated as absent. This keeps the log append-only while making the data logically gone, and a compaction pass later reclaims the physical bytes. The tombstone itself becomes the audit evidence that the delete happened.
• Crypto-shredding. The decisive trick for "I cannot reach every physical copy." Encrypt each user's data under a per-user key. To delete the user, you do not chase down every copy in every backup and log segment — you delete the one key. Every ciphertext copy, wherever it lives, instantly becomes unrecoverable noise. You turned a deletion that had to touch N copies into a deletion that touches one key. This is how you honor erasure against immutable logs and weeks-old backups without rewriting them.
• Bounded retention. The log does not have to be infinite. Lesson 20 wanted retention "long enough to replay any view you'd ever rebuild" — but that is a budget, not "forever." A 90-day or 1-year retention window means old personal data ages out automatically; deletion partly becomes "wait for expiry" rather than "hunt and destroy." Bounded retention is the cheapest privacy mechanism there is, and it is a one-line policy decision.
• Propagate the delete to every derived view. The same fan-out that lesson 20 used to keep views fresh is the fan-out that keeps them forgotten. The delete (or tombstone) flows down the ordered log to every consumer: the search index drops the document, the analytics job excludes the user from the next recompute, the cache evicts. Because each view is a recomputable function of the log (lesson 20, §1), and each tracks its offset, you can prove the delete reached it — re-run the view and audit that the entity is absent.

4 · Auditability and accountability: the same lineage, reused

Lesson 20 built lineage (knowing which source records and which code produced each derived value) and audit logs (an independent record you can recompute and compare against) to detect drift and repair views. Those exact tools are what make rights and accountability enforceable — they were never only about correctness.

• Lineage makes deletion and access provable. You cannot honor "delete everything about me" or "show me everything you have about me" if you do not know where a person's data flowed. The data-flow graph that let you rebuild a drifted view is the same graph that enumerates every place a delete must reach (§3). A view you cannot put on that graph is a view you cannot prove you deleted — which is the privacy version of lesson 20's "forgotten second source of truth."
• Audit logs make accountability concrete. When a model makes a decision that affects a person — a loan denied, content demoted — accountability means being able to answer "why?" The audit and lineage trail ("this decision used these features, computed from these events, under this model version") is the same machinery you built to debug skew, now answering an accountability question. Because the pipeline is auditable, "we couldn't have known" is rarely an honest defense.
• The audit log is itself personal data. A subtlety: the audit trail that proves you deleted a user may itself contain that user's identifiers. The usual answer is to log the fact and proof of deletion (entity ID hash, timestamp, scope) rather than the deleted content, and to give the audit log its own bounded retention.

5 · The GDPR/CCPA rights as operations your architecture must support

The data-protection rights are not abstractions — each is a concrete query or operation, and your architecture either can run it or cannot. Designing for them means making sure each is a supported access pattern (lesson 2's lesson, applied to law), not a heroic one-off SQL safari every time a request arrives.

Notice that three of the four rights are made easy or hard by the same two decisions you already made for correctness: a single system of record with a clear fan-out (rectification and erasure are just writes that propagate), and lineage that enumerates every store (access and erasure need to know where to look). Privacy-by-design is, to a large degree, the good distributed-data design you already wanted — keyed by subject, single source of truth, recomputable views, known lineage. The CCPA adds a distinct one — the right to opt out of sale/sharing — which, like consent withdrawal (§2), becomes a per-subject flag joined at processing time.

6 · Surveillance, feedback loops, and data as power

Law is the floor, not the ceiling. The harder problems are the ones no statute fully covers, and they arise precisely because the system works well. Keep this grounded: these are design consequences to weigh, not slogans to recite.

• Surveillance risk. The same enrichment that improves a recommender — joining behavioral streams across sources (lesson 19's stream joins) to build a rich per-person profile — is, at scale and aimed at people, surveillance. The capability is value-neutral; the decision to collect and retain a given signal is not. Data minimization (§2) is the direct lever: the signal you never collected cannot be turned into surveillance, leaked, or subpoenaed.
• Feedback loops. A predictive system that acts on its predictions shapes the very behavior it then measures. A recommender that surfaces an item makes it more likely to be clicked, which becomes a training label that makes it surface the item more — the model is now partly predicting its own past outputs, not user preference. Predictive-policing and content-ranking systems are the canonical cautionary cases. The data-system tell-tale: your training labels are contaminated by your own serving decisions (a cousin of lesson 20's label-leakage), and event-time lineage is what lets you even detect it.
• Bias amplification. A model trained on derived feature tables inherits whatever skew lives in the events; a feedback loop then amplifies it over successive retraining cycles. Because the pipeline is auditable (§4), this is detectable — and therefore your responsibility to look for.
• Data as power. A system that retains and acts on personal data concentrates influence over the people in it. The engineer who builds the log is not a neutral pipe: what you choose to log, how long you keep it, what you join, and what a derived decision is allowed to do are design choices with consequences. The point is not to moralize in code review; it is to recognize that the lag budgets, idempotency keys, and lineage you built for correctness are the same levers that honor deletion, limit retention, and keep the system accountable.

Checkpoint exercise

Where this points next

We have now closed the loop the orientation lesson opened: the "Where is truth?" artifact carries not just a system of record and a freshness budget but an owner, a deletion path, a retention clock, and audit evidence — privacy and accountability live in the same artifact as correctness because they are designed at the same time. The 2nd edition of DDIA promotes this material from a closing section to a chapter of its own, and the next lesson — 26 · Current DDIA: What the 2nd Edition Adds — is the signpost to it, alongside what else the new edition adds (consensus reframed, the cloud-native shift of lesson 24, and updated data-system landscape). It is where you take this track's vocabulary back to the source and see how the field itself moved.

Interview prompts

• Why call law and ethics an "architecture input" rather than a compliance step? (§1 — they constrain the schema, retention, and deletion path with the same standing as an SLO; retrofitting deletion onto a system built to remember is archaeology, while designing for it is a column in the source-of-truth artifact.)
• What is data minimization and how does it change the schema? (§2 — collect/retain only what a stated purpose needs; each field is a deletion obligation and breach blast-radius, so you start from the purpose and admit only the fields it requires, rather than "log everything because it's cheap.")
• Why does GDPR's right to erasure collide with lesson 20's design, and how do you resolve it? (§3 — lesson 20's defining feature is an immutable log you recompute every view from, i.e. a system built to never forget; resolve with tombstones, crypto-shredding the per-user key, bounded retention, and propagating the delete to every derived view.)
• What is crypto-shredding and what problem does it solve? (§3 — encrypt each user's data under a per-user key; to delete the user, delete the one key, and every ciphertext copy in every backup and log segment becomes unrecoverable at once — turning a deletion that had to reach N copies into one that touches one key, collapsing the worst-case window from the backup-retention period to seconds.)
• How do lineage and audit logs serve privacy, not just correctness? (§4 — lineage enumerates every store a person's data flowed to, which is exactly what access and erasure requests must reach; audit logs answer "why did this decision happen?" for accountability — the same machinery built to detect drift.)
• Name the four data-protection rights and what operation each maps to. (§5 — access = a query across every subject-keyed store; rectification = a write to the SoR that propagates via the log; erasure = the full deletion path; portability = the access query plus serialization to a standard format. Three of four are made easy by single-source-of-truth, keyed-by-subject, recomputable design.)
• What is a feedback loop and why is it a data-system problem? (§6 — a predictive system that acts on its predictions shapes the behavior it then measures; serving decisions contaminate the training labels (a cousin of label leakage), so the model predicts its own past outputs and amplifies bias each retraining cycle. Event-time lineage and audit are what let you detect it.)