Part 12 · Applied & interviews
Interview Case: Model Registry
Lesson 32 solved the dual-write trap: never write your source database and a derived search index in two separate steps — make one of them the log and let the other follow. This case applies that same discipline to a different artifact. A model registry is the system of record for trained ML models: the large immutable weights sitting in blob storage, the small critical metadata (version, metrics, lineage, stage) describing them, and — the part that bites in production — a single pointer that says which version is live right now. The crux is not storing files. It is making "promote version N to production" an atomic, linearizable flip that every serving replica agrees on, while keeping every artifact immutable so a rollback is just pointing back at a version that still exists, byte-for-byte.
New capability: design a registry that separates a strongly-consistent metadata store from a durable blob store, makes artifacts content-addressed and immutable, exposes one linearizable production pointer per model so the whole serving fleet flips atomically, records enough lineage to answer "what produced this model", and rolls back safely in seconds.
1 · The prompt and what to ask first
Prompt. "Design a model registry. It is the system of record for our trained ML models. Teams push model versions; the registry stores the weights and the metadata about each version; and there is a notion of which version is currently serving production traffic. Make promotion and rollback safe, and make sure the whole serving fleet agrees on the live version."
The weak candidate starts drawing an S3 bucket and a Postgres table. The strong candidate first separates the two halves of the problem — a small but critical metadata store and a large but simple artifact store — and then asks the questions that determine the consistency model.
The single most important clarification is the one about the production pointer's consistency. If the answer is "a few seconds of staleness is fine" you can serve the pointer from an eventually-consistent cache. If the answer is "every replica must agree the instant promotion returns" — the usual answer for production model serving — you need a linearizable register backed by consensus, and that decision shapes the whole design.
2 · Back-of-envelope sizing
Numbers turn vague requirements into a concrete design. Take a mid-size ML org.
Metadata. Each version's metadata row (version, metrics, lineage refs, stage, checksums) is a few KB. 20,000 × ~4 KB ≈ 80 MB. The entire metadata store fits in RAM many times over. This is the key asymmetry: 40 TB of artifacts, 80 MB of metadata. They do not belong in the same engine.
Production pointers. One small record per model — "model M → version V" — so 500 pointers, a few hundred bytes each. Tiny. This is the linearizable state, and it is essentially nothing in size.
Promotion frequency (write rate). Even a busy org promotes maybe 50 times a day across all models — under 0.001 writes/sec to the pointer set. Promotion is rare. Consensus is famously slow (a round-trip among a quorum, see L17), but at this write rate its cost is irrelevant.
Read QPS (the serving fleet). Here is the real load. Suppose 2,000 serving replicas across all models. If each replica polled "what is production for my model" every 5 s, that is 2,000 / 5 = 400 reads/sec on the pointer — modest, but it grows linearly with the fleet and adds up-to-5-second staleness. If instead each replica watches the pointer (a coordination service push on change), steady-state read load on the source of truth drops to near zero and a flip propagates in well under a second. The asymmetry — rare writes, fleet-scale reads, sub-second propagation wanted — points straight at a watched coordination register, not a polled database.
Artifact read bandwidth. A replica fetches weights only on cold start or a flip, not per request: 2,000 replicas each pulling a 2 GB artifact during a fleet-wide promotion is 4 TB of egress in a burst. That is a real number — it argues for caching artifacts on the serving hosts and staggering the roll, not a thundering-herd pull from the blob store the instant the pointer flips.
The sizing settles three design choices before any diagram: (a) artifacts and metadata get separate stores; (b) the production pointer is a tiny linearizable register with rare writes and is read via watches, not polling; (c) a flip triggers a bounded artifact-fetch burst that must be staggered.
3 · Design walk
Build it from the track's mechanisms. Four components, each with a job it is good at and a consistency need it must meet.
3.1 · Immutable, content-addressed artifacts (L03, L20)
Weights go in a blob/object store (S3, GCS) that gives durability and throughput but no transactions — and it does not need them. The discipline that makes the registry safe is immutability: a registered artifact's bytes never change. The cleanest way to enforce that is content addressing — name each artifact by the hash of its bytes (e.g. sha256:9f86d0...), exactly the append-only / immutable-log thinking from L03. The hash is the address, so the same bytes always land at the same key, a different model can never overwrite an existing one, and a download can be verified against its name. The version's metadata row stores that hash; "the weights for version V" is a content hash that resolves to bytes that cannot have drifted. This also gives free dedup (two versions with identical weights share storage) and makes lineage tamper-evident.
3.2 · Strongly-consistent metadata store (L13, L16)
Metadata — version number, metrics, the artifact hash, lineage refs, stage, checksums, who/when — lives in a small relational store (Postgres) that gives you transactions (L13) and is easy to query ("best F1 among production-eligible versions of model M"). Registering a version is one transaction: write the metadata row referencing the already-uploaded, content-addressed artifact. Crucially, register the artifact first (upload bytes, get the hash), then commit the metadata row that points at it. That ordering — durable artifact before the row that references it — means the metadata never points at bytes that do not exist, the registry's version of "no dual write" (L32): one store is the follower of the other, never two independent writes that can half-fail.
3.3 · The linearizable production pointer (L16, L17)
This is the heart of the case. "Which version is live for model M" is a single mutable value that the whole fleet must agree on. The non-negotiable property is linearizability (L16): the pointer behaves as one register, and the moment a promotion's write returns, every subsequent read — from any replica — sees the new version; there is never a window where two replicas serve different "production" versions because one read a stale copy. A normal asynchronously-replicated database (L08) does not give you this: a follower can lag and hand a serving host the old version after the flip "succeeded".
The right home for the pointer is a coordination service — etcd or ZooKeeper — which is a consensus system (L17) purpose-built for exactly this: a tiny amount of strongly-consistent state, linearizable writes via a quorum, atomic compare-and-set, and watches that push a notification to every interested client the instant the value changes. (Linearizable reads need a small care: etcd serves them through the Raft leader by default, while ZooKeeper reads can be served stale from a follower unless you issue a sync first — but the watch-driven flow below means replicas are pushed the new value rather than relying on a fresh read.) Promotion is a single linearizable compare-and-set: "set /models/M/production from version 41 to version 42, only if it is still 41." It succeeds atomically or fails — never half-applies. Watches mean the 2,000 replicas learn about the flip in sub-second time without polling (the sizing in §2). Because promotion is rare (§2), the cost of consensus is irrelevant; what you buy is that the fleet can never disagree about the live version.
Read the flow as: artifact bytes are uploaded and addressed by hash (1); the metadata row is committed referencing that hash (2 on the write side); promotion is a single linearizable compare-and-set on the coordination service (2 on the pointer side); every replica is watching and is pushed the new version (3); each replica then fetches the new weights by hash from the blob store, staggered to avoid a stampede (4). The only linearizable operation in the whole system is the pointer flip; everything else is immutable bytes or rarely-changing metadata.
3.4 · Audit, lineage, and rollback (L20)
Lineage answers "what produced this model" — which training-data snapshot, which code commit, which hyperparameters, which parent version. This is derived-data thinking (L20): a model is derived from data + code, and recording the inputs makes the derivation reproducible and the system auditable. Because artifacts are content-addressed (§3.1) and the data snapshot and code commit are themselves immutable references, the lineage is a tamper-evident chain — you can recompute the model from the recorded inputs and confirm you get the same hash.
Audit of promotions falls out almost for free: log every pointer change as an append-only record — (model, from_version, to_version, who, when). That log is the answer to "what was production on date D" and "who promoted the bad model", and it is itself an immutable log (L03). Rollback is then trivial and safe: because every prior version's artifact is immutable and still present (never overwritten, never mutated), rolling back is just another compare-and-set flipping the pointer back to a version that still exists byte-for-byte. There is no "restore from backup" and no risk that the old weights drifted — immutability is what makes rollback a sub-second pointer flip rather than a recovery operation.
| Component | Job | Consistency need |
|---|---|---|
| Artifact / blob store | Hold large immutable weights, durably, at throughput | Durability + immutability (content-addressed); no transactions needed |
| Metadata store | Versions, metrics, lineage, stage; rich queries | Strong consistency / transactions; tiny data, registered after its artifact |
| Production pointer | The one live version per model the fleet reads | Linearizable, consensus-backed, atomic compare-and-set + watches |
| Audit / lineage log | Who promoted what; what produced each model | Append-only / immutable; derived-data provenance |
4 · Failure timeline — split-brain promotion
The failure that separates a good design from a fragile one: two promotions racing, or a stale-read replica, producing a fleet that disagrees about the live version.
Root cause of the broken version: the pointer was treated as ordinary replicated data, so concurrent writes did last-writer-wins (a lost update) and lagging followers served stale reads. Mitigation: the pointer is a linearizable register with atomic compare-and-set on a coordination service — concurrent promotions are serialized (one wins, the other is told it lost and must retry against the new value), reads cannot go stale after a successful write, and watches push the single agreed value to the fleet. The conditional compare-and-set is the same fencing idea from L15: an action that must not stomp a newer state checks the version it expects before it commits.
5 · Answer rubric
Good answer (baseline)
- Separates the large artifact store (blob, durable, throughput) from the small metadata store (strong consistency, queryable) — does not jam both into one engine.
- Makes artifacts immutable and versioned; never overwrites a registered model's bytes.
- Has an explicit production pointer per model and a promote operation distinct from registering a version.
- Supports rollback by re-pointing at a prior version, and records who promoted what.
Better answer (senior signal)
- Names linearizability explicitly and puts the pointer in a consensus-backed coordination service (etcd/ZooKeeper), with atomic compare-and-set for promotion so concurrent flips can't lose updates.
- Uses watches to push flips to the fleet sub-second and justifies it against polling with the read-QPS sizing.
- Makes artifacts content-addressed (hash = name) for immutability, verification, and dedup; registers the artifact before the metadata row (no dual write).
- Treats lineage as derived-data provenance (data snapshot + code commit + parent) and the promotion log as an append-only audit; rollback is a sub-second flip because prior bytes are immutable.
- Quantifies: 40 TB artifacts vs 80 MB metadata; rare writes vs fleet-scale reads; staggers the post-flip artifact-fetch burst.
Red flags
- Storing weights as a
BLOBcolumn in the metadata DB, or putting tiny critical metadata in the blob store — wrong store for each. - Mutable artifacts ("just re-upload to the same key") — destroys reproducibility and makes rollback unsafe.
- Putting the production pointer in an async-replicated database and assuming reads are fresh — invites the split-brain timeline.
- A non-atomic promote (delete-then-write, or two separate writes) that can leave the pointer in a half-applied state.
- Polling the pointer at high frequency from thousands of replicas with no watch mechanism, or fetching artifacts in a thundering herd on flip.
- No lineage / no audit log — cannot answer "what produced this" or "who promoted the bad model".
Likely follow-ups
- "How do you do a canary / gradual rollout instead of an instant flip?" (pointer holds a weighted map of versions, or per-shard pointers; still linearizable per key.)
- "What if the coordination service is unavailable?" (reads served from each replica's last-known watched value — stay serving the current version; promotions block, which is the safe failure for a CP system.)
- "How do you stop a 2,000-replica fetch burst on flip?" (cache artifacts on hosts, pre-stage the new version before flipping, stagger/rate-limit the roll.)
- "How do you guarantee a model is reproducible from lineage?" (content-addressed data snapshot + pinned code commit + recorded hyperparameters; recompute and compare hashes.)
- "How do you handle multi-region serving?" (per-region pointer that follows a global one, or a globally linearizable pointer with the latency cost made explicit.)
Checkpoint exercise
Where this points next
The registry is a system of record with a tiny linearizable core and a large immutable periphery — the consistency burden concentrated in one pointer. The next case inverts that profile. A metrics / observability dashboard has almost no strong-consistency requirement but enormous write fan-in: millions of high-cardinality time-series samples per second pouring in from the same serving fleet, stored columnar and rolled up, queried fresh-vs-historical with approximate aggregates. Lesson 34 builds that ingest-and-query path — where the hard problems are write throughput, downsampling, and hot-recent-vs-cold-historical storage rather than a single atomic flip.
Interview prompts
- Why split artifacts and metadata into two stores? (§1, §2 — 40 TB of large immutable weights wants a durable high-throughput blob store with no transactions; 80 MB of critical metadata wants a strongly-consistent queryable engine. Opposite needs; one engine serves neither well.)
- What property must the production pointer have, and where does it live? (§3.3 — linearizability: the moment a promotion returns, every replica's read sees the new version. Put it in a consensus-backed coordination service (etcd/ZooKeeper), not an async-replicated DB whose followers lag.)
- How is promotion made atomic against concurrent flips? (§3.3, §4 — a single compare-and-set "set M from 41 to 42 only if still 41"; concurrent promotions are serialized, the loser is told it lost and retries — no lost update, no half-applied state.)
- Why content-address the artifacts? (§3.1 — naming by hash enforces immutability (same bytes, same key; no overwrite), gives download verification and free dedup, and makes lineage tamper-evident.)
- How do 2,000 replicas learn about a flip without hammering the source of truth? (§2, §3.3 — watches: the coordination service pushes the change to every watching client sub-second, so steady-state read load is near zero versus polling, which scales with the fleet and adds staleness.)
- Why is rollback safe and fast here? (§3.4 — artifacts are immutable, so every prior version still exists byte-for-byte; rollback is one compare-and-set back to it, not a restore — no risk the old weights drifted.)
- What does lineage record and why? (§3.4 — the training-data snapshot, code commit, hyperparameters, and parent version; it is derived-data provenance that makes a model reproducible and the registry auditable.)
- What happens if the coordination service goes down? (§5 — replicas keep serving their last watched version (reads from local cache); promotions block. Blocking writes while keeping reads is the safe failure mode for a CP coordination layer.)