Part 12 · Applied & interviews

Interview Case: Global, Multi-Region User Profile Store

Lesson 29 was a full case too — the social home timeline — and its hard problem was fan-out: one write touching millions of followers. This case keeps the senior-interview shape (clarify, size, design, break, grade) but swaps the pressure. The profile store is read-dominated and the writes are tiny, so fan-out is a non-issue. The pressure here is geography: users are spread across continents, every read must be fast locally, yet a profile edited in Frankfurt must eventually look the same when read in São Paulo — and when a user edits their own profile they must see the change immediately. That is the multi-region consistency problem, and it forces the CAP and PACELC trade-offs into the open: during a network partition you must choose stale-but-available or correct-but-unavailable, and even with no partition, a strongly consistent read still has to pay the cross-region round trip. This case makes every one of those choices explicit and quantitative.

DDIA source

Synthesizes the multi-region threads from Martin Kleppmann, Designing Data-Intensive Applications — leader-based replication and read-your-writes (Ch.5), multi-leader and leaderless conflict resolution / last-write-wins / version vectors (Ch.5), partitioning (Ch.6), and the consistency spectrum and linearizability cost (Ch.9). The CAP / PACELC framing and the profile-store sizing are our own synthesis; the conclusions are faithful to the book.

Linear position

Prerequisite: Replication and lag / read-your-writes (lesson 08); multi-leader and leaderless conflict handling, last-write-wins vs version vectors (lesson 09); partitioning a 2-billion-row dataset (lesson 11); the consistency spectrum and what linearizability costs (lesson 16).
New capability: Walk a multi-region store from prompt to graded answer — size it with real numbers, make the per-region replica + partition topology concrete, state the CAP behavior during a partition and the PACELC latency cost when there is none, and defend read-your-writes and concurrent-edit handling against an interviewer's follow-ups.

How to read this case

Six numbered sections, the shared case template. (1) The prompt, the requirements, and the clarifying questions a strong candidate opens with. (2) Back-of-envelope sizing — 2 billion profiles, the read/write split, storage, and per-region p99 budget. (3) The design walk, building the store from this track's mechanisms with an ASCII multi-region diagram. (4) One concrete failure timeline — a cross-region partition during a self-edit. (5) The answer rubric: Good / Better / Red flags / Likely follow-ups. (6) Takeaway plus interview prompts with answers. No widget — geography is taught by a diagram and arithmetic, not a slider.

1 · The prompt, requirements, and first questions

The prompt. "Design the user-profile store for a global product with about 2 billion accounts. A profile is the small bag of fields every other service needs: display name, handle, avatar URL, bio, locale, a few preference flags, privacy settings. Almost every page load and almost every backend service reads a profile; users edit their own occasionally. Users live on every continent and expect the app to feel instant wherever they are. Make reads fast everywhere and keep the data coherent."

That last sentence is the whole interview. "Fast everywhere" and "coherent" pull in opposite directions the moment your users span continents, and the candidate's job is to surface that tension rather than hand-wave it.

Functional requirements

Read a profile by user id (the dominant path, called from everywhere). Write/update your own profile. Read your own profile right after you edit it and see the edit (read-your-writes). Profiles are independent — there are no cross-profile transactions, no joins, no "transfer between two accounts".

Non-functional requirements

Low-latency local reads on every continent (single-digit-to-low-tens of milliseconds at the store). Very high read availability — a profile read should almost never fail. Durability of edits once acknowledged. Bounded staleness: a remote edit should converge globally within a small, stated budget (seconds, not minutes).

Out of scope (say so)

The social graph and timeline (that was lesson 29), auth/sessions, media blob storage (avatar bytes live in object storage / a CDN; the profile holds only the URL). Scoping these out keeps the discussion on the one hard thing: multi-region consistency.

Clarifying questions a strong candidate asks first

What is the read:write ratio, really? "Occasionally edited, read everywhere" suggests something like 1000:1 or more. This single number decides the whole architecture — it is what licenses aggressively replicating reads to every region.
How current must a read of someone else's profile be? If a friend reads your bio two seconds after you change it and sees the old one, is that acceptable? Almost always yes — profiles tolerate eventual consistency (all replicas converge if writes stop) for third-party reads. That single concession is what unlocks fast local reads.
How current must a read of your own profile be? This is the hard exception: a user who just changed their avatar and reloads MUST see the new one, or it looks broken. So we need read-your-writes (a session guarantee: you always see your own prior writes) even while everyone else is eventually consistent. Naming this asymmetry early is the strongest signal in the whole case.
Can the same user edit from two regions at once? Rare for one human, but it happens (a phone on cellular in one region, a laptop on VPN in another; or a support agent editing on the user's behalf). So we need a concurrent-write conflict story — even if conflicts are rare, "rare" is not "never".
What is the regional failure expectation? If a region or the link between regions goes down, do we keep serving local reads (possibly stale) or refuse? This is the CAP question, and the answer is almost certainly "stay available, serve stale".

2 · Back-of-envelope sizing

Pin numbers before drawing boxes. Assume 2 billion profiles, deployed across 4 regions (say NA, EU, APAC, SA), with users roughly evenly split, 500 million served from each region.

Worked sizing — storage, QPS, and the p99 budget

Storage. A profile is small structured data — name, handle, bio, a few URLs and flags. Call it 2 KB serialized.
• Raw dataset = 2,000,000,000 profiles × 2 KB = 4 TB.
• We replicate a full copy to every region (this is the whole point — local reads), so 4 regions hold 4 TB × 4 = 16 TB of profile data globally, before within-region replication. With 3-way replication inside each region for durability, that is 16 TB × 3 = 48 TB on disk total. Trivial by storage standards — profiles are tiny. Storage is not the constraint; latency and consistency are.

Read QPS. Suppose 1 billion daily active users, each triggering on the order of 50 profile reads/day (every page render, every backend service that needs a name or avatar). That is 50,000,000,000 reads/day ≈ 580,000 reads/sec globally, with peak maybe 3× average, so ~1.7 million reads/sec at peak. Split across 4 regions: ~430,000 reads/sec per region at peak. This is large but very cacheable — the same hot profiles are read constantly.

Write QPS. If each active user edits their profile, say, once per month, that is 1,000,000,000 / (30 × 86,400) ≈ 390 writes/sec globally. Round up to ~1,000 writes/sec peak. The read:write ratio is therefore roughly 1,700,000 / 1,000 = 1700:1 — overwhelmingly read-heavy, exactly as suspected. This ratio is the license to optimize relentlessly for reads and treat writes as a rare, expensive-is-fine event.

Latency budget. Target store-side p99 read of 10 ms within a region. That is comfortably met by a local read from an in-region replica or cache — local network round trips are sub-millisecond. The number that matters is the one we must not pay on the read path: a cross-region round trip is roughly EU↔APAC ≈ 150–250 ms, NA↔SA ≈ 100–120 ms. If a read had to cross an ocean, p99 would blow past 200 ms — 20× the budget. So the design rule writes itself: reads must be served locally; only writes and replication may cross regions.

Replication-lag budget. Edits propagate to other regions asynchronously. Budget convergence at p99 ≤ 2 s globally under healthy links — small enough that a friend almost never sees a stale bio, large enough that we never block a write on a slow cross-ocean link.

3 · Design walk — building it from the track's mechanisms

The sizing dictates the shape: a full copy of the data in each region, reads served locally, writes propagated cross-region asynchronously. Now make each layer concrete with the mechanisms this track built.

Partitioning inside a region (L11). 4 TB is small, but 430,000 reads/sec per region wants horizontal spread. Hash-partition by user_id across, say, a few dozen partitions per region. The dominant query is "read by user id", so the shard key is the query key — every read is a single-partition point lookup, never a scatter-gather. There are no cross-profile transactions, so no operation ever spans partitions. Profiles have no celebrity hot-key problem on writes (you only edit your own), and hot reads (a famous account read constantly) are absorbed by the cache layer below, not by splitting keys.

Replication, the crux choice (L08, L09). The dataset must exist, fully, in every region. There are two defensible cross-region topologies, and naming the trade is the core of the answer:

Single global leader region, read replicas elsewhere (L08)

One region (say EU) holds the leader for every partition; the other regions hold async read replicas. Reads are local everywhere (fast). Writes from APAC must travel to EU and back — a 150–250 ms write, but writes are rare (1000/sec) so that is fine. The big win: no write conflicts ever, because a single leader serializes all writes — last-write-wins and version vectors never come up. The cost: a write from a far region is slow, and if the leader region is partitioned away, the whole world is read-only until failover.

Multi-leader, one leader per region (L09)

Every region accepts local writes (a local leader) and ships them to the others. Both reads and writes are local everywhere — lowest latency, and a region survives isolation by continuing to accept its own writes. The price is the one L09 warned about: two regions can write the same profile concurrently, producing a conflict the system must resolve (LWW or version vectors, below). You trade "writes never conflict" for "writes are always local and a region is never blocked".

For a profile store the multi-leader topology is usually the better fit, precisely because profiles are independent and conflicts are rare: a user editing their own single profile from two regions at the same instant is a genuinely uncommon event, and the conflict — when it does happen — is small and field-level. We pay a tiny conflict-resolution complexity to get local writes and survive region isolation. (For a store where writes must be globally serialized — say a model-registry "promote to prod" pointer, lesson 33 — you would instead pick the single-leader, even linearizable, design. Profiles do not need that.)

Concurrent-edit conflict resolution (L09). With multi-leader, two regions can hand us two versions of the same profile. Two strategies, and a strong candidate names both and picks per field:

Last-write-wins (LWW). Stamp each write and keep the one with the higher timestamp; the other is silently dropped. Simple, but it discards a concurrent edit, and it leans on clocks — and clocks across regions are not perfectly synchronized (L15: clock skew), so "last" can be wrong. Acceptable for low-stakes fields where losing one of two simultaneous edits is harmless (a bio typo race).
Version vectors / per-field merge. Track a version per replica so the system can detect a true concurrent edit (neither write happened-before the other) rather than guess by clock. Then merge at the field grain — if region A changed the avatar and region B changed the bio, keep both; only a true same-field conflict needs a resolution rule (LWW within that field, or surface both to the user). This is more work but loses no data and does not trust cross-region clocks. Because a profile is a small bag of independent fields, field-level merge is natural here.

The senior move is "version vectors to detect, field-level merge to resolve, LWW only within a genuinely conflicting field" — it acknowledges that LWW alone throws away writes and that clocks lie, while keeping the common case (edits to different fields) lossless.

Read-your-writes, the asymmetry that makes this case (L08, L16). Everyone tolerates eventual consistency for other people's profiles, but a user reading their own profile right after editing must see the edit, or the app looks broken. Replication lag (L08) means a naive local read could hit a replica that has not yet applied the user's own write. Three mechanisms, cheapest first:

Read from the region that owns your write. With multi-leader, your edit is applied to your home region's leader synchronously before acknowledgment, and you keep reading from that region. Your own writes are always visible locally; only other regions lag. This handles the common case for free.
Write-timestamp / version token. When you write, return the version (or a logical timestamp) of your write. On subsequent reads, the client sends that token; the store serves only a replica that has applied at least that version, else waits or routes to the leader. This is read-your-writes made explicit and is robust even if you roam between regions or read from a lagging replica. It is the L16 idea: a session guarantee enforced by a monotonic version, not by global linearizability (which we deliberately do not pay for).
Sticky cache invalidation. On your own write, eagerly invalidate (or update) the cache entry for your profile in your region so your next read does not serve a stale cached copy. Cheap and essential — the cache is the most likely source of a stale self-read.

Caching (the read amplifier). In front of each region's partitioned store sits a read cache (the same hot profiles are read constantly at 430,000 reads/sec). A cache hit serves in ~1 ms. On a profile write, invalidate that profile's key in the writing region immediately (for read-your-writes) and let other regions' caches expire by short TTL plus an async invalidation message riding the same replication stream. Bounded staleness lives partly here: the cross-region convergence budget (2 s) is replication lag plus remote cache TTL.

4 · Failure timeline — a cross-region partition during a self-edit

The instructive failure for this system is the one the prompt's "coherent" word is hiding: a network partition between regions, exactly while a user is editing their own profile. This is where CAP stops being a slide and becomes a decision.

t0 APAC user edits avatar. Write goes to APAC's local leader, acked. (local, fast) t1 APAC leader begins async fan-out of the edit to NA / EU / SA. t2 *** PARTITION *** the link from APAC to the other regions drops. The edit is durable in APAC but cannot propagate. t3 APAC user reloads -> reads APAC's local store -> SEES new avatar. (read-your-writes HOLDS, because we read from the same region that took the write.) t4 An EU user opens the APAC user's profile -> reads EU's local store -> sees the OLD avatar. <-- STALE, but the read SUCCEEDS. (AP choice: available + stale) t5 Partition heals. APAC's buffered edit replicates to NA / EU / SA. t6 Convergence: all regions now show the new avatar (within the lag budget once the link is back). Cache-invalidation messages riding the stream evict the stale EU cache entry.

The CAP / PACELC trade, made explicit

CAP (during a partition). When regions cannot talk, you must choose: serve the available stale read (the EU user sees the old avatar at t4) or refuse the read to guarantee consistency. A profile store overwhelmingly chooses AP — available and partition-tolerant, sacrificing strong consistency: showing a 30-seconds-old bio is fine; showing an error page because a transatlantic link blipped is not. The multi-leader topology even keeps writes available in every reachable region during the partition (APAC's edit at t0 succeeds), at the cost of a possible conflict to reconcile at t5 (handled by version-vector merge from §3).

PACELC (the part candidates forget). CAP only describes the partition case. PACELC adds: else — when there is no partition (the normal case) — you still trade Latency against Consistency. Even on perfectly healthy links, a strongly consistent read (one guaranteed to reflect every region's latest write) would have to coordinate across regions and pay that 150–250 ms cross-ocean round trip on every read — 20× our 10 ms budget. So in the normal case we again choose L over C: serve fast local reads that may be a couple of seconds stale, rather than slow globally-consistent ones. The store is PA/EL — available under partition, low-latency otherwise — with read-your-writes carved out as the one consistency guarantee we do pay for, cheaply, via a per-session version token rather than global linearizability.

5 · Answer rubric

Good answer (baseline)

Replicates a full copy to every region and serves reads locally; recognizes the read:write ratio is huge and optimizes for reads.
States that third-party profile reads can be eventually consistent, which is what makes local reads possible.
Names read-your-writes as a requirement for self-reads and proposes a mechanism (read from your home region, or a version token).
Picks a topology (single-leader or multi-leader) and can say why writes may cross regions while reads do not.
Sizes storage and QPS with plausible numbers and concludes that storage is not the bottleneck — latency and consistency are.

Better answer (senior signal)

Frames the whole problem as the consistency-vs-latency trade and states both CAP (partition: AP, stale-but-available) and PACELC (no partition: still pay cross-region latency for strong reads, so choose L) explicitly.
Treats read-your-writes as an asymmetric guarantee: eventual for others, session-monotonic for self, enforced by a cheap per-session version token rather than global linearizability.
For multi-leader, gives a real conflict story: version vectors to detect concurrent edits, field-level merge to resolve, LWW only within a truly conflicting field — and notes LWW alone drops writes and trusts unsynchronized clocks (L15).
Quantifies the lag budget (convergence p99 ~2 s) and the cross-region RTT that the read path must never pay.
Routes the failure case correctly: read-your-writes still holds during a partition because the self-read hits the write's home region.

Red flags

Proposes synchronous cross-region replication or a globally linearizable read on the hot read path — blows the latency budget 20× and misses that profiles do not need it.
Says "just use strong consistency everywhere" with no awareness of its cross-region latency cost (the PACELC blind spot).
Picks multi-leader but has no concurrent-write conflict story, or reaches for LWW without noticing it silently discards data and depends on synchronized clocks.
Treats read-your-writes as automatic — assumes a local read always reflects the user's own just-made edit, ignoring replication lag and stale cache.
During a partition, chooses to reject reads (CP) for a profile store, where stale-but-available is clearly correct.
Obsesses over storage capacity (4 TB) and never engages the actual hard problem, geography.

Likely follow-ups

"A user edits from two regions in the same second — walk me through exactly what each region stores and what the merged result is." (Version vectors + field merge.)
"How does a user who just edited in EU and then flew to APAC still see their own change?" (Version token carried by the client; APAC waits for or routes to a replica at that version.)
"Quantify the staleness a third party can observe, and where it comes from." (Replication lag + remote cache TTL; bound it.)
"What changes if this were a model-registry promote-pointer instead of a profile?" (Now writes must be globally serialized: single-leader / linearizable pointer, CP for that operation — see lesson 33.)
"Add a fifth region. What moves?" (Another full copy + a leader; rebalance in-region partitions, L11; convergence budget unchanged.)
"How do you stop a partition from causing permanent divergence?" (Buffer un-acked cross-region writes; replay on heal; conflict-merge on convergence.)

6 · Takeaway and interview prompts

Where is truth?

System of record: for a given profile, the leader that accepted the latest write — under multi-leader, that is the user's home-region leader for that field, with the per-field version vector as the authoritative record of "what version each replica holds." There is no single global master; truth is the merged latest version across regions. Copies / derived views: the async cross-region replicas in the other three regions, and the per-region read caches in front of them. Freshness budget: convergence p99 ≤ 2 s on healthy links (replication lag plus remote cache TTL); read-your-writes is exempt — your own write is visible to you immediately via home-region pin or version token. Owner: the profile service owns the store; each region's leader owns writes for sessions homed there. Deletion path: a delete (or field clear) is a write like any other — applied at the home leader, version-stamped, and replicated; remote caches are evicted by the invalidation message riding the replication stream. Reconciliation/repair: after a partition heals, buffered writes replay and version vectors detect true concurrent edits, resolved by field-level merge (LWW only inside a genuinely conflicting field); anti-entropy re-syncs a lagging or rebuilt replica from the others. Evidence of correctness: version vectors prove which writes a replica has seen (so convergence is checkable, not assumed); monitoring replication lag against the 2 s budget and comparing a profile's version across regions shows the copies are tracking the record.

Takeaway

A global profile store is read-dominated (here ~1700:1), so the architecture is "a full copy in every region, every read served locally, only writes and replication cross regions" — because a cross-region round trip is 150–250 ms against a 10 ms budget, the read path must never leave the region. The hard part is consistency, and the case is won by stating the trade explicitly: CAP says that during a partition you choose AP — serve stale-but-available reads (and, with a multi-leader topology, keep accepting local writes) rather than refuse — while PACELC says that even with no partition you still trade latency against consistency, so you again choose low-latency local reads over globally-strong ones. The one consistency guarantee you do buy is read-your-writes, carved out asymmetrically (eventual for others, session-monotonic for self) and enforced cheaply by reading from the write's home region or by a per-session version token — never by global linearizability. Multi-leader buys local writes and region-isolation survival at the price of concurrent-edit conflicts, resolved by version vectors to detect plus field-level merge to keep both edits, with LWW only inside a genuinely conflicting field (and the warning that LWW drops writes and trusts unsynchronized clocks).

Interview prompts

Why can a profile store serve fast reads on every continent but a strongly-consistent store cannot? (§2, §4 — local reads hit an in-region replica in ~10 ms; a globally-consistent read must coordinate across regions and pay the 150–250 ms cross-ocean RTT on every read. PACELC: with no partition you still trade L vs C, and profiles choose L.)
State the CAP choice this system makes during a partition, and why. (§4 — AP: serve stale-but-available reads (and keep taking local writes under multi-leader) rather than reject, because a slightly-old bio is fine but an error page for a link blip is not. CP would be wrong here.)
Read-your-writes for your own profile but eventual consistency for others — how? (§3 — asymmetric guarantee: read from the region that took your write, or carry a per-session version token so the store serves only a replica that has applied your write; cache-invalidate your own key on write. No global linearizability needed.)
Two regions edit the same profile concurrently — single-leader vs multi-leader, and how do you resolve it? (§3 — single-leader serializes all writes so conflicts can't happen but far writes are slow and a leader-region outage goes read-only; multi-leader keeps writes local but can conflict, resolved by version vectors to detect + field-level merge, LWW only within a conflicting field.)
Why is plain last-write-wins a risky conflict resolver across regions? (§3 — it silently discards the losing write and decides "last" by timestamp, but cross-region clocks are skewed (L15), so the wrong write can win. Use version vectors to detect true concurrency and merge at field grain instead.)
Size the store and say what the bottleneck is. (§2 — 2B × 2 KB = 4 TB raw, ×4 regions ×3 in-region replicas ≈ 48 TB, trivial; ~1.7M reads/sec peak vs ~1000 writes/sec. Storage is not the constraint — local read latency and cross-region consistency are.)
What would you change if this were a "promote model to prod" pointer instead of a profile? (§3, §5 — that write must be globally serialized, so single-leader / linearizable pointer and CP for that operation; you'd give up local writes for correctness. Profiles deliberately don't need that — see lesson 33.)