Part 12 · Applied & interviews

Case: Social Home Timeline

Lesson 06 introduced the social home timeline as this track's canonical load/latency example — the running illustration of how to describe load and budget tail latency. This is the full worked case. Lesson 28 gave us two things it leaned on: a library of failure timelines (how async replication loses writes, how a cache stampede topples a database) and a set of quantitative drills (Little's Law, fan-out amplification, partition sizing, storage growth). Here we spend both at once. We take a full senior-level system-design prompt — the social home timeline, the canonical Twitter question — and answer it end to end using only mechanisms the track already built: storage engines, replication, partitioning, the home-timeline cache, and a log. The whole case turns on one decision (fan-out-on-write vs fan-out-on-read) and one villain (the celebrity hot key from lesson 11). We will size it with real numbers and verify the arithmetic as we go.

DDIA source

This case is built directly on Martin Kleppmann, Designing Data-Intensive Applications — the Twitter home-timeline example is the running illustration of "describing load" in Chapter 1 (the fan-out problem), and the design draws on Chapter 5 (replication), Chapter 6 (partitioning / hot keys), and Chapter 11 (stream processing / the timeline-as-derived-data view). The interview framing, the numbers, and the rubric are our synthesis.

Linear position

Prerequisite: Lesson 28 (Failure timelines and quantitative drills) — you can read an ASCII failure timeline and do back-of-envelope QPS, fan-out, storage, and latency-budget math. We also lean hard on lesson 11 (partitioning and the celebrity hot key) and the caching ideas from lesson 28's stampede timeline.
New capability: Drive a complete senior interview from prompt to rubric — clarify, size, design from named mechanisms, name one failure and its fix, and know exactly what separates a passing answer from a hire-strong one on this specific problem.

The plan

Six sections, the shared interview-case template. (1) The prompt, the functional and non-functional requirements, and the clarifying questions a strong candidate asks before writing anything. (2) Back-of-envelope sizing — users, read and write QPS, the follower fan-out distribution, timeline storage, and a p99 latency budget — every number checked. (3) The design walk: fan-out-on-write vs fan-out-on-read vs the hybrid, the home-timeline cache, and how the celebrity breaks pure push, with an ASCII architecture diagram and cross-links to the home lesson for each mechanism. (4) One failure timeline — a celebrity tweet melts the fan-out pipeline — and its mitigation. (5) The answer rubric: Good / Better / Red flags / Likely follow-ups. (6) Takeaway, a Checkpoint variant to design, and interview prompts with answers.

1 · The prompt and the questions to ask first

The prompt (as an interviewer would give it). "Design the home timeline for a Twitter-like service. A user follows other accounts; when they open the app, they see a reverse-chronological feed of recent posts from everyone they follow. Posting and reading should both feel instant. Scale it to hundreds of millions of users."

A weak candidate starts drawing boxes. A strong one first pins down what "the system" must and must not do, because the entire design hinges on the read/write ratio and on whether the feed must be perfectly fresh.

Functional requirements

Post a tweet — append a short message authored by a user (the write path).
Read home timeline — for the logged-in user, return recent posts from all accounts they follow, newest first (the read path; this is the hard one).
Follow / unfollow — maintain the social graph that defines whose posts land in a timeline.
Out of scope to keep focus: search, notifications, DMs, the ranked "For You" feed (we build the simpler reverse-chronological feed; ranking is an orthogonal layer on top).

Non-functional requirements

Read-latency dominated. Timeline reads are far more frequent than writes and must be fast — target p99 < 200 ms for a timeline fetch.
Eventual consistency is acceptable. A new tweet appearing in followers' timelines a few seconds late is fine; this is not a bank ledger. That single relaxation unlocks almost every optimization below.
High availability over strict freshness — a slightly stale timeline beats an error page.
Massive read fan-out, write amplification on post. One post may need to reach millions of timelines.

Clarifying questions a strong candidate asks

What is the read/write ratio? (Reads dominate writes — we derive roughly 15:1 on average in section 2, steeper at peak — which is the whole reason fan-out-on-write is even considered.)
How fresh must the timeline be? Real-time, or is a few seconds of lag acceptable? (Seconds is fine — this is what lets us precompute.)
What is the follower distribution? Is it uniform, or is there a long tail with celebrities at millions of followers? (Heavily skewed — and this skew, the lesson 11 hot key, is the crux of the whole problem.)
Reverse-chronological or ranked? (Reverse-chron for this design; ranking is a layer we can add later.)
Do we need to support reading the full history, or just the recent window? (Recent window for the home timeline; deep history is a separate fetch from the tweet store.)

The interviewer's answers set every number in section 2. Asking them is itself a strong signal — it shows you know that the design depends on the load shape, not the feature list.

2 · Back-of-envelope sizing

Pin down the load before choosing a mechanism. We will use round, defensible numbers and verify each step (the drills from lesson 28).

Assumptions (state them out loud)

  • 300,000,000 total users; 150,000,000 daily active (DAU).
  • Each DAU posts 2 tweets/day and opens their timeline ~30 times/day (read).
  • Follower distribution: average 200 followers; long tail up to a celebrity at 100,000,000 followers.
  • A tweet is ~300 bytes of text plus metadata; call the timeline entry ~300 bytes when materialized (tweet id + author id + timestamp, fattened with a little denormalized text).

Write QPS (tweets posted)

Writes per day = 150,000,000 DAU × 2 = 300,000,000 tweets/day. Spread over 86,400 seconds:

300,000,000 / 86,400 = 3,472 tweets/sec (avg write QPS) peak ~= 3x avg ~~ 10,000 tweets/sec

Verify: 3,472 × 86,400 = 299,980,800 ≈ 300,000,000. Good. Write QPS is modest — about 3.5k/s average. Hold that thought; the writes are cheap, it is what each write triggers that hurts.

Read QPS (timeline opens)

Reads per day = 150,000,000 × 30 = 4,500,000,000 opens/day:

4,500,000,000 / 86,400 = 52,083 reads/sec (avg read QPS) peak ~= 3x avg ~~ 156,000 reads/sec

Verify: 52,083 × 86,400 = 4,499,971,200 ≈ 4.5 billion. Good. Read:write ratio ≈ 52,000 / 3,472 ≈ 15:1 on average, and worse at peak — and each read wants tens of recent posts merged from potentially hundreds of followees. This asymmetry is the entire argument for doing work at write time so reads stay cheap.

Fan-out: the number that decides the design

The cost of fan-out-on-write is: for every tweet, write one copy into each follower's materialized timeline. So the fan-out write rate is write QPS times the average follower count:

fan-out writes/sec = tweets/sec × avg_followers = 3,472 × 200 = 694,400 timeline-inserts/sec (avg) peak ~= 3x ~~ 2,000,000 timeline-inserts/sec

Verify: 3,472 × 200 = 694,400. So a 3.5k/s write stream amplifies into ~700k/s of timeline inserts — a 200× write amplification. Large, but spreadable across a partitioned cache cluster. This is the price of cheap reads, and for the average account it is a price worth paying.

The celebrity: where pure fan-out-on-write breaks

Now apply the same formula to one celebrity post with 100,000,000 followers:
1 tweet × 100,000,000 followers = 100,000,000 timeline-inserts for a single tweet.

At even a modest celebrity posting rate, this dwarfs the entire baseline fan-out. A few thousand high-follower accounts posting concurrently can demand hundreds of millions of inserts in a burst — the fan-out pipeline backs up, every follower's timeline goes stale, and the cluster melts. This is exactly the lesson 11 hot-key problem: the work for one key (one author) cannot be spread, because it is fundamentally one author's post multiplied by an enormous fan-out. Hashing the timeline by user spreads the destinations fine; it does nothing about the volume one celebrity generates. Pure push does not scale to the head of the distribution. The fix (section 3) is the hybrid: do not fan out celebrity posts at all — merge them in at read time.

Timeline storage

Materialize the recent window only: keep, say, the last 800 entries per user's home timeline (enough for many screens of scroll; older reads fall through to the tweet store).

per-user timeline = 800 entries × 300 bytes = 240,000 bytes ~= 240 KB all active users = 240 KB × 150,000,000 DAU ~= 36 TB (one copy) with replication 3x ~= 108 TB

Verify: 800 × 300 = 240,000 bytes; 240,000 bytes × 150,000,000 users = 36,000,000,000,000 bytes = 36 TB; × 3 = 108 TB. That is a large but ordinary in-memory + flash cache footprint, sharded across a few thousand cache nodes. The full tweet corpus (the system of record) is separate and larger, but it is written once and read rarely on the home path.

p99 latency budget

Target p99 < 200 ms for a timeline read. With a precomputed timeline this is easy; the budget is dominated by the cache hit, not by any merge:

timeline read p99 budget (200 ms total) ├─ network in / auth / routing ~ 20 ms ├─ fetch materialized timeline (cache) ~ 10 ms (one partition, point read) ├─ merge in live celebrity posts ~ 30 ms (pull from a few hot authors) ├─ hydrate tweet bodies (batch get) ~ 40 ms └─ serialize + network out ~ 20 ms total ~ 120 ms (headroom under 200 ms)

The whole point of fan-out-on-write is that the expensive merge happened at write time, so the read is mostly a single cache point-read (lesson 11: a query that includes the shard key — here user_id — routes to one partition). Contrast a pure fan-out-on-read design where the same fetch must scatter-gather across hundreds of followees' tweet lists and merge-sort them inline — that easily blows the 200 ms budget at the tail, because p99 tracks the slowest of hundreds of sub-queries.

3 · Design walk — push, pull, and the hybrid that wins

There are exactly two pure strategies, and a hybrid that takes the best of each. We build all three from track mechanisms.

Fan-out-on-write (push)

When a user posts, immediately insert the tweet id into the materialized home timeline of every follower. Reads become a single cheap point-read of a precomputed list. Reads are O(1); writes are O(followers). Great for the average account; catastrophic for celebrities (the 100M-insert burst above).

Fan-out-on-read (pull)

Store only each author's own tweet list. On read, fetch the tweet lists of everyone the user follows and merge-sort by time. Writes are O(1); reads are O(followees) scatter-gather. Trivial for posting; reads are slow and tail-latency-heavy for users who follow many accounts.

Hybrid (the answer)

Push for ordinary authors; do not push for celebrities. At read time, take the precomputed timeline and merge in the recent posts of the (few) celebrities the user follows, pulled live. Cheap reads and bounded write fan-out. This is what real Twitter converged on.

Why the hybrid is forced by the numbers. Section 2 showed both extremes fail at one end of the follower distribution: push fails at the head (celebrity write burst), pull fails for users who follow many accounts (read scatter-gather). The distribution is bimodal, so the strategy must be too — split the author population at a follower threshold and treat the two halves differently.

The celebrity threshold is a tunable knob

Define an author as a celebrity when their follower count exceeds some threshold T (say T = 100,000). Below T: fan out on write. At or above T: do not fan out; their posts are pulled at read time. The vast majority of accounts (avg 200 followers) are well below T, so almost all writes still get the cheap-read benefit. The handful above T are exactly the accounts whose fan-out would melt the pipeline. A typical user follows only a few celebrities, so the read-time merge pulls from a small number of hot authors — bounded work, unlike pulling from all followees. The threshold trades write amplification against read-merge cost; tune it from the live follower histogram.

The write path (post a tweet)

POST /tweet | v +------------------+ | Tweet service | 1. append to tweet store (system of record) +------------------+ (LSM-backed, partitioned by tweet_id -> L04/L11) | v author followers >= T ? / \ NO (ordinary) YES (celebrity) / \ v v +-------------------+ +------------------------+ | Fan-out worker | | DO NOT fan out. | | (async, off a | | Mark author "hot"; | | log -> L19) | | read path pulls live. | +-------------------+ +------------------------+ | | for each follower f: v insert (tweet_id, ts) into home_timeline[f] (home-timeline cache, partitioned by follower user_id -> L11)

Mechanisms, cross-linked: the tweet store is the durable system of record — an LSM-tree store (lesson 04) partitioned by tweet id (lesson 11), each partition itself a replica set (lesson 08). The fan-out runs asynchronously off a log: the tweet service appends a "tweet posted" event and fan-out workers consume it (the stream-processing pattern, lesson 19 — the timeline is derived data, a materialized view of "tweets joined with the follow graph," lesson 20). Doing fan-out asynchronously is what keeps the post request fast: posting returns as soon as the tweet is durable and the event is logged, not after a million inserts finish.

The read path (get home timeline)

Step 1 is the home-timeline cache — the precomputed materialized view, sharded by the reader's user_id so the read is a single-partition point lookup (the lesson 11 performance contract: the shard key is the read key). Step 2 is the hybrid's pull half, and it is cheap precisely because we only pull the celebrities a user follows, not all followees. Steps 1 and 2 together are why the design beats both pure strategies: the heavy merge over ordinary authors was already paid at write time, and the only live merge is over the small celebrity set.

Consistency: it is eventual, and that is fine (lesson 16)

The home-timeline cache is derived data that lags the tweet store by however long the async fan-out takes — typically seconds. A tweet is durable the instant the post returns (it is in the tweet store), but it appears in followers' timelines only after the fan-out worker processes it. That lag is the eventual consistency we negotiated in section 1 (lesson 16). The one place users notice: a user should see their own just-posted tweet immediately (read-your-writes). Handle that on the read path by always merging the user's own recent posts in client-side, rather than waiting for fan-out to reflect them — the same read-your-writes concern lesson 08 raised for replica lag.

4 · Failure timeline — a celebrity tweet stalls the fan-out pipeline

Suppose the celebrity threshold is misconfigured (set too high, or a fast-growing account crosses 100M followers before it is reclassified). A pure-push fan-out then fires for that account. Here is what the on-call engineer sees.

Root cause. One log event expanded into 100M units of work that cannot be parallelized away — the lesson 11 hot key, now expressed as a hot event on the fan-out log. Ordinary tweets queue behind it (the consume rate fell below the produce rate, so the backlog grows without bound — the CDC/backlog drill from lesson 28). Every follower's timeline goes stale because the workers are pinned on the celebrity's inserts.

Mitigation

Immediate: reclassify the account as a celebrity (above T), drop its in-flight fan-out, and let the read path pull its posts live instead. The 100M-insert job evaporates.
Structural: (1) Enforce the celebrity threshold before enqueuing fan-out, not after — never let a high-follower account enter the push path. (2) Make the threshold dynamic, driven by the live follower histogram, so a fast-growing account is reclassified automatically. (3) Give celebrity-scale events a separate, rate-limited lane so a misclassification cannot starve ordinary fan-out (priority isolation on the log). (4) Cap and shed: if fan-out backlog exceeds a bound, demote the offending author to pull. The general lesson — the same one lesson 11 taught for hot keys — is that you must detect the hot key and route it down a different path; you cannot out-scale a single key by adding machines.

5 · Answer rubric

What an interviewer is actually scoring on this specific problem.

Good answer (baseline)

Clarifies the read/write ratio and freshness requirement before designing.
Identifies that reads dominate and proposes fan-out-on-write to make reads a cheap precomputed lookup.
Sizes write QPS, read QPS, and timeline storage with reasonable numbers.
Separates the durable tweet store (system of record) from the home-timeline cache (derived).
Notes the feed can be eventually consistent.

Better answer (senior signal)

Recognizes pure push breaks for celebrities and derives the 100M-insert burst from the fan-out formula — unprompted.
Proposes the hybrid with a tunable celebrity threshold, and explains the read-time merge over only the celebrities a user follows.
Ties the hot author explicitly to the partitioning hot-key problem (lesson 11) and explains why adding machines does not help.
Runs fan-out asynchronously off a log (derived-data / stream view) so the post request stays fast; handles read-your-writes for the user's own posts.
Gives a p99 latency budget and shows the merge is cheap because it was precomputed.

Red flags

Jumps to boxes-and-arrows without asking the read/write ratio or freshness.
Proposes pure fan-out-on-read for the whole system and never confronts the read tail-latency for users who follow many accounts.
Proposes pure fan-out-on-write and waves away the celebrity case ("we'll just add more workers") — the one thing you cannot scale by adding workers.
Makes the post request block on the full synchronous fan-out, so posting a tweet takes seconds.
Demands strong consistency / a transactional timeline when eventual is plainly fine, and pays a huge cost for it.
Confuses partitioning the timeline (spreads destinations) with solving the celebrity volume (it does not).

Likely follow-ups

"A celebrity gains 50M followers overnight — what changes?" (Dynamic threshold; reclassify to pull; section 4.)
"How do followers see a tweet that was posted while their timeline was being fanned out?" (Async lag + read-your-writes merge.)
"Now make it a ranked feed, not reverse-chron." (Ranking layer on top of the candidate set the timeline produces; reuse pull + score.)
"How do you handle unfollow / deleted tweets in precomputed timelines?" (Filter at read time, or lazily clean; do not try to delete from millions of timelines synchronously.)
"Size the fan-out worker fleet." (From the ~700k inserts/sec avg, ~2M peak — a partition-count / throughput drill.)

Checkpoint exercise

Try it — design the variant

Redesign for Instagram-style image posts with a ranked feed instead of reverse-chronological text. (a) The media is large (a 2 MB image vs a 300-byte tweet) — explain why you still fan out only references (post ids), never the bytes, and where the bytes live (object store / CDN). (b) A ranked feed means reads must score candidates, not just merge by time — show how the home-timeline cache becomes a candidate set that a ranking model consumes at read time, and what that does to your p99 budget vs section 2. (c) Engagement is even more skewed than follower count (a viral post draws orders of magnitude more reads than an ordinary one) — identify the new hot key on the read path and propose a fix (hint: lesson 11 cache / key-split). (d) State which numbers from section 2 change and re-derive the storage figure for fanning out references only.

Where this points next

The timeline case stressed one mechanism above all — the hot key from partitioning, and the read/write asymmetry that makes precomputation worth it. The next case keeps the user as the entity but inverts the pressure: a global profile store (lesson 30) where the hard part is not fan-out volume but geography — serving low-latency reads from many regions while a user's own edits must be visible to them immediately (read-your-writes, lesson 08/lesson 16) and concurrent edits from different regions must be reconciled (lesson 09). Same template — prompt, sizing, design, failure, rubric — pointed at the multi-region consistency trade-off (CAP/PACELC) instead of the fan-out one.

Where is truth?

System of record: the tweet store (durable, LSM-backed, partitioned by tweet id) plus the follow graph — together they fully determine every timeline. Derived views: the home-timeline cache (a materialized view of "tweets joined with the follow graph," one precomputed list per reader). Freshness budget: seconds — the async fan-out lag — with read-your-writes preserved for the author's own posts by client-side merge. Owner: the timeline/fan-out service owns the cache; the tweet service owns the record. Deletion path: deletes and unfollows hit the tweet store / graph (the truth) immediately; the precomputed timelines are not mutated synchronously across millions of followers — instead they are filtered at read time against the current tweet store and graph, and lazily cleaned. Reconciliation/repair: a timeline is fully rebuildable by replaying the tweet log against the follow graph, so a corrupt or lost cache shard is repaired by recompute, never by trusting the cache. Evidence of correctness: the read-time filter against the record guarantees a deleted tweet never renders; spot-checking a rebuilt timeline against a fresh recompute (and monitoring fan-out backlog against its freshness budget) shows the view tracks the truth.

Takeaway

The social home timeline is a read-heavy, eventually-consistent fan-out problem, and its whole design follows from two numbers: reads dominate writes ~15:1 (so precompute at write time to make reads cheap), and the follower distribution is bimodal (so no single strategy fits both ends). Fan-out-on-write (push a tweet id into every follower's materialized timeline) gives O(1) reads but O(followers) writes; fan-out-on-read (pull and merge followees' tweets) gives O(1) writes but slow scatter-gather reads. The celebrity — a lesson 11 hot key, 100M inserts for one tweet — breaks pure push, and you cannot fix a single hot key by adding machines. The answer is the hybrid: push for ordinary authors below a tunable follower threshold, pull celebrities' posts and merge them in at read time. Run the fan-out asynchronously off a log (the timeline is a derived materialized view) so posting stays fast, serve reads from a home-timeline cache sharded by reader id (a single-partition point read), and accept seconds of staleness while preserving read-your-writes for the author's own posts.

Interview prompts

Fan-out-on-write vs fan-out-on-read — what does each cost? (§3 — push precomputes each follower's timeline so reads are an O(1) point lookup but writes are O(followers); pull stores only each author's own tweets so writes are O(1) but reads scatter-gather and merge O(followees), with bad tail latency for users who follow many accounts.)
Why does pure fan-out-on-write break for celebrities, and why can't you just add workers? (§2–§4 — one celebrity post with 100M followers is 100M timeline inserts; that work is for a single key (one author) so it cannot be spread across partitions — the lesson 11 hot-key problem. Adding workers does not help because the volume is inherent to one key; you must route it down a different path: don't push it, pull it at read time.)
How does the hybrid decide push vs pull, and why is the read-time merge cheap? (§3 — split authors at a tunable follower threshold T: below T push on write, at/above T don't fan out and pull live at read. The merge is cheap because a typical user follows only a few celebrities, so the live pull is over a small set, not all followees, while ordinary authors were already precomputed.)
Why run fan-out asynchronously off a log, and what consistency does that give? (§3 — so the post request returns as soon as the tweet is durable and the event is logged, not after up to millions of inserts; the timeline is then a derived materialized view (lesson 19/20) that lags by seconds — eventual consistency, which we negotiated as acceptable, with a read-time merge of the user's own posts to preserve read-your-writes.)
Walk the p99 read budget and say why precomputation is what makes it fit. (§2 — ~120 ms of a 200 ms budget: a single-partition cache point read for the precomputed timeline, a small live merge of celebrity posts, and a batch hydrate of tweet bodies. The expensive O(followees) merge was paid at write time, so the read avoids the scatter-gather over hundreds of followees that would blow the tail in a pure-pull design.)