Part 12 · Applied & interviews
Interview Case: Metrics / Observability Dashboard
The previous case (lesson 33) built a model registry — a small, append-only system of record where every byte matters, the "promote to prod" pointer is linearizable, and losing a write is unacceptable. This case turns the dial all the way to the opposite pole. A metrics and observability backend ingests millions of time-series at once, is overwhelmingly write-heavy, and — crucially — tolerates approximation and loss: a dropped sample or a p99 computed from a sketch is fine if the dashboard stays fresh. The same toolbox (partitioning, columnar storage, the durable log, materialized aggregates) builds it, but the invariant flips from integrity to timeliness. That contrast is the heart of the interview.
New capability: design a write-heavy, high-cardinality time-series store end to end — partition by series id, ingest through a log, store columnar and time-partitioned, roll up to cheap downsampled tiers, age data from hot to cold, and answer dashboard queries (including p99 over millions of series) with approximate aggregates inside a query-latency budget.
1 · The prompt and what to ask first
The prompt. "Design the storage and query backend for a metrics and observability platform. Services across the fleet emit time-series — CPU, request latency, error counts, queue depth, and thousands of custom application metrics — tagged with labels like host, region, service, endpoint, and status code. Engineers open dashboards that plot these over the last hour, the last day, or the last quarter, and alerting rules evaluate them continuously. Make ingest keep up at fleet scale and make dashboards load fast." This is a textbook write-heavy, read-light, append-mostly workload — almost the inverse of the model registry.
Functional requirements. Ingest tagged samples (a metric name plus a set of label key-value pairs plus a timestamp and a float value) from millions of sources; store them durably enough; serve range queries ("series X over the last 6 hours") and aggregations ("p99 latency by region over the last day", "sum of error rate across all hosts") fast; support retention with cheap long-term history.
Non-functional requirements. Ingest must absorb a write-heavy fan-in without dropping more than a tiny, bounded fraction even under spikes. Dashboard query p99 must sit inside a few hundred milliseconds for recent windows. Old data must get cheap — a quarter of history cannot cost the same per point as the last hour. And the system explicitly tolerates lossy and approximate answers: a missing scrape or a p99 from a sketch is acceptable; freshness and availability beat exactness.
- What is the cardinality? The number of distinct series (metric name × every label combination) dominates the whole design. "1,000 metrics" is trivial; "1,000 metrics × 10,000 hosts × 20 endpoints" is ten million series and a different system. Always pin down cardinality before anything else.
- Is this a system of record, or telemetry? If a regulator or a billing pipeline reads it, integrity is non-negotiable and the design changes. For ops telemetry, we may drop and approximate — confirm we are in the telemetry world (cross-link L20 timeliness-vs-integrity).
- What query shapes matter? Mostly recent dashboards and alerts (last minutes to hours) with occasional long-range historical zoom-outs? That justifies hot/cold tiers and rollups. If long-range raw-resolution scans were common, the storage plan would differ.
- What resolution and retention per tier? Raw at 10-15 s steps for a few days, then 1-min and 1-hour rollups kept for months or years?
- Are high-percentile latencies needed? If yes, we must ingest histograms, not raw scalars, because you cannot recover a p99 by averaging pre-aggregated means.
2 · Back-of-envelope sizing
Pin numbers down first; they drive every later choice. Assume a fleet emitting 10,000,000 active series (10 million distinct metric+label combinations), each scraped every 15 s.
Raw storage, naive. A naive row is ~16 bytes (8-byte timestamp + 8-byte float) plus per-row series identification — call it 50 bytes uncompressed. 700,000 pts/s × 50 B × 86,400 s/day = ~3 TB/day. Over 7 days of raw that is ~21 TB — before replication. Unacceptable to keep at that cost for long.
Raw storage, compressed columnar. Time-series compress brilliantly: delta-of-delta on timestamps (the step is nearly constant) and XOR-of-floats (Gorilla-style) on values routinely hit ~1.3-2 bytes per point. At 1.5 B/point: 700,000 × 1.5 × 86,400 = ~91 GB/day of raw. Seven days raw ≈ ~635 GB, times a replication factor of 3 ≈ ~1.9 TB. Columnar + delta encoding bought a ~33× reduction over the naive row. (This is the L04 columnar win, applied to time.)
Rolled-up storage. Downsample raw 15 s to 1-min (4× fewer points) for 30 days, and to 1-hour (240× fewer points) for 1 year. 1-min tier: 91 GB/day ÷ 4 × 30 ≈ ~683 GB. 1-hour tier: 91 GB/day ÷ 240 × 365 ≈ ~138 GB. A full year of history thus costs less than a single week of raw — old data got cheap, which was a hard requirement.
Query p99 budget. "p99 latency by region, last 6 hours" over the raw tier would touch 6 h × 3,600 ÷ 15 = 1,440 points per series × however many series match the label filter. If the filter matches 50,000 series that is 72,000,000 points scanned — too slow for a 300 ms budget. Hitting the 1-min rollup instead drops it to 360 pts/series × 50,000 = 18,000,000, and a 1-hour rollup with a pre-merged histogram drops it to ~6 pts/series × 50,000 = 300,000 — comfortably inside budget. The rollup tier is not just a storage saving; it is what makes the read budget achievable.
The sizing already dictates the architecture: 700k writes/sec forces an ingest buffer and partitioning; 3 TB/day naive forces columnar compression; the 300 ms read budget over millions of series forces rollups and pre-aggregated histograms.
3 · Design walk — build it from the track's mechanisms
Read the pipeline left to right: a write-heavy fan-in lands in a durable log, is partitioned by series id, is written columnar and time-partitioned, is continuously rolled up into cheaper tiers, and ages from hot local storage to cold object storage. Each stage maps to a lesson.
3.1 Ingest: a log absorbs the fan-in
Millions of agents push samples into a stateless ingest tier that parses each sample, resolves its label set to a stable series id (a hash of metric name + sorted labels), and appends to a partitioned durable log (the L19 event log: an ordered, append-only, replicated log split into partitions). The log is the shock absorber — it decouples a bursty 700k-points/sec write rate from the storage tier's flush cadence, and on a storage-node restart the consumer simply replays from its last offset. We are explicitly not running a transaction or a quorum write per sample; that would be system-of-record machinery this workload does not need. A bounded buffer that may shed load under extreme spikes is the right call here — timeliness over integrity (L20).
3.2 Partition by series id
The log topic and the storage shards are both partitioned by hash(series-id) (L11): each series lives on exactly one shard, so all writes and all reads for that series route to one place — a single-hop point lookup, no cross-shard coordination on the write path. Hashing the series id spreads the ~700k writes/sec evenly and tames the timestamp hot-end trap (range-partitioning by time would send every current write to the newest partition — the classic L11 hot end). The residual risk is a single hot series: one extremely high-frequency metric whose lookups all land on one shard. The L11 fix transfers directly — front it with a cache, or split that one series across sub-keys. A query that filters by label but not by an exact series (most dashboard queries) is a deliberate scatter-gather: it fans out to all shards holding matching series and merges, which is fine for a read-light workload but is why we keep shard count and per-shard scan cost low via rollups.
3.3 Columnar, time-partitioned storage
Within a shard, data is written columnar and time-partitioned (L04). Recent writes accumulate in an in-memory head block backed by a WAL (the L03 write-ahead log for crash recovery), then flush to immutable time-bounded blocks (e.g. 2-hour blocks) that LSM-style compaction later merges. Storing each series' timestamps and values as separate compressed columns is what unlocks the encodings from the sizing: delta-of-delta on the near-constant 15 s timestamp step and XOR-of-floats on slowly-changing values, together giving ~1.5 bytes/point — the 33× reduction over naive rows. Columnar also means a query reads only the columns and time-ranges it needs, not whole rows.
3.4 Rollup / downsampling into materialized aggregates
A background job continuously downsamples raw blocks into coarser tiers — raw 15 s → 1-min → 1-hour — and at each step stores not just an average but a materialized aggregate (the L04 data-cube idea applied to time): min, max, sum, count, and a histogram sketch per bucket. These are precomputed answers, so a "last quarter, hourly" query reads a few hundred pre-merged points instead of scanning millions of raw ones. Crucially, you cannot recover a p99 by averaging means — so the rollup must carry an aggregable representation of the distribution, which leads to the next piece.
3.5 Approximate aggregates for percentiles over cardinality
Computing an exact p99 latency "across all hosts in a region" would require collecting every raw sample from tens of thousands of series and sorting — impossible inside 300 ms. Instead the system stores approximate, mergeable sketches:
Mergeability is the whole point: each sketch is stored per series-per-bucket in the rollup tiers, and a scatter-gather query simply merges the sketches returned by each shard. This is the L20 timeliness-vs-integrity trade made concrete — we accept a bounded approximation in exchange for answering p99 over ten million series inside a tight budget.
3.6 Hot / cold tiering
Recent data (last few hours, raw resolution) lives hot on local SSD or in memory where dashboards and alerts hammer it; older blocks age out to cold object storage (immutable, cheap, slower) once they have been rolled up. The query layer transparently spans both. This mirrors the LSM tiering instinct from L04 and keeps the expensive fast tier small while history stays affordable.
Contrast with a system of record
The model registry (lesson 33) is a system of record: every write must survive, the promote pointer is linearizable (L16), and integrity is the non-negotiable invariant. This metrics store flips every one of those. It tolerates dropped samples, computes percentiles from sketches, never needs a linearizable write, and treats timeliness and availability as the invariants. Same mechanisms (log, partitioning, columnar storage, materialized aggregates), opposite priorities — naming that flip out loud is the senior signal the interviewer is listening for (L20).
4 · Failure timeline — ingest fan-in overload
The defining failure of a write-heavy fan-in is a coordinated write spike that outruns the storage tier — for example, a fleet-wide incident causes every service to emit a burst of error and latency metrics at once, and a new noisy label (a per-request id mistakenly used as a label) explodes cardinality.
Mitigations. (1) A cardinality limiter at the ingest tier rejects or drops series from labels whose distinct-value count blows past a threshold — the single most important guard, since unbounded cardinality is the number-one way these systems die. (2) Per-tenant / per-priority ingest quotas so a runaway service sheds its own low-value series first, not everyone's. (3) The log buffer turns a hard outage into graceful lag — writes keep being accepted while consumers catch up, and bounded load-shedding drops the least-important samples rather than failing the whole pipeline. Because the workload tolerates loss, deliberately dropping a bounded fraction to protect freshness and availability is the correct behavior, not a bug — exactly the timeliness-vs-integrity choice from L20.
5 · Answer rubric
Good answer (baseline)
- Identifies the workload as write-heavy, high-cardinality, read-light time-series.
- Partitions by series id (L11) and uses columnar storage (L04) for compression.
- Separates recent hot data from old cold data, and downsamples old data to keep it cheap.
- Gives at least one real sizing number (series count, points/sec, or storage/day).
Better answer (senior signal)
- Leads with cardinality as the first clarifying question and the driver of the whole design.
- Puts a partitioned log (L19) in front of storage to absorb the fan-in and enable replay.
- Stores mergeable sketches (t-digest / histogram / HLL) so p99 and distinct-counts work over millions of series within budget.
- Explicitly names the timeliness-vs-integrity flip (L20) versus a system of record, and justifies dropping/approximating.
- Shows the rollup tier doing double duty: cheaper storage and a smaller read scan that meets the query p99 budget.
Red flags
- Range-partitioning by timestamp — sends every current write to one hot partition (the L11 hot end).
- Storing raw samples forever in a relational table with a row per point; no compression, no rollup.
- Claiming exact p99 by averaging pre-aggregated means — mathematically impossible; reveals no grasp of percentiles.
- Treating it as a system of record (per-sample transactions, quorum writes, linearizable inserts) — massive over-engineering for telemetry.
- Ignoring cardinality; no plan for a label that explodes the series count.
Likely follow-ups
- "A single metric becomes a hot series — what happens and how do you fix it?" (L11 hot key: cache or split.)
- "How do you compute p99 across 50,000 series in 200 ms?" (Mergeable sketches over the rollup tier, not raw scans.)
- "What stops one team's bad deploy from taking down ingest for everyone?" (Cardinality limits + per-tenant quotas + load shedding.)
- "What if a storage node crashes mid-flush?" (Replay from the log offset; WAL recovers the head block.)
- "How would the design change if this fed billing?" (Becomes a system of record — integrity wins; no dropping/approximation.)
Checkpoint exercise
user_id label and active series jumps to 40,000,000 — walk the failure timeline and name the two ingest-side guards that contain it. (d) State, in one sentence each, why partitioning by series id (not by timestamp) and storing histograms (not raw means) are both forced by the requirements.Where this points next
This case closes the applied tour: across lessons 29-34 the same handful of mechanisms — the durable log, partitioning, replication, columnar storage, materialized aggregates, and the timeliness-vs-integrity dial — re-combined to build six very different systems, the metrics store sitting at the lossy-and-approximate extreme. The track's arc, drawn from the first edition of Designing Data-Intensive Applications, still holds entirely. But the field has moved since 2017. The final lesson steps back to ask what the 2nd edition adds — cloud/serverless storage-compute separation, event sourcing, vector search, durable execution, local-first sync, formal methods, and a dedicated ethics chapter — and where each extends the foundations you just used.
Interview prompts
- What is the single most important clarifying question for a metrics backend, and why? (§1 — cardinality: the count of distinct series drives storage, partitioning, query cost, and the failure modes; everything else is downstream of it.)
- Why partition by series id rather than by timestamp? (§3.2 — hashing series id spreads ~700k writes/sec evenly and avoids the L11 timestamp hot end where every current write piles onto the newest partition; series-keyed reads stay single-hop.)
- How do you answer "p99 by region over the last day" across millions of series in under 300 ms? (§2, §3.5 — query the coarse rollup tier and merge mergeable quantile sketches (t-digest / histograms) per shard; never sort raw samples and never average means.)
- Why does this store tolerate loss while a model registry does not? (§3.6, §5 — telemetry's invariant is timeliness and availability, not integrity (L20); a dropped scrape or an approximate p99 is acceptable, so load-shedding and sketches are correct, whereas a registry's every write must survive.)
- What makes old data cheap without losing usefulness? (§2, §3.4 — downsampling raw → 1-min → 1-hour materialized aggregates (L04), so a year of history costs less than a week of raw while still serving long-range dashboards.)
- A new label explodes the series count overnight — what breaks and what saves you? (§4 — ingest rate and hot-tier memory spike, consumers lag, query p99 climbs; a cardinality limiter drops the runaway label and per-tenant quotas shed low-priority series, while the log buffer turns the burst into graceful lag.)
- Why must the rollup tier store histograms, not just averages? (§3.4, §3.5 — percentiles are not recoverable from means; only a mergeable distribution representation (sketch / bucket histogram) lets a scatter-gather merge per-series results into a region-wide p99.)