Design metrics monitoring and alerting

A metrics system is a write-heavy time-series database wearing a human contract. Its real adversary isn't request volume — it's cardinality: the number of distinct series you're forced to keep alive, which one careless label can blow up by a thousand-fold overnight.

0. The hinge

The hinge is the sentence that turns a generic "build monitoring" prompt into a forced architecture. Here it is: storage and query cost are driven by series count, not sample count, and series count is set by label cardinality — so the design must bound cardinality at ingestion or it dies. A candidate who leads with "we'll shard the writes and use a TSDB" has skipped the hinge; the interviewer is waiting to hear you reason about the cardinality bomb before you draw boxes.

1. Clarify the contract

Treat the prompt as a product contract before a component diagram. The system must:

• Ingest metrics (counters, gauges, histograms) from thousands of hosts and services, via scrape (pull) or push.
• Store time-series samples efficiently, with recent data at high resolution and old data downsampled.
• Query dashboards over both recent (seconds–hours) and historical (weeks–years) windows.
• Evaluate alert rules reliably on a schedule, tolerating ingestion delay without false pages.
• Bound cardinality and cost — reject or relabel runaway dimensions.

And explicitly what it need not do: it is not a system of record. Monitoring data is lossy-tolerant — dropping 0.1% of samples during an incident is acceptable; an unavailable monitoring system during that same incident is not. It is also not a tracing or logging system; high-cardinality per-request detail belongs in traces/logs (different storage model), and conflating the two is exactly how the cardinality bomb gets planted. This relaxed durability is what lets us favour append-optimised, eventually-consistent storage over a transactional database.

2. Put numbers on it — the cardinality bomb

The whole case lives or dies on one calculation, so do it explicitly. Cardinality is multiplicative across labels:

series = ∏ (distinct values of each label) , summed over each metric name

Consider a deceptively innocent metric, http_requests_total, with a per-user label:

Two hundred million series from one metric. Now price the churn. Each series emits a sample on every scrape; at one scrape every 15 s and ~8 bytes per stored sample (timestamp delta + compressed value, roughly Gorilla-style):

samples/day = 200M series × (86,400 s / 15 s) = 200M × 5,760 ≈ 1.15 × 10¹² samples/day

raw sample bytes = 1.15 × 10¹² × 8 B ≈ 9.2 TB/day (before compression)

But the samples are the cheap part — Gorilla compression squeezes them to ~1–2 bytes amortised. The killer is the index. Every one of the 200M series needs an inverted-index posting (label → series-id) plus an in-memory head chunk. Index overhead is on the order of hundreds of bytes to a kilobyte per series; even at a conservative ~500 B/series that is 200M × 500 B ≈ 100 GB of index alone, churning every time series appear/disappear (a process called churn, e.g. when users are active for only part of the day). Call it ~100 GB/day of index churn just to track which series exist — independent of how many samples each holds. That index has to fit largely in RAM to keep queries fast, so this metric alone would demand a multi-hundred-GB memory footprint per shard.

Now bound the label instead. Replace user_id with a tier label (free / pro / enterprise):

Same observability question — "how is request volume split across endpoints and customer tiers?" — answered by 60 series instead of 200 million. The information you actually act on is almost always aggregate; per-user breakdowns belong in logs or traces, queried on demand, not in a metric that must materialise every combination forever. This single contrast is the entire lesson. Whenever a label's distinct-value count is unbounded or grows with traffic (IDs, emails, URLs, full error messages), it does not belong in a metric.

3. Define the surface area

Keep the API small and aggregate-shaped; it must not leak the shard or rollup layout.

4. Model the data

The data model is the first architecture. A sample is tiny; its identity is everything.

A series is identified by the hash of its full label set — its fingerprint. The fingerprint is the partition key (more on this below) and the index key. Samples for a series are stored as a compressed, time-ordered run; the index maps each label name=value to the set of fingerprints carrying it, so a query like requests{tier="pro"} intersects posting lists to find matching series, then scans their sample runs over the requested time range.

5. Linearized design

Walk an event in the order it actually moves, and the first bottleneck — cardinality — surfaces on its own.

1. Collect. Agents/scrapers pull from targets (stable fleets) or a push gateway accepts pushes (short-lived jobs). Pull gives the system control over scrape cadence and a free up/down signal; push suits batch jobs that die before any scrape.
2. Validate & relabel. At ingestion, enforce a per-tenant series budget and drop or rewrite high-cardinality labels (request_id, raw url). This is the cardinality firewall — the cheapest place to stop the bomb.
3. Shard by fingerprint. Hash the label set to route every sample for a series to the same node, so a series is contiguous and queryable without cross-node scatter.
4. Hot write-store. Recent samples land in an append-optimised, in-memory head block, periodically flushed to immutable on-disk blocks. Writes are sequential; reads of "now" are cheap.
5. Downsample into rollup tiers. A background job rolls raw → 5 m → 1 h aggregates, each with its own retention. Long-range dashboard queries read the coarse tier and touch a fraction of the data.
6. Evaluate alerts. A scheduler runs each rule as an instant_query on aligned windows, applies a for: duration, and persists alert state before routing notifications.

6. Deep dives

6.1 Cardinality is the dominant cost driver

Section 2 priced it; here is why it dominates. A TSDB's cost has two terms: the per-sample cost (compressed, tiny — Gorilla gets ~1.37 bytes/sample) and the per-series cost (index posting, head chunk, label strings — hundreds of bytes, mostly RAM-resident). Sample cost scales with scrape frequency; series cost scales with cardinality. Because the per-series constant is two-to-three orders of magnitude larger and lives in memory, cardinality wins the cost equation almost always.

The trap is that cardinality is multiplicative and silent. Adding a single label that looks harmless — request_id on an error counter — turns N series into N × (requests) series, because every request is a new value. There is no error, no rejection by default; the series count just climbs, the index outgrows RAM, head-block memory balloons, and one node OOMs. This is why the firewall lives at ingestion (DDIA Ch. 6, partition by series fingerprint, also gives us the natural enforcement point): you cap series-per-tenant and reject or relabel before the write ever touches storage, because once a bad label is written, removing it still leaves the index churn already paid.

6.2 Time-series storage is append-heavy and time-ordered — which is LSM-friendly

Samples arrive in roughly monotonic timestamp order and are essentially never updated in place. That access pattern is the textbook fit for a log-structured merge tree (DDIA Ch. 3): buffer writes in an in-memory structure (the "head"), flush to immutable sorted blocks, and compact/merge blocks in the background. There are no random in-place updates, so there is no B-tree write amplification from page splits; sequential append throughput is the whole game.

The model is also columnar within a series (Ch. 3's column-oriented storage): a series is a column of (Δt, value) pairs, delta-of-delta encoded for timestamps and XOR-encoded for values, so a CPU-cheap, highly compressible run. Old immutable blocks are exactly what the downsampler reads to build rollup tiers, and exactly what we can age out by simply deleting whole blocks past retention — no tombstones, no vacuum. The relaxed durability from §1 is what permits this: we can lose the un-flushed head on crash (replayed from a short write-ahead log, or just tolerated) rather than pay for synchronous fsync per sample.

6.3 Alert on symptoms (SLO burn), not causes

The third deep dive is about the human contract. There are two ways to write an alert rule, and the choice decides whether the pager is trusted or ignored.

• Cause alerts page on internal mechanisms: "CPU > 80%", "queue depth > 1000", "a replica restarted". Most of these never reach a user — CPU at 85% with latency fine is not an incident. Page on enough of them and humans learn the pager is noise and stop reading it. That is the failure: an ignored pager is worse than no pager.
• Symptom / SLO-burn alerts page on user-visible pain: "error rate exceeds the SLO budget", "p99 latency > 300 ms for 5 min". They fire only when something a user would notice is actually wrong, regardless of the underlying cause.

The disciplined form is multi-window burn-rate alerting on an error budget. If your SLO is 99.9% (budget = 0.1% errors over 30 days), define burn rate = observed error rate ÷ budget rate. A fast-burn rule pages when you're burning the monthly budget in hours (e.g. burn > 14.4 over a 1 h and 5 m window — the short window confirms it's current, the long window suppresses single-spike noise); a slow-burn rule opens a ticket when you'd exhaust it in days. The windowed evaluation here is DDIA Ch. 11 stream processing: the evaluator runs each rule over a sliding time window on a fixed cadence, which is precisely a windowed aggregation over the sample stream, and the for: duration plus hysteresis is how we suppress flapping at the window edge.

7. Trade-offs

The sharpest one is flexible labels vs strict schema. Teams love arbitrary labels for debugging, and a strict schema feels bureaucratic — until someone labels by request_id and the cluster OOMs. The senior resolution is not "ban labels" but "flexible labels behind a budget": allow any label, but enforce a per-metric, per-tenant series cap at ingestion that rejects new series once exceeded and pages the owning team. You get debugging freedom with a hard ceiling on the blast radius — the same admission-control philosophy as foundation lesson 15, applied to series instead of requests.

8. Failure modes

9. What to say in the interview

Related foundation lessons & cases

This case implements foundation lesson 16 · Observability — that lesson defines metrics/SLOs as a concept; here we build the system that stores and evaluates them. The ingestion pipeline is built on the partitioned append-log of C23 (distributed message queue) — samples flow in over the same kind of append-only, fingerprint-partitioned log. Sharding by series fingerprint is lesson 07 · Partitioning's hot-shard problem applied to time series. And the alerting half exists to defend the latency/error SLOs introduced in foundation lesson 02 · Latency & throughput — burn-rate alerts are how you notice the utilisation cliff before users do.