Observability & operability
You cannot operate what you cannot see. A request now crosses a dozen services (02's fan-out, 09's async hops) — when it is slow or wrong, there is no single process to attach a debugger to. The job is to ask arbitrary, unplanned questions of a live system from its outputs alone.
A distributed system fails in ways you did not predict — the unknown-unknowns. Monitoring answers questions you wrote down in advance: predefined dashboards and alerts for known failure modes. Observability is the stronger property that you can investigate the novel failure — reconstruct what happened and why — without shipping new code to collect new data. Monitoring is a subset; the gap between them is exactly the set of incidents you did not anticipate, which in a mature system is most of them.
The three pillars: something, where, why
There is no single signal that answers every question cheaply. The three pillars exist because they trade cardinality (how many distinct things you can distinguish) against cost in opposite directions. You need all three, and they answer different questions.
| Pillar | What it is | Cardinality / cost | Answers | Bad at |
|---|---|---|---|---|
| Metrics | Numeric time series: counters, gauges, histograms, sampled at fixed intervals | Low & bounded cardinality; cheap (a histogram is a few floats per scrape); aggregatable | How many, how fast, is it trending; basis of dashboards & alerts | "Why this one request?" — it has thrown away per-request identity |
| Logs | Discrete timestamped events, ideally structured (key=value, not free text) | High cardinality, high detail; expensive at volume; costly to index | What exactly happened in this case — the forensic record | Aggregate trends; counting from logs is slow and pricey |
| Traces | The causal path of one request across services as a tree of spans sharing a trace-id | High cardinality per trace; sampled because full fidelity is unaffordable | Where did the latency go; which hop failed in a fan-out | Rare events if head-sampled; not a counting tool |
Metrics tell you something is wrong. Traces tell you where. Logs tell you why. An interviewer who hears you reach for the right pillar per question — not "log everything and grep" — knows you have run systems. Logging at metric volume is the classic way to set fire to the observability bill (see cardinality, below).
SLI, SLO, SLA, and the error budget
Reliability is not a feeling; it is a measured quantity with a target. Four terms, often muddled, are actually a clean ladder. They tie directly to 02's percentiles (the SLI is usually a tail threshold) and 11's nines.
| Term | Definition | Example |
|---|---|---|
| SLI | The measured indicator — a ratio of good events to valid events | fraction of requests served in < 200 ms and 2xx |
| SLO | The internal target on the SLI over a window; set tighter than the SLA | 99.9% over a rolling 28 days |
| SLA | The external contract with penalties (refunds, credits) if breached | 99.5% / month or money back |
| Error budget | 1 − SLO — the allowed failure, a spendable quantity | 0.1% of 28 days ≈ 40 min |
The error budget is the brilliant reframe. "Be more reliable" is an argument that never ends — reliability and velocity look like enemies. "You have 40 minutes of badness this month" is a budget. You can spend it: ship faster, take launch risk, run a risky migration. Burn it too fast and the policy is automatic — freeze launches until the rolling window refills. This aligns the dev team (who want to ship) and the SRE team (who want to be safe) on one number instead of pitting them against each other.
budgetminutes = (1 − SLO) × windowminutes ⇒ 99.9% over 28 d = 0.001 × 40320 ≈ 40 min
| SLO | Error budget (per 30 d ≈ 43200 min) |
|---|---|
| 99% | 1% → ~432 min ≈ 7.2 h |
| 99.9% ("three nines") | 0.1% → ~43 min |
| 99.99% ("four nines") | 0.01% → ~4.3 min |
| 99.999% ("five nines") | 0.001% → ~26 s |
SLO = 99.9% over a 28-day window ⇒ budgeted error rate = 0.1%, total budget ≈ 40.3 min of "bad time" (40320 valid-minutes × 0.001). Suppose a bad deploy pushes the live error rate to 2%.
burn-rate multiple = current error rate ÷ budgeted error rate = 0.02 ÷ 0.001 = 20×
At 20× you are spending the whole 28-day budget in 28 / 20 = 1.4 days. If only 40% of the budget remains, time-to-exhaustion ≈ 0.40 × 1.4 d ≈ 13.4 h. That is a fast burn: page now. A 1.5× burn that drains the budget over many days is a slow burn: open a ticket, do not wake anyone.
Alerting: page on symptoms, not causes
The cardinal rule: alert on user-facing symptoms (SLO burn), not on internal causes. A single CPU spike or one node going unhealthy is a cause — and under 11's redundancy it is very likely self-healing. Paging on it produces pager fatigue: the on-call learns to ignore the alert, and misses the real one. Page on "error rate is burning the budget fast" — a symptom the user actually feels.
Burn-rate alerting & the method acronyms
- Multi-window burn-rate alerts. A fast burn (e.g. ≥14.4× over 1 h and 5 min) pages now; a slow burn (e.g. ≥3× over 6 h) files a ticket. The dual window suppresses flapping: the short window must also be hot for the page to fire.
- RED for request-driven services: Rate, Errors, Duration. Three signals per service is enough to know if it is healthy.
- USE for resources (CPU, disk, queue): Utilization, Saturation, Errors. Saturation is the leading indicator — queue depth rises before latency does (recall 02's queueing).
"A dashboard with 200 graphs and not one alert that matters." Lots of graphs is not observability — it is a wall you stare at after a customer tells you something broke. If a graph is not tied to a question you would ask during an incident, or to an SLO, it is decoration.
Distributed tracing mechanics & the cost driver
A trace-id (and a root span-id) is generated at the edge — 12's gateway — and propagated via headers (e.g. W3C traceparent) through every synchronous hop. Across 09's async boundaries it rides in the message metadata, so the consumer continues the same trace minutes later. Each service records its own span (start, duration, status, attributes) and ships it; a backend assembles the spans into a tree by parent-id.
Sampling is unavoidable — keeping every trace at scale costs as much as the system itself.
- Head-based: decide at ingress (e.g. keep 1%). Cheap and stateless, but you discard the decision before you know if the trace was slow or errored — so you miss the rare interesting ones.
- Tail-based: buffer all spans of a trace, decide after seeing the whole thing — keep it if it errored or breached a latency threshold, drop the boring 99%. Catches the interesting traces, but needs a buffering tier (more infra, memory, and per-trace state).
Cardinality is the bill. A label's cardinality is its number of distinct values; storage scales with the product of label cardinalities. Put user_id on a metric with 10M users × 5 other labels and you have created tens of millions of time series for one metric — the database falls over and the invoice explodes. High-cardinality identity (user-id, request-id) belongs in logs and traces (which you sample), never as a metric label (which you must keep for all time). Observability is often a double-digit percentage of total infra spend, so you sample and budget it like any other capacity.
Interactive: error-budget burn calculator
Set an SLO, a window, and the success rate you are currently observing over that window. Read off the budget you were granted, how much you have burned, what remains, the burn-rate multiple, and how long the remainder lasts at this rate.
Interview prompts you should be ready for
- What is the difference between monitoring and observability? (senior answer) Monitoring answers questions you defined in advance (known failure modes via dashboards/alerts); observability is the property that you can answer new questions about behaviour from existing outputs without shipping code. Monitoring ⊂ observability; the value is in the novel incident.
- When do you use a metric vs a log vs a trace? (senior answer) Metric for "how many / how fast / trending" — cheap, aggregatable, bounded cardinality, drives alerts. Trace for "where in the fan-out did latency/failure occur" — per-request span tree, sampled. Log for "what exactly happened in this case" — structured, high detail, expensive. Metrics say something is wrong, traces say where, logs say why.
- Walk me through SLI / SLO / SLA / error budget. (senior answer) SLI = measured ratio (good/valid events); SLO = internal target on it over a window, set tighter than the SLA; SLA = external contract with penalties; error budget = 1 − SLO, the allowed failure. The budget turns reliability into a spendable quantity that gates launch risk.
- Why alert on symptoms rather than causes? (senior answer) Causes (CPU spike, one bad node) are noisy and often self-heal under redundancy; paging on them causes pager fatigue and missed real incidents. Page on user-facing SLO burn via multi-window burn-rate rules: fast burn pages, slow burn tickets.
- What is a burn-rate alert and why two windows? (senior answer) Burn rate = observed error rate ÷ budgeted error rate; ≥14.4× drains a 30-day budget in ~2 days. Two windows (long to define severity, short as a "still happening now" guard) stop the alert flapping when the spike has already passed.
- Head-based vs tail-based sampling — trade-offs? (senior answer) Head decides at ingress: cheap, stateless, but discards before knowing if the trace was interesting, so it misses rare errors. Tail buffers the whole trace then keeps errored/slow ones: catches the signal but needs a stateful buffering tier and more infra/memory.
- A metric's storage cost just exploded — what happened? (senior answer) Someone added a high-cardinality label (user_id, request_id, URL with IDs). Series count scales with the product of label cardinalities; unbounded labels create millions of series. Move per-request identity to logs/traces (sampled); keep metric labels bounded.
- How do you propagate context across an async boundary? (senior answer) Carry trace-id/span-id in message metadata/headers (e.g. W3C traceparent), not just the sync call chain. The consumer extracts it and continues the same trace, so a Kafka-delayed step minutes later still appears in one trace tree.
Instrument for unknown-unknowns: metrics for "something" (cheap, bounded, alertable), traces for "where" (per-request, sampled), logs for "why" (structured, forensic). Express reliability as an SLO and spend its error budget; alert on symptom burn-rate, never on causes. Watch cardinality — it is the bill — and propagate trace context across every hop, sync and async, so a single request stays one story end to end.