Debugging CPU, memory, and IO incidents
An incident is system design run backwards under a stopwatch. You are not asked to build the system — you are asked to find, in minutes, the one saturated resource that explains a symptom users are already feeling, and to stop the bleeding before you understand why it started.
The hinge: why incidents are non-linear
Engineers reach for incidents as if they were puzzles — "what changed, what's the root cause?" — and burn ten minutes reading code while the queue fills. The forcing fact is that systems near saturation behave nothing like systems at rest. Reuse lesson 02's utilisation cliff directly: model the hot resource as an M/M/1 queue, utilisation ρ = λ/μ, and response time is the idle service time multiplied by 1/(1−ρ).
| ρ (utilisation) | 1/(1−ρ) — wait multiple | What an operator sees |
|---|---|---|
| 0.5 | 2× | Healthy. Dashboards flat and boring. |
| 0.8 | 5× | p99 starting to fray; still "green". |
| 0.95 | 20× | Latency obviously bad; CPU graph "only" at 95%. |
| 0.99 | 100× | Effective outage. A 4% load bump did this. |
Read the jump from ρ=0.95 to ρ=0.99: only 4% more load, but the wait multiple goes from 20× to 100× — a 5× latency increase. This is the whole reason incidents feel like cliffs rather than slopes, and why "CPU is only at 95%, that's fine" is the most expensive sentence in the room. And M/M/1 is the optimistic case — real, bursty workloads (the M/G/1 variance factor) make the blow-up start even earlier. The operational consequence: your job in the first five minutes is to identify which resource has crept up the curve, not to explain how it got there.
Clarify the contract — what incident response must do
Treat the incident process as a product with a hard SLA on its own latency (time-to-mitigate). It must:
- Detect from the symptom, not the cause — alert on what the user feels (SLO burn: latency, error rate, availability), not on every internal counter.
- Localise the blast radius fast — region / shard / tenant / deploy version — so you mitigate the smallest blast radius that contains the damage.
- Mitigate before it understands — restore service first (rollback, shed, flag off), root-cause after.
- Preserve evidence — a timeline, deploy markers, a heap dump or profile, so the postmortem is possible.
- Not become the load — debugging a saturated system must cost near-zero on the hot path (this is DDIA Ch. 8's partial-failure world: the act of probing can tip a degraded node over).
And explicitly what it need not do: it need not find the root cause to succeed. A rolled-back, healthy service with an unexplained bug is a resolved incident and an open investigation — those are two different clocks. Conflating them is the classic junior error.
Put numbers on it
The numbers that steer the response, not the code:
Concretely: a service instance that comfortably serves at μ = 500 req/s sitting at λ = 475 req/s is at ρ=0.95 (20× idle latency). A retry storm or a 5% traffic bump takes it to λ=495, ρ=0.99, 100× latency — a 5× degradation from a 4% trigger. The same arithmetic says debug tooling must be cheap: if introspection adds even 5% load to a node at ρ=0.95, you push it to ρ≈1.0 and the queue diverges. That is why you sample (1-in-1000 traces, replica-side log queries), never full-fidelity capture on the hot path.
The diagnostic surface — small and symptom-shaped
The API of incident response is four read/act operations. Notice none of them is "read the source code."
| Operation | What it answers |
|---|---|
SLO / golden dashboards | Is the user hurting, and how much budget is left? (the symptom) |
blast-radius query | Which region / shard / version / tenant is affected? (the cut) |
saturation panels | Which resource is near 1.0: CPU, mem, disk/net IO, queue depth, connection pool? |
mitigate action | Rollback / traffic-shift / shed / feature-flag — the lever (lesson 13). |
The data model behind them is a small, intentional set of signals: an SLI time-series per user-facing flow, a resource-utilisation series per node, a deploy-version marker overlaid on every graph, and a trace span sampled at <1%. The deploy marker is the cheapest, highest-leverage object on the list — most incidents correlate with a change, and an annotated timeline turns "when did this start?" into a glance.
Linearized design — the symptom-first decision tree
Walk an incident the way a good responder actually walks it: from the outside in, symptom → radius → resource → lever. Each step narrows the next.
- 1. Start at the SLO burn. Page on user-visible symptoms (latency p99, error rate, availability), never on raw CPU. CPU at 90% with happy users is not an incident.
- 2. Cut the blast radius. Is it one region? one shard? one tenant? one deploy version? The radius is the single most diagnostic fact — a version-scoped symptom is almost certainly a deploy; a shard-scoped one is a hot key (lesson 07).
- 3. Find the saturated resource. Within the radius, which signal is near 1.0 — CPU, memory, disk/net IO, queue depth, or connection-pool occupancy? This is where you locate the cliff.
- 4. Correlate, lightly. Overlay deploy markers, config changes, traffic shifts, dependency health. One glance, not a code review.
- 5. Mitigate. Pull the cheapest lever that restores the SLO: rollback the deploy, shift traffic off the bad region, shed load, flip the feature flag (lesson 13).
- 6. Capture, then investigate. Snapshot the timeline and evidence; root-cause runs on a separate clock, after recovery.
Deep dives
Symptom-first, not cause-first
The instinct is to start from a hypothesis ("I bet it's the new query") and go looking for confirming evidence — which is exactly how you spend twenty minutes confirming a theory that was wrong. The discipline is to start from what the user feels and let the system tell you where it hurts. The SLO burn is the symptom; the blast radius is the bisection; the saturated resource is the diagnosis. You only form a causal theory at step 4, after the data has already pointed at a resource and a radius. This is DDIA Ch. 1's reliability framing made operational: you defined an SLO precisely so that, in an incident, "is this bad?" is a measured fact and not a debate — and so that the alert that wakes you is tied to user pain, not to an internal counter that may be perfectly benign at 90%.
Mitigate before you root-cause
These are two clocks. The mitigation clock runs against the SLO error budget and is measured in minutes; the investigation clock runs against engineering understanding and can take days. Conflating them — refusing to roll back until you understand the bug — burns budget you cannot get back. Rollback, traffic-shift, and load-shedding (lesson 13) all restore service without requiring understanding: a rollback returns you to a known-good version regardless of why the new one broke; shedding drops the marginal load that pushed ρ over the cliff regardless of what the marginal request was doing. Understanding comes after, from the evidence you preserved.
The one caveat that earns senior points: rollback is not always free. If the broken deploy ran a forward-only schema migration, rolling back the code against the new schema can corrupt data — which is why migrations must be backward-compatible (expand/contract), so the code and schema can move independently. Name that exception and you've shown you know rollback is a lever with its own failure mode, not a magic button.
The non-obvious saturations: flat CPU, slow service
The reason "look at CPU first" is junior advice is that the two most common production saturations do not show up on the CPU graph at all. A thread is not consuming CPU while it is blocked waiting — for a connection, for a lock, for a downstream response. So:
- Connection-pool exhaustion. Every worker is blocked waiting to borrow a DB connection from a pool of, say, 50. CPU sits at 20% because the threads are parked, not running — yet latency is vertical because the pool itself is the M/M/1 queue at ρ→1. The saturated resource is the pool, and the only signals that reveal it are pool-occupancy and wait-for-connection time, never CPU. (Lesson 02's Little's Law sizes the pool: in-flight = λ·W; if the pool is below that, requests queue in front of it.)
- Downstream latency. A dependency slows from 5 ms to 200 ms. Your service's CPU is flat — it is just waiting — but its own latency tracks the dependency, and its in-flight count climbs (Little's Law again: same λ, bigger W, more concurrency held), eventually exhausting threads. This is DDIA Ch. 8's partial failure: a degraded dependency, not a dead one, is the dangerous case, because health checks pass while latency rots. The fix is a timeout + circuit breaker (lesson 13) so you fail fast instead of holding threads.
The unifying lesson: latency is the symptom, and the saturated resource may be a queue you forgot was a queue — a connection pool, a thread pool, a downstream's own queue. CPU, memory, and disk are only three of the resources that can hit ρ=1.
Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Rollback vs debug live | Fast, understanding-free recovery | Less live evidence; risky if migration is forward-only | Symptom correlates with a recent deploy version |
| Sampling vs full logging | Near-zero hot-path overhead | May miss a rare event | Hot incident — the system can't afford the probe |
| Load-shed vs scale-out | Shed: instant ρ drop. Scale: more capacity | Shed drops real users; scale is slow and can hide the bug | Shed to survive now; scale only if it's a genuine traffic rise |
| Fail open vs fail closed | Open: availability. Closed: protection/safety | Open risks leakage/bad data; closed is an outage | Open for non-critical deps; closed where correctness > uptime |
The sharpest one is rollback vs debug live. Engineers resist rollback because it feels like giving up the crime scene — and they're not wrong that a rolled-back system tells you less. But the queueing math wins the argument: every minute at ρ=0.99 is 100× latency for real users, and the rollback buys that back in one action without needing a single hypothesis to be correct. Preserve evidence cheaply (a heap dump, a few sampled traces, the deploy marker) then roll back. The crime scene is reconstructed from artefacts, not from leaving the building on fire.
Failure modes
| Failure | Mitigation |
|---|---|
| Debug query overloads production | Run heavy queries on a read replica; sample traces <1%; cap/timeout expensive introspection. The probe must never tip ρ→1. |
| Rollback corrupts a migration | Backward-compatible (expand/contract) migrations so code and schema move independently. |
| Alert storm | Alert on symptoms (SLO burn) not causes; suppress derivative alerts; route by service ownership. |
| No baseline | Golden dashboards with deploy markers, kept current — you can't spot an anomaly without a "normal". |
| Retry storm amplifies the incident | Exponential backoff + jitter and circuit breakers (lesson 13); retries are added λ straight onto the cliff. |
Interview Q&A
- Latency tripled, CPU is flat, error rate is normal — where do you look? (senior answer) Not at CPU — flat CPU with high latency means threads are blocked, not busy. I look at queue depth, connection-pool occupancy and wait-for-connection time, and downstream dependency latency. The most likely culprits are connection-pool exhaustion (the pool is the M/M/1 queue at ρ→1) or a downstream that slowed from milliseconds to hundreds of milliseconds, holding threads via Little's Law (in-flight = λ·W grows as W grows). Mitigate with a timeout + circuit breaker so we fail fast instead of holding threads.
- Why mitigate before root-causing? (senior answer) Two different clocks. Mitigation runs against the error budget in minutes; investigation runs against understanding in days. Rollback and shedding restore service without requiring a correct hypothesis — a rollback returns to known-good regardless of the bug. At ρ=0.99 every minute is 100× latency for real users, so I buy that back first and investigate from preserved evidence.
- The dashboard shows CPU "only" at 95%, so we're fine, right? (senior answer) No — that's lesson 02's cliff. At ρ=0.95 response time is already 20× idle, and going to ρ=0.99 (a mere 4% more load) makes it 100× — a 5× degradation from a tiny trigger. 95% utilisation is not headroom, it's the edge of the cliff. Real workload variance makes it worse, so I plan capacity around ρ≈0.5–0.7.
- How do you debug a service at 99% CPU without making it worse? (senior answer) Carefully — at ρ=0.99 even a 1% probe overhead can push it to ρ≈1.0 and diverge the queue. So I move heavy work off the hot path: query logs on a replica, sample traces at <1%, read existing metrics rather than capturing new full-fidelity data. Better still, shed load first to drop ρ, then introspect the now-breathing node.
- How do you localise an incident quickly? (senior answer) Bisect by blast radius before touching code. Is it one region, one shard, one tenant, one deploy version? A version-scoped symptom is almost certainly the deploy (roll back); a shard- or tenant-scoped one is a hot key (lesson 07); a fleet-wide one points at a shared dependency or a traffic shift. The radius tells me both the cause class and the right lever.
- Memory is climbing steadily — leak or not? (senior answer) Distinguish unbounded growth from bounded-but-large. A true leak grows with no ceiling and survives load drops; cache growth plateaus at a configured cap; a queue backlog grows because consumption fell behind production (a downstream slowdown) and recovers when it catches up. I look at heap vs RSS, allocation rate, GC frequency, and eviction behaviour. The mitigation differs: leak → roll back / restart; backlog → fix the consumer or shed producers.
- When is rolling back the wrong move? (senior answer) When the deploy ran a forward-only schema migration — rolling the code back against the new schema can corrupt data. That's why migrations should be expand/contract (backward-compatible), so code and schema move independently and rollback stays a safe, free lever. If I can't roll back safely, I mitigate another way: feature-flag the new path off, or shift traffic.
- How do you keep retries from amplifying an incident? (senior answer) Retries are extra λ added straight onto a queue that's already at the cliff — a tight retry loop turns a blip into a thundering herd. The defences are exponential backoff with jitter, a retry budget, and circuit breakers that stop calling a degraded dependency entirely (lesson 13). During the incident I check whether the "traffic spike" is real demand or my own clients retrying.
Related lessons
This case is the operational payoff of three foundation lessons. It reuses rather than re-derives them. From lesson 02 (latency & queueing) comes the entire premise — the 1/(1−ρ) cliff that makes incidents non-linear, and Little's Law that explains why a slow downstream silently exhausts thread and connection pools. From lesson 13 (fault tolerance) come the mitigation levers themselves — rollback, load-shedding, circuit breakers, backoff-with-jitter — which restore service without requiring a root cause. From lesson 16 (observability) comes the telemetry this whole method runs on: SLO-based alerting (symptom, not cause), golden dashboards with deploy markers, and sampled traces cheap enough to use on a hot system. DDIA Ch. 1 grounds the SLO/reliability framing and Ch. 8 the partial-failure reality that a degraded dependency is more dangerous than a dead one.