Microservices, containers, deployment, and Lambda
Every infrastructure choice is a choice about who owns the failure. A boundary you draw in the architecture is a boundary you must defend at runtime — splitting a service splits the transaction, and going serverless trades server management for a cold-start tax you pay in the latency tail.
0. The interview hinge
The hinge — the moment the prompt stops being generic and the architecture becomes forced — is this: a boundary that buys deploy independence also destroys the local transaction that used to span both sides. If you split checkout from inventory so each team ships on its own cadence, you can no longer wrap "decrement stock" and "create order" in one ACID transaction. You bought deploy granularity and you now owe a saga or a two-phase commit (DDIA Ch. 9). The candidates who fail this case are the ones who draw the box-and-arrow diagram without naming that debt.
BEGIN…COMMIT solved for free. The mirror-image mistake is "serverless scales to zero, so put everything in Lambda" — which collapses the instant you hit sustained high QPS or a latency-sensitive endpoint where the cold-start tail is unacceptable. Both are real tools; both have a regime where they stop making sense, and naming that regime is the senior signal.
1. Clarify the contract
Treat this as a product contract before a component diagram. The system must:
- Decompose along ownership boundaries — services map to teams and data domains, not to database tables.
- Package deployably — every release is an immutable artifact with pinned dependencies, identical from laptop to prod.
- Deploy safely with bounded blast radius — a bad release must be caught on a small slice of traffic and rolled back automatically.
- Scale capacity to load — including scaling to zero for spiky, event-driven work.
- Know when each tool stops paying — when a split becomes a distributed monolith, when serverless economics flip.
And explicitly what it need not do: it need not split every domain into its own service on day one (a modular monolith is a perfectly senior answer for an early product), and it need not make every endpoint serverless. The contract is about choosing boundaries deliberately, not maximising the number of them.
2. Put numbers on the shape
The numbers that steer this architecture are all about start-up time and the tail it produces. Start-up cost is the hidden axis behind every choice here:
The cold-start number is the load-bearing one. A warm Lambda dispatches in about a millisecond; a cold one — fresh container, runtime init, your init code, dependency load — costs 100–800 ms depending on language and package size. That gap is invisible in the average and lethal in the tail. Make it concrete: you serve 1000 req/s, and suppose 1% of invocations are cold (a steady trickle of scale-up and idle-recycling). That is 10 cold starts per second. At p99 you are looking at the 10th-slowest request out of every thousand — exactly the population the cold starts live in. The tail is not a rounding error; it is your p99.
Let's compute the effective p99 directly. Take warm latency = 5 ms, cold-start penalty added on top of warm. The effective p99 is whichever request sits at the 99th percentile, and the cold population determines it:
| Cold-start % | Cold penalty | Cold reqs / s (of 1000) | Sits at percentile | Effective p99 |
|---|---|---|---|---|
| 0.1% | +400 ms | 1 | p99.9 | ~5 ms (warm) — cold hides above p99 |
| 1% | +400 ms | 10 | p99 boundary | ~405 ms — the cold tail is p99 |
| 1% | +150 ms | 10 | p99 boundary | ~155 ms |
| 5% | +400 ms | 50 | p95–p99 all cold | ~405 ms at p99, and now p95 ≈ 405 ms too |
Read the second row carefully: 1% cold at +400 ms turns a 5 ms service into a 405 ms p99 — an 80× degradation at the percentile your SLO actually measures. This is the single most important fact about serverless latency, and it is why "1% cold starts, who cares" is wrong: 1% lands precisely where p99 is read. Below ~0.1% the cold tail hides above p99 and you are fine; above ~1% it dominates. The mitigation (provisioned concurrency / keep-warm) is really just "pay to push the cold fraction below the percentile you report."
3. Define the surface area
Keep the surface small and express the operation, not the topology:
| API / operation | Why it exists |
|---|---|
POST /orders (service endpoint) | The product operation; the caller never learns which services it fans out to. |
GET /healthz & /readyz | Liveness vs readiness — the rollout controller gates on ready, the orchestrator restarts on not-live. |
POST /deployments {version, strategy} | Trigger a rolling / blue-green / canary rollout; the control plane owns the funnel. |
invoke(event) (function) | Stateless, event-shaped unit of work — no assumption about what ran before it. |
4. Model the data — and the boundary it implies
The data model is the first architecture. The rule that prevents the distributed monolith: a service owns its data and no other service touches that store directly. The moment two services share a table, they are one service that merely happens to deploy as two — coupled at the schema, unable to evolve independently. So the artifacts here are about isolation and immutability:
5. Linearized design
Walk a release through the system — that is where the boundaries and their costs show up in order:
- 1. Draw boundaries along domains and teams, not tables. Start with a modular monolith; extract a service only when a piece needs independent deploy or scaling (lesson 17).
- 2. Package each service as an immutable image — same artifact, laptop to prod. Config is injected, never baked, so "config drift" can't happen silently.
- 3. Ship through a canary: route ~5% of traffic to the new version, hold for a metrics window, gate on error rate / latency, then ramp to 100% — or auto-roll-back.
- 4. Route via service discovery + load balancing; every inter-service call gets a timeout, a retry budget, and a circuit breaker (lesson 13).
- 5. For bursty / event-driven work with no warm-state assumptions, use serverless — and measure its cold-start fraction against your p99 SLO before committing latency-sensitive paths to it.
- 6. Emit metrics, traces, and deploy markers at every boundary so a regression is attributable to a specific release (lesson 16).
The first bottleneck this exposes is the canary metrics gate: it is only as good as the window you wait. Too short and a slow-burn regression (a memory leak, a tail that only appears under cache-cold conditions) escapes to 100%; too long and every deploy crawls. That tension is the heart of the rollout deep dive below.
6. Deep dives
(1) The distributed-monolith anti-pattern
This is the trap that makes microservices net-negative, and it is worth deriving precisely. Suppose you have a monolith where placing an order does:
BEGIN; decrement inventory; insert order; charge balance; COMMIT;
One transaction. It either all happens or none of it does; the database's atomicity and isolation (DDIA Ch. 7) do the hard work for free. Now you split into orders, inventory, and billing services — a reasonable instinct, three teams, three deploy cadences. What did you actually do? You replaced three local statements that shared one transaction with three network calls that cannot. There is no BEGIN…COMMIT across three databases on three machines. So now:
- Inventory is decremented, then billing fails — you've sold stock you didn't charge for. Partial failure (DDIA Ch. 8) is now your problem, not the database's.
- To recover you need a saga: a sequence of local transactions with explicit compensating actions (release the inventory hold if billing fails), or a heavyweight two-phase commit that blocks and couples availability. This is precisely the distributed-transaction machinery DDIA Ch. 9 spends a chapter warning is hard — see lesson 12 for the saga/outbox mechanics.
The distributed monolith is the failure to notice this: you split the code into services but left the data and the transaction entangled — services still call into each other synchronously for every operation, still share a database, still must be deployed together because a schema change breaks all of them. You have paid the full cost of microservices (network latency, partial failure, distributed coordination — DDIA Ch. 8's cascades and timeouts) and bought none of the benefit (you still can't deploy or scale them independently). The diagnostic question: "can each service be deployed, scaled, and fail independently?" If no, you have a distributed monolith, and you should either re-merge or do the harder work of actually decoupling the data — DDIA Ch. 11's event-driven, async-decoupled style, where services react to events on a log instead of synchronously calling each other, is the way out.
(2) Serverless economics and cold-start sensitivity
Serverless is priced per-invocation × duration × memory, and its superpower is scale-to-zero: an endpoint that fires 100 times a day costs ~nothing when idle, with no server to patch. The economics are brilliant for spiky, low-duty-cycle work. They invert for sustained high QPS: a function billed per GB-second running flat-out 24/7 is simply a more expensive way to rent the CPU you'd get cheaper on a reserved container or VM — past some QPS crossover, the always-on instance wins on cost and on latency.
And latency is the second axis. From §2: warm ≈ 1 ms, cold ≈ 100–800 ms, and at 1000 req/s with 1% cold the effective p99 is ~405 ms — an 80× hit at the percentile your SLO reads. The mitigation menu, in order of cost: (a) shrink the package and lazy-init (cuts the cold penalty itself); (b) keep-warm pings (crude, fights idle-recycling); (c) provisioned concurrency — pre-warmed instances you pay for whether used or not, which is just buying your way back below the cold-fraction-that-matters threshold. Note that (c) erodes the scale-to-zero economics: if you must keep N instances warm to hit your p99, you are paying for an always-on fleet of N and have re-derived a container service the expensive way. That convergence is the tell that serverless has stopped paying — which is the sharpest senior question below.
(3) Rollout strategy and blast radius
The whole point of a deployment strategy is to bound the blast radius of a bad release — the fraction of traffic exposed before you can detect and reverse the damage. Three strategies, escalating in safety and cost:
| Strategy | Blast radius before detection | Rollback speed | Capacity cost |
|---|---|---|---|
| All-at-once | 100% — everyone sees the bug at once | Re-deploy old version (minutes) | None extra |
| Rolling | ~1/k at a time as instances cycle | Pause + roll back remaining | None extra |
| Canary | ~5% (the canary slice) for the metrics window | Shift 5%→0% instantly | Slightly more than the slice |
| Blue/green | 0% until cutover, then 100% (atomic flip) | Flip back instantly | 2× — full duplicate fleet |
Canary is the workhorse: route 5% of traffic to the new version, hold for a ~30-minute window so slow-burn regressions and traffic-cycle effects surface, gate on error rate and p99 against the baseline, then ramp 25→50→100% or auto-roll-back. The cost is honest: a bad deploy still hurts 5% of users for 30 minutes, and the rollout itself is slow. The window length is a real bias/latency trade — shorter catches less, longer ships slower — and it must be long enough to span at least one cache-warm cycle and your slowest-firing endpoint, or a leak escapes the gate. Blue/green is the choice when you cannot tolerate any degraded slice and can afford a duplicate fleet for the cutover instant.
7. Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Monolith / modular monolith | One ACID transaction, no network between modules, one deploy | Whole app ships together; can't scale parts independently | Early product, one team, unclear domain boundaries |
| Microservices | Independent deploy / scale / fault isolation per domain | Network latency, partial failure, sagas instead of transactions | Stable boundaries, multiple teams, parts scale differently |
| Containers vs VMs | ~seconds vs ~minutes to start; identical artifact everywhere; dense packing | Shared-kernel isolation (weaker than VM); orchestrator complexity | Service fleets that scale and roll often |
| Canary vs all-at-once | Blast radius capped at the canary slice; auto-rollback on metrics | Slower rollout; the slice still sees the bug for the window | Production services with real SLOs |
| Serverless vs long-running service | Scale to zero, no server management, pay per use | Cold-start tail in p99; per-GB-s cost flips at high QPS; runtime limits; statelessness | Spiky / event-driven / low-duty-cycle, latency-tolerant work |
The sharpest trade-off to narrate is microservices vs monolith, because its cost is the one candidates forget: you are not trading "simple" for "scalable," you are trading one local transaction for a distributed-coordination problem (DDIA Ch. 9). A modular monolith gives you clean internal boundaries and ACID transactions and one deploy — it is the right default until a specific module demonstrably needs to deploy or scale on its own. Extract that one module; don't shatter the whole app on principle. "We split it into 40 microservices" is a confession, not a credential, unless each of the 40 actually deploys and scales independently.
8. Failure modes
| Failure | Mitigation |
|---|---|
| Bad deploy reaches all users | Canary gate on error/p99 with automated rollback; blast radius capped at the 5% slice. |
| Service dependency cascade | Per-call timeouts, retry budgets (not unbounded retries), circuit breakers — DDIA Ch. 8 partial failure / cascades (lesson 13). |
| Partial failure mid-transaction | Saga with compensating actions or outbox-driven async; never assume a cross-service write is atomic (lesson 12). |
| Config drift | Immutable images + externalised, versioned config; the artifact that passed staging is byte-identical in prod. |
| Cold-start latency blows the SLO | Measure cold fraction vs p99; provisioned concurrency / keep-warm, or move the hot path to an always-on service. |
| Retry storm under partial outage | Exponential backoff + jitter, retry budgets, circuit breakers to stop the thundering herd (lesson 13). |
9. Senior interview Q&A
- When does serverless stop making sense? (senior answer) Three regimes. Sustained high QPS: per-GB-second billing running flat-out 24/7 costs more than a reserved container past a crossover point — you're renting CPU the expensive way. Cold-start-sensitive latency: at 1000 req/s with 1% cold and a +400 ms penalty, effective p99 ≈ 405 ms, 80× warm — unacceptable for a tight SLO, and provisioned concurrency to fix it just re-derives an always-on fleet. Long-running or stateful work: execution-time caps, no warm local state, and statelessness make it the wrong tool for anything that wants to hold a connection, a model in memory, or run for minutes. Serverless is for spiky, latency-tolerant, stateless, low-duty-cycle work.
- What is a distributed monolith and how do you spot one? (senior answer) Services split in code but coupled in data and release — they share a database, call each other synchronously for every op, and must deploy together. You pay all the microservice costs (network, partial failure, DDIA Ch. 8 cascades) and get none of the benefit. Diagnostic: can each service deploy, scale, and fail independently? If no, re-merge or actually decouple the data via events (DDIA Ch. 11).
- You split one ACID transaction across three services — now what? (senior answer) You've lost the database's atomicity, so you owe a saga (local transactions + compensating actions) or two-phase commit (blocking, couples availability). DDIA Ch. 9 is a chapter on why this is hard. The senior move is to draw the boundary so the transaction stays inside one service wherever possible, and only accept the saga where the split genuinely pays.
- Why canary at 5% for 30 minutes — why not faster? (senior answer) 5% bounds the blast radius; the window must span at least one cache-warm cycle and your slowest-firing endpoint so a slow-burn regression (leak, cold-path tail) surfaces before 100%. Too short and it escapes the gate; too long and every deploy crawls. The window length is a bias/latency trade you tune to your traffic shape.
- Containers vs VMs vs serverless — what's the start-time spectrum and why care? (senior answer) VM ~minutes, container ~seconds, warm Lambda ~1 ms, cold Lambda ~100–800 ms. It matters because start time sets how fast you scale to a spike and, for serverless, becomes your p99 tail. Containers are the default for service fleets: fast enough to autoscale and roll, identical artifact everywhere, dense packing.
- How do you stop a dependency failure from cascading? (senior answer) Timeouts on every call (an un-timed call is an outage waiting to happen), retry budgets not unbounded retries, circuit breakers to fail fast when a dependency is down, and backoff-with-jitter to avoid retry storms. This is DDIA Ch. 8's partial-failure reality; lesson 13 has the mechanics.
- Blue/green vs canary? (senior answer) Blue/green is an atomic flip between two full fleets — instant rollback, zero degraded slice, but 2× capacity for the cutover. Canary trickles 5% through a metrics gate — cheap, but the slice sees the bug for the window. Use blue/green when no degraded slice is tolerable and you can afford the duplicate; canary otherwise.
Related lessons
This case sits next to C43 (cloud choice) — its sibling: C42 decides the runtime boundaries and rollout, C43 decides whose hardware they run on; read them together. The decomposition discipline here is foundation lesson 17 (architecture decomposition) — where to draw the service boundaries in the first place. The cascade and circuit-breaker machinery is lesson 13 (fault tolerance), the answer to DDIA Ch. 8's partial failure. And the saga/outbox you owe the moment you split a transaction is lesson 12 (distributed transactions), DDIA Ch. 9. Observability with deploy markers (lesson 16) is what makes the canary gate actually attributable to a release.