System architecture & service decomposition
Every lesson so far moved one lever inside a system. This one is about the system's shape: how many deployable pieces it is, where the lines between them fall, and how those pieces talk. The headline you must internalise before any of the tactics: a monolith is the correct default; you split a service out only when a real seam is under pressure — and every split trades a code problem for a distributed-systems problem.
Architecture is not a technology choice, it is an org-and-failure-domain choice that happens to be expressed in technology. The question is never "monolith or microservices?" in the abstract — it is "which boundaries in this system are under enough pressure (a team that can't ship independently, a component that must scale alone, a fault that must be contained) to be worth paying a network, a lost transaction, and an extra thing to operate?" If no boundary is under pressure, the answer is one deployable. Splitting before the pressure exists is the most common senior-level mistake.
1. Start from the monolith — it is the default, not the embarrassment
Recall the series' starting machine (lesson 01): one server, one database. As an architecture, a monolith — one deployable artefact, all modules in one process — has properties that are pure gift, and you give them up the moment you split:
- In-process calls are nanoseconds. A function call between modules is ~1–100 ns and cannot fail in the network sense. There is no serialization, no retry, no timeout, no partial failure.
- One database means real transactions. A single BEGIN…COMMIT gives you ACID across the whole write — atomic, consistent, isolated. One source of truth; no two copies to reconcile (lesson 09).
- Refactoring is cheap and safe. Move a boundary, rename an interface, change a shared type — the compiler/test suite catches it across the whole codebase in one commit. You are not coordinating a rollout across teams.
- One thing to deploy, monitor, and reason about. One log stream, one trace, one place a request lives. The operational surface is minimal.
The monolith's two failures are exactly the series' two failures: it eventually can't scale (one process, one DB) and it is a single failure domain (one bad deploy or memory leak takes everything). Those are real — but they are not reasons to split on day one. They are reasons to keep the code modular so you can split later, cheaply, exactly where it hurts.
The senior default is not "monolith forever" — it is a modular monolith: one deployable, but internally organised into modules with clean, enforced interfaces and (ideally) separate schemas per module. You get nanosecond calls and ACID now, and a pre-cut seam to extract a service later when one module is under genuine pressure. Modularity is a code discipline that is free; a service boundary is a distributed-systems cost you defer until it pays for itself.
2. Why split into services — and what each split costs
You split when a benefit is worth its paired cost for that specific boundary. State both. The right-hand column is the one juniors forget.
| Benefit of splitting | The cost you take on |
|---|---|
| Independent deploy & ownership per team — a team ships on its own cadence without a fleet-wide release | Versioned contracts & backward-compatibility forever; cross-team coordination moves from the compiler to API reviews and on-call rotations |
| Scale only the hot component — give the one CPU-bound service more replicas without paying for the rest | The call into it is now a network call: latency tax + tail risk (lesson 02), plus a new tier to capacity-plan with the ladder method (lesson 01) |
| Fault isolation — a crash or memory leak in one service doesn't take the others down | Only if you actually isolate it (timeouts, circuit breakers, bulkheads, lesson 13); a naive synchronous split spreads failure instead of containing it |
| Polyglot — use the right language/runtime/datastore per service | N build systems, N deploy pipelines, N sets of libraries and CVEs; harder to move engineers between services |
| Bounded cognitive load — a team holds one small service in its head, not a million-line monolith | No one holds the whole system in their head; end-to-end behaviour now lives in the gaps between services |
Three of those costs are large enough to name on their own, because they recur in every interview:
- You lose ACID; you inherit distributed transactions. A write that used to be one local transaction now spans services and databases. Two-phase commit blocks and doesn't scale, so you reach for sagas + idempotency + the outbox pattern (lesson 12) and accept eventual consistency (lesson 09) as a product decision.
- Every cross-boundary call is a network call. Nanoseconds become milliseconds, and a synchronous chain multiplies both the latency tax and the failure probability (the widget below makes this concrete). The ladder you measured in lesson 01 gets longer.
- The operational and observability burden grows super-linearly. A request now hops services, so you cannot debug it from one log — you need distributed tracing, per-service SLOs, and correlation IDs across the fan-out (lesson 16).
Microservices trade a code problem for a distributed-systems problem. In a monolith, the hard problems are "this codebase is large and tightly coupled" — solvable with refactoring, modules, and tests. The moment you split, those become "this call can be slow, time out, retry, arrive twice, or see a stale copy," and "this write is no longer atomic." You have not removed difficulty; you have relocated it from the compiler (which is fast and reliable) to the network (which is slow and unreliable). That trade is sometimes worth it. It is never free.
Conway's law — the real reason systems get split
Conway's law: an organisation's systems mirror its communication structure. Three teams will, left alone, produce three services with the seams falling on team boundaries — because a within-team change is cheap and a cross-team change is expensive, so the architecture drifts to minimise cross-team work. The senior reading is the inverse: choose the team structure you want the architecture to have, because the architecture will follow it. If you want one cohesive service, don't hand it to two teams; if you want two independent services, give them to two teams with a clean contract between them. Service decomposition is, in large part, org design.
3. Where the boundaries go — bounded contexts, not technical layers
The single most common mistake is splitting by technical layer — a "UI service," a "business-logic service," a "database service." That maximises coupling: a single user-facing change touches all three and they must deploy in lockstep. You have the cost of a split with none of the independence.
Split instead along business capabilities / bounded contexts — a vertical slice that owns a coherent piece of the domain end to end (Orders, Payments, Inventory, Identity). The test is the classic pair:
- High cohesion — things that change together live together. A change to "how orders work" should touch one service.
- Low coupling — services interact through a small, stable contract, not by reaching into each other's internals or sharing a schema.
Database-per-service — no shared database. Each service owns its tables and is the only code that touches them; everyone else goes through its API. The instant two services share a database, you have re-coupled them through the schema: a migration in one breaks the other, and you've lost the independent-deploy benefit you paid for. A shared database is also a hidden series dependency for availability (lesson 13) and a denormalisation/modelling decision you no longer control alone (lesson 04). If two "services" must share a write database, they are one service wearing two hats — merge them. (Read-only replicas, or a separate reporting / analytics store fed asynchronously, are fine — those don't couple the write path, and the CQRS pattern in §6 leans on exactly that.)
The distributed-monolith anti-pattern
The failure mode that gets the worst of both worlds. A distributed monolith is a set of services that:
- must be deployed together (a release is a coordinated lockstep of all of them), or
- chat synchronously on nearly every request (Service A can't answer without three blocking calls to B, C, D), or
- share a database or a sprawling set of shared models.
You paid the full microservices tax — network hops, lost transactions, N pipelines, distributed debugging — and kept the coupling the split was supposed to buy you out of. It is strictly worse than the monolith you started with. The tell is operational: "we can't deploy X without also deploying Y," or "one slow service makes everything slow." If that is true, your boundaries are wrong — they are not where the system actually wants to flex.
4. Layers & tiers — the stateless/stateful split
Independent of how many services you have, every web architecture stratifies into tiers by what holds state, because state is what makes scaling hard (lesson 05):
- Stateless edge & app tier — holds no per-request state between calls, so any replica can serve any request and you scale it by adding boxes behind a load balancer (lesson 05). This tier is cheap to scale; that's why you push work toward it and keep state out of it.
- Stateful data tier — databases, caches, queues. This is where partitioning, replication, and consistency live; it is the genuinely hard part of every design.
- API gateway — the front door. One place for TLS termination, authentication, rate limiting and load shedding (lesson 15), and routing to the right service. It keeps cross-cutting policy out of every individual service.
- Backend-for-frontend (BFF) — a thin service per client type (web, mobile, partner API) that aggregates and reshapes downstream calls into exactly what that UI needs. It stops you from contorting one generic API to serve a chatty mobile screen and a wide desktop dashboard at once — at the cost of one more deployable per client.
5. Communication styles — synchronous vs asynchronous
Once there is more than one service, how they talk is the decision that determines your coupling and your failure behaviour. There are two families, and the choice is per-interaction, not per-system.
Synchronous request/response (REST, gRPC)
Caller sends a request and blocks for the reply (lesson 03). Simple, easy to reason about, easy to trace — and it is the right default for a read that needs an answer now. But it couples the caller to the callee in time: if the callee is down or slow, the caller is down or slow. In a synchronous chain, failure cascades — one sick service backs up its callers, which back up theirs, until the whole chain is exhausted (the retry-storm and circuit-breaker material of lesson 13). It also multiplies latency and failure probability with depth, which is exactly what the widget computes.
Asynchronous, event-driven (queues, logs, pub/sub)
The producer emits an event/message and does not wait; a broker holds it; consumers process it when they can (lesson 11). This decouples in time: the consumer can be down for a minute and the producer neither knows nor cares — the broker buffers, and the consumer catches up. That is the resilience win, and it lets each side scale and fail independently. The cost: the result is eventually consistent (the effect lands "soon," not "now"), and a flow that spans several async hops is harder to trace and reason about — there is no single stack to read, only a trail of events.
| Dimension | Synchronous (REST/gRPC) | Asynchronous (events/queue) | Cost of the choice |
|---|---|---|---|
| Coupling | Temporal — both up at once | Decoupled — broker buffers | Async needs a broker you now operate |
| Consistency | Read-your-write, immediate | Eventual (lesson 09) | Async: must design for stale/in-flight state |
| Failure behaviour | Cascades down the chain | Contained — retry from the queue | Sync: needs timeouts/breakers (lesson 13) |
| Latency on the critical path | Sum of all hops (blocking) | Off the critical path | Sync: tail risk compounds with depth |
| Traceability | One trace, easy | Trail of events, harder | Async: needs correlation IDs (lesson 16) |
| Delivery | Exactly-ish (one call) | At-least-once → must be idempotent | Async: idempotency keys everywhere |
Orchestration vs choreography
When a business flow spans several services, who drives it?
- Orchestration — a central coordinator (an "order saga" service) explicitly calls each step and handles compensation. Buys: the whole flow is in one place, easy to see and to debug. Costs: the orchestrator is a coupling hub and a thing that can fail; it knows about everyone.
- Choreography — no central brain; each service reacts to events and emits its own ("OrderPlaced" → Payments reacts and emits "PaymentTaken" → Inventory reacts…). Buys: maximal decoupling, easy to add a new reactor. Costs: the end-to-end flow exists nowhere explicitly — it's emergent, and very hard to follow when it breaks.
This is the same axis as sagas in lesson 12. Rule of thumb: orchestrate when the flow is complex and you need to see it whole (most money flows); choreograph when steps are genuinely independent and you value adding consumers without touching a coordinator.
6. Event-driven patterns — and when each is over-engineering
Three named patterns sit on top of the event-driven style. Each is powerful and each is a complexity tax — say when it's worth it and when it isn't.
Event-driven architecture (EDA)
Services communicate primarily by emitting and reacting to events rather than calling each other directly. Buys: decoupling, resilience, easy fan-out to new consumers. Costs: eventual consistency and the loss of a single readable flow. Worth it when you have many independent reactors to one fact (one "UserSignedUp" event drives email, analytics, provisioning, and fraud checks) and they must scale and fail independently.
CQRS — Command Query Responsibility Segregation
Split the write model from the read model: commands mutate a normalised, validated write store; a separate read-optimised projection is built (often via events) and serves queries. This is the architectural form of the denormalisation you met in lesson 04 and the read-side projection a cache embodies in lesson 06 — a second, query-shaped copy of the data. Buys: reads and writes scale and are modelled independently; the read side can be wide, denormalised, and fast. Costs: two models to keep in sync and a propagation lag (the read side is eventually consistent with the write side). Worth it when the read and write workloads are genuinely different — a high read:write ratio with complex query shapes the write schema can't serve efficiently. Over-engineering when reads and writes are symmetric CRUD; then it is two models and a sync pipeline for no benefit, and a plain cache-aside read path (lesson 06) does the job.
Event sourcing
Don't store current state — store the append-only log of events that produced it, and derive current state by replaying them. Buys: a perfect audit trail (you have every change, not just the result), time-travel/replay, and the ability to build a brand-new projection from history. Costs: substantial — you must version event schemas forever, replay can be slow (so you snapshot), "just query current state" becomes non-trivial, and the whole team must think in events. Worth it where the history is the product or is legally required (ledgers, accounting, audit-heavy domains). Over-engineering for ordinary CRUD where you only ever need "what is the value now" — you've taken on a hard storage model to answer a question a row would have answered.
Both are frequently cargo-culted onto systems that don't need them. The honest interview answer is: "I'd reach for CQRS only when the read and write workloads diverge enough that one model can't serve both well, and for event sourcing only when the event log is itself a requirement (audit/ledger). For a typical CRUD service, both are complexity I haven't earned — I'd start with one model and a read replica or cache." Naming when not to use a pattern is what distinguishes a senior answer.
7. Failure domains & blast radius
Decomposition is also failure-domain design: deciding, for each shared dependency, how much can die together. The unit of correlated failure is the blast radius, and physical infrastructure nests it:
- Rack — shares power and a top-of-rack switch; one rack can vanish.
- Availability zone (AZ) — an isolated datacentre with independent power/cooling/network; the standard unit you spread replicas across so an AZ outage is survivable (the parallel-availability win of lesson 13).
- Region — a geographic cluster of AZs; surviving a region loss means active-active or active-passive multi-region, expensive and worth it only for the highest tiers.
Two architectural patterns shrink the blast radius deliberately:
- Cell-based architecture — partition the whole stack (gateway → services → data) into independent "cells," each serving a slice of users/tenants. A bad deploy or a poison-pill request damages one cell, not everyone — so a 100% outage becomes a "1-of-N cells" outage. The cost is routing complexity and per-cell overhead.
- Bulkheads (lesson 13) — isolate resources (thread pools, connection pools, replica sets) per dependency or per tenant so one saturated thing can't starve the rest, exactly like watertight compartments in a ship.
The unifying idea: a shared component is a shared fate. Architecture decides what shares fate with what — and a good decomposition makes sure that no single failure (a bad deploy, a hot tenant, a dead AZ) can take the whole system with it.
8. The trade calculator: a synchronous chain's hidden tax
Take a request that fans through a chain of services synchronously — each must call the next and wait. Two costs grow with the chain length, and both are easy to underestimate until you compute them. Added latency is (hops − 1) × RTT (the in-process calls you replaced cost ~0). End-to-end availability is the product of the per-service availabilities, ∏ Aᵢ — the series-availability law from lesson 13: needing all of them up multiplies their uptimes, so nines erode fast. Meanwhile the benefit you bought — independent deployability / team parallelism — rises with service count. Move the sliders and watch the trade.
9. The senior synthesis
Pull it together into the judgement an interviewer is actually probing:
- Architecture is org design + failure-domain design as much as it is technology. Conway's law says the system will mirror the teams; blast-radius thinking says it will share fate along its shared components. Decide both deliberately.
- Start as a modular monolith. One deployable, clean internal modules, a schema per module. You keep nanosecond calls and ACID, and you keep the option to split.
- Extract a service when a real seam is under pressure — a team that can't ship independently, a component that must scale or fail alone, a fault you must contain — not preemptively. Each extraction trades a code problem for a distributed-systems problem; do it only where the trade pays.
- Default to synchronous for "I need the answer now," asynchronous to decouple and absorb load. Reach for CQRS and event sourcing only when the read/write split or the audit log is a genuine requirement.
- Make every boundary own its data and talk through a versioned contract. A shared database means you didn't actually split.
This is the lens you carry into the next two lessons. Lesson 18 classifies a prompt into its recurring shape (CRUD, feed, queue, streaming…); the decomposition vocabulary here tells you, for that shape, how many services it wants and how they should talk. The capstone (lesson 19) then walks a full design where you'll make exactly these calls under interview pressure.
Interview prompts you should be ready for
- "Monolith or microservices for a new product with a 4-engineer team?" (senior answer: monolith — specifically a modular monolith. With one small team there's no Conway's-law pressure to split and no proven scaling hot spot; a monolith keeps in-process calls, ACID, and cheap refactoring. I'd enforce module boundaries and a schema-per-module so I can extract a service later, exactly where pressure appears, instead of paying the network/transaction/ops tax up front.)
- "Where do you draw service boundaries?" (senior answer: along business capabilities / bounded contexts, never technical layers. The tests are high cohesion — things that change together live together — and low coupling — a small stable contract between them. And each service owns its data: database-per-service, no shared schema, or you haven't really decoupled.)
- "What does splitting a monolith into services actually cost?" (senior answer: it trades a code problem for a distributed-systems problem. In-process nanosecond calls become network calls with a latency tax and tail risk; you lose ACID and inherit sagas/idempotency and eventual consistency; and observability/ops burden grows super-linearly — you now debug across a fan-out. You take those costs to buy independent deploy, isolated scaling, and fault containment — only worth it where a boundary is genuinely under pressure.)
- "What's a distributed monolith and how do you spot one?" (senior answer: services that must deploy together, chat synchronously on every request, or share a database — so you paid the full microservices tax and kept the coupling. It's strictly worse than a monolith. The tell is operational: 'we can't release X without Y' or 'one slow service makes everything slow.' The fix is to move the boundary to where the system actually flexes, or merge them back.)
- "When is CQRS or event sourcing worth it — and when is it over-engineering?" (senior answer: CQRS when read and write workloads genuinely diverge — high read:write ratio with query shapes the write schema can't serve — accepting two models and a sync lag. Event sourcing when the event log itself is the requirement: ledgers, audit, replay. Both are over-engineering for ordinary CRUD, where one model plus a read replica or cache-aside is simpler and sufficient. Knowing when not to use them is the point.)
- "Synchronous REST/gRPC or asynchronous events between two services?" (senior answer: per-interaction, not per-system. Synchronous when the caller needs the answer now and the flow is simple — but it couples in time and cascades failure, so it needs timeouts and circuit breakers. Asynchronous to decouple and absorb load: the broker buffers so the consumer can lag, failures retry from the queue — at the cost of eventual consistency, idempotent consumers, and harder tracing.)
- "How does your architecture relate to your team structure?" (senior answer: Conway's law — the system will mirror how teams communicate, because within-team changes are cheap and cross-team ones aren't. The senior move is the inverse: pick the team topology you want the architecture to have. One cohesive service → one team; two independent services → two teams with a clean contract. Decomposition is largely org design.)
- "A single bad deploy took down 100% of users. How would you have limited the blast radius?" (senior answer: failure-domain design. Spread replicas across AZs so no zone is a single point; partition the stack into cells so a bad deploy or poison request hits one cell, not everyone — turning a total outage into 1-of-N; and bulkhead resources per dependency/tenant so one saturated pool can't starve the rest. A shared component is a shared fate; architecture decides what shares it.)
A monolith — one deployable, one database — is the correct default: nanosecond calls, real ACID transactions, cheap refactoring, one thing to operate. You split into services only where a real seam is under pressure (a team that must ship independently, a component that must scale or fail alone, a fault to contain), and every split trades a code problem for a distributed-systems problem: network calls with a latency tax and tail risk, lost transactions and eventual consistency, and a heavier ops/observability burden. Put boundaries on business capabilities with database-per-service (a shared DB means you didn't split); avoid the distributed monolith (lockstep deploys, synchronous chatter); choose sync vs async per interaction; and reach for CQRS / event sourcing only when the read/write split or the audit log is a genuine requirement. Architecture is org design and failure-domain design wearing a technology costume — start as a modular monolith and extract a service when, and only when, the pressure is real. Next, classify the prompt's shape in 18 · System design archetypes.