Design Gmail or a distributed email service
Email's defining promise is brutally simple: once your SMTP server says "250 OK," that message must never be lost. Everything else — search, spam filtering, attachment storage — bends around that one durability invariant.
250 OK, the sending MTA deletes its copy and stops retrying. From that instant the message exists only on your side. Lose it and there is no recovery — the sender believes it was delivered, the recipient never sees it, and nobody knows. So the forcing constraint here is not throughput or latency; it is "accepted ⇒ durable," and it must hold before any expensive, fallible processing runs.This single rule reorganizes the whole design. The naïve pipeline — receive mail, scan it for spam and malware, then store it — is wrong, because spam scanning is the most expensive and most likely-to-crash stage in the system. If you return 250 OK and then the filter OOMs, you have already promised durability for a message you just lost. The correct ordering is inverted: durably persist first, acknowledge, then scan asynchronously. The rest of this case derives the consequences of that inversion, and a second one almost as important: of the 5 PB of data we are about to store, ~90% is attachments, and the cheapest petabyte is the one you never write twice.
1. Clarify the contract
Treat the prompt as a product contract before a box diagram. The system must:
- Receive and send mail over standard SMTP (interop with the entire internet — we don't control the other MTAs).
- Store messages durably per mailbox, surviving retries, partial failures, and reordering.
- Full-text search a user's mailbox (subject, body, sender, attachment filenames).
- Filter spam and malware — adversarial, CPU-heavy, and continuously evolving.
- Handle attachments up to ~25 MB without bloating the per-message mailbox record.
And, just as important, what it need not do. Search results may lag the mailbox by seconds — a message that arrived 3 seconds ago not yet being searchable is annoying, not a correctness bug. Spam classification may be revised after delivery (a message can move to Spam later). We are not building real-time chat: end-to-end delivery latency of seconds is fine. These relaxations are what let us push the slow, fallible work off the acknowledgement path.
2. Put numbers on the shape
The two numbers that drive the architecture are the storage split and the inbound retry behaviour. Start with storage. Take 1M users, each with a 5 GB mailbox quota:
1{,}000{,}000 users × 5 GB = 5{,}000{,}000 GB = 5 PB
Now decompose it. Empirically email storage is dominated by attachments — call it ~90% attachment bytes, ~10% headers + bodies + index:
attachments ≈ 0.90 × 5 PB = 4.5 PB | text + metadata ≈ 0.10 × 5 PB = 0.5 PB
That 4.5 PB is the budget that matters, and it has a beautiful property: attachments are immutable blobs, and the same blob is sent to many people. A 4 MB slide deck mailed to a 500-person company is stored 500 times unless you dedup it. Content-hash dedup — key the blob by SHA-256 of its bytes, store once, reference-count — routinely cuts attachment storage by a large factor on mailing-list and corporate traffic. This is why attachments must live in a content-addressed object store, not inline in the message, and it is the single highest-leverage decision in the design.
The second number governs ingress capacity. Suppose steady inbound is ~5,000 msg/s. Email's retry semantics (DDIA Ch. 8: SMTP is built on the assumption of unreliable, retrying peers) mean a remote MTA that couldn't reach you for an hour does not spread its backlog out — it dumps it the moment you come back. A single large sender releasing a queued backlog can drive a 10× spike:
peak ingress ≈ 10 × 5{,}000 = 50{,}000 msg/s
You cannot scale the durable-write path to absorb every conceivable storm cheaply — and you don't have to, because SMTP gives you a legitimate "not now" answer. The protocol lets you return a 4xx temporary failure, and a well-behaved sender will back off and retry later. That turns the retry storm from a capacity problem into an admission-control problem (foundation lesson 15): defend the durable write path, shed the overflow with a 4xx, and let the network's own retry machinery smooth the spike out for you.
3. Data model & API
The data split mirrors the durability and dedup decisions. Three distinct stores, each with a different access pattern and consistency need:
| Entity | Lives in | Why |
|---|---|---|
| Message (envelope + headers + body, with attachment refs) | Mailbox store, partitioned by user | The canonical copy. Source of truth; everything else is rebuildable from it. |
| Attachment blob (keyed by content hash) | Content-addressed object store | Immutable, deduplicated, ref-counted. The 4.5 PB lives here, written once per distinct blob. |
| Search index (inverted index per user) | Search store, partitioned by user | Derived, lossy-tolerant, rebuildable. Lags the mailbox; never the source of truth. |
Keep the API a product surface, not a leak of the internals:
| API / operation | Why it exists |
|---|---|
SMTP RCPT/DATA → 250 / 4xx / 5xx | The protocol contract. 250 is the durability promise; 4xx is the backpressure escape hatch. |
GET /mailbox?label=&cursor= | Paginated read of the canonical mailbox — always serves committed mail. |
GET /search?q= | Reads the inverted index; may not yet reflect mail from the last few seconds. |
GET /attachment/{hash} | Content-addressed fetch from the object store; permission-checked against the referencing message. |
4. Linearized design
Walk a single inbound message through the system in the order events actually occur. The ordering is the whole point: notice exactly when we send 250 OK.
- 1. Accept at edge SMTP. A remote MTA connects. We run cheap pre-acceptance checks (greylisting, connection rate limits, RBL lookups) — these can return a
4xxand cost us nothing downstream. - 2. Durably persist the raw message. Write the raw MIME to a replicated, durable queue/log. Large attachment parts are streamed to the object store under their content hash; the message record keeps only references.
- 3. Acknowledge — return
250 OK. This is the commitment point. We only emit it after step 2's durable write is acknowledged. From here the message is ours and cannot be lost. - 4. Async: spam + malware pipeline. A separate, isolated fleet consumes from the durable log, classifies, and tags. It is CPU-heavy and adversarial — and crucially it runs after the ack, so a crash here re-processes from the log; it never loses mail.
- 5. Async: index build. Another consumer tokenizes headers/body/filenames and updates the per-user inverted index (DDIA Ch. 3).
- 6. Commit to mailbox store. The cleaned, classified message lands in the recipient's mailbox, deduplicated by
(message-id, recipient).
The first bottleneck this exposes is the durable-write path at step 2: it is the only synchronous, capacity-bound stage on the acknowledgement path, which is precisely why the storm in §2 is dangerous and why backpressure lives at step 1, ahead of it.
5. Deep dives
5.1 "Accepted means durable" — persist before you process
Spell out the failure the ordering prevents. Imagine the wrong pipeline: accept → spam scan → store → ack. A 50k msg/s storm hits, the spam fleet is CPU-bound, a classifier process OOMs mid-batch and drops 2,000 in-flight messages. If we had already acked them we've silently lost mail — the cardinal sin. If we ack only after store, then under load we hold open SMTP connections through the slow scan, the sender times out, and re-sends — now we have duplicates and a connection pileup.
The correct pipeline puts a durable, replicated log between acceptance and processing (DDIA Ch. 11, stream processing: the log is the buffer and the source of replayable truth). Write to the log, get an ack from a quorum of replicas (foundation lesson 08), then return 250 OK. Everything downstream — spam, index, mailbox commit — is a consumer of that log. A consumer crash is now harmless: it resumes from its committed offset and re-processes. We have traded "fast but lossy" for "the ack is a real promise," which is the only acceptable trade for email.
5.2 Search index lag is tolerable; lost mail is not
This is the consistency asymmetry that lets the design breathe. The mailbox store is the canonical copy and is read-your-writes consistent: a message you just received is immediately in GET /mailbox. The inverted index (DDIA Ch. 3) is a derived view built by an async consumer, and it is allowed to lag — a few seconds between "mail arrives" and "mail is searchable" is invisible to almost everyone and harmless to the rest.
Because the index is derived and lossy-tolerant, two operational superpowers follow. First, if the index is corrupted or its schema changes, you don't have a data-loss incident — you rebuild it from the canonical mailbox, which is the source of truth. Second, you can re-index lazily or with cheaper hardware, decoupled from the latency-sensitive write path. The rule to state in the interview: never let a derived store become a source of truth. Mailbox reads are authoritative; search is best-effort-eventually-correct.
5.3 Spam filtering is adversarial and CPU-heavy — isolate it
Spam classification is the opposite of the durable-write path in every way: it is expensive (ML models, attachment sandboxing, URL detonation), it is adversarial (attackers craft inputs specifically to blow up your CPU or evade detection), and it is the stage most likely to fail. Run it in-line and an attacker has a direct lever on your mailbox availability: send pathological mail, starve the CPU, and legitimate delivery stalls behind it.
So isolate it as its own fleet consuming from the log, with its own capacity, its own queue, and its own backpressure. If the spam fleet falls behind, mail still gets accepted and stored durably (step 3 doesn't depend on it); messages simply sit untagged in a holding state slightly longer. The blast radius of a spam-pipeline failure is "classification is delayed," not "mail is lost" and not "the mailbox is down." This is the same principle as bulkheading a fragile dependency (foundation lesson 13): a malicious or overloaded slow path must not be able to starve the safe path.
5.4 Inline vs. object store, made concrete
Why not just store the attachment bytes in the message row? Because the mailbox store is your hot, per-user, latency-sensitive path, and attachments are huge and shared. Inlining a 4 MB blob into a partitioned mailbox row means every read of that mailbox row drags 4 MB, replication copies 4 MB per replica per recipient, and the dedup opportunity vanishes. Pushing it to a content-addressed object store keeps mailbox rows tiny (an envelope + a hash), lets the 4.5 PB live on cheap object storage, and stores each distinct blob exactly once regardless of how many of the 500 recipients got it.
6. Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Store raw MIME vs parsed body | Audit, exact protocol fidelity, reprocessing after a parser bug | More storage; raw MIME is bulkier than a normalized record | Always keep raw for inbound — protocol correctness and re-parse safety outweigh the bytes |
| Sync index vs async index | Search is instantly consistent with the mailbox | Index write is on the delivery critical path; couples mailbox availability to index health | Only tiny mail systems; at scale async + rebuildable wins |
| Attachment inline vs object store | Simpler single-read; no second fetch | Mailbox bloat, no dedup, replication amplification of huge blobs | Only for tiny inline images; anything > a few KB goes to the object store |
| Aggressive spam rejection vs quarantine | Less storage burned on abuse; floods rejected cheaply | False positives silently lose legitimate mail — the worst outcome | Reject (5xx) only known-bad senders; quarantine the uncertain |
The sharpest of these is the reject-vs-quarantine call, because it collides directly with the durability invariant. Outright rejecting borderline mail at SMTP time (a 5xx) is cheap and saves storage, but a false positive means a legitimate message is bounced and gone — which violates the spirit of "don't lose mail." The senior move is to be generous about accepting (accept-and-quarantine, where the message is durably stored but hidden in a Spam folder) and stingy only about pre-acceptance rejection of senders you are confident are malicious. You can always re-classify a stored message later; you can never un-bounce one you refused.
7. Failure modes
| Failure | Mitigation |
|---|---|
| Remote SMTP retry storm (10× ingress) | Defend the durable-write path with admission control: greylist new senders, rate-limit per connecting MTA, and return 4xx on overflow so the sender's own retry logic spreads the backlog out. Never drop accepted mail to shed load. |
| Spam fleet overloaded / crashing | Isolated consumer with its own backpressure; mail still accepted and stored, classification simply lags. Resume from committed log offset on restart. |
| Search index corruption | Rebuild from the canonical mailbox — the index is derived, never the source of truth. |
| Attachment malware discovered post-delivery | Scan in the async pipeline; maintain revocation state on the content hash so all referencing messages are blocked from download at once (dedup makes revocation one operation). |
| Duplicate delivery | Idempotent commit keyed by (message-id, recipient) (DDIA Ch. 12) — the same message accepted twice resolves to one mailbox entry. |
8. Interview Q&A
- A malicious sender retries the same message 10,000 times after you 4xx'd them — how do you defend? (senior answer) Layer three defences. (1) Greylisting: a first-contact 4xx that a legitimate MTA retries after a delay but a dumb spammer often abandons — it costs the attacker round-trips for free. (2) Backpressure / per-sender rate limits at the edge so the retries are rejected cheaply, before the durable-write path, and never queue downstream work. (3) Idempotent accept keyed by
(message-id, recipient): even if the duplicates get past the edge and are accepted, they collapse to one mailbox entry (DDIA Ch. 12), so the attacker cannot amplify storage or notifications. The key insight: a 4xx is an invitation to retry, so the defence is making each retry cheap to reject and making a successful retry a no-op. - Why persist before spam scanning instead of scanning first? (senior answer) Because
250 OKis a durability promise the sender relies on by deleting its copy. Spam scanning is the most expensive and crash-prone stage; if we ack and then a scanner OOMs, we've lost acknowledged mail. Durably log first, ack, then scan as an async consumer — a crash there just replays from the offset. "Accepted ⇒ durable" must hold before any fallible processing. - What does "durable" actually require at the ack point? (senior answer) Replicated and fsync'd to a quorum, not just written to one node's page cache. "Accepted ⇒ survives the loss of any single node." A single-node write that dies before replicating loses acknowledged mail exactly like no write at all (lesson 08's quorum semantics).
- How do you store 5 PB economically? (senior answer) Recognize ~90% (4.5 PB) is attachments, which are immutable and frequently identical across recipients. Content-hash (SHA-256) them into an object store, store each distinct blob once, ref-count it. A 4 MB deck to 500 people is 4 MB, not 2 GB. Mailbox rows then hold only an envelope plus a hash, keeping the hot per-user store small and replication cheap.
- Why is it OK for search to lag but not the mailbox? (senior answer) The inverted index (DDIA Ch. 3) is a derived view; the mailbox is canonical. A few seconds of search lag is invisible and harmless, and if the index corrupts you rebuild it from the mailbox. Never invert that — a derived store must not become a source of truth, or a rebuild becomes a data-loss event.
- Reject spam at SMTP time or accept and quarantine? (senior answer) Reject (5xx) only senders you're confident are malicious; for the uncertain, accept-and-quarantine into a Spam folder. A false-positive rejection bounces legitimate mail irrecoverably, which betrays the durability promise. You can re-classify a stored message any time; you can never un-bounce a refusal.
- A user reports a missing message that the sender swears was delivered — how do you investigate? (senior answer) Trace the message-id through the durable log first: if it's in the log, the ack was real and the bug is downstream (spam quarantine, index, or mailbox commit) and recoverable. If it's not in the log but we returned 250, that's the cardinal failure and an incident — it means the ack-before-durable invariant was violated somewhere. The log is your audit trail precisely because everything else is derived from it (DDIA Ch. 11).
9. Related foundation lessons
The async spam and index pipelines are textbook applications of lesson 11 (async messaging) — the durable log decouples the fast ack from the slow, fallible consumers. The three-store split (canonical mailbox / content-addressed blobs / derived index) is lesson 04 (data modeling & storage) applied to wildly different access patterns. The retry-storm defence is lesson 13 (fault tolerance) — backpressure and bulkheading the spam fleet so a slow path can't starve the safe one — and is sized as the admission-control problem of lesson 15 (rate limiting & edge), where the 4xx is the cheap early reject. And because "mail must never silently vanish," lesson 16 (observability) matters acutely: you need to be able to trace any message-id through the log and prove the ack-before-durable invariant held.