Design large file upload and serving

A 10 GB upload over a home connection takes an hour. Networks drop, browsers close, laptops sleep — so the only sane design treats failure as the normal case and never makes the user start over.

This is the anchor case for resumable multipart upload, chunk dedup, and idempotent commit. Other cases lean on the machinery here: C14 (video transcode) and C15 (file sync) are consumers of an upload pipeline; C16 (object storage) provides the durable substrate underneath it. So we teach the upload protocol in full and let those cases reference it.

1. Why failure dominates the design

Start with the transfer time, because it sets the whole problem. Take a 10 GB file over a 20 Mbps uplink — a generous home connection. 10 GB = 80,000 Mb (using 8 bits/byte and 1000-based units), so the floor is:

80,000 Mb ÷ 20 Mbps = 4,000 s ≈ 67 min

Call it ~68 minutes in ideal conditions, longer in practice. That is not a request; it is a session that has to survive Wi-Fi blips, a closed lid, a roaming handoff, a process restart. Treat the whole upload as one atomic HTTP PUT and a single dropped packet at minute 66 costs the user the entire hour. That is the forcing function: the transfer is too long to be all-or-nothing.

The fix is to slice the file into independently-retryable parts. With 8 MB parts, a 10 GB file is:

10 GB ÷ 8 MB = 10,240 MB ÷ 8 MB = 1,280 parts

Now put a failure rate on it. Suppose each part PUT fails with probability p = 0.5% (a transient TCP reset, a 5xx from the store, a timeout). Expected failed parts over the whole upload:

1,280 × 0.005 = 6.4 parts ≈ 51 MB re-uploaded

So across a 10 GB upload you expect to re-send about 6–7 parts (~50 MB) — under 0.5% of the file — instead of restarting all 10 GB on the first hiccup. That is the entire economic argument for resumable chunking, in two numbers.

2. Clarify the contract

Treat the prompt as a product contract before a box diagram. The system must:

• Upload large files over unreliable networks — chunked, with parts retried independently.
• Resume after interruption — the client can ask "which parts do you already have?" and send only the rest.
• Verify integrity — guarantee the bytes that land are the bytes the client meant to send.
• Serve downloads efficiently — from object store / CDN, never proxied byte-by-byte through app servers.
• Reclaim abandoned uploads — expire sessions and garbage-collect orphaned parts.

And explicitly what it need not do: it is not a transcoding pipeline (that is C14, downstream of commit), not a delta-sync engine (C15), and the app server is not a byte proxy. The app issues credentials and records metadata; the bytes flow client → object store directly. Naming these non-goals out loud is what keeps the design small.

3. Data model & API

The data model is the first architecture. Keep the API expressing the product operation, not the storage layout.

The session record is the spine:

This is DDIA Ch. 4 (encoding) territory: the manifest — the ordered list of parts and their checksums — is the canonical encoding of "what this file is." Assembly and verification both read from it, so its integrity is the file's integrity.

4. Control plane vs data plane

The single most important structural decision: the app server never touches the file bytes. It runs the control plane — create the session, mint signed URLs, validate, publish. The data plane — the actual gigabytes — flows from client straight to the object store.

Why this split is non-negotiable: if the app proxied bytes, a few thousand concurrent 10 GB uploads would saturate app-server NICs and turn a stateless, cheap-to-scale tier into a bandwidth bottleneck — the exact failure the trade-off table below warns against. By handing the client a signed URL (a time-boxed, scoped credential to PUT one specific object key), the store does the heavy lifting and the app stays a thin metadata service. This is the same control/data-plane separation that lets C16's storage fleet scale independently of any front-end.

5. Linearized design — follow one upload through

1. Create. Client calls create with file size and a content hash. Server allocates an objectKey, picks a part size, writes an open session with expiresAt, returns the uploadId. No bytes yet.
2. Resume check. On (re)connect the client calls list parts. The server returns the part numbers already stored. The client uploads only the missing ones — this is the line that turns "restart 10 GB" into "send the 6 parts you lost."
3. Upload parts (data plane). For each missing part the client PUTs the chunk to its signed URL, directly to the object store, including a per-part checksum. Parts go in parallel (say 4–8 at once) to fill the bandwidth-delay product. The store records each part under (uploadId, partNumber) and returns an ETag/checksum.
4. Complete. Client calls complete with the full manifest. Server validates: all N parts present? each stored checksum match the manifest? If yes, it instructs the store to assemble and atomically publish the object key. If no, it returns the list of bad/missing parts and stays open.
5. Serve. Downloads go through a short-lived signed URL, CDN-fronted for shareable content, so app servers never proxy download bytes either (caching).
6. Reclaim. A background sweep aborts sessions past expiresAt and garbage-collects their orphaned parts so abandoned uploads do not leak storage forever.

The first bottleneck this exposes is not bandwidth — that is the store's problem now — it is the commit step's correctness under retries, which is where the rest of the lesson lives.

6. Deep dives

6.1 Chunk size: a three-way tension

Part size is the one tuning knob that touches everything. It trades off three quantities at once:

The optimum is tied to the network failure rate. Expected wasted bandwidth from retries scales roughly as N · p · partSize, but since N = fileSize / partSize, that is fileSize · p for the re-send volume — note part size cancels there. What part size actually controls is the granularity of each loss (you re-send in whole-part units) versus the fixed overhead of more parts (each part costs a request, a signed URL, a metadata row, a round trip). So: on a lossy link, go smaller — you want each loss to be cheap and you can't afford a giant part to keep failing mid-flight. On a clean link, go larger — amortize the per-part overhead. 8 MB is a sane default precisely because at p≈0.5% the retry waste (~50 MB) is already negligible, so you bias toward fewer parts and lower overhead. (Note S3's hard limit of 10,000 parts also forces part size up for very large files: a 1 TB object needs parts ≥ ~100 MB.)

6.2 Direct-to-store upload via signed URLs

A signed URL is a capability: the control plane computes an HMAC over (object key, HTTP method, expiry, optional content constraints) using a secret the store also knows. The store validates the signature on arrival — no app-server involvement, no long-lived credentials handed to the client. This is what physically realizes the control/data-plane split: the app grants the right to write one specific part for the next few minutes, and the bytes flow on a path the app never sees.

The senior nuance is scoping. A part-upload URL should be scoped to exactly (bucket, key, partNumber), a single PUT, with a short expiry. If you over-scope (e.g. a wildcard write URL), a leaked URL becomes a write primitive over your bucket. The download side is symmetric: short-lived signed GET URLs, optionally with a CDN in front (caching) so the store isn't the egress bottleneck either. The failure-as-normal stance applies here too — if a URL expires mid-upload (the part took longer than the expiry on a slow link), the client just re-fetches a fresh URL from the control plane and retries the part. Resume already handles this.

6.3 Integrity — a 200 OK does not prove the right bytes landed

This is the subtlest part and the one juniors skip. A successful HTTP PUT means "the store accepted some bytes for this key." It does not mean those are the bytes the client intended: a truncated body that still got a 200, a buggy proxy that re-chunked, a client bug that read the wrong file offset, bit-rot between accept and durability — all return 200. Transport success is not data correctness.

So integrity is enforced with checksums at two levels, mirroring DDIA Ch. 4's "the encoding is the contract":

• Per-part checksum. The client computes a hash (CRC32C/MD5/SHA-256) of each chunk and sends it with the PUT. The store verifies on write and stores the checksum. A part whose bytes don't match its declared checksum is rejected — caught at minute 30, not at publish.
• Final manifest check. At complete, the client's manifest lists every part number and its expected checksum. The server verifies that the stored set exactly equals the manifest — every part present, every checksum matches, no extras — before assembling. Optionally it computes a whole-file hash (a hash-of-hashes, like S3's multipart ETag) and compares to the content hash the client declared at create.

This double check protects against both client bugs (wrong bytes sent) and storage bugs (wrong bytes stored), and it is what makes the commit trustworthy.

6.4 Idempotent, atomic commit (the hard part)

Two retry hazards, both DDIA Ch. 11 idempotency patterns:

Idempotent part PUT. A client times out on part 47, doesn't know if it landed, and re-sends it. Because parts are keyed by (uploadId, partNumber), the re-send simply overwrites/coincides with the same logical slot — uploading part 47 twice is indistinguishable from uploading it once. No duplicate, no corruption. This is the chunk-dedup property: the address of a part is a deterministic function of its identity, not of when it arrived.

Atomic publish. complete is the linearization point. It must be all-or-nothing: validate the full manifest, and only then flip the object from completing to published in a single atomic step (a conditional metadata write / compare-and-set on session state). A second complete on an already-published session returns the same result — it is a no-op, not a re-publish. Until publish succeeds, no reader can resolve the object key; after it succeeds, all readers see the whole file. There is no window in which a download sees a half-assembled object.

7. Trade-offs

The sharpest one is proxy vs direct. The reflex "route it through the app so we can scan and authorize every byte" feels safe and is almost always wrong at scale: it converts a stateless tier into a multi-gigabit pipe and makes your app the SPOF for every upload. The senior move is to keep the data plane direct-to-store and move the controls off the hot path — scan asynchronously after upload but before publish (the object is invisible until the scan and manifest both pass), and authorize at URL-issue time rather than per byte. You get the safety without making the app proxy 10 GB streams.

8. Failure modes

9. Interview Q&A

Related lessons

This case anchors resumable multipart for the set: C14 (transcode) consumes a completed upload as its input and reuses the same chunk/idempotency machinery on the segment level; C15 (file sync) is the delta-aware sibling — same resumable upload, plus block-level diffing — so it references the protocol here rather than re-deriving it. The durable substrate the parts and manifest live on is C16 (object-storage durability). The whole stance — treat interruption as the normal case, retry the small unit safely — is foundation lesson 13 (fault tolerance) applied to a long-lived transfer, and the small, intentional surface follows 03 (API design). Edge serving leans on 06 (caching).