Part 5 · Distribution

Multitenancy, Request Routing, and Sharding Operations

Lesson 11 gave you the mechanics of splitting data by key: range vs hash partitioning, the shard key as a performance contract, the celebrity hot key, and why hash(key) mod N is a disaster for rebalancing. That was the algorithm. This lesson is the operations on top of it — running a real cluster that serves many customers at once without one starving the others, getting every request to the partition that owns its key while partitions are moving underneath you, and moving shards across machines in production without taking an outage. We can already split data by key; now we run a real multi-tenant cluster and move shards without an outage.

1 · Multitenancy models: the isolation/cost spectrum

A tenant is one isolated customer of a shared system — one company on your SaaS app, one team in your feature store, one project in your vector index. Multitenancy is running many tenants on the same infrastructure. The whole point is to amortize cost: a thousand small tenants sharing one cluster is far cheaper than a thousand clusters. The whole danger is that "sharing" means a fault, a load spike, or a data leak in one tenant can reach another. Every multitenancy decision trades exactly that — density (cheap) against isolation (safe).

There are three canonical models, in increasing isolation and cost:

Most real systems are hybrid and tiered: the long tail of small tenants shares one pool (shared table), mid-size tenants get a schema each, and a handful of whales get dedicated clusters. The model is not global — it is a per-tenant tier you can promote a tenant through as it grows. The promotion path (shared row → own schema → own cluster) is exactly the hot-tenant mitigation of §3.

2 · The noisy neighbor: when one tenant starves the rest

The defining failure of dense multitenancy is the noisy neighbor: one tenant consumes a shared resource — CPU, disk I/O, connections, cache, storage — so aggressively that co-located tenants are starved, even though those tenants did nothing wrong. It is the hot-key problem of lesson 11 raised to tenant granularity, and it is invisible until it isn't: tenants A through Y see latency spikes and timeouts because tenant Z ran an unbounded query or a backfill.

Mitigations all share one idea: turn an implicitly shared resource into an explicitly budgeted one, per tenant.

• Per-tenant quotas and rate limiting. Cap each tenant's qps, connections, query cost, and storage. A token bucket per tenant_id at the routing tier (§4) lets bursts through but enforces a sustained ceiling. The worked number above is the whole argument.
• Resource isolation at the OS / scheduler. When tenants run in their own processes or pods, pin CPU and memory with cgroups (the Linux kernel feature that caps a process group's CPU shares and memory) or Kubernetes pod limits, so a tenant physically cannot exceed its slice. A memory cap turns a runaway tenant's OOM into that pod's restart, not a shared-node crash.
• Storage quotas. Cap bytes per tenant so one tenant's unbounded growth cannot fill the shared disk and stall everyone's writes.
• Fair queueing / admission control. Schedule shared work round-robin across tenants rather than first-come-first-served, so a flood from one tenant cannot monopolize the queue.
• Promote the heavy tenant out (§3) — move it to its own schema or cluster so it has no neighbors to be noisy toward.

3 · The hot tenant: the celebrity problem at tenant granularity

Lesson 11's hot key was one shard-key value drawing a disproportionate share of traffic. A hot tenant (a "whale") is the same pathology one level up: a single tenant whose data or load dwarfs the others. If tenant_id is the leading part of the shard key, a whale's entire dataset and traffic hash to one partition — exactly the celebrity problem, and hashing alone cannot spread one tenant across partitions because all its rows share the same tenant_id.

The mitigations parallel the hot-key fixes from lesson 11, now applied to a tenant:

• Dedicated shard / cluster. Promote the whale to its own partition set or its own cluster (the cluster-per-tenant tier of §1). It now has the whole machine to itself and no neighbors — the cleanest fix, and why tiering exists. This is the tenant-scale analogue of "front the hot key with its own capacity."
• Split the tenant. If even one tenant exceeds a single partition, sub-partition within the tenant on a second key — e.g. shard key (tenant_id, user_id) so the whale's rows spread across many partitions while small tenants still co-locate on one. This is the compound-key idea from lesson 11.
• Key-salting. For a hot write target inside a tenant (a per-tenant global counter, a shared inbox), append a small random salt 0..k to spread writes across k partitions and merge on read — the random-suffix trick from lesson 11, scoped to the tenant. Cost: reads of that value fan out to the k salted sub-keys.

4 · Request routing: finding the partition that owns a key

Once data is partitioned and partitions can move (rebalancing, §5), a client with a key faces a question: which node owns this key right now? This is service discovery for data, and lesson 11 named three shapes. Here is what each costs operationally.

Routing-table staleness is the operational hazard. The partition map is not static — a rebalance (§5) changes which node owns a range. Any client or proxy that caches the map can hold a stale entry and send a request to a node that no longer owns the key. Good systems handle this with a redirect-and-refresh protocol: the wrongly-addressed node replies "not mine, owner moved" (or forwards), and the client invalidates its cached map and refetches. The window between "partition moved" and "every cache learned" is the staleness window, and it is precisely why the cut-over step of a rebalance must be atomic and observable (§5).

Local vs global secondary indexes — the ops view

Lesson 11 introduced the two secondary-index layouts; the operational question is simply which queries stay single-shard. A local (document-partitioned) index lives beside the rows it indexes: writes touch one shard, but a query on the indexed field must scatter-gather across every shard (no single shard knows all the matches). A global (term-partitioned) index is partitioned by the indexed term: a query on that field hits one shard (single-shard read), but a write may update terms on several shards and the index lags. For multitenancy this often resolves cleanly: index by tenant_id (or make it the leading term) so a tenant's secondary lookups stay within that tenant's shard — single-shard, no cross-tenant fan-out. A query that omits tenant_id is the one that scatters, and you should know it is rare before you allow it.

5 · Rebalancing operations: don't trigger a storm

Rebalancing — moving partitions between nodes when you add or lose machines — was introduced mechanically in lesson 11 (fixed partitions, consistent hashing, dynamic splitting, so only ~1/N of data moves). The operational question is when and how fast the moves run, and who decides.

Manual vs automatic. Automatic rebalancing — the system detects imbalance and moves partitions on its own — is convenient and dangerous. The danger is the rebalance storm: an automatic rebalancer reacts to a transient signal (a node that is slow because it is busy serving traffic) by moving partitions off it, which makes the new owners busy, which the rebalancer reads as more imbalance, so it moves more — a feedback loop where rebalancing traffic starves live traffic and cascades. A node that looks overloaded during a brief spike, or one that is briefly unreachable during a failover (lesson 09), can trip a full data migration when nothing was actually wrong. We model exactly this cascade in lesson 28's drills. For this reason DDIA recommends keeping a human in the loop: the system proposes a rebalance plan, an operator approves it. And every automatic mover must throttle — cap the bytes-per-second and the number of partitions moving at once — so migration never out-competes the workload it exists to serve.

Checkpoint exercise

Where this points next

We can now run a multi-tenant cluster: data split by key, tenants isolated by quota and tier, requests routed to the right partition even as shards move, and rebalances executed from a runbook instead of a panic. But notice what we have not made safe — what happens when one logical operation must touch more than one key, on possibly different partitions, while concurrent clients and crashes claw at it. A transfer between two tenants' balances, an order plus its inventory row: these must behave as a single all-or-nothing unit even though partitioning has scattered the keys across machines. That is the next pressure — transactions and isolation (lesson 13): what atomicity and isolation actually promise, the ladder of isolation levels, and the anomalies each level does or doesn't prevent, with partition boundaries making all of it harder.

Interview prompts

• Walk the three multitenancy models and their trade. (§1 — shared table with tenant_id is densest/cheapest but weakest isolation; schema-per-tenant adds logical separation and per-tenant backup at more metadata cost; cluster-per-tenant gives the strongest isolation and smallest blast radius but is most expensive. Real systems tier tenants across the spectrum.)
• What is a noisy neighbor and how do you stop it? (§2 — one tenant saturating a shared resource and starving co-located tenants; fixed by per-tenant quotas/rate limits at the routing tier and cgroup/pod resource limits, converting a cluster outage into one tenant's 429.)
• One tenant runs 40k qps on a 100k cluster of 100 tenants — what happens, and what does a 2k-qps quota change? (§2 — demand hits 119k vs a 100k ceiling, the pool saturates and all 99 others miss SLOs; a 2k quota caps that tenant so total demand stays ~81k under ceiling and no one else is affected.)
• How is a hot tenant the same problem as a hot key, and how do you fix it? (§3 — a whale's rows share one tenant_id so they all hash to one shard, just like a celebrity key; fix with a dedicated shard/cluster, sub-partitioning on a second key, or key-salting — the tenant-scale analogues of lesson 11's hot-key fixes.)
• How does a client find which node owns a key, and what breaks during a rebalance? (§4 — a routing tier/proxy, a coordination service holding the authoritative map, or gossip; the hazard is a stale cached routing table after a move, handled by the wrong node redirecting and the client refreshing its map.)
• What is a rebalance storm and how do you avoid one? (§5 — automatic rebalancing reacting to a transient spike/failover and cascading into mass data movement that starves live traffic; avoid with a human-in-the-loop approved plan and a throttle capping migration bandwidth/concurrency.)
• Give the safe rebalance runbook and the invariant behind it. (§5 — add nodes, mark the minimum partitions, copy in the background while old owners serve, cut over routing atomically, verify, decommission last; invariant: copy first and never delete the source until the destination is proven.)