Rate limiting & the API edge

Every request enters through a front door. That door is the cheapest place on Earth to enforce cross-cutting concerns and to keep the load behind it bounded — so that's exactly where you put the cop.

First principle

A request that never touches your business logic costs almost nothing to reject. A request that reaches a saturated database costs you the database — and every other request queued behind it. So the optimal place to drop excess load is as early and as cheaply as possible: the API gateway. Rate limiting is just admission control — the planned, fair sibling of lesson 13's emergency load shedding. It converts unbounded incoming load into a bounded stream the backend can actually serve.

The API gateway: the front door

Before "rate limiting," name the thing doing it. The API gateway is a single logical entry point that sits between clients and your services and handles the concerns that every request needs, so that no individual service has to:

TLS termination — decrypt once at the edge instead of in every service.
Authn / authz — verify the token / API key once; pass a trusted identity inward.
Routing & composition — map /v2/orders to the right service; fan out and merge for BFF-style aggregation.
Request / response shaping — protocol translation, header injection, payload size caps.
Observability hooks — one place to emit per-request metrics, traces, and logs (lesson 16).
Rate limiting — the focus here.

The trade-off you must name out loud

Centralising these concerns protects and simplifies the backends — but the gateway is now an extra network hop on every request and a critical-path component. If it dies, the whole API dies. So it must itself be replicated, stateless where possible, and fast (lesson 13: don't let the door become the SPOF). The gateway buys you leverage at the cost of being the single most important box to keep alive.

Concretely, the gateway is a function in front of your fleet: edge(req) → {reject early | forward}. A rejected request costs maybe a TLS handshake plus a counter lookup — call it tens of microseconds and zero downstream resources. The same request, admitted into a saturated system, might consume a connection, a thread, a database query, and seconds of tail latency that delays a hundred well-behaved requests behind it. That asymmetry — cheap to reject, expensive to over-admit — is the entire economic argument for the edge.

Why rate limit — it's about protecting the system, not punishing users

Rate limiting is rarely about being mean to a client. The real motivations, in rough order of importance:

Keep backends below saturation. Lesson 02's utilisation cliff: as ρ→1, queueing latency blows up superlinearly. A limiter caps arrivals so ρ stays in the safe zone.
Fairness across tenants. One noisy client (or a runaway retry loop) should not starve everyone else. Per-tenant limits give isolation.
Abuse / DoS / scraper defence. Reject floods cheaply at the edge before they cost real resources.
Cost control. Bound calls to expensive downstreams (a paid third-party API, an LLM, a query that scans a partition).

Note what's not on that list: "stop people from using your product." If your limits are routinely rejecting legitimate users, the answer is usually more capacity or higher limits, not a tighter clamp. The limiter is a safety valve sized to protect supply, not a tool to ration demand below what you can actually serve. A useful test: every 429 you emit should map to either real abuse or genuine backend pressure. A flood of 429s with the backend at 30% utilisation means your limit is mis-sized, not that the limiter is "working."

Sizing the limit from first principles

Before choosing an algorithm, derive the number. Start from the backend's safe capacity, not from a round number that "feels right." Suppose a service instance can serve 500 req/s at p99 latency before lesson 02's queue blows up, you run 10 instances, and you want to stay at 70% utilisation for headroom against spikes and failures (13). Safe aggregate ≈ 500 · 10 · 0.7 = 3500 req/s. If you have ~700 active tenants and want rough fairness, a baseline per-tenant sustained rate of 3500 / 700 = 5 req/s is the floor; premium tiers get more, and you over-subscribe deliberately because not all tenants are active at once. The limit is a budget allocation, not a moral judgement — re-derive it whenever capacity or tenant count changes.

The algorithms

Five canonical algorithms. The axes that matter: how they treat bursts, how much memory they cost per key, and how accurate they are at the window boundary.

Algorithm	How it works	Burst behaviour	Memory / key	Accuracy
Fixed window	One counter per calendar window (e.g. per minute); increment, reject above limit, reset at window edge.	Allows a spike at each edge.	O(1) — one int.	Poor: boundary burst lets through up to 2× the limit.
Sliding window log	Store the timestamp of every request; count those within the trailing window.	Exact — no edge artifact.	O(requests) — every timestamp.	Exact, but memory explodes under load.
Sliding window counter	Keep current + previous fixed-window counts; weight the previous by how much it still overlaps the trailing window.	Smooths the boundary.	O(1) — two ints.	Near-exact; cheap. Common production choice.
Token bucket	Bucket of capacity B refills at r tokens/s; each request spends a token; reject (429) if empty.	Allows bursts up to B, caps sustained rate at r.	O(1) — tokens + timestamp.	Exact on average rate; burst-friendly.
Leaky bucket	Requests enter a queue that drains at a constant rate r; overflow is dropped.	No downstream bursts — output is a smooth r.	O(queue depth).	Smooths output; adds queueing delay.

The boundary-burst flaw, concretely

Suppose the limit is 100 req/min with a fixed window. A client sends 100 requests at 00:00:59 and another 100 at 00:01:00. Both windows individually pass — but the backend saw 200 requests in one second. That's 2× the intended limit, delivered in a microburst. Sliding-window counter fixes this by blending the two windows.

accepted(t) ≈ count_cur + count_prev · ( 1 − elapsed_in_window / window )

The mental model for the two bucket algorithms is worth memorising: token bucket = "allow bursts, cap the average"; leaky bucket = "enforce a smooth constant rate, no bursts." In practice most APIs prefer token bucket because short bursts are usually fine and even desirable — a user loading a page fires 30 requests at once and that's normal. Leaky bucket earns its place specifically when the downstream is fragile: a third-party API with its own hard limit, a legacy mainframe, a payment processor — anything where you'd rather add a few milliseconds of queueing delay than ever exceed a smooth rate.

One subtlety worth saying out loud: token bucket and the sliding-window counter are computationally equivalent in the steady state — both converge accepted-rate to r. The difference is in the transient. Token bucket stores "credit" you can spend in a lump; the window counter spreads admission more evenly. If you want a client to be able to save up for a big batch job, token bucket; if you want maximally smooth admission, the counter.

Token-bucket math

Two parameters define everything:

Capacity B (tokens) — the maximum burst you'll absorb instantly.
Refill rate r (tokens/s) — the sustainable long-run throughput.

Each arriving request takes one token; if the bucket is empty it's rejected with 429. Over any long interval the accepted rate converges to min(λ, r) regardless of how spiky λ is — that's the whole point. The bucket starts full, so the first burst of up to B is admitted at once; beyond that, admission is gated by refill.

Worked example

Set B = 100 tokens, r = 20 tokens/s. Backend can sustain ~20 req/s per tenant comfortably (02 headroom).

Idle client suddenly sends 100: bucket is full → all 100 admitted instantly (the allowed burst). Bucket now empty.
Then a steady 50 req/s arrives: only 20/s refill, so 20/s pass and 50 − 20 = 30 req/s are rejected. Sustained accepted rate = min(50, 20) = 20.
Time for the initial burst capacity to drain under that 50/s: the deficit is λ − r = 30 tokens/s drained from a 100-token bucket → B / (λ − r) = 100 / 30 ≈ 3.3 s of "free burst" before steady rejection kicks in.
Recovery: after the flood stops, the bucket refills at 20/s, fully replenishing in B / r = 100 / 20 = 5 s, ready for the next burst.

Net: the tenant gets a generous burst allowance and a hard cap on average load. The backend never sees more than 20 sustained req/s from this key.

clients API GATEWAY (replicated, stateless) services ───────── ┌────────────────────────────────────────────┐ ─────────── app ──┐ │ TLS term → authn/authz → RATE LIMIT → route │ ┌──▶ orders app ──┼───────▶ │ ┌───────── token bucket per API key ──────┐ │ ──┼──▶ users bot ──┘ │ │ tokens: ▣▣▣▣▣▣▢▢▢▢ refill r/s │ │ └──▶ billing │ │ spend 1 / req · empty ⇒ 429 + Retry-After │ │ │ └───────────────────────────────────────────┘ │ └────────────────────────────────────────────┘ accepted rate ⟶ min(λ, r) rejected (429) ⟶ max(0, λ−r) in steady state

Follow one request through the front door

The client opens a TCP connection and completes the TLS handshake — which terminates here, at the gateway, not at the backend. The orders service never sees raw bytes off the wire; it only ever talks to the trusted edge.
The gateway extracts the API key / token and verifies authn / authz. An unauthenticated or unauthorised caller is rejected right here — we don't spend a token, and we don't route anything inward.
For that key, the gateway runs an atomic token-bucket check. If the bucket is empty, it returns 429 with Retry-After immediately — having touched nothing downstream. No connection, no thread, no query.
A token is spent and the request is routed to the orders service. This is the first step that consumes real backend resources — a connection, a worker thread, a DB query.
The response is shaped (headers, payload caps) and metrics / traces are emitted (16) before it's handed back to the client.

The punchline: steps 1–3 are deliberately the cheap part — a handshake and a counter lookup, tens of microseconds and zero downstream cost. Only step 4 costs backend resources. That is exactly why the limiter (the "cop") stands at step 3, before the expensive work, never after it.

Distributed rate limiting

Everything above assumed a single bucket. Now make it real: you have a fleet of N gateway nodes behind a load balancer, autoscaling up and down, and a client's requests land on different nodes. Where does the counter live? This is where the interview earns its "senior" label, and the recurring theme is accuracy vs. coordination cost.

Approach	How	Accuracy	Cost / risk
(a) Per-node local	Each node holds its own bucket with limit L; no coordination.	Approximate. Effective global limit ≈ N·L and drifts with N as you autoscale; a client spread evenly across nodes can do N·L.	Cheapest — zero network hops, no shared state. Over-permissive; you set L = limit/N and hope the LB spreads evenly.
(b) Centralised shared counter	One store (Redis) holds the counter; nodes do an atomic `INCR` / a Lua script / a `CL.THROTTLE` (redis-cell) per request.	Accurate — single source of truth, exact global limit.	Adds a network hop per request (sub-ms but real), makes the store a dependency / SPOF (13) and a hot key (07) for a popular tenant.
(c) Hybrid (local + global budget)	Local token buckets that periodically borrow a slice of a global budget and sync residuals back.	Eventually-accurate — small transient over/under-shoot between syncs.	Low coordination (sync every 100ms–1s, not per request). More code; this is lesson 09's eventual-consistency trade-off applied to counters.

Picking sensibly

If your limit is a soft guard against saturation, (a) local is often enough — being 2× off rarely matters and you pay nothing. If the limit is a hard contractual quota (billing, a paid tier), you need (b) centralised accuracy — then protect the Redis with replication and accept the hop. At very high RPS where the hot key or the hop hurts, move to (c) hybrid. Don't reach for Redis reflexively; ask "how wrong can I afford to be?" first.

Why the centralised path needs atomicity

The naive centralised implementation — read counter, compare to limit, write counter+1 — has a textbook race: two nodes read 99 (limit 100), both decide they're under, both write 100, and you've admitted 101. Under high concurrency this leaks badly. The fix is to make check-and-increment atomic on the store: Redis INCR is atomic, but to also apply the limit and TTL in one round trip you ship a Lua script (Redis executes it atomically server-side) or use a purpose-built module like redis-cell's CL.THROTTLE, which implements a token bucket (GCRA) in one atomic call and returns remaining tokens and retry-after for free. The lesson generalises: a distributed counter is only correct if the read-modify-write is atomic where the data lives.

token bucket over time (B=10, r=2/s, λ spikes to 6/s for 5s) tokens ▲ 10 ┤█▙ refill 2/s ↘ 8 ┤█ ▜▖ burst of 10 admitted instantly 6 ┤█ ▜▖ then deficit (6−2)=4/s drains the bucket 4 ┤█ ▜▖ 2 ┤█ ▜▙▁▁▁▁▁▁▁ empty ⇒ admit only 2/s, reject 4/s (429) 0 ┤█__________▔▔▔▔▔▔▔▔▔▔________ refills back to 10 in B/r=5s after spike └┬────┬────┬────┬────┬────┬──▶ t (s) 0 1 2 3 4 5

The client contract

Rejecting a request is only half the job — you must tell the client how to behave so it doesn't make things worse. The standard contract:

HTTP 429 Too Many Requests on rejection.
Retry-After header — seconds (or a date) until the client may retry. A well-behaved client waits this long.
Rate-limit headers on every response: RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset — so clients self-throttle before hitting 429.
Good clients then apply exponential backoff with jitter (lesson 13) — never a tight retry loop, which turns your limiter into a thundering-herd amplifier.

There's a sharp distinction between 429 (Too Many Requests) and 503 (Service Unavailable): 429 means "you specifically have exceeded your quota — slow down," while 503 means "the service as a whole is overloaded" (lesson 13's shedding). Clients can and should treat them differently — a 429 says back off on this key, a 503 says the whole service is hurting. Emitting the right one is part of the contract. And whatever you do, the rejection path must be cheaper than the success path; if computing "should I reject?" itself touches the database, an attacker can DoS you through your own limiter.

Where to key the limit is its own decision: per API key (best — stable identity tied to a tenant/plan), per user, or per IP. IP is the fallback when there's no identity, but beware the NAT / shared-IP pitfall: an entire office, a carrier-grade NAT, or a corporate proxy shares one IP, so an IP limit can throttle thousands of innocent users together while a distributed botnet sails under it. Layer keys: a coarse per-IP limit for anonymous traffic plus a precise per-key limit for authenticated traffic. Also consider request coalescing / dedup at the edge — collapse identical concurrent reads into one downstream call.

Two more dimensions appear in good designs. First, limit granularity: a single global per-key limit is crude; production systems usually limit per endpoint class, because a cheap GET /health and an expensive POST /report shouldn't draw from the same budget. The natural extension is weighted / cost-based limiting — each request spends a number of tokens proportional to its estimated cost (GitHub's API uses exactly this, charging more "points" for heavier GraphQL queries). Second, where the limiter physically lives: an in-process check at your own gateway is one option, but you can also push limiting to a CDN / WAF edge (Cloudflare, Fastly) so abusive traffic is rejected on a different continent before it ever reaches your region — cheapest of all, at the cost of the edge having coarser identity information.

Token-bucket simulator

Set capacity B, refill rate r, and incoming rate λ. Watch the steady-state split and how long the burst budget lasts.

Sustained accepted = min(λ, r)

—

Burst absorbed instantly = B

—

Steady reject = max(0, λ−r)

—

Time to drain bucket

—

Interview prompts you should be ready for

Why put the rate limiter at the gateway and not in each service? (senior answer) It's the cheapest, earliest place to drop excess load before it costs real resources, and it centralises a cross-cutting concern so services stay simple. The cost is an extra hop and a critical-path component that must be replicated — don't let the door become the SPOF.
Token bucket vs. leaky bucket — when each? (senior answer) Token bucket allows bursts up to B and caps the average at r — right when short bursts are normal (page loads, batch fan-out). Leaky bucket queues and drains at a constant r, producing a smooth downstream feed with no bursts, at the cost of queueing delay — right when the downstream can't tolerate spikes at all.
Explain the fixed-window boundary-burst flaw. (senior answer) A client can send the full limit at the end of one window and again at the start of the next, delivering 2× the limit in a tiny interval. Sliding-window counter fixes it by weighting the previous window's count by its remaining overlap — O(1) memory, near-exact.
How do you rate limit across N autoscaling gateway nodes? (senior answer) Three options on an accuracy-vs-coordination spectrum: per-node local (free, approximate, effective limit drifts as N·L), centralised Redis counter (accurate, adds a per-request hop plus a hot-key/SPOF risk), or hybrid local buckets that periodically borrow from a global budget (eventually accurate, low coordination). Choose by how wrong you can afford to be.
What does a good 429 response look like? (senior answer) Status 429 with Retry-After and RateLimit-Limit/Remaining/Reset headers so clients self-throttle and back off with jitter. Without that contract clients hammer in a tight loop and amplify the overload.
Should you key limits on IP? (senior answer) Only as a fallback for anonymous traffic. Shared NAT/CGNAT means one IP can be thousands of users, so IP limits both over-punish innocents and under-catch distributed attacks. Prefer per-API-key/per-user; layer a coarse per-IP cap underneath.
How does rate limiting relate to load shedding (lesson 13)? (senior answer) Same goal — keep the backend below the saturation cliff — different timing. Rate limiting is planned, per-key admission control enforced continuously; load shedding is the emergency reaction when you're already overloaded. A good system does both: the limiter is the steady gatekeeper, shedding is the circuit breaker.
Your limiter's Redis goes down — now what? (senior answer) Decide the failure mode deliberately. Fail-open (allow traffic) keeps the API up but removes protection; fail-closed (reject) protects backends but causes an outage. Usually fall back to per-node local limits (degraded accuracy, still bounded) and alert — that's the hybrid design paying off.

Takeaway

The edge is where unbounded demand meets bounded supply. An API gateway makes that boundary cheap to defend: reject early, reject for the right reasons (protect the backend, not punish the user), and tell the client how to behave. Token bucket — burst up to B, average down to r — is the default; sliding-window counter is the cheap-and-accurate counter. The hard part is distribution: every accuracy gain costs coordination, so size the mechanism to how wrong you can afford to be, and never let the front door become the single point of failure.