Design ad click event aggregation
Two numbers must come out of the same firehose of clicks: a dashboard number that is fast but allowed to be wrong, and a billing number that is slow but must be exactly reproducible. The whole architecture is the discipline of keeping those two paths honest about which one they are.
0. The hinge: the reconciliation gap, in dollars
Before any boxes, put the forcing number on the table. Suppose the system takes 1 billion clicks/day. Empirically a small tail of events arrive badly late — say 0.3% land more than 5 minutes after the click actually happened (mobile retries, offline buffering, cross-region replication lag). That tail is:
1{,}000{,}000{,}000 · 0.003 = 3{,}000{,}000 late events/day
Those 3 million clicks are real, billable events. The realtime dashboard, which closes its window after ~5 minutes so it can stay fresh, has already published totals that omit them. So at any instant the dashboard's running total and the eventual billing total disagree by up to 3M clicks/day. The interview lives in that delta: what do you show on the dashboard, what do you bill, and how do you reconcile them so finance trusts the number?
Two of those deserve their arithmetic shown. Average rate is 10^9 / 86{,}400 ≈ 11{,}574 events/s; ad traffic is diurnal and campaign-bursty, so size ingest for a 10× peak ≈ 116 k/s, not the average. Raw-log storage is 10^9 · 200\text{ B} · 3 \text{ (RF)} · 30 \text{ d} = 1.8 \times 10^{13}\text{ B} ≈ 18\text{ TB} — cheap and non-negotiable, because the raw log is the thing that makes billing reproducible (see deep dive 2). You do not get to drop it to save money.
1. Clarify the contract
Treat the prompt as a product contract before a diagram. The system must:
- Ingest click beacons at ~116 k/s peak with minimal synchronous work at the edge — the collector's only job is "make it durable, fast."
- Deduplicate retried/replayed clicks by a deterministic key so one human click is billed once.
- Aggregate by (campaign, ad, publisher, time-window) for both dashboards and billing.
- Serve a fresh dashboard (seconds of lag) that is explicitly allowed to be approximate.
- Produce a billing total that is exactly reproducible from the raw log — bit-for-bit on replay.
- Absorb late and out-of-order events and correct already-published numbers.
And — just as important — what it need not do: it need not make the dashboard perfectly accurate (we will label it estimated), it need not bill in real time (billing runs on a daily/hourly cadence), and it need not decide fraud inline (it must only preserve the raw evidence so fraud can be adjudicated later). Cutting those three "musts" is what makes the design tractable.
2. Data model & API
The data model is the first architecture. Keep the API expressing the product operation, never the internal queue or shard layout.
| API / operation | Why it exists |
|---|---|
POST /clk (beacon) | Edge ingest. Carries the deterministic event key (see below) plus campaign/ad/publisher ids and the client event-time. |
GET /agg?campaign&window | Dashboard read — returns the stream estimate, flagged provisional until finalised. |
GET /billing?campaign&day | Billing read — returns the batch-reconciled, finalised total with a version/run id. |
The records that matter:
The single most load-bearing field is the deterministic event key. A click beacon carries something like key = hash(ad_id, user_id_or_device, click_ts_ms, nonce) generated on the client at click time. Because the key is derived from the click's own immutable attributes, the same physical click always produces the same key — whether it arrives once, or three times after a retry. That property is what lets dedupe be deterministic and replay be reproducible (deep dive 3).
3. Linearized design
Follow one click from finger to invoice. The split into two paths happens exactly once, at the durable log, and never blurs after that.
- 1. Edge collector. The beacon hits the nearest collector. It does the minimum: validate shape, stamp a receive-time, and append to a durable partitioned log. No aggregation, no dedupe, no DB write on the hot path. Acknowledge as soon as the append is durable.
- 2. Durable raw log. A partitioned append-only log (Kafka-style) partitioned by a key that co-locates a campaign's events — this is the immutable source of truth and the substrate both downstream paths read. (Partitioned-log design is anchored in C23; the IO mechanism — sequential writes, batching, zero-copy — in C26.)
- 3a. Stream path → dashboard. A stream processor consumes the log, dedupes within a bounded recent-key set, and folds events into short event-time windows (e.g. 1-minute tumbling). Windows close on a watermark and emit a provisional aggregate to a fast serving store. Lag: seconds. Accuracy: approximate (misses the late tail).
- 3b. Batch path → billing. On a cadence (hourly/daily), a batch job reads the same raw log from offset 0 of the period, applies the same deterministic dedupe, and computes the exact per-window totals. Output is a versioned finalised aggregate. Lag: hours. Accuracy: exact and reproducible.
- 4. Correction log (feedback). When batch finalises a window, it diffs against what the stream published and writes a correction (delta + new version). The dashboard store applies corrections so a finalised window stops being provisional; downstream consumers see "this number was revised, here's why."
4. Deep dives
Deep dive 1 — Event-time vs processing-time, and why watermarks decide when a window closes
Every event has two clocks. Event-time is when the click happened on the device; processing-time is when your pipeline saw it. They diverge, sometimes wildly: a phone clicks at 12:00:00, goes offline, and the beacon lands at 12:09:30. If you aggregate by processing-time you put that click in the 12:09 minute — wrong campaign-minute, wrong dashboard cell, wrong everything. Billing must aggregate by event-time, because that is when the value was actually created. This is the central distinction of DDIA Ch. 11's stream-processing chapter.
But aggregating by event-time raises a question the stream cannot avoid: when is the 12:00 minute "done"? You can always receive another straggler. The answer is a watermark — a moving assertion of the form "I believe I have now seen all events with event-time ≤ W." A common heuristic: W = \max(\text{event-time seen}) - \delta, where δ is a slack (say 5 minutes) sized from the observed lateness distribution. When the watermark passes the end of a window, the stream closes the window and emits a provisional result — that's how it stays fresh without waiting forever.
The late event at 12:09:30 belonging to the 12:00 window is exactly the 0.3% tail. The stream has two honest choices for it: drop it (cheap, but the dashboard stays permanently short — only acceptable for low-value analytics) or emit a retraction/correction (the window's total is revised upward). For billing we never drop: the batch path re-reads the whole raw log for the day, so by construction it sees every late event regardless of when it arrived, and produces the corrected truth. The watermark is therefore a freshness/completeness knob: a larger δ closes windows later (fresher data is slower but more complete); a smaller δ is faster but corrects more often. You tune δ from the measured lateness CDF so that, say, 99.7% of events beat the watermark — which is precisely why 0.3% don't.
Deep dive 2 — Why the dashboard can be approximate but billing must replay (the Lambda architecture)
The two paths are not redundant; they have different correctness contracts, and conflating them is the classic failure. The dashboard answers "roughly how is this campaign doing right now?" — a 0.3% transient error is invisible to a human watching a trend line, and freshness is the product. Billing answers "what exactly does this advertiser owe?" — here a 0.3% error is a lawsuit, and reproducibility is the product.
Reproducibility has a precise technical meaning here: given the immutable raw log and a deterministic computation, replaying the batch job at any future time produces byte-identical billing totals. That requires three things to hold, and the design exists to enforce all three:
- The input is immutable and complete. The raw log is append-only with RF=3 and 30-day retention; nothing mutates or deletes an event in place. Replay from offset 0 of a day always reads the same bytes.
- The computation is deterministic. Dedupe uses the deterministic event key (no wall-clock, no random tie-breaks); aggregation is by event-time (not "whenever we processed it"); the job is pure over its input.
- The output is versioned. Each batch run stamps its results with a run id, so a re-run is comparable to the prior one and corrections are auditable, not silent overwrites.
This is DDIA Ch. 10's batch philosophy (a job is a deterministic function from immutable input files to output) married to Ch. 11's stream layer — the Lambda architecture: stream gives latency, batch gives correctness, and the batch result is authoritative wherever they disagree. The modern critique (Kappa: "just make the stream reprocessable and drop the batch tier") is a legitimate senior aside, but the conceptual point it preserves is the same — billing must be recomputable from an immutable log, whether that recompute is a separate batch job or a stream replay. The split is about contracts, not about owning two clusters.
{window:12:00, +6, version:final}. The dashboard store applies it; the cell flips from estimated to finalised; billing invoices the 10,000. The gap existed, was bounded (≤0.3%), was visible, and was closed deterministically. That sentence is the whole case.Deep dive 3 — Deduplication by a deterministic event key
Retries are not optional: the collector acks only after a durable append, so any network hiccup makes a well-behaved client resend the same beacon — at-least-once delivery is the only safe default (this is the delivery-semantics problem anchored in C27). Without dedupe, every retry inflates the billable count. The fix is a deterministic event key: because the key is a hash of the click's own immutable attributes (ad, device, click timestamp, nonce), the duplicate carries the same key, and dedupe is just "have I already counted this key for this window?"
"Deterministic" is the load-bearing word, and it's why dedupe must not be a wall-clock or arrival-order decision. On replay six months later the batch job will encounter the duplicates in a possibly different order, yet it must reach the identical total — only a content-derived key gives that. The stream path keeps a bounded recent-key set (a rolling window of seen keys, often a Bloom filter or a keyed state store with TTL) — bounded because it can't remember a billion keys forever; that boundedness is acceptable precisely because the stream is allowed to be approximate. The batch path dedupes exactly over the full day by grouping on the key, which it can afford because it isn't latency-bound. Note the deliberate non-goal: dedupe collapses identical events (same key = same click retried); it does not try to decide whether two different clicks from the same user are fraud. Fraud is a separate adjudication that runs offline over the preserved raw evidence — the pipeline's job is only to keep that evidence, not to judge it inline.
5. Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Stream + batch (Lambda) | fresh dashboard AND reproducible billing | two code paths over one log; correction logic | money is on the line — the default here |
| Drop late events vs correct windows | dropping → stable, simple aggregates | permanent revenue undercount (the 0.3%) | drop only for low-value analytics; never for billing |
| Exact dedupe (full state) vs probabilistic (Bloom) | exact → billing correctness | state/storage cost; Bloom risks false-positive drops | exact for billing; Bloom is fine on the approximate stream |
| Small watermark δ vs large δ | small → fresher windows | more corrections, larger stream/billing gap | tune δ to the lateness CDF (e.g. cover 99.7%) |
| Edge validation vs central validation | edge → cheap early rejection of malformed beacons | policy changes need edge redeploy | only obvious shape/abuse filtering at the edge |
The sharpest trade-off is the second row, and it is where juniors quietly lose money. "Drop late events" looks attractive because it makes the stream's windows immutable and the code simple — close the window on the watermark, never touch it again. But applied to billing it silently discards the 3M late clicks/day, a permanent undercount that publishers will eventually notice and dispute. The correct posture is asymmetric: the dashboard may drop-or-approximate within a window because it is labelled provisional, but the billing path must never drop, because its whole reason to exist is to re-read the immutable log and recover exactly those stragglers. Keeping that asymmetry explicit — same data, two contracts — is the senior move.
6. Failure modes
| Failure | Mitigation |
|---|---|
| Collector / region outage | Client buffers beacons locally and retries with backoff; DNS/edge failover to a healthy collector. Late-arriving buffered events are exactly the tail the watermark + batch path are built to absorb. |
| Duplicate / replayed clicks | Deterministic event key + dedupe (stream: bounded key set; batch: exact group-by). Retries are billed once. |
| Aggregate store corruption / bad deploy | Aggregates are derived, not source. Rebuild any window by replaying the immutable raw log through the (deterministic) batch job. |
| Stream processor falls behind / crashes | Dashboard goes stale (acceptable, labelled). Resume from committed log offset; batch path still produces correct billing independently. |
| Watermark too aggressive | More events land late → bigger gap and more corrections. Re-tune δ from the observed lateness distribution; batch still backstops correctness. |
| Dimension / cardinality explosion | Enforce a campaign-metadata schema and an allow-list of aggregation labels; reject unknown dimensions at ingest. |
7. Interview Q&A
- Prove your billing total is exactly reproducible after a full pipeline replay. (senior answer) Three invariants. (1) The input is an immutable append-only raw log — replay from offset 0 of the period reads identical bytes every time. (2) The computation is deterministic: dedupe keys are content-derived (no wall-clock, no arrival-order tie-breaks), aggregation is by event-time, the batch job is a pure function of its input. (3) Output is versioned by run id so reruns are auditable, not silent overwrites. Given those, replaying the batch job in a year yields byte-identical totals. Idempotent application of those totals downstream is its own problem — that's the delivery-semantics machinery in C27.
- Why two pipelines instead of one fast one? (senior answer) Different correctness contracts. The dashboard wants freshness and tolerates a 0.3% transient error; billing wants exact reproducibility and tolerates hours of latency. You can't satisfy both over the same data at the same instant because the late tail hasn't arrived yet. Lambda: stream for latency, batch for correctness, batch wins on disagreement.
- Event-time or processing-time, and why does it matter for money? (senior answer) Event-time — when the click happened — because that's when value was created. Processing-time would bin a click that arrived 9 minutes late into the wrong campaign-minute, corrupting both the dashboard cell and the bill. Watermarks let the stream decide a window is "probably complete" so it can stay fresh, while the batch path re-reads everything by event-time for the exact answer.
- What's the gap between the dashboard number and the bill, quantitatively? (senior answer) The late tail. At 1B clicks/day and a 0.3% >5-min-late fraction, that's 3M events/day the stream's provisional windows miss until batch backfills — on a $0.50 CPC campaign doing 2M/day, about $3k/day understated until finalised. It's bounded, measurable, and closed by the correction log; I'd label the dashboard "estimated, finalises at T+24h."
- How do you dedupe, and why deterministically? (senior answer) A key hashed from the click's immutable attributes (ad, device, click-ts, nonce) generated client-side, so a retried beacon carries the same key. Determinism is mandatory because replay may encounter duplicates in a different order yet must reach the identical total. Stream uses a bounded TTL'd key set (Bloom is fine since it's approximate); batch dedupes exactly by group-by over the full period.
- A duplicate retry and a fraud burst both look like repeated clicks — handle both? (senior answer) No — deliberately. Dedupe collapses identical events (same key). Two different clicks from one user is a fraud question, adjudicated offline over the preserved raw evidence on a different cadence. The pipeline's job is to keep the evidence, not to judge inline; conflating the two corrupts the billable count.
- Your aggregate store gets corrupted by a bad deploy — recovery? (senior answer) Aggregates are derived, never source. Drop them and replay the immutable raw log through the deterministic batch job to rebuild any window exactly. This is the entire reason we pay ~18TB to keep 30 days of raw events — the log is the recovery substrate, not just an audit trail.
- How do you size the watermark slack δ? (senior answer) From the measured lateness CDF, not a guess. Pick δ so that the desired fraction (say 99.7%) of events beat the watermark; that directly sets how big the stream/billing gap is (the remaining 0.3%) and how often you emit corrections. Smaller δ = fresher but noisier; larger δ = slower but more complete. Batch backstops correctness regardless of δ.
Related foundation lessons
This case sits on the streaming spine. The raw log is the C23 partitioned append-log substrate (and C26 for why that log is fast); at-least-once ingest means duplicates, so reproducible billing leans on the delivery-semantics / dedupe machinery anchored in C27 (we show only the case-specific deterministic key here). The event-time/watermark/windowing mechanics and the stream-vs-batch (Lambda) duality are foundation lesson 11 (async messaging & streams), with the batch-reconciliation half drawing on DDIA Ch. 10. Idempotency at the boundary is foundation lesson 12; the dashboard's SLOs/metrics tie back to 16.