Streaming pipeline ops - Kafka, Flink, ETL, and feature freshness

The book treats real-time recommendation as a data-engineering system: Kafka buffers events, Spark/Flink compute windows, Redis/HBase/Hive store features, and quality monitors keep training and serving aligned. This lesson linearizes that material into the operational path an interviewer expects.

1 · Start from the feature freshness contract

Do not begin with "use Kafka and Flink." Begin with the freshness target. A short-video ranker may need second-level session features, minute-level item hotness, hourly creator statistics, and daily long-term user portraits. Each has a different cost and failure mode.

2 · Draw the end-to-end data path

The book's ETL framing maps directly onto this path. Extract is event collection. Transform is validation, cleaning, sessionization, window aggregation, and feature construction. Load is writing to online stores and offline warehouses. Recommendation adds one hard requirement: the online value and offline training value must mean the same thing at the same timestamp.

3 · Choose Lambda or Kappa consciously

The book contrasts real-time and offline processing. In practice you will see two architecture shapes:

A common short-video answer: Kafka is the data bus, Flink handles second/minute features, Spark/Hive produces daily portraits and training snapshots, and a feature store provides one interface to both online and offline values.

If the interviewer asks the book's Ray version, keep the same contract but swap the compute layer: Ray Serve receives or exposes feature requests, Ray actors maintain sharded state, the object store reduces copy overhead, named actors hold long-lived feature state, async checkpoints provide recovery, and feature sharding spreads hot users/items across workers. Ray is a flexible distributed execution option; Kafka/Flink remains the more common streaming interview baseline.

4 · Kafka design is about ordering, scale, and replay

Kafka is not the compute engine. It is the durable, scalable event buffer that decouples producers from consumers. Its main design decisions are topic boundaries, partition keys, retention, schema, and consumer-group behavior.

4.1 · Size the partition count from throughput, not from a guess

The partition count is the one Kafka number an interviewer can check arithmetically, so derive it instead of saying "a lot." A single partition has a practical write+consume ceiling of roughly 10 MB/s (it is one append-only log served by one broker thread and drained by one consumer in the group). So the floor is set by raw throughput and the parallelism you need:

Worked example. A click-and-play firehose peaks at 50 MB/s. Throughput alone needs 50 / 10 = 5 partitions. But the Flink job that drains it runs at parallelism 8, and a partition can only feed one consumer in a group at a time — so 5 partitions would idle 3 of those 8 slots. Take the max: 8. Then pre-provision ~20% headroom for a promo spike, so create 10. Going much higher is not free: every partition is open file handles, replication traffic, and one more entry the controller tracks during a rebalance, so thousands of near-empty partitions slow leader election and lengthen rebalances. Match the count to P = ⌈ T_peak / c_partition ⌉ with consumer-parallelism and a spike buffer, not to a round number.

Three more decisions ride on the same throughput thinking:

• Compression. Enable compression.type=snappy on the producer. Recommendation event payloads, especially ones carrying serialized video or multimodal feature vectors, are large and compress well; Snappy trades a little CPU for materially less network and disk, which is what actually lifts the per-partition ceiling under bursty traffic.
• Consumer-group-per-use isolation. Give each downstream a separate group.id — feature computation, A/B logging, and offline archival read the same topic through independent groups. A slow A/B-logging sink then cannot stall feature computation, and one group's rebalance does not pause the others. Sharing a group across unrelated jobs couples their failure modes.
• Strong-consistency streams go the other way. A model-parameter or counter update that must be globally ordered wants a single partition with high replication, not many partitions — you are buying ordering and durability, not parallelism.

5 · Flink turns events into stateful features

The book's Flink sections emphasize event time, windows, state, watermarks, joins, and exactly-once semantics. Translate them to recommender features:

Watermarks answer: how long will the job wait for late events before closing a window? State TTL answers: how long should old behavior remain in memory? Checkpoints answer: how do we recover after failure without losing state or double-processing offsets? Watermark semantics and the end-to-end exactly-once mechanism live in 17 · real-time streaming; this section is about how the stateful operator is actually built.

5.1 · How a sliding window is actually built in Flink

"Keyed state and a window" is the headline; the implementation is a KeyedProcessFunction plus a state primitive plus a timer. The shape an interviewer wants to see:

The two state choices trade memory for CPU. ListState stores every raw event and recomputes on each timer fire — simple, but a hot user with thousands of events per minute makes every fire O(n) and the state large. MapState<behaviorType, count> keeps running aggregates so the timer just reads a handful of counters — O(1) per fire and far smaller — at the cost of not being able to recompute arbitrary windowed statistics later. Default to MapState for count/CTR-style features; reach for ListState only when you genuinely need the raw sequence (e.g. a Transformer behavior encoder).

Bound it with TTL. Keyed state never expires on its own — a user who churns leaves their state resident forever, and across millions of dead keys that is the classic streaming memory leak. Attach a StateTtlConfig (say 24h, updated on write) so idle keys are reclaimed. The event-time timer evicts within a live key's window; TTL reclaims the whole key once it goes quiet. You need both.

5.2 · RocksDB state backend and the checkpoint cost

For sessionization or hour-scale windows the total keyed state can run to tens of GB — far past the JVM heap. Switch to the RocksDB state backend: state lives in an embedded LSM store on local disk with the hot working set in memory, so the job is bounded by disk, not heap, and dodges GC pauses on large state. The price is per-access serialization (every read/write crosses the JVM↔RocksDB boundary), so for small state the heap backend is faster — pick RocksDB by state size, not reflexively.

Checkpoint cost is the failure mode to name. A full checkpoint snapshots all keyed state to durable storage (HDFS/S3) every interval; as state grows, checkpoint duration grows with it, and a checkpoint that takes longer than its interval starves the next one and stalls progress. Two levers: incremental checkpoints (RocksDB ships only the SST files changed since the last checkpoint, turning an O(total-state) snapshot into O(delta)), and aggressive TTL/eviction so total state stays bounded in the first place. Worked intuition: 40 GB of state full-checkpointed every 60 s against a sink writing ~200 MB/s would need ~200 s just to flush — three intervals deep, permanently behind; the same job on incremental checkpoints ships only the ~hundreds of MB that actually changed and finishes in seconds. A jumping checkpoint duration in the Flink UI is therefore an early warning that state is unbounded or the sink is blocking — long before lag is visible.

6 · Serving a fast-changing encoding: the dynamic encoding store

One streaming output the lessons gesture at but rarely mechanize is the encoding for fast-changing categoricals — trending hashtags, freshly uploaded items, new creators. The feature definition is owned by 35 · feature engineering; here we make the serving side concrete, because a new value appearing every few seconds breaks any static lookup table.

Two-tier store: Redis hot, HBase cold. Keep the active value→embedding (or value→id) map in a Redis cluster for sub-millisecond reads, and back the long-tail and historical mappings in HBase. An LRU eviction on the Redis tier bounds memory — without it the dictionary grows unbounded as new values arrive, the same leak as un-TTL'd Flink state. A read misses Redis, falls through to HBase, and warms the hot tier.

New-value path. A value seen for the first time has no trained embedding yet. Assign it a temporary hash code immediately so serving never blocks, and asynchronously train/backfill a dedicated embedding that replaces the placeholder once ready. Use consistent hashing for that placeholder so the same new value maps to the same code on every node — otherwise two servers encode "trending tag #X" differently and the model sees noise.

Double-buffer swap. When a batch job rebuilds the encoding table, do not mutate the live one in place — readers would see half-updated state. Hold two tables, write the refresh into the idle buffer, then atomically flip a pointer. Reads always hit a complete, self-consistent table; the swap is the lookup-table analogue of the embedding-table double-buffer used in serving.

Dynamic numeric normalization, three tiers. The mirror problem for continuous features whose distribution drifts (watch-time during a holiday, a viral spike): a normalizer fit offline goes stale. Layer it. Offline T+1 computes global P_1/P_{99} as a fallback. Near-line, a Flink job recomputes sliding-window (say 6h) quantiles every ~10 min and writes them into a Redis Sorted Set, whose rank operations give fast quantile lookup at serving time. Online, normalize over the user's last ~100 behaviors for personalization. Smooth the published stats with an EMA so a single spike does not jolt every feature, and on a near-line timeout degrade to the offline P_1/P_{99} — the same labeled-fallback principle as the freshness cascade in section 10.

7 · Exactly-once is end to end, not a slogan

The book asks how Kafka-to-Flink pipelines achieve exactly-once. The precise answer is that each link must participate:

1. Kafka source offsets are captured in Flink checkpoints.
2. Flink keyed state and window state are checkpointed consistently.
3. The sink is transactional or idempotent.
4. Feature-store writes include feature name, entity key, window timestamp, and version.
5. Readers use overwrite-if-newer or deterministic window IDs to tolerate retries.

If your sink is Redis, you may not get textbook exactly-once. You can still get operationally safe behavior with idempotent keys, deterministic window IDs, timestamps, and monotonic writes.

8 · Optimize throughput and offline storage together

The book's Spark/Hive material matters because real-time systems do not eliminate offline systems. Training, backfill, replay, audits, and daily user portraits still depend on efficient batch storage.

For Spark Streaming vocabulary, know both generations: old DStream jobs use Direct Stream consumption from Kafka plus updateStateByKey or mapWithState for cross-window user state; newer Structured Streaming uses DataFrame semantics, watermarks, and checkpointed sinks. In either case, the output usually lands in Redis for low-latency features or HBase/Hive for larger and offline-facing state, with TTLs to prevent unbounded growth.

The batch storage layout is where most teams quietly lose hours of training-pipeline time, so the Hive row deserves its mechanism. See 29 · data engineering for the columnar / partition-pruning fundamentals; the points below are the feature-table specifics.

8.1 · Hive feature-table storage tuning

Two-level partition: date, then a user hash bucket. Partition by dt=yyyyMMdd at the top level and by user_bucket = hash(user_id) % 50 underneath. The date level gives partition pruning for any time-windowed read; the user-bucket level does two things: it spreads load evenly even when a few power users dominate the raw logs (a plain user_id partition would skew badly and blow past partition limits), and it lets a user-level training join read just the matching bucket instead of scanning the whole day. Sizing matters — a too-fine bucket count recreates the small-files problem, a too-coarse one under-prunes. With 50 buckets and a 90-day retention you hold 90 × 50 = 4,500 partitions, comfortably inside the metastore budget below.

ORC + ZSTD over ORC + Snappy. Use ORC (columnar, so a feature read touches only the columns it needs) compressed with ZSTD. ZSTD gives a meaningfully better compression ratio than Snappy at a decompression speed close enough that read latency barely moves — so you pay less for storage and less for the IO that dominates a wide feature scan. Push it further per column type: dictionary encoding for sparse high-cardinality categoricals, and Delta+ZSTD double-compression for dense float arrays (embeddings). A ZSTD dictionary trained on your feature distribution can add roughly another 30\% over generic ZSTD.

Bloom filters for the point lookup. Set orc.bloom.filter.columns=user_id. The filter lets a reader skip entire ORC stripes that cannot contain the target user, turning a "find this user's rows" query from a full-partition scan into a few stripe reads — the difference between seconds and sub-second on a large daily partition.

Fight the small-files problem on cold data. Streaming-style or many-task writes produce thousands of tiny files per partition; each one is a separate open/seek and a metastore listing, so read throughput collapses even though the bytes are small. Set hive.optimize.sort.dynamic.partition=true so dynamic-partition writes are sorted and emit fewer, larger files, and periodically run ALTER TABLE ... COMPACT 'major' on cold partitions to merge the accumulated small files into one. This is the offline analogue of Flink's "bound the state" — left unmanaged, file count grows without limit.

When you need near-real-time partitions, switch table formats. Plain Hive partitions are append/overwrite only, so an incremental upsert (e.g. correcting a user's daily aggregate as late events land) means rewriting the partition. Hudi or Iceberg add row-level upsert/merge-on-read and built-in small-file compaction, letting the same table serve both the daily batch and a minute-scale incremental feed — at the cost of operating a more complex table format. Reach for them when the freshness contract from section 1 pushes a batch table toward near-real-time.

9 · Debug Kafka backlog and Flink backpressure linearly

The book asks about Kafka message accumulation and Flink backpressure. Use a fixed debugging path:

9.1 · The concrete commands and the tripwire

The ladder above is the reasoning; in the room name the actual instruments. Measure lag with kafka-consumer-groups.sh: kafka-consumer-groups.sh --describe --group feature-calc prints, per partition, the current offset, the log-end offset, and the LAG between them. That per-partition column is exactly what tells the first two branches of the ladder apart — lag spread evenly across partitions means the consumer side is underprovisioned or the sink is slow; lag piled on one partition means a hot key (one user/item/creator hashing to that partition). For the Flink side, backpressure surfaces in the Flink Web UI as a per-operator indicator (the back-pressured operator is the one downstream of the red one), and rising checkpoint duration in the same UI flags the state/sink problem from section 5.2 before lag is even visible.

Two fixes worth naming because they map to specific causes. A hot key (lag on one partition, a slow key in the Flink metrics) is fixed by two-stage aggregation (local pre-aggregate then global combine) or by splitting the key with salting / consistent-hash so one viral item no longer monopolizes a single task — the same hot-key remedy as the symptom table, now tied to the metric that revealed it. Large messages (video or multimodal feature payloads) inflating throughput are tamed with Snappy compression on the producer, as in section 4.1 — compression is both a throughput lever and a lag remedy.

10 · Design degradation before the incident

Realtime features fail. A feed should degrade to less fresh personalization, not fail open with corrupt values.

Every feature read should carry a timestamp or freshness bit. A stale-but-labeled feature is easier for the model and monitors to reason about than a stale feature pretending to be live.

Interview prompts you should be ready for