Part 2 · Meaning over time

Encoding, Schemas, and Evolution

Lessons 03–04 built the storage engine — how a write becomes durable in an append-only log or LSM tree, and how a read stays fast through indexes and columnar layout. But all of that assumes the bytes on disk and on the wire mean the same thing to the program that wrote them and the program that reads them. Data must also survive deploys and schema change over time: the writer and the reader are routinely different versions of your own code, deployed days apart. This lesson is about the boundary where an in-memory object turns into bytes — encoding — and the discipline that keeps those bytes legible as the code that produces and consumes them keeps changing under you.

DDIA grounding

Arc adapted from Designing Data-Intensive Applications, Chapter 4 (Encoding and Evolution). Original synthesis, not a reproduction. The book's three dataflow modes — through databases, through services, through asynchronous messages — structure the second half here, recast for ML feature and training pipelines.

Linear position

Prerequisite: Lessons 01–04 — you can pick a data model (relational / document / graph) and design around an access pattern, and you know data is written to a storage engine as bytes (in a log, a B-tree page, or an SSTable) that outlive the process — and the code — that wrote them. You know a record has fields.
New capability: You can choose an encoding for a given boundary (storage, RPC, message log), and you can evolve a record's schema during a rolling deploy without a coordinated big-bang migration — adding, removing, and renaming fields safely in both directions.

The plan

Five moves. (1) Establish why in-memory objects must become bytes, and why language-native serialization is a trap for anything that outlives one process. (2) Compare text formats (JSON/XML/CSV) against binary schema formats, and hand-encode one tiny record with field tags to see where the bytes go. (3) Define backward and forward compatibility and walk concretely through add / remove / rename — what's safe, what breaks, and why tags and defaults make it safe; contrast tag-based (Protobuf/Thrift) with writer/reader-schema (Avro). (4) Place encoding in the three dataflow modes — database, service, message log. (5) Tie it to ML: a feature record's schema is a model's input contract, and a model trained on v1 features silently breaks on v3 data.

1 · Why bytes, and why not language-native serialization

In memory, your data is objects, structs, dictionaries, rows, or tensors — pointer graphs the CPU can chase in nanoseconds. None of that survives leaving the process. To put data on disk, send it over a socket, or append it to a log, you need a self-contained sequence of bytes. Encoding (a.k.a. serialization or marshalling) is the translation from the in-memory representation to a byte sequence; decoding (deserialization, parsing) is the reverse. The byte form has no pointers, no class identity, no heap — just a layout that both sides must agree on.

Every language ships a built-in serializer: Python pickle, Java Serializable, Ruby Marshal. They are one-line convenient and a long-term trap. They tie the bytes to one runtime and one class definition, so another language can't read them; they often embed arbitrary-class instantiation, which is a remote-code-execution hole when you decode untrusted input; versioning is an afterthought, so a renamed field can throw at load time; and they are CPU- and space-inefficient. The rule that organizes this whole lesson: data outlives code. Bytes you write today will be read by a program you haven't written yet, and read by an old program you forgot to retire. A serialization format is a contract across time and across languages, and language-native serializers honor neither axis.

2 · Text vs. binary schema formats — and where the bytes go

The first fork is whether the format is self-describing text or carries a separate schema — an explicit, named declaration of each field, its type, and its identity, defined once and shared by writer and reader.

JSON

Human-readable, ubiquitous, schema optional. Field names repeated in every record. No int/float distinction, no bytes type, no fixed precision. Validation lives in app code.

XML

Verbose text with attributes and namespaces; rich schema (XSD) tooling. Heavyweight; mostly legacy / document interchange now.

CSV

Tabular, trivially readable, no nesting and no types — everything is a string. Quoting/escaping ambiguities bite constantly. Fine for flat dumps, dangerous as a contract.

Protocol Buffers / Thrift

Binary, schema-required. Each field carries a numeric tag, not its name. Compact, strongly typed RPC contracts. Need a shared .proto/IDL and a codegen step.

Avro

Binary, schema-required, but no tags or field numbers in the bytes. Encoder and decoder reconcile a writer's schema with a reader's schema. Built for large data files and evolving pipelines.

Parquet / Arrow

Columnar binary: bytes grouped by column, not by row, so analytical scans touch only the columns they need. Storage/scan format, not an RPC or mutable-row format (lesson 04).

Text formats pay their cost in size and looseness: JSON repeats "timestamp" as 11 bytes in every single record, and a number could be an int, a float, or — past 2^53 — a silently-rounded float. Binary schema formats drop the names entirely. Here is the whole point, hand-encoded. Take this record and a Protobuf-style schema:

schema (.proto) record ----------------- ------ 1: user_id int64 user_id = 1001 2: is_active bool is_active = true 3: nickname string nickname = "ada" JSON encoding (50 bytes, names included every time): {"user_id":1001,"is_active":true,"nickname":"ada"} Protobuf-style binary (10 bytes): each field = (tag, type) + value 08 E9 07 field 1 (user_id), varint -> 1001 10 01 field 2 (is_active), varint -> 1 (true) 1A 03 61 64 61 field 3 (nickname), length 3 -> "ada" The first byte of each field packs the TAG and the wire type: 0x08 = (1 << 3) | 0 -> tag 1, varint 0x10 = (2 << 3) | 0 -> tag 2, varint 0x1A = (3 << 3) | 2 -> tag 3, length-delimited

The field tag — the small integer that identifies a field in the encoded bytes — is the load-bearing idea. The name nickname never appears in the binary; only tag 3 does. That is most of the size win (10 bytes vs. 50), and, as the next section shows, it is also the entire mechanism behind safe evolution.

3 · Backward and forward compatibility, field by field

Why does evolution even need two directions? Because deploys are not atomic. A rolling upgrade replaces server instances a few at a time, so for minutes-to-hours OLD and NEW code run simultaneously, reading and writing the same store and the same message stream. A mobile client may be months stale. A data lake holds records written years ago. So you need both:

Backward compatibility — NEW code can read data written by OLD code. (Newer reader, older data. The newer reader must tolerate the absence of fields it now expects.)
Forward compatibility — OLD code can read data written by NEW code. (Older reader, newer data. The older reader must tolerate fields it has never heard of, and ignore them.)

Forward compatibility is the harder, subtler one: it means code must be safe against the future. Now the three edits, in a tag-based format (Protobuf/Thrift):

Edit	Safe?	Why / how
Add a field	Yes, if optional with a default	New field gets a fresh, never-reused tag. Old readers see an unknown tag and skip it (forward-compat). New readers reading old data find the field missing and fill the default (backward-compat). A new required field breaks both — old data lacks it, old writers never set it.
Remove a field	Yes, if it was optional	New writers stop emitting tag N; new readers ignore it. Never reuse tag N for a different field — an old record still carries the old meaning at that tag. Removing a required field breaks old readers that still demand it.
Rename a field	Yes — it's free	The name lives only in the schema, never in the bytes. Keep the tag the same and the wire format is byte-identical; rename `nickname`→`handle` and every old record decodes unchanged. Changing the tag, though, is a remove-plus-add and must follow those rules.
Change a field's type	Risky	Some widenings are safe (Protobuf int32→int64 for in-range values); most are lossy or reinterpret bytes. Treat as add-new-tag + deprecate-old.

The unifying invariant: tags are immutable identity; names are just documentation. Two laws keep tag-based evolution safe forever — never change a field's tag, and never reuse a retired tag. Now Avro, which is built differently.

Avro has no tags and no field numbers in the bytes — it encodes values back-to-back in schema order, with nothing marking where one field ends and the next begins. Decoding is therefore impossible without knowing the exact schema that wrote the bytes: the writer's schema. The reader supplies its own reader's schema (the shape it wants), and the Avro library resolves the two by field name: fields present in both are matched by name; a field only in the reader's schema is filled from its declared default (backward-compat); a field only in the writer's schema is read and skipped (forward-compat).

TAG-BASED (Protobuf/Thrift) WRITER+READER SCHEMA (Avro) bytes carry tag numbers bytes carry only values, in order reader matches fields by TAG reader matches fields by NAME rename = free (tag unchanged) rename needs an alias (name is the key) add = pick a new unused tag add = give the new field a default writer/reader schemas can writer's schema MUST travel with the drift independently data (file header) or be in a registry

This is why Avro suits big data files and Kafka-style logs: write the writer's schema once in the file header (or reference a registry ID), then millions of records carry pure values. The trade is that you must transport the writer's schema; tag-based formats embed enough self-description to skip that, at the cost of a tag byte per field.

4 · Three ways data flows — and where evolution bites

DDIA's frame: encoded data crosses process boundaries in exactly three modes, and each stresses compatibility differently.

AThrough a database. The writer is your code now; the reader may be your code in five years — or your code now reading a row written by last month's binary. A single database often holds records at many schema versions at once. This is the purest "data outlives code" case: backward-compat is mandatory, and a careless reader that re-saves a row can silently drop fields it didn't understand unless the format preserves unknown ones.

BThrough a service (REST / RPC). Client and server deploy independently, so the request/response encoding must be compatible both ways across versions. Old clients keep calling; new servers keep answering. RPC (Protobuf/gRPC, Thrift) leans on tag-based schemas precisely so a v2 server can add a response field that v1 clients ignore.

CThrough an asynchronous message / log. A producer writes to a queue or append-only log (lesson 03's durable log, reused by streams in lesson 19); consumers read later, maybe much later, maybe several different consumer versions at once. The message is decoupled in time, so forward-compat matters most — a slow consumer must tolerate fields newer producers added.

Modes B and C differ in a way worth naming, because each constrains compatibility differently. REST and RPC are both synchronous request/response: the caller blocks until the callee answers, so the two are coupled in time — but they are still independently deployed, so the encoding must survive a version gap in both directions (old client to new server, new client to old server). RPC's flavor of the trap is that it dresses a network call as a local function call, which hides latency, partial failure, and exactly this version skew; tag-based schemas (gRPC/Thrift) are how it keeps a v2 server's extra response field invisible to a v1 client. A message broker (an intermediary like Kafka or RabbitMQ that stores messages until a consumer reads them) removes the time coupling entirely: producer and consumer need not be up at the same moment, so the same message may be decoded by several consumer versions over hours or days — which is why forward compatibility dominates here. The actor model (Akka, Erlang, Orleans) is the same one-way message-passing discipline pulled inside one application: independent actors communicate only by sending immutable messages, never by sharing memory, so during a rolling upgrade an actor's mailbox can hold messages encoded by both the old and new code — making the message encoding a versioned contract just like a cross-service one.

All three modes share one operational reality that forces the two-way discipline of §3: the rolling upgrade. Rather than stopping the whole fleet and swapping every instance at once (a "big-bang" deploy that means downtime and no rollback), production systems replace instances a handful at a time. For the duration — minutes to hours — the old and new versions run simultaneously, writing to the same database, calling the same services, and sharing the same message log. This is the concrete reason you need backward AND forward compatibility at the same time, not one or the other: in that window a new instance reads bytes an old instance just wrote (needs backward-compat), and an old instance reads bytes a new instance just wrote (needs forward-compat). Neither direction is hypothetical — both are happening on the same cluster, in the same minute. The same is true for client/server skew (mode B, where a new server must answer old clients while old servers still answer new clients mid-rollout) and for replayed log records (mode C). Plan every schema change as expand → migrate → contract so both versions stay legible throughout.

Worked number — the cost of a non-atomic deploy

You run 50 service instances and roll a new version 5 at a time, draining ~2 minutes each: the cluster is mixed-version for ~20 minutes. At 8,000 requests/second, that's roughly 8,000 × 60 × 20 = 9,600,000 requests served while old and new code coexist. If the new version had added a required field to the response, every request an old client touched in that window — call it half, ~4.8M — risks a decode failure. With the field added as optional + default, that number is zero: old clients skip the unknown tag, new clients fill the default. One word in the schema, optional, is the difference between a clean deploy and a 20-minute partial outage.

5 · ML tie: a feature record's schema is the model's input contract

Everything above lands hard in ML infra, where schema drift doesn't crash — it silently corrupts. A feature record (the encoded vector of inputs for one entity, written to a feature store, a training-example file, or a TFRecord/Parquet shard) is exactly the kind of long-lived data this lesson is about: training reads records written months ago; serving reads records written milliseconds ago; both must agree with the model's input signature.

Consider a model trained on v1 features. Over two quarters the pipeline evolves: v2 renames user_age to age_years; v3 drops device_os and adds session_embedding. Now run the v1 model on v3 data:

model input contract (v1) data produced by pipeline (v3) user_age : float age_years : float (renamed!) device_os : string session_embedding: float[64] (added) clicks_7d : int clicks_7d : int (unchanged) (device_os removed) If the loader matches by NAME: user_age not found -> 0.0 / null device_os not found -> 0.0 / null Two features quietly become constant. No exception. The model still returns a score. AUC just drops, and nobody knows why.

The model didn't error because the encoding "succeeded" — it just resolved missing fields to defaults, exactly as backward-compat is supposed to. The lesson: a rename in a feature pipeline is only free if the model's input contract is versioned alongside it. So treat feature definitions, dataset schemas, and a model's input signature as data contracts: version them, validate at the ingestion boundary, and record the schema version in both the dataset snapshot and the model registry. When serving code and training code disagree on schema, you have train/serve skew even though both programs run cleanly — the most expensive class of ML bug precisely because nothing throws.

Failure modes

Language-native serialization as a wire/storage contract. Locks bytes to one runtime and one class; an RCE vector on untrusted input; breaks the moment a class is renamed.
Adding a required field. Old data lacks it and old writers never set it — breaks both directions. New fields must be optional with a default.
Reusing a retired tag. An old record still carries the old meaning at that tag; the new reader misinterprets ancient bytes as the new field.
Read-modify-write that drops unknown fields. An old reader decodes, re-saves, and silently erases fields written by newer code. Preserve unknown fields on round-trip.
Avro without the writer's schema. No tags means the bytes are undecodable alone; lose the schema (file header or registry) and the data is gibberish.
Silent feature drift. A renamed/removed feature resolves to a default; no exception, AUC quietly decays — train/serve skew.

Decision checklist

What is the boundary — durable storage, RPC, or async log? Choose the format for it.
Is every new field optional with a sensible default?
Are field tags treated as immutable, and are retired tags never reused?
For Avro: how does the writer's schema travel — file header or schema registry?
Do readers preserve unknown fields across a read-modify-write?
Is the schema version stamped into dataset snapshots and the model registry?
Is there ingestion-time validation that fails loud on a contract mismatch instead of defaulting silently?
Does the rollout follow expand → migrate readers → contract, with retention windows respected before removal?

Checkpoint exercise

Try it

Design a safe migration of a feature record from user_age (int) to birth_year (int), consumed by both a training pipeline and a live-serving model, across a rolling deploy. Specify: (1) which format you'd use and why; (2) the field tags before and after; (3) the expand-contract sequence — what gets written and read at each phase so that an old serving binary and a new training job never disagree; (4) what the model registry must record so a model trained in phase 1 can't be silently fed phase-3 data with a constant-zero feature.

Where is truth?

For an evolving encoded record, the system of record is the schema — the .proto/IDL or the Avro schema in the registry — paired with the immutable field tags; that, not any one byte stream, is what defines what a record means. The copies / derived views are every encoded instance on disk, on the wire, and in the message log, written at many schema versions at once. The freshness budget is the rolling-deploy window plus the retention horizon of the oldest reachable record — both old and new code must stay legible across it. The owner is whoever controls the schema registry and the tag namespace. The deletion path is deprecate-then-retire a field, never reuse its tag. The reconciliation/repair path is re-encoding old records forward, or resolving writer-vs-reader schemas by tag (Protobuf) or by name (Avro). The evidence it is correct is round-trip tests across version pairs plus ingestion-time schema validation that fails loud instead of defaulting silently.

Where this points next

We now have a record's bytes and a discipline for evolving them across deploys — a contract that holds true over time. But we've been silent about what those bytes are measured against once real users arrive: what stays correct when components fail, what stays fast enough when load grows, and what stays changeable as humans keep editing. That's the yardstick. Lesson 06 defines it: reliability, scalability, and maintainability — the three non-functional properties every later mechanism buys one of and bills another. Encoding kept meaning stable through change; next we face demand, and learn to grade designs by which promise they protect and which they tax.

Takeaway

Encoding turns pointer-laden in-memory objects into a self-contained byte sequence — a contract across both languages and time, which language-native serializers honor on neither axis. Text formats (JSON/CSV) repeat field names and stay loosely typed; binary schema formats drop the names, encoding each field as a numeric tag plus value, which is both the size win and the evolution mechanism. Because rolling deploys run old and new code at once, encodings must be backward compatible (new code reads old data) and forward compatible (old code tolerates new data): add fields only as optional-with-default, never reuse a tag, and renames are free because names live in the schema, not the bytes. Tag-based formats (Protobuf/Thrift) match by tag and let writer/reader schemas drift; Avro matches by name and must ship the writer's schema. Data flows through databases, services, and message logs, each stressing a different direction of compatibility. In ML, a feature record's schema is the model's input contract — version it, or a rename quietly becomes train/serve skew that drops your metric with no error to chase.

Interview prompts

Why is Python pickle a poor choice for data you store long-term or send between services? (§1 — ties bytes to one runtime and class definition, is an RCE risk on untrusted input, and has no real versioning story; encoding is a contract across languages and time.)
How does a binary schema format encode a record more compactly than JSON? (§2 — it omits field names entirely, writing a small numeric tag + wire type + value per field; the name lives only in the shared schema.)
Define backward vs. forward compatibility and say which a rolling deploy needs. (§3 — backward = new code reads old data; forward = old code tolerates new data; a rolling upgrade runs both versions at once, so it needs both.)
Walk through adding, removing, and renaming a field safely in Protobuf. (§3 — add = new optional tag with default; remove = stop writing an optional field but never reuse its tag; rename = free, the tag is unchanged and names aren't in the bytes.)
Contrast how Protobuf and Avro stay evolvable. (§3 — Protobuf matches fields by immutable tag and is self-describing enough to skip; Avro carries no tags, matches by name, and must transport the writer's schema via file header or registry.)
What are the three dataflow modes and which compatibility direction does each stress? (§4 — database: long-lived data, backward-compat + preserve unknown fields; service/RPC: both directions across independently deployed client/server; async log: forward-compat for lagging consumers.)
A model trained on v1 features scores fine on v3 data but its AUC drops — what happened? (§5 — a renamed/removed feature resolved to a default, so a feature went silently constant; no error, just train/serve skew. Version the input contract and validate at ingestion.)