Part V - AI applications

RAG on Kubernetes

Lesson 15 taught you to share GPUs fairly between latency-sensitive inference and bulk batch work without one starving the other. RAG is the first application where that tension lives inside one product: a heavy write-time pipeline that reads, chunks, and embeds a corpus, and a thin read-time path that must answer a user in well under a second. The two pipelines are coupled by choices they must agree on — the same embedding model, the same chunking, the same index — yet they run as completely different Kubernetes workloads. This lesson treats RAG not as "a model plus a database" but as two systems that silently break each other when their assumptions drift apart.

1 · RAG is two pipelines that must agree

RAG = Retrieval-Augmented Generation. Instead of relying only on what a language model memorized during training, you retrieve the most relevant chunks of your own documents at query time and paste them into the prompt, so the model answers from fresh, attributable, private context. The model is unchanged; the system around it does the work.

That system is really two pipelines with very different shapes. The write path (ingestion) runs offline: it reads source documents, splits them into chunks (chunking — cutting a long document into passages small enough to embed and retrieve individually), turns each chunk into a vector with an embedding model (a model that maps text to a fixed-length vector — say 768 or 1536 floats — so that semantically similar text lands nearby in that space), and stores those vectors in a vector database (a store that indexes embeddings for fast approximate nearest-neighbor (ANN) search — finding the closest vectors to a query vector without the prohibitive cost of comparing against every stored vector exactly). The read path (query) runs online, on the hot path: it embeds the user's question, searches the vector DB for the nearest chunks, optionally re-scores them with a reranker, assembles a prompt, and calls the LLM to generate an answer with citations.

The mental model: a vector DB is a derived-data product, like a search index or a materialized view, not a pile of embeddings. It is computed from sources by a specific recipe. The instant the read path uses a different recipe than the one that built the index, the two pipelines are speaking different languages — and nothing crashes. This is why the book treats RAG as an application architecture: the model is one component, and the hardest bugs live in the consistency between write and read.

2 · The write path — ingestion as Jobs and CronJobs

Ingestion is bulk GPU/CPU work that processes a whole corpus. It must never run inside the serving Deployment: a re-ingestion of millions of documents would consume the exact GPU capacity your users' queries need, and lesson 15's lesson applies directly — give ingestion lower priority and its own quota so it yields to online inference.

Kubernetes mapping: a one-time backfill is a Job; periodic refresh ("re-ingest changed docs nightly") is a CronJob; a complex multi-stage flow with retries and fan-out is a workflow engine (Argo Workflows, or a queue with worker Deployments). The embedding step needs model-serving capacity but lives off the hot path, so it can run on cheaper or spot GPUs and tolerate restarts — embedding is a pure function of (chunk, model version), so a preempted spot pod simply re-embeds the chunks it lost on restart with no corruption.

The embed step is the throughput bottleneck, and batching is the lever. Embedding models are small relative to an LLM, but a corpus is large, so the write path lives or dies on tokens-per-second. A single GPU running an embedding model in batches of 256 chunks might sustain ~200 chunks/s once the batch is full; at batch size 1 it manages a fraction of that because the GPU sits idle between requests. So ingestion deliberately accumulates chunks into large batches before calling the model — the opposite of the read path, which embeds one query at minimum latency. The two paths use the same model with opposite batching strategies, which is one more reason they are different workloads. For a 10M-chunk corpus at 200 chunks/s, the embed step alone is 10,000,000 ÷ 200 = 50,000 s ≈ 13.9 GPU-hours on one GPU — the same figure section 4 will charge you for any full re-ingestion.

Index writes are upserts, not appends. When a source document changes, you do not pile new vectors on top of the old ones — you re-chunk it, embed the new chunks, and upsert by a stable key (document ID + chunk index) so the new vectors replace the old, and you delete chunks for passages that vanished. Make this idempotent: re-running ingestion on an unchanged document must be a no-op (hash the chunk text and skip if the stored hash matches), so a retried or duplicated Job does not double-index the corpus. Without idempotency, every retry inflates the index and your nightly CronJob slowly fills the disk with stale duplicates.

apiVersion: batch/v1
kind: CronJob
metadata: { name: rag-reindex }
spec:
  schedule: "0 2 * * *"            # nightly, off-peak
  jobTemplate:
    spec:
      template:
        spec:
          priorityClassName: batch-low   # yields to online inference (lesson 15)
          containers:
          - name: ingest
            image: rag-ingest:emb-v3      # pinned embedding-model version
            env:
            - { name: EMBED_MODEL,  value: "text-embedding-v3" }
            - { name: CHUNK_TOKENS, value: "512" }
            - { name: CHUNK_OVERLAP, value: "64" }
            - { name: INDEX_SCHEMA, value: "schema-v3" }
            resources: { limits: { nvidia.com/gpu: 1 } }
          restartPolicy: OnFailure

Notice the version stamps live in the spec: the embedding model, chunk policy, and index schema are pinned together. The image tag emb-v3 and the schema schema-v3 move as one. That pinning is the whole game — it is what lets the read path declare which contract it expects and refuse to query an index built under a different one.

3 · The read path — query under a latency budget

The query path is a latency-sensitive Deployment behind a Service. Every stage spends part of a budget — say a 700 ms target for time-to-first-token. A typical breakdown:

The reranker earns its latency by fixing a weakness of vector search. ANN retrieval scores each candidate by embedding similarity computed independently — the query was turned into one vector and each chunk into another, long before they ever met — so the score never sees the two pieces of text side by side. That is fast and gives good recall (the right chunk is usually somewhere in the top-50) but coarse precision (the single best chunk may be ranked 12th). A reranker is a cross-encoder: it feeds the query and one candidate into a model together, so attention can compare them token by token, and emits a sharper relevance score. The pattern is over-fetch then re-score: pull top-50 from ANN (cheap), rerank all 50 (one batched model call), keep top-5 for the prompt. You trade ~100 ms for materially better precision in what the model finally sees. Drop the reranker when the budget is tight and recall@5 is already high; keep it when wrong context is expensive — a confidently-cited but irrelevant chunk is worse than no chunk.

Two refinements worth naming. Hybrid search runs a keyword/lexical index (BM25) alongside the vector index and fuses the two rankings, because pure embedding search misses exact-match tokens — error codes, SKUs, names — that share no semantic neighborhood with the query. And top-k is a knob, not a constant: too small and the right chunk never enters the candidate set (a recall miss the reranker cannot fix); too large and you pay more rerank latency and risk diluting the prompt with marginal chunks. Tune k on the same fixed eval set you use to track recall.

Two non-negotiables on the read path. First, access-control filters must run at search time, not after: the vector query carries the user's permission tags so the ANN search only considers chunks the user is allowed to see. Filtering after retrieval is both a leak risk and a correctness bug — it risks the model reading and citing a document the user cannot access, and it also wrecks top-k, because if 8 of your top-10 ANN hits are filtered out post-hoc you are left with 2 chunks when the model needed 5. Push the filter into the index so the top-k you ask for is the top-k of permitted chunks. Second, every retrieved chunk carries its provenance, so the generated answer cites a real document and version, and an auditor can trace any sentence back to its source — a RAG answer with no resolvable citation is indistinguishable from a hallucination.

4 · The consistency trap and the cost of getting it wrong

Here is the failure that defines RAG operations. Suppose someone upgrades the embedding model on the ingestion side from v2 to v3 — better quality, everyone is happy — but the query service still embeds questions with v2. Now the stored vectors live in v3's space and the query vectors live in v2's space. Cosine similarity between two vectors from different embedding spaces is meaningless: it still returns a number, the search still returns "nearest" chunks, the answer still looks plausible. Nothing throws. Retrieval quality just quietly collapses, and you discover it weeks later from a slow drip of bad answers.

The clean way to upgrade an embedding model, then, is a dual-index cutover: build a new index (schema-v3) with the new model while the live query path keeps serving the old one (schema-v2), and flip an alias only once the new index is fully built and its quality validated on the eval set. The query service is pinned to a model version and an index alias together, so it never embeds a v3 question against a v2 index. Cutover is one atomic alias swap; rollback is the reverse swap. This is the same pattern as a blue-green deploy, applied to derived data — and it is exactly the checkpoint exercise below.

The same coupling makes index loss expensive. Because the index is derived data, you can in principle rebuild it — but rebuilding means re-embedding the entire corpus, and that is not cheap.

Therefore: operate the vector DB like the stateful system it is. It runs as a StatefulSet — the Kubernetes controller for pods that own durable, individually-identified storage, where each replica gets a stable name (vdb-0, vdb-1) and its own PersistentVolume that survives reschedules, unlike the interchangeable, disk-less pods of a Deployment. That stable identity is what lets a vector shard keep its data when its pod restarts. On top of it you need the full stateful discipline:

An untested restore is the difference between a 60-second snapshot recovery and the multi-hour re-ingestion incident above. The dollar cost of re-embedding is trivial; the cost is the capacity and the downtime, on the very GPUs your users are queued behind.

5 · Failure modes and what reveals them

Checkpoint exercise

Where this points next

RAG gives a model trustworthy context for a single question-and-answer turn. But an agent does not stop at one retrieval — it plans, calls tools, observes results, and loops, and retrieval becomes just one tool it can choose to invoke (often several times, with refined queries). Lesson 17, Agents on Kubernetes, adds the machinery for that loop: long-running stateful sessions, tool-call orchestration, concurrency and memory limits, and the scheduling implications of workloads that fan out unpredictably. The write/read consistency discipline you built here stays — the agent's retrieval tool is exactly this RAG read path — but now it sits inside a controller that decides when to retrieve.

Interview prompts

• Why is RAG "two pipelines" and what binds them? (§1 — a throughput-bound write path and a latency-bound read path, coupled by a contract: the same embedding model, chunker, index schema, and provenance, versioned together.)
• Why does ingestion belong in Jobs/CronJobs instead of the serving Deployment? (§2 — it is bulk GPU work; at batch priority with its own quota it yields to online inference instead of starving the query latency budget, per lesson 15.)
• What does a reranker buy and why does it cost latency? (§3 — ANN search ranks coarsely by embedding similarity; a cross-encoder reads query+candidate together to re-score the top-k for precision, ~100 ms for materially better final context.)
• Describe the embedding-model mismatch failure and why it is dangerous. (§4 — upgrading the embedding model on write but not read puts stored and query vectors in different spaces; similarity still returns numbers, so quality degrades silently with no error.)
• Why is an untested vector-DB restore worse than it sounds? (§4 — the index is derived data, so loss without a tested snapshot forces re-embedding the whole corpus, e.g. ~14 GPU-hours for 10M chunks — minutes become a multi-hour incident on shared GPUs.)
• Where must access-control filtering happen, and why? (§3 — at search time, so the index only returns permitted chunks; filtering after retrieval risks the model reading or citing documents the user cannot access.)
• Why measure retrieval and generation quality separately? (§5 — they fail independently; tracking recall@k on a fixed eval set distinct from answer quality tells you whether a regression is in retrieval or in the model.)
• How do you upgrade an embedding model with no quality regression and no downtime? (§4 — a dual-index cutover: build a new index with the new model while serving the old one, validate recall on the eval set, then swap an alias atomically and pin the query service to (model version + index) together; rollback is the reverse swap.)
• Why does ingestion batch embeddings while the query path does not? (§2 — ingestion is throughput-bound, so it accumulates large batches (e.g. 256) to keep the GPU saturated (~200 chunks/s); the read path embeds one query at minimum latency. Same model, opposite batching — one reason they are separate workloads.)
• Why add lexical/BM25 hybrid search on top of vector search? (§3 — pure embedding search misses exact-match tokens (error codes, SKUs, names) that share no semantic neighborhood with the query; fusing a keyword ranking with the vector ranking recovers them.)