Part V - AI applications

Agents on Kubernetes

Lesson 16 gave a model the ability to read trustworthy context: a retrieval path that pulls facts into the prompt and a provenance trail that says where each fact came from. An agent goes one step further — it can act. It calls tools, books things, writes to systems, and sometimes hands work to other agents. The moment an agent can act, the platform must answer a new set of questions that RAG never had to: on whose behalf is this action taken, with what authority, through which tool, recorded where, and resumable how? This lesson maps those answers onto Kubernetes primitives so that tool access, identity, and state are explicit objects you can audit — not implicit behavior hidden inside the model's loop.

1 · An agent is a workload with authority

A RAG service is, at bottom, a function: text in, text out. It reads from a vector store but it does not change the world. An agent is different. It runs a loop — observe, decide, call a tool, observe the result, decide again — and the tools it calls send email, open pull requests, charge cards, delete files, spin up infrastructure. The defining property is not intelligence; it is authority. An agent is a workload that has been granted the ability to act on systems beyond itself.

Concretely, the loop is the thing that distinguishes an agent from a chatbot. A single-shot model call is a function with no lasting consequence. The agent loop turns that function into a process with memory and side effects:

while not done:
    obs    = read(context, last_tool_result)   # observe
    plan   = model(obs)                         # decide  (model RECOMMENDS)
    if plan.action.needs_gate:                  # costly/irreversible/sensitive?
        wait_for_human_approval()               # explicit workflow state
    token  = exchange(user_token, agent_svid,   # downscope to least privilege
                      scope=plan.action.scope)
    result = tool_call(plan.action, token)      # ACT  (platform DECIDES)
    checkpoint(run_id, step, obs, result)       # persist before next step

Every line of that loop touches a platform surface — the gate, the token exchange, the tool call, the checkpoint. That is why agents cannot be treated as "just another microservice with a clever prompt": each iteration is a place where authority is exercised and state must survive.

That single property is what makes agents a platform problem rather than an application detail. A stateless web service needs a Deployment, a Service, and an HPA. An agent needs all of that plus four surfaces that a stateless service never had:

The mental model for the whole lesson: Kubernetes already gives you the substrate for all four — workload identity, network boundaries, persistent volumes and external stores, events and Jobs, and admission/network policy. The agent framework (LangGraph, CrewAI, an ADK runtime, the model server's own tool loop) must fit inside those boundaries, not invent its own shadow versions of them. When a framework keeps identity in an environment variable, state in the chat transcript, and tool access in a hard-coded list, it has reimplemented the platform badly. The job of this lesson is to push each surface back onto the primitive that already exists for it.

2 · The three identities you must keep separate

The most common and most dangerous mistake in agent platforms is collapsing three distinct identities into one credential. Name them and keep them apart:

These answer three different questions — who is asking, what is acting, and what is it allowed to do right now — and conflating any two of them is how authority leaks. If the agent simply reuses the user's identity for everything, the agent inherits the user's full permissions (over-broad). If it uses only its own service identity for everything, it loses the user context needed for per-user authorization and audit. The correct shape keeps all three and combines them at the moment of a tool call: this user, through this agent, is permitted this narrow capability.

SPIFFE/SPIRE: issuing the agent's service identity

SPIFFE (Secure Production Identity Framework For Everyone) is a standard for naming workloads — a SPIFFE ID looks like a URI, e.g. spiffe://prod.example/ns/agents/sa/research-agent. SPIRE is the reference implementation that runs as a server plus a per-node agent: it attests a pod (verifies it really is the workload it claims, using the node, its ServiceAccount, and selectors) and then issues it a short-lived SVID (SPIFFE Verifiable Identity Document) — typically an X.509 certificate or JWT that rotates every few minutes. The point: the agent pod gets a cryptographic, automatically rotated identity tied to what it is, not a long-lived secret baked into an image. A downstream tool or another agent can verify the caller's SPIFFE ID via mTLS before deciding anything. This is the Kubernetes-native way to make "service identity" a real, verifiable thing rather than a string in a config map.

3 · The central failure: token passthrough vs. token exchange

Here is the security mistake that defines this lesson. A user authenticates at the gateway with a token that carries their full permissions. The agent needs to call a tool on the user's behalf. The lazy implementation simply passes the user's token through to the tool — and to the next tool, and to any sub-agent. This is token passthrough, and it is a textbook confused deputy: the agent, a deputy acting for the user, now wields the user's entire authority for a task that needed a sliver of it. If the agent is compromised, if a tool is malicious, or if the model is prompt-injected into calling the wrong tool, that broad token is right there to be abused. The agent ends up with more authority than the task — or even the user, in that narrow context — should have.

The fix is token exchange: at the moment of a tool call, the agent presents the user token and its own service identity to a token service (an OAuth 2.0 Token Exchange endpoint, RFC 8693), which downscopes them into a fresh, least-privilege, short-lived token good for exactly one tool and one operation. The user token never reaches the tool; the tool only ever sees a narrow capability. Compare the two request paths:

PASSTHROUGH (confused deputy)
  user --[full user token]--> gateway --[same full token]--> agent
       --[same full token]--> tool A
       --[same full token]--> tool B        <-- every hop holds full authority

EXCHANGE (least privilege)
  user --[user token]--> gateway (authn) --> agent (own SPIFFE identity)
       agent + user token --> token-exchange service
                          --> [scoped: tool=email, action=send, ttl=60s] --> tool
       different tool call --> token-exchange --> [scoped: tool=calendar, read, ttl=60s] --> tool
                                                  ^ user token never leaves the boundary

A note on placement and scaling. Token exchange adds one round-trip per tool call to the token service — typically a few milliseconds against a cached or co-located issuer, negligible next to the tool call and the model inference it sits between. The cost is operational, not latency: you must run and trust a token-exchange service (or your cloud's STS) and configure each tool to accept only its own narrow scope. That is the trade — a small standing service in exchange for collapsing the blast radius. The token service and the SPIRE server are themselves privileged workloads; lock them down with NetworkPolicy and run them in their own namespace, because whoever can mint tokens or identities owns the whole agent fleet.

4 · Bind tools through MCP or an explicit gateway

An agent's tools should not be ad-hoc HTTP calls scattered through application code. MCP — the Model Context Protocol — is a standard for schema-bound tool and capability discovery and invocation: an MCP server advertises a typed list of tools (name, input schema, output schema, description), the agent discovers them at runtime, and every call is validated against that schema. The value on a platform is that the set of capabilities is explicit and inspectable: you can list what an agent can reach, version it, and put a policy in front of it, instead of grepping code for `requests.post`.

On Kubernetes, an MCP server is just another workload — a Deployment behind a Service, reachable only by the agents that need it (lock the path down with NetworkPolicy so a compromised agent cannot reach tools outside its job). The agent calls the MCP server; the MCP server holds the scoped tool credentials minted by token exchange and makes the actual outbound call. That gives you a single chokepoint to enforce schema validation, rate limits, allow-lists, and audit logging — the "explicit tool gateway" pattern. The MCP tool list is itself an attack surface: a malicious or over-broad tool description can steer the model, so the catalogue of advertised tools needs the same review discipline as any other privileged interface.

The single-chokepoint shape also makes the network policy trivially expressible. The agents call the MCP gateway and nothing else outbound; the gateway is the only workload with egress to the actual tool endpoints. A minimal NetworkPolicy that pins the gateway as the sole talker to a tool looks like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: tools-via-gateway-only, namespace: agents }
spec:
  podSelector: { matchLabels: { role: tool-backend } }
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - podSelector: { matchLabels: { app: mcp-gateway } }  # ONLY the gateway
    # agents have no path here; a compromised agent cannot reach the tool directly

5 · Externalize state: checkpoints and reconstructable A2A

An interactive agent that answers in two seconds can keep its working state in memory. A long-running agent — a research run that takes twenty minutes, an automation workflow that spans hours waiting on a human approval — cannot. Pods are ephemeral: a node drain, an OOM kill, a rolling update, or a spot reclaim restarts the pod at any time. If the agent's loop state lived only in memory, the entire multi-step run is lost and must start over — re-paying for every tool call and model token already spent.

The fix is the same one RAG taught for vector stores: state lives outside the pod, and the agent checkpoints after each meaningful step. A checkpoint is a durable snapshot of the run — its step index, accumulated observations, pending tool calls, and any human-gate status — written to an external store keyed by a run ID. On restart, the agent loads the latest checkpoint and resumes from the next step. Pick the store by access pattern (a production system usually uses all three):

run R-9af1, checkpoint after step 4
{
  run_id:   "R-9af1",  user: "u-204",  agent_svid: "spiffe://.../research-agent",
  step:     4,
  status:   "awaiting_human_approval",      # <-- a real workflow state, not a chat line
  observations: [ ...step 1-4 results... ],
  pending_tool_call: { tool: "spend", args: {amount: 4200}, needs_gate: true },
  audit:    [ {step:3, tool:"search", scoped_token:"...", ts:...}, ... ]
}
# pod restarts -> load latest checkpoint for R-9af1 -> resume at step 5 (or the gate)

On Kubernetes this maps cleanly: an interactive agent is a Deployment behind a Service; a long-running agent is often a Job (or a worker consuming a queue) that checkpoints to the external store; an ambient/triggered agent is an event-driven worker or CronJob. A human-in-the-loop gate becomes an explicit workflow state — awaiting_human_approval above — that the run sits in until an approval event arrives, not a chat message the model might ignore. Put these gates wherever an action is irreversible, costly, or policy-sensitive (deletes, spend above a threshold, anything touching production).

Checkpointing alone is not enough; resume must be idempotent. Consider the worked failure: the agent calls the "charge card" tool, the charge succeeds, and the pod is killed before it can write the checkpoint recording that success. On restart it loads the last checkpoint, sees the charge as still pending, and charges again — a double-charge born of a crash, not a model error. The fix is to make the side-effecting step carry an idempotency key (the run ID + step index), so the tool deduplicates a repeated call, and to record the intent to act before acting and the outcome after. This is exactly why the checkpoint stores pending_tool_call as a distinct field: on resume the agent first asks "did this already happen?" rather than blindly re-issuing it. Externalized state buys resumability; idempotent steps buy correct resumability.

A2A must be reconstructable

A2A — agent-to-agent communication — is what happens when a coordinator agent delegates a sub-task to a specialist agent. Mechanically it is ordinary service-to-service communication, but with stronger contracts: discover the peer's capability, send a typed task, track its status, collect the result, and preserve the original user's request context across the hop. The failure to avoid is opacity: if delegated calls are unlogged, a multi-agent failure cannot be reconstructed afterward — you see a bad final answer and have no way to know which agent, on which sub-task, with which inputs, went wrong. So A2A rides on the same primitives as everything else: each agent exposes a Service, NetworkPolicy limits who may call whom, the caller's SPIFFE identity tells the receiver who is asking, the downscoped token (not the user's full token) authorizes the sub-task, and a shared trace ID + structured message log ties every hop back to the originating user request. Without those, a multi-agent system is just distributed hidden state.

6 · One tool call, all three identities at once

Pull the pieces together with a single concrete call. A user asks a research agent to "email the Q3 summary to finance." Trace the identities and tokens at the moment the agent invokes the email:send tool:

1. user u-204 authenticates at gateway        -> user token (40 scopes, ttl 1h)
2. gateway forwards request to research-agent  -> agent runs as spiffe://.../research-agent
3. agent decides: send_email(to=finance, body=summary)
4. agent -> token-exchange(user=u-204, agent_svid, scope="email:send", aud="email-tool")
                                               -> scoped token (1 scope, ttl 60s)
5. agent -> mcp-gateway  (mTLS: gateway verifies agent's SPIFFE ID)
6. mcp-gateway validates args vs email:send schema, checks allow-list,
   presents scoped token to email-tool        -> email sent
7. checkpoint(run, step, intent+outcome, scoped_token_id) -> audit row
   trace_id ties steps 1-7 to user u-204's original request

Read off the three identities: the user is u-204 (step 1), the service identity is the agent's SPIFFE ID (steps 2, 5), and the delegated tool authority is the 60-second single-scope token (step 4). At no point does the user's 40-scope token leave the gateway boundary, no action runs without a schema-validated, policy-checked, audited path, and the whole chain is reconstructable from one trace ID. If a single one of these is missing — passthrough at step 4, no SPIFFE check at step 5, no checkpoint at step 7 — the corresponding failure mode below is what you get.

Failure modes and checklist

Checkpoint exercise

Where this points next

This lesson deliberately stayed on the platform side of the boundary — identity, tools, state, gates. The orchestration side — how the agent loop, routing, tool selection, and multi-agent coordination are actually designed — is the subject of a whole separate track. If you want the application-level depth behind the patterns named here, the Agentic Systems track covers the agent design patterns in full, and its MCP lesson goes deeper on the protocol than the platform framing above. Read this lesson as "where those patterns land on Kubernetes" and that track as "how to design the patterns themselves."

You now have the four agent surfaces mapped onto Kubernetes primitives: service identity via SPIFFE/SPIRE, least-privilege tool access via token exchange and MCP, externalized checkpointed state, and explicit human gates. Lesson 18 — Capstone: Production GenAI Platform — assembles every layer of the track into one reference platform: GPU scheduling and serving from Parts II-III, autoscaling and routing, RAG from lesson 16, and the agent identity/state machinery from this lesson, wired together with the observability and policy that make it operable. Treat the capstone as the place where these isolated patterns become a single system you could put on call for.

Interview prompts

• What changes when a workload becomes an agent rather than a stateless service? (§1 — it gains authority to act on external systems, which adds four surfaces a stateless service lacks: identity, bounded tools, externalized state, and human gates — each mapped onto a Kubernetes primitive.)
• Name the three identities an agent platform must keep separate and why. (§2 — user identity (who is asking), service identity (what the agent is — a SPIFFE/SVID), and delegated tool authority (a narrow scoped token for one call). Conflating any two leaks authority or loses audit context.)
• Explain token passthrough vs. token exchange. (§3 — passthrough forwards the user's full token to tools, a confused deputy with huge blast radius; exchange downscopes user token + agent identity into a least-privilege, short-lived, single-tool token so the user token never reaches the tool.)
• What do SPIFFE and SPIRE give you on Kubernetes? (§2 — SPIFFE names workloads as URIs; SPIRE attests a pod and issues a short-lived, auto-rotating SVID (X.509/JWT) so the agent has a verifiable cryptographic service identity instead of a baked-in secret.)
• What does MCP solve, and what does it not solve? (§4 — it makes tool capabilities schema-bound, discoverable, and inspectable; it does not authorize calls — the platform still checks identity and presents the downscoped token, and the tool list itself is an attack surface to review.)
• Why must long-running agent state be externalized and checkpointed? (§5 — pods are ephemeral (drains, OOM, preemption); in-memory state is lost on restart, re-paying every tool call and token. Checkpoints keyed by run ID let the agent resume from the next step, with an audit trail.)
• Why must A2A communication be logged with identity and trace context? (§5 — otherwise a multi-agent failure is opaque and cannot be reconstructed; SPIFFE identity, scoped tokens, and a shared trace ID tie every delegated hop back to the originating user request.)