Algorithm — the plugin interface

The only piece of the system that's actually different across PPO, GRPO, RLOO, DAPO, and Dr.GRPO. Everything else is identical. This lesson defines the interface; Part II walks each algorithm in depth.

The two-method contract

In this framework, an Algorithm answers exactly two questions:

1. Advantage: Given a group of trajectories with rewards, what advantage does each token get?
2. Loss: Given a batch with advantages, old log-probs, and reference log-probs, what is the loss as a function of the current policy's log-probs?

That's it. The rollout engine doesn't know which algorithm is running. The trainer doesn't either. Swapping GRPO for DAPO is one file change — exactly the design pressure that makes algorithm.py a plugin and not a tangle of conditionals throughout the trainer.

class Algorithm(Protocol):
    def compute_advantages(self, groups: list[list[Trajectory]]) -> None:
        """Write traj.advantage on every Trajectory in place."""

    def compute_loss(self, batch: Batch, new_logp: Tensor) -> tuple[Tensor, dict]:
        """Return (scalar_loss, metrics_dict). new_logp flows gradient."""

Why this is one role, not five

Look at any two algorithms in RL/algorithms/ side by side — GRPO vs DAPO, or REINFORCE vs Dr.GRPO. They differ in:

• What the advantage is. (Raw reward? r − mean? Divided by std? Leave-one-out? GAE from a value head?)
• What the loss looks like. (Multiply A · log π? Use a PPO ratio? Clip symmetrically? Asymmetrically? Normalize per-token or per-sequence?)

They don't differ in: how rollouts are sampled, what the reference is for, where gradients flow, when weights sync. Those live in other roles. The whole point of factoring algorithm out is that everything algorithmic sits behind these two methods, and the trainer / rollout / controller never need to know about it.

The shared signature — what flows in

By the time compute_advantages is called, every Trajectory has:

The algorithm's job is to fill in one more field — traj.advantage — then later, when given a packed Batch and the trainer's fresh new_logp, produce a scalar loss.

Interactive · feel the advantage assignment

Three rollouts. You set their rewards. The table shows what advantage each algorithm assigns. This is a preview of lesson 11's widget — read the depth there. The point here: this table is the only thing different across algorithms.

The loss — same skeleton, different fillings

Once advantages are written, every algorithm produces its loss the same way:

# Simplified from RL/framework/algorithm.py — the shared skeleton
log_ratio = (new_logp - batch.old_logp) * mask
ratio     = torch.exp(log_ratio)

# ── ONE OF THESE LINES is the algorithm-specific part ──
# REINFORCE: pg_tok = -batch.advantage * new_logp
# GRPO / PPO: pg_tok = -torch.min(ratio*A, clip(ratio, 1-eps, 1+eps)*A)
# DAPO:       same min, but clip uses asymmetric (1-eps_low, 1+eps_high)
# Dr.GRPO:    same min, but A is built without /std

pg_loss = _masked_mean(pg_tok, mask)                    # token-level
kl_loss = beta * _masked_mean(compute_kl_k3(new_logp, ref_logp), mask)
loss    = pg_loss + kl_loss

The KL anchor (lesson 03) is added by every algorithm — it doesn't really "belong to" any of them but we apply it inside compute_loss because new_logp and ref_logp are already in the batch. Same arithmetic, one fewer call site for the masking.

Where to read each algorithm