[2.4.2] RLVR

Colab: exercises | solutions

Please send any problems / bugs on the #errata channel in the Slack group, and ask any questions on the dedicated channels for this chapter of material.

If you want to change to dark mode, you can do this by clicking the three horizontal lines in the top-right, then navigating to Settings → Theme.

Links to all other chapters: (0) Fundamentals, (1) Transformer Interpretability, (2) RL.

Introduction

🚧 Work in progress: this material is new and still being polished, so don't treat it as final.

So far we've been using RL mostly to change the behaviour of the model, like generating sentences with more periods in them, or outputting sentences with positive sentiment.

We now introduce Reinforcement Learning from Verifiable Rewards (RLVR), a method to teach the model new capabilities that it either couldn't do before, or would greatly struggle with before. We do this by supplying problems for the model to solve, having the model generate a long chain-of-thought before answering, automatically grading the answer against a verifier, and reinforcing the rollouts that lead to the correct answer via some RL algorithm.

To achieve this, the reward function needs access to some sort of ground truth (the answers to set of benchmark questions, a calculator, a python interpreter with a battery of unit tests, a formal proof verifier, etc.) that can be used to verify the answer. Unlike RLHF where the reward function is some sort of reward model that's a proxy for human preferences, and is thus vulnerable to reward hacking, the reward function here is (usually) immune to reward hacking, as the only possible way to receive positive reward is to generate a rollout that leads to the correct answer.

For this reason, we usually don't care much about the rollouts from the model remaining in-distribution of what the base model would say. Almost by definition, if the model is learning a new capability that it didn't have before, rollouts that succeed must be out-of-distribution from the base model.

Optional Reading:

  • From REINFORCE to Dr. GRPO (Blog Post)
    • Covers a quick history of MDPs, Policy Gradient, baseline functions, ReMax, RLOO, PPO, GRPO and Dr. GRPO
    • Dr. GRPO (GRPO with minor changes) is the algorithm we use today.

It's pretty clear that if we train a model using RLVR from scratch (that is, regular RL, without any pre-training, against an unhackable reward function), the model learns new capabilities that it couldn't possibly have had before. We've seen this on the previous days of RL: an uninitialized network clearly cannot balance a pole, nor could there be any elicitation of capabilities as with randomly generated weights, there is no capability to elicit. Now, pre-trained models already have lots of abilities, and need to be at least somewhat capable of the task at hand (otherwise there would never be good rollouts to reinforce). Whether RLVR fine-tuning of an LLM is really teaching the model something new, or merely eliciting a dormant capability, is still up for debate. At the end of this section we'll explore this claim and find the honest answer is unclear: given enough samples the base model often catches up or exceeds the performance of the RL'd model, yet RL still improves the model's one-shot performance which is in practice what we care about. Does mere sample efficiency count as "teaching" the model something new? Does RL do anything real, or does it just "wake up" skills hidden in the base model? I leave to you to decide.

Content & Learning Objectives

A note on what you'll build vs what's provided

As in [2.4.1] RLHF, we fine-tune a Hugging Face base model (Qwen/Qwen3-0.6B-Base) with LoRA, but this time we use the peft library's LoRA rather than the LoRALinear you wrote there. To keep the focus on the RL ideas:

  • You build the parts that are RLVR: the verifiable reward, the Dr. GRPO objective, the difficulty controller that keeps every question near a 50% solve rate so the reward stays maximally informative, and the rollout and learning phases of the trainer.
  • We provide the plumbing that isn't part of the lesson: the task generator and its difficulty ladder, the answer parser, loading the model/tokenizer, applying a LoRA adapter with peft, fast sampling, the Dr. GRPO advantages, micro-batching, and evaluation.

1️⃣ The verifiable problem

The task that we will be training on is multi-digit multiplication, which is trivial to verify with a calculator, while allowing us to easily adjust the difficulty up or down as need be. We provide the task generator and a parser that extracts the answer from the rollout; you'll write the reward function that scores it.

Learning Objectives
  • Understand what makes a task suitable for RLVR (verifiable, tunable difficulty, large input space)
  • Understand how the task generator, difficulty ladder and answer parser fit together, and implement the verifiable reward

2️⃣ GRPO with verifiable rewards

Agents learn when the questions are of the right difficulty that they sometimes get it right, so then we can positively reinforce the correct answers, and negatively reinforce the incorrect answers. We achieve this with a dynamic curriculum that adjusts the difficulty of problems up as the agent improves.

Learning Objectives
  • See why a verifiable reward lets us drop the KL-to-reference term, and what Dr. GRPO changes vs GRPO
  • Understand why a ~50% solve rate makes the reward maximally informative for group-relative advantages
import contextlib
import os
import random
import re
import sys
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable

from liveplot import LivePlot
from tqdm.auto import tqdm

# Must be set before torch initialises CUDA; reduces allocator
# fragmentation, which matters when the trainer is running near the 16GB limit of an A4000.
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")

import numpy as np
import torch as t
from eindex import eindex
from jaxtyping import Float, Int
from peft import PeftModel
from torch import Tensor
from transformers import PreTrainedTokenizerBase

# Make sure exercises are in the path
chapter = "chapter2_rl"
section = "part42_rlvr"
root_dir = next(p for p in Path.cwd().parents if (p / chapter).exists())
exercises_dir = root_dir / chapter / "exercises"
section_dir = exercises_dir / section
if str(exercises_dir) not in sys.path:
    sys.path.append(str(exercises_dir))

from part42_rlvr import tests, utils

# With PyTorch's default of one thread per core, on machines that report many more cores than they let you use
# (common in cloud containers), CPU-side work spends longer waiting on threads than computing. Adjust if needed.
t.set_num_threads(min(4, t.get_num_threads()))

device = t.device("mps" if t.backends.mps.is_available() else "cuda" if t.cuda.is_available() else "cpu")

# Optional speed-up: Liger's fused RMSNorm / SwiGLU / RoPE kernels make the learning phase ~15-25% faster (same results up
# to bf16 rounding). They're Triton kernels, so only on GPUs where those work (pre-Blackwell); otherwise we skip them.
try:
    if device.type == "cuda" and t.cuda.get_device_capability()[0] < 10:
        from liger_kernel.transformers import apply_liger_kernel_to_qwen3

        apply_liger_kernel_to_qwen3(rope=True, rms_norm=True, swiglu=True, cross_entropy=False, fused_linear_cross_entropy=False)
        print("Applied Liger kernels!")
except ImportError:
    print("Skipping Liger kernels...")
    pass


MAIN = __name__ == "__main__"
SLOW = False  # set True to run the slow optional experiments (the pass@k comparison takes ~20 min on an A40)