1️⃣ Multi-digit multiplication

Learning Objectives

• Understand what makes a task suitable for RLVR (verifiable, tunable difficulty, large input space)
• Understand how the task generator, difficulty ladder and answer parser fit together, and implement the verifiable reward

In our harness a task is a function (rng, difficulty) -> (prompt, answer), where answer is the ground truth the reward will check. We'd like a task that is genuinely hard for the base model, but not impossible (we need the base model to already get non-zero performance so it has some learning signal to latch onto). Tasks where we can dynamically adjust the difficulty up and down at will are ideal, so we can find a sweet-spot of "the base model can only just barely do it, and RL can reinforce the rollouts that succeed". Multi-digit multiplication is a good task for this: The model is already aware of the concept of multiplication and how to do it, but struggles with larger numbers. A large input space of numbers to multiply also gives us confidence that the model isn't just memorizing what it has learned from pre-training.

Below is our task generator: given an rng and the difficulty (how many digits each operand has), generate a prompt asking the model to multiply two numbers, and the answer the verifier will check against.

# Difficulty ladder: (name, range of a, range of b). Base-model sampled pass@1 (Qwen3-0.6B-Base, 768-token cap, n=64):
# Measured success rates on the first six rungs: 0.56 / 0.44 / 0.30 / 0.16 / 0.08 / 0.05
DIFFICULTY_LADDER = [
    ("3x2", (100, 999), (10, 99)),
    ("4x2", (1000, 9999), (10, 99)),
    ("3x3", (100, 999), (100, 999)),
    ("4x3", (1000, 9999), (100, 999)),
    ("4x4 (b<3000)", (1000, 9999), (1000, 2999)),
    ("4x4", (1000, 9999), (1000, 9999)),
    ("4x5", (1000, 9999), (10000, 99999)),
    ("5x5", (10000, 99999), (10000, 99999)),
]


def mult_byhand(rng: random.Random, difficulty: int = 0) -> tuple[str, int]:
    """Generate a multiplication problem.

    Args:
        rng: Random number generator.
        difficulty: Index into `DIFFICULTY_LADDER`.

    Returns:
        The prompt, and the answer a * b.
    """
    _, (a_lo, a_hi), (b_lo, b_hi) = DIFFICULTY_LADDER[difficulty]
    a, b = rng.randint(a_lo, a_hi), rng.randint(b_lo, b_hi)          # e.g. 100..999 x 10..99 on the first rung
    prompt = f"Compute {a} * {b}. Reason step by step, then give the final answer as \\boxed{{N}}.\n"
    return prompt, a * b

Performance of the base model

Before training, let's check the base model is mediocre but not hopeless at the given problem, so that at least some rollouts get rewarded. Larger Qwen models solve multi-digit multiplication reliably, but our small Qwen/Qwen3-0.6B-Base base can't reliably solve the problem. We'll play with this for the moment.

The prompt we use is a f-string of the format

f"Compute {a} * {b}. Reason step by step, then give the final answer as \\boxed{{N}}.\n"

which when printed for a problem with a = 948 and b = 39 looks like

Compute 948 * 39. Reason step by step, then give the final answer as \boxed{N}.

The trailing newline matters. A base model just continues the document: without it, the completion carries on the same line (" To", " The", and ~10% of the time " What", i.e. another question), and pass@1 on 3×2 drops from 0.56 to 0.50 (3×3: 0.24 → 0.19). You might like to adjust the difficulty (the rung of DIFFICULTY_LADDER) and see how the base model's success rate falls: it is usually right on 3×2, wrong more often than not on 3×3, and almost never right on 4×4.

Exercise - Qwen3 playground

Play around with the various Qwen models (especially Qwen3-0.6B-Base, as that's the one we'll be fine-tuning) to try and generate rollouts to solve this task. Convince yourself that this is a difficult task for Qwen3-0.6B-Base to solve, and that even with very strong prompting or a few shot examples, it still struggles to get the correct answer given sufficient difficulty (I find that 4x4 multiplication can be quite taxing for the base model). You might like to get a frontier LLM to help generate suitable prompts for you.

If you run the following command,

cd chapter2_rl/exercises/part42_rlvr/playground
python app.py           # Qwen/Qwen3-0.6B-Base on port 8088 by default

it should serve on localhost:8088 a browser tool which you can use to play around with generations from models in the Qwen family. VSCode/Cursor should prompt you with a popup that the remote machine is serving something on port 8088 (which should already be forwarded to your local machine). If not, follow this guide to forward the port manually, or run this command in a LOCAL terminal on your machine (that is, not the VSCode/Cursor terminal, but actually go open a separate terminal on your device and run the command) to forward the port manually.

ssh -N -L 8088:localhost:8088 my-arena-server-name

If that still doesn't work (or you're on Colab, sorry!), then you can sample from the base model directly with Hugging Face instead. prompt is the prompt generated by the task generator above, and example is an (optional) worked example that gets prepended to the prompt as a few-shot demonstration.

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B-Base")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B-Base", dtype=t.bfloat16).to(device)

a = 948
b = 39
prompt = f"Compute {a} * {b}. Reason step by step, then give the final answer as \\boxed{{N}}.\n"

example = """Let's compute 17 x 34 by hand. We multiply all cross terms:
17 x 34 = (10 + 7) x (30 + 4) = 10 x 30 + 10 x 4 + 7 x 30 + 7 x 4 = 300 + 40 + 210 + 28 = 578.
Therefore the answer is 578. \\boxed{578}."""

inputs = tokenizer(example + "\n\n" + prompt, return_tensors="pt").to(device)
with t.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=1.0, top_p=0.95,
                         num_return_sequences=10, pad_token_id=tokenizer.pad_token_id)
completions = tokenizer.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
for c in completions:
    print(c, "\n---")

Compute 123 × 62 by hand. Let's compute the partial products.
123 × 2 = (100 + 20 + 3) × 2 = 200 + 40 + 6 = 246.
123 × 60 = (100 + 20 + 3) × 60 = 6000 + 1200 + 180 = 6000 + 1380 + 7380
7380 + 246 = 7626.
Therefore 123 × 62 = 7626.
\boxed{7626}

Compute 123 * 62 by hand using long multiplication. Multiply 123 by each digit of 62 to get the partial products, then add them up. Do NOT write or run any code — show the arithmetic by hand, step by step. Give the final answer as \boxed{N}.
Let's compute the partial products, one digit of 62 at a time.

Partial product 1 — multiply 123 by the units digit 2:
123 × 2 = (100 + 20 + 3) × 2
        = 100×2 + 20×2 + 3×2
        = 200 + 40 + 6
        = 246.

Partial product 2 — multiply 123 by the tens digit 6 (which is worth 60):
123 × 60 = (100 + 20 + 3) × 60
         = 100×60 + 20×60 + 3×60
         = 6000 + 1200 + 180
         = 7380.

Now add the partial products 246 + 7380 by columns, right to left, carrying when a column exceeds 9:
  units:      6 + 0 = 6,                   write 6, carry 0
  tens:       4 + 8 = 12,                  write 2, carry 1
  hundreds:   2 + 3 = 5, plus carry 1 = 6, write 6, carry 0
  thousands:  0 + 7 = 7,                   write 7, carry 0
Reading the written digits from the thousands column down to the units gives 7626.

Check by partial sums: 7380 + 246 = 7380 + 200 + 40 + 6 = 7580 + 40 + 6 = 7620 + 6 = 7626.
Therefore 123 × 62 = 7626.
\boxed{7626}

We follow the convention of requiring the model to provide the answer in the form \boxed{N}, where N is the answer, and rewarding it for doing so. This then makes it easy for the verifier to extract out the answer from the model's output.

Here we have the function to extract out the boxed integer from the model's completion. It's a regex to find the last \boxed{N} in the text, and extracts and returns the integer N inside, returning None if no boxed integer is found.

def extract_boxed_int(text: str) -> int | None:
    """Return the integer in the last `\\boxed{N}` of `text`, or None."""
    # Matches \boxed{42}, \boxed{ -7 }, etc. Only plain integers are captured.
    pattern = r"\\boxed\{\s*(-?\d+)\s*\}"

    matches = re.findall(pattern, text)
    if not matches:
        return None

    last_match = matches[-1]
    return int(last_match)

tests.test_extract_boxed_int(extract_boxed_int)

The reward function is simple: as per RLHF, the reward on every timestep is zero until the model has completely generated the full rollout (or got cut off early due to the max token limit). The reward on the last timestep is then

You can try shaping the reward later on to be more complex to give the model a hint when it's close to the correct answer, but for now we'll keep it simple and use a hard 0 or 1 reward for the correct answer, plus a formatting bonus if the answer was wrong but correctly formatted. Using a more complex reward function runs the risk of the model learning to game the reward function instead.

Exercise - the verifiable reward (int_reward)

Implement the reward for one GRPO group: the G completions sampled for the same problem, scored against its answer. Like the reward functions in RLHF, it returns a tensor, here of shape (G,). For each completion:

• 1.0 if the boxed integer exactly equals the answer.
• 0.1 if there's a well-formed boxed integer but it's wrong.
• 0.0 otherwise.

A for-loop is fine for this exercise. Use extract_boxed_int from earlier to extract out the answer (if it exists) from a given completion.

def int_reward(completions: list[str], answer: int) -> Float[Tensor, " G"]:
    """Reward each completion of one problem: 1.0 if correct, 0.1 if boxed but wrong, else 0.0."""
    raise NotImplementedError()


tests.test_int_reward(int_reward)

def int_reward(completions: list[str], answer: int) -> Float[Tensor, " G"]:
    """Reward each completion of one problem: 1.0 if correct, 0.1 if boxed but wrong, else 0.0."""
    rewards = []
    for text in completions:
        boxed = extract_boxed_int(text)
        if boxed is None:
            rewards.append(0.0)
        elif boxed == answer:
            rewards.append(1.0)
        else:
            rewards.append(0.1)
    return t.tensor(rewards)