[2.3.2] - Goal Misgeneralisation
Please send any problems / bugs on the #errata channel in the Slack group, and ask any questions on the dedicated channels for this chapter of material.
If you want to change to dark mode, you can do this by clicking the theme toggle button (sun / moon icon) in the top navigation bar.
Links to all other chapters: (0) Fundamentals, (1) Transformer Interpretability, (2) RL, (3) LLM Evaluations, (4) Alignment Science.
System RequirementsGPU🟩 OptionalTime⏱️ GPU <1 minColab🟢 YesDownload⬜ <10 MBKeys🔑 None
- 🖥️ GPU: 🟩 Optional, peak ~0.9 GiB.
- ⏱️ Runtime (ex. downloads/API): GPU <1 min · CPU ~13 min
- ☁️ Colab? 🟢 Yes
- 💾 Downloads: ⬜ Under 10 MB
- 🔑 API keys: None
- 🐏 RAM: ~2.8 GB at peak
GPU times measured on a 16 GB RTX A4000. Compare every day →
Introduction
When we train an agent with reinforcement learning, we communicate what we want it to do through a reward function. Today is about the two most important ways this can go wrong:
-
Goal misspecification (also known as reward hacking, specification gaming or less commonly, outer misalignment): the agent finds a way to achieve high reward in a way that doesn't match the behaviour the operator intended. The reward function is optimised, but the intent behind it is not, as the reward function was not a perfect specification of the desired behaviour.
-
Goal misgeneralisation: the agent learns a goal that is consistent with the reward function on the training distribution, but which diverges from the intended goal under distribution shift. The agent remains capable in the new environments, it just capably pursues the wrong thing.
Goal misgeneralisation is the more subtle of the two: even if you could perfectly specify a reward function that fully encapsulates all of the desires you have of the agent, the agent might internalize the wrong goal. From a black-box perspective, an agent that has learned the wrong goal can be indistinguishable from one that has learned the intended goal (while on the training distribution).
We'll explore both of these failure modes hands-on, in a simple grid-world environment called the pottery shop: a small robot shares a shop floor with fragile urns and piles of broken shards, and we'd like it to clean the shards up without smashing anything in the process. You'll train agents with PPO, watch them learn to hack a naively-designed reward function, fix the specification using potential shaping and penalties, and then discover that even a well-specified reward function doesn't protect you from goal misgeneralisation when the deployment environment is out of distribution.
A highly optimized implementation of PPO (Proximal Policy Optimization) and the network architecture is already provided for you for training. Instead, today you'll be working one level up: designing environments and reward functions, and studying the behaviour of the agents they produce.
For a lecture on the material today, which gives the high-level picture before you dive into the exercises, see below.
Attribution Statement
These exercises are adapted with permission from a lab by Matthew Farrugia-Roberts, originally written for the AI Safety and Alignment course at the University of Oxford (MT 2025). The original lab and supporting library, written in JAX, can be found at github.com/matomatical/reward-lab. This codebase has been ported here to PyTorch for the ARENA programme.
This section first appeared as the goal misgeneralisation day of the Iliad Intensive (materials on GitHub; by David Quarel, based on the lab above), shared under the CC BY 4.0 licence.
Content & Learning Objectives
1️⃣ Environments
We introduce the environment we'll be working with today: the pottery shop.
Learning Objectives
- Understand how the pottery shop environment is implemented as batched tensor operations
- Build intuition for the dynamics of the environment
2️⃣ Reward Functions
We introduce the third element of reinforcement learning, the reward function, and use the provided PPO implementation to train an agent in the pottery shop.
Learning Objectives
- Interpret a reward function: what behaviour does it incentivise, and what behaviour was it meant to incentivise?
- Train an agent with the provided PPO implementation
- Interpret a trained agent's behaviour, and spot when it diverges from the designer's intent
3️⃣ Specification Gaming
The agent you just trained doesn't do what the reward designer had in mind. We diagnose the problem quantitatively using reward functions as behavioural probes, then fix the specification with potential shaping and an urn-breaking penalty.
Learning Objectives
- Recognise specification gaming / reward hacking when you see it
- Write reward functions that act as behavioural probes for misbehaviour
- Understand and implement potential shaping, a way to add hints to a reward function without allowing the agent to reward hack.
- Combine shaping terms and penalties into a fixed specification, and verify the fix
4️⃣ Generalisation
We move from a single fixed layout to a distribution of procedurally generated layouts, and train an agent that has to generalise across them.
Learning Objectives
- Understand generalisation in RL as a special case of generalisation in machine learning
- Train an agent on procedurally generated environments
- Probe how a trained policy behaves on layouts it has never seen
5️⃣ Goal Misgeneralisation
The generalising agent has learned to carry shards... to where the bin used to be. We pin down this example of goal misgeneralisation, measure it, and explore the role of the training distribution.
Learning Objectives
- Understand the definition of goal misgeneralisation, and the distinction between the intended objective and the behavioural objective
- Write a reward function encoding the behavioural objective, and use it to demonstrate goal misgeneralisation quantitatively
- Understand how broadening the training distribution can fix goal misgeneralisation — and why this isn't always available in practice
☆ Bonus
Theoretical and open-ended extensions: proving that potential shaping preserves the ordering over policies, finding situations where breaking an urn is still worth it, and more.
Readings (optional)
Outer alignment and specification gaming (sections 2️⃣–3️⃣)
- Outer alignment (LessWrong wiki). A short definition: the problem of specifying a reward function that captures what we actually want. Also known as reward misspecification.
- Specification gaming: the flip side of AI ingenuity (Krakovna et al., 2020) and its list of examples. Dozens of real agents finding unintended ways to score reward, much like our urn-smashing robot.
- Policy invariance under reward transformations (Ng, Harada & Russell, 1999). The original potential shaping paper, behind the fix in section 3️⃣. It shows that potential-based shaping never changes the optimal policy, and that no other kind of shaping has that guarantee.
Inner alignment and goal misgeneralisation (sections 4️⃣–5️⃣)
- Inner alignment (LessWrong wiki). The definition, how it relates to outer alignment, and a curated reading list. It treats goal misgeneralisation under distribution shift as an inner alignment failure.
- The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment (Robert Miles, video). An accessible introduction to inner alignment and mesa-optimisers. A good first stop if the terms are new.
- Goal Misgeneralization in Deep Reinforcement Learning (Langosco, Koch, Sharkey, Pfau, Orseau & Krueger, ICML 2022). The paper that named and first demonstrated goal misgeneralisation in RL. In their CoinRun experiment the coin always sits at the end of the level during training, so the agent learns "go right" rather than "get the coin". That's the big sibling of the bin-in-the-corner agent you'll meet in section 5️⃣.
- We Were Right! Real Inner Misalignment (Robert Miles, video). A walkthrough of the Langosco et al. results.
- Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals (Shah, Varma, Kumar, Phuong, Krakovna, Uesato & Kenton, 2022). More examples, including outside RL, and the argument in the title: fixing the reward function (as we do in section 3️⃣) doesn't stop the agent learning the wrong goal. It defines goal misgeneralisation without assuming the agent is a mesa-optimiser.
Going further
- Risks from Learned Optimization: Introduction (Hubinger, van Merwijk, Mikulik, Skalse & Garrabrant, 2019). The sequence that introduced mesa-optimisers and the inner/outer alignment split. Inner Alignment: Explain like I'm 12 Edition (Rafael Harth) is a friendlier summary of it.
- Outer vs inner misalignment: three framings (Richard Ngo). Why the inner/outer line is blurrier than it looks.
- Reward is not the optimization target (Alex Turner). Argues that reward shapes which behaviours get reinforced, and a trained agent needn't be trying to maximise it. Worth keeping in mind when you interpret what your agents learned.
Setup code
Colab users: select a GPU runtime before running anything (Runtime → Change runtime type → GPU). The code below picks up a GPU automatically if one is available; the later training runs are several times slower on CPU.
import arena_lib
exercises_dir, section_dir = arena_lib.setup("2.3.2")
import functools
from typing import Callable
import matplotlib.pyplot as plt
import numpy as np
import torch as t
from jaxtyping import Float, Int
from torch import Tensor
from tqdm.auto import tqdm
import part32_goal_misgen.evaluation as evaluation
import part32_goal_misgen.potteryshop as potteryshop
import part32_goal_misgen.ppo as ppo
import part32_goal_misgen.tests as tests
import part32_goal_misgen.utils as utils
from part32_goal_misgen.agent import ActorCriticNetwork
from part32_goal_misgen.evaluation import RewardFunction
from part32_goal_misgen.potteryshop import Action, Environment, Item, State
from liveplot import LivePlot
device = t.device("cuda" if t.cuda.is_available() else "mps" if t.backends.mps.is_available() else "cpu")
# On CPU, PyTorch's default of one thread per core hurts on big machines: the rollout loop is 64
# sequential environment steps on small tensors, so beyond ~8 threads the synchronisation overhead
# outweighs the parallelism (measured: 8 threads is fastest, 16 is ~30% slower, 48 is several
# times slower). This has no effect when running on a GPU.
if device.type == "cpu" and t.get_num_threads() > 8:
t.set_num_threads(8)
DISCOUNT_RATE = 0.995 # the shaped rewards below use this; keep it equal to `discount_rate` in `ppo.train_agent`