5️⃣ Goal Misgeneralisation

Learning Objectives
  • Understand the definition of goal misgeneralisation, and the distinction between the intended objective and the behavioural objective
  • Write a reward function encoding the behavioural objective, and use it to demonstrate goal misgeneralisation quantitatively
  • Understand how broadening the training distribution can fix goal misgeneralisation — and why this isn't always available in practice

⚠️ Make sure you have finished the previous section before reading further.

If all has gone to plan, you should have found at least one example of goal misgeneralisation. Let's move forward with the following example. When the policy is tested on environment layouts where the bin is outside of the corner, we expect to see the following:

  1. the policy generalises in terms of its ability to pick up shards, carry them to a destination, and drop them there; but
  2. the policy fails to generalise in terms of its behavioural goal of carrying the shards to the location of the bin.

In this section, we will unpack this example and explore the effect of the procedural environment generator on the way the policy generalises its goal.

Exercise - elements of goal misgeneralisation

Difficulty: 🔴🔴🔴⚪⚪
Importance: 🔵🔵🔵🔵🔵
You should spend up to 20-25 minutes on this exercise.

Here is an excerpt from Langosco et al. (2022) where the authors informally define goal misgeneralisation:

2.1 Defining Goal Misgeneralisation \ A deep RL agent is trained to maximize reward $R : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$, where $\mathcal{S}$ and $\mathcal{A}$ are the sets of all valid states and actions, respectively. Assume that the agent is deployed under distributional shift; that is, an aspect of the environment (and therefore the distribution of observations) changes at test time. Goal misgeneralization occurs if the agent now achieves low reward in the new environment because it continues to act capably yet appears to optimize a different reward $R' \neq R$. We call $R$ the intended objective and $R'$ the behavioral objective of the agent.

Your next task is to line up the elements of this definition with our case of goal misgeneralisation:

  1. What is the distribution shift? Provide an example of an environment from the test distribution, env_shift (you can use one of the environments you designed for the previous exercise).
  2. What is the behavioural objective in this case? Identify it and then write a reward function proxy that encodes this behaviour. (The tests count a drop at (0, 0) whether or not the bin happens to be there.)
  3. Use evaluation.evaluate_behaviour and the histogram code from section 3 to show that the policy achieves low return under the training reward function but high return under the behavioural objective in env_shift.
env_shift = ...
def proxy(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
    raise NotImplementedError()


tests.test_env_shift(env_shift)
tests.test_proxy(proxy)
reward_fns = [reward2, proxy]
return_vecs = [
    evaluation.evaluate_behaviour(
        env=env_shift,
        net=net3,
        reward_fn=r,
        generator=t.Generator().manual_seed(1),
    )
    for r in reward_fns
]

utils.plot_return_hists(
    reward_fns,
    return_vecs,
    title="net3 on env_shift (bin moved): low reward2 (intended) but high proxy (corner-drop) = misgeneralisation",
)
Solution (discussion)
  1. The distribution shift: during training, the bin was always in the top-left corner (0, 0); at test time, the bin is somewhere else. The example env_shift we suggest above has the bin in the top-right corner.

  2. The behavioural objective: "carry shards to the top-left corner and drop them there". During training, this objective and the intended objective ("carry shards to the bin") were perfectly correlated: the policy had no way to distinguish them, and the simpler/more learnable one won out. The proxy reward function above encodes it: +1 for dropping shards at (0, 0) (wherever the bin is), and 0 otherwise.

  3. The histograms should show the policy achieving much lower return under reward2 than it does in-distribution (it almost never actually bins anything in env_shift) while scoring solidly positive return under proxy: the agent is still competently optimising something, it's just not the thing we trained for. (If reward2 shows a small constant positive return, about $0.5\gamma^{64} \approx 0.36$, that isn't binning: evaluate_behaviour stops after 64 steps while the agent is still holding shards, so the shaping potential it was paid on pickup is never paid back. This is the finite-trajectory caveat from the optional shaping proof.)

Note: We call the behavioural objective proxy because it's correlated with the training reward function on the training environment distribution.

Discussion on goal misgeneralisation

There would be no way to tell apart two agents, one of which has learned the behavioural objective, and one that has learned the intended objective, without either some sort of white-box method that allows us to examine the internals of the model to see what the agent is optimising for, or observing the behaviour of the agents on out-of-distribution environments. Here, it's quite easy to infer what the out-of-distribution behaviour might be, but in general this is an open problem, as it's not a priori always clear what out-of-distribution inputs might look like. A sufficiently capable agent that is aware that it is in training might deliberately act in accordance with the intended objective during training to prevent being updated by training dynamics, and then pursue its own goals once in deployment. Anthropic observed this phenomenon in 2024.

Exercise - implement the distribution shift

Difficulty: 🔴🔴🔴⚪⚪
Importance: 🔵🔵🔵🔵⚪
You should spend up to 10-15 minutes on this exercise.

In the previous two exercises, you manually generated environment layouts in which the policy misgeneralises.

Your next task is to automate the construction of these environments by writing a new procedural environment generator that samples from a broader distribution of environments.

In particular, write a function generate_shift, a modification of generate from above, that randomises not only the item and robot spawn locations, but also the bin location.

Hint

An easy way to do this is to incorporate the bin placement into the same without-replacement sample as the robot and the other items: permute all size**2 cells (don't reserve cell 0), and use the first sampled cell for the bin, the second for the robot, and the rest for the items.

def generate_shift(
    size: int,
    num_shards: int,
    num_urns: int,
    num_envs: int,
    generator: t.Generator | None = None,
) -> Environment:
    raise NotImplementedError()

    return Environment(
        init_robot_pos=robot_pos,
        init_items_map=items_map,
        bin_pos=bin_pos,
    )


tests.test_generate_shift(generate_shift)
Solution
def generate_shift(
    size: int,
    num_shards: int,
    num_urns: int,
    num_envs: int,
    generator: t.Generator | None = None,
) -> Environment:
    num_cells = size**2

    # sample bin, robot and item positions without replacement, by taking the
    # first few cells of a random permutation of all grid cells
    num_positions = 1 + 1 + num_shards + num_urns
    perm = t.rand(num_envs, num_cells, generator=generator).argsort(dim=1)
    positions = perm[:, :num_positions]
    rows, cols = positions // size, positions % size
    bin_pos = t.stack((rows[:, 0], cols[:, 0]), dim=-1)
    robot_pos = t.stack((rows[:, 1], cols[:, 1]), dim=-1)

    # create item map
    items_map = t.zeros((num_envs, size, size), dtype=t.long)
    batch = t.arange(num_envs)[:, None]
    items_map[
        batch,
        rows[:, 2 : 2 + num_shards],
        cols[:, 2 : 2 + num_shards],
    ] = Item.SHARDS
    items_map[
        batch,
        rows[:, 2 + num_shards :],
        cols[:, 2 + num_shards :],
    ] = Item.URN

    return Environment(
        init_robot_pos=robot_pos,
        init_items_map=items_map,
        bin_pos=bin_pos,
    )

You can visually check your generator with the following code — the bin should now appear all over the grid:

envs = generate_shift(
    size=6,
    num_shards=3,
    num_urns=4,
    num_envs=32,
    generator=t.Generator().manual_seed(1),
)
utils.display_envs(
    envs,
    grid_width=8,
    title="Shifted distribution from generate_shift(): bin now appears anywhere on the grid",
)

Training out of distribution

In principle, an easy way to fix goal misgeneralisation is to train a policy in a broader distribution of levels, like that generated by generate_shift as opposed to generate. If we train a new policy using this new environment generator, we should see goal misgeneralisation decrease. This task is harder than the last one (the agent now has to find the bin rather than always heading for the corner), so it takes longer: expect the return to stay near zero for the first ~50 steps, climb over the next ~150, and level off around step 250. If your return curve hasn't levelled off by the end, train for another hundred steps or so before drawing conclusions.

size = 6

net4 = ActorCriticNetwork.init(
    obs_height=size,
    obs_width=size,
    net_channels=16,
    net_width=64,
    num_conv_layers=5,
    num_dense_layers=2,
    num_actions=len(Action),
    generator=t.Generator().manual_seed(1),
)

net4 = ppo.train_agent_multienv(
    gen=functools.partial(
        generate_shift,
        size=size,
        num_shards=4,
        num_urns=2,
    ),
    net=net4,
    reward_fn=reward2,
    num_train_steps=256,
    seed=1,
)
reward_fns = [reward2, proxy]
return_vecs = [
    evaluation.evaluate_behaviour(
        env=env_shift,
        net=net4,
        reward_fn=r,
        generator=t.Generator().manual_seed(1),
    )
    for r in reward_fns
]

utils.plot_return_hists(
    reward_fns,
    return_vecs,
    title="net4 on env_shift (bin moved): reward2 recovers and proxy collapses to ~0 = fix",
)

Mitigating goal misgeneralisation in practice

In practice, we might want to prevent goal misgeneralisation in a situation where:

  1. We only have access to a given distribution of environments analogous to generate;
  2. We face some unknown distribution shift in deployment, analogous to evaluating in environments sampled from generate_shift; but
  3. We don't have access to generate_shift during training!

In this case, the simple solution of just training in environments from generate_shift might not be available. Finding ways to mitigate this kind of goal misgeneralisation is an active research area.

Conclusion

Hopefully you have gained an appreciation for the relationship between a designer's intention, a reward function, and a policy's behaviour in reinforcement learning:

  • Goal misspecification/reward hacking/outer alignment failure: In training environments, if there are behaviours that score higher return than the designer's intended behaviours according to the reward function, then the policy will pursue those instead.
  • Goal misgeneralisation/inner alignment failure: In out-of-distribution environments, the behaviour of the policy is not necessarily determined by what the reward function would have incentivised: rather it comes down to the inductive biases of the agent architecture. A goal that is "simpler", and that was perfectly correlated with the intended goal during training, will often be the same goal that the policy will continue to pursue later in environments where the intended goal, and the learned goal, diverge.