☆ Bonus

Suggestions for further exploration, in rough order of effort:

  • Everyone has a price, revisited. If you skipped the optional exercise in section 3, go back to it: find a concrete layout in which breaking an urn is return-maximising even under reward2, and verify your prediction by training an agent in that layout. Can you design a reward function under which breaking urns is always suboptimal, no matter the layout?
Solution

Seal the shards behind a solid wall of urns, so the only way to reach them is to smash through:

# column 3 is urns top to bottom; a 3x3 pile of shards is sealed behind it
env_wall = Environment(
    init_robot_pos=t.tensor((3, 1), dtype=t.long),
    init_items_map=t.tensor(
        (
            (0, 0, 0, 2, 0, 1, 1, 1),
            (0, 0, 0, 2, 0, 1, 1, 1),
            (0, 0, 0, 2, 0, 1, 1, 1),
            (0, 0, 0, 2, 0, 0, 0, 0),
            (0, 0, 0, 2, 0, 0, 0, 0),
            (0, 0, 0, 2, 0, 0, 0, 0),
            (0, 0, 0, 2, 0, 0, 0, 0),
            (0, 0, 0, 2, 0, 0, 0, 0),
        ),
        dtype=t.long,
    ),
    bin_pos=t.tensor((0, 0), dtype=t.long),
)

The detour-vs-smash comparison from section 3 (with env_price swapped for env_wall) gives detour +0.000 smash +1.416: no shard is reachable without breaking an urn, and the best plan breaks the one at (0, 3).

No fixed penalty fixes this. Put enough shards behind the wall and the reward for binning them outweighs any finite cost for breaking an urn. To make breaking always suboptimal you'd need a penalty that grows with what's behind the wall, or urns the robot can't break.

A trained agent never learns it anyway. Training PPO here (ppo.train_agent(env_wall, net, reward_fn=reward2, ...)) for 512–2048 steps across several seeds, the agent learns nothing: reward2, reward_break and a bin-only probe (like the reward_bin helper in the reward_shaped reference solution) all stay near zero. The −2 comes first and the +1s come only after a long, precise sequence of moves, so PPO learns to avoid urns long before it ever stumbles on the payoff. Shaping, a curriculum, an exploration bonus, or starting the robot behind the wall would help.

So a behaviour being optimal under the reward is necessary but not sufficient for an agent to learn it: the agent also has to find it.

  • Break the agent. Our reward2 agent clears essentially the whole shop in the fixed layout. The trainer in part32_goal_misgen/ppo.py gets there by reusing each batch of rollouts for several epochs of minibatch updates with a fairly high learning rate (lr=0.003 in ppo.train_agent). Pass num_epochs=1, num_minibatches=1 to ppo.ppo_train_step and/or change the optimiser's learning rate to 0.001 in ppo.train_agent, and retrain: how many piles does the agent clean now, and how much longer does it take? (A single update per batch and lr 0.001 typically gets stuck cleaning one pile.)

  • Other shaping potentials. The inventory potential is the simplest useful potential. Try a distance-based potential (e.g. negative distance from the robot to the nearest shards while empty-handed, or to the bin while loaded). Does it speed up learning? Does it stay un-hackable, as the theory guarantees?

  • Partial shift. Instead of training on generate_shift (bin anywhere), try training on a distribution where the bin appears in two of the four corners. How does the policy generalise to the other two corners? What does this tell you about the inductive biases of the network?

  • Measure misgeneralisation as a function of diversity. Sweep the fraction of training environments that come from generate_shift vs generate (e.g. 0%, 1%, 10%, 50%), and plot the proxy return on shifted environments as a function of this fraction. How much diversity is enough?

  • Read the literature. Langosco et al. (2022) and Shah et al. (2022) catalogue goal misgeneralisation examples (the CoinRun example is a direct big sibling of our bin-in-the-corner example); Krakovna et al.'s specification gaming list is a fun collection of real-world reward hacks (blog post); and Ng, Harada & Russell (1999) is the original potential shaping paper, proving that potential shaping is the only form of shaping that never changes the optimal policy.

  • Competence is not alignment. Write a probe reward_pickup that scores +1 whenever the robot picks up a pile of shards (regardless of where it later drops them), and use evaluate_behaviour to show that under the bin-moved shift, net3 keeps reward_pickup HIGH (it is still a competent shard-collector) even as reward2 collapses.