3️⃣ Specification Gaming
Learning Objectives
- Recognise specification gaming / reward hacking when you see it
- Write reward functions that act as behavioural probes for misbehaviour
- Understand and implement potential shaping, a way to add hints to a reward function without allowing the agent to reward hack.
- Combine shaping terms and penalties into a fixed specification, and verify the fix
Note: Finish the previous exercise before reading further.
If all has gone to plan, you should be looking at an example of goal misspecification (also known as reward hacking), in which the reinforcement learning algorithm found a better way to increase expected return than the class of valid solutions the reward designer had in mind.
The two kinds of discrepancies we expect to see in this example are:
- The agent prefers to repeatedly pick up and drop shards compared to picking them up once and taking them to the bin; and
- The agent is willing to break urns to find new shards to pick up.
Why are these behaviours incentivised?
Each pickup is worth +1, dropping shards on the floor costs nothing,
and a broken urn is just a fresh supply of shards to farm pickups from.
Depending on your layout and the randomness of training, the second behaviour may
be less prominent in the final policy than the first: once the agent has found a
pile of shards to farm with pickup-drop cycles, it never
needs to break an urn. But notice that nothing in reward1
discourages urn-breaking either: the agent smashes through any urns
on its path with total indifference, and we'd certainly rather it didn't.
In this section, we will redesign the reward function to prevent these unintended behaviours from being incentivised.
The inverse reward hypothesis
Recall the reward hypothesis from above. While this hypothesis is usually invoked in the context of designing an agent using reinforcement learning by identifying an appropriate reward, it applies more broadly as stated.
In particular, the reward hypothesis applies to the behaviour of existing agents. Suppose we have a system that is exhibiting some sort of coherent behaviour. Then (by the reward hypothesis), that behaviour can be well thought of as maximising expected return for some reward function.
One could call the above corollary the inverse reward hypothesis, following Stuart Russell (1998) and Andrew Ng and Stuart Russell (2000), who introduced the problem of inverse reinforcement learning (that of taking a policy and extracting from it a non-trivial reward function for which that policy maximises expected return).
Caveat: For any policy $\pi$, the constant zero reward $R_0(s,a,s') = 0$ trivially has the property that $\pi$ maximises expected return for $R_0$. What we would really desire is something more specific: a reward function $R_\pi$ such that $\pi$ maximises expected return for $R_\pi$, but policies that have learned different behaviour do not.
Exercise - write reward functions as behavioural probes
We won't trouble ourselves with inverse reinforcement learning algorithms today. Instead, let's just take a step in this direction by quantifying the misbehaviour of our agents using reward functions that incentivise each problematic behaviour. We won't use these reward functions for training, but we can use them to measure to what extent we are inadvertently training for these behaviours.
Your next task is to write one reward function that measures the extent to which an agent is engaging in each problematic behaviour:
- First, write a reward function
reward_dropthat assigns+1every time the agent drops a shard (other than in the bin). - Second, write a reward function
reward_breakthat assigns+1every time the agent breaks an urn.
Remember that these functions operate on batched transitions and should return one reward per transition (look back at
reward1for the idiom). There are a couple of ways to implement each of these, since the information is redundantly represented across the state, action, and next state: any correct implementation is fine.
Hint for reward_drop?
A drop (outside the bin) happens exactly when:
- the robot is not at the bin position, and
- the cell below the robot is empty (otherwise
PUTDOWNdoes nothing), and - the robot is holding shards, and
- the action is
PUTDOWN.
All four of these conditions are functions of state and action alone.
To compare positions per-environment, the boolean
(state.robot_pos != state.bin_pos).any(dim=-1) is true if either the x or y coordinates of
the robot and the bin don't match i.e. the robot is not at the bin.
Hint for reward_break?
An urn breaks exactly when the robot's new position (i.e. next_state.robot_pos)
is a cell that contained an urn before the transition and contains shards after it.
You can gather the item at the robot's new position from both state.items_map
and next_state.items_map.
Help! I'm having trouble with the batched indexing for reward_break?
The thing you want is a vector of the item beneath the robot, batched across each parallel
environment i.e. a vector item_below_robot : (B,) such that
```python no-copy item_below_robot[b] = state.items_map[b, next_state.robot_pos[b, 0], next_state.robot_pos[b, 1]]
for each environment `b`.
We could of course do this as a loop:
```python
item_below_robot = t.empty(num_envs)
for b in range(num_envs):
item_below_robot[b] = state.items_map[b, next_state.robot_pos[b, 0], next_state.robot_pos[b, 1]]
but this can be slow for a large number of environments, and is not something that we can accelerate with a GPU.
We can achieve this batched indexing with the following code:
batch_idx = t.arange(num_envs, device=state.inventory.device)
item_below_robot = state.items_map[
batch_idx,
next_state.robot_pos[:, 0],
next_state.robot_pos[:, 1],
]
which is a standard idiom for batched indexing. Note that
item_below_robot = state.items_map[
:,
next_state.robot_pos[:, 0],
next_state.robot_pos[:, 1],
]
is NOT the same thing, because it will instead collect
item_below_robot[b] = state.items_map[:, next_state.robot_pos[b, 0], next_state.robot_pos[b, 1]]
for each environment b, forming a matrix of shape (num_envs, num_envs) of which
the diagonal is what you actually wanted.
def reward_drop(state: State,
action: Int[Tensor, "B"],
next_state: State
) -> Float[Tensor, "B"]:
raise NotImplementedError()
def reward_break(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
raise NotImplementedError()
tests.test_reward_drop(reward_drop)
tests.test_reward_break(reward_break)
Solution
def reward_break(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
batch = t.arange(state.inventory.shape[0], device=state.inventory.device)
item_below_robot_after_transition = next_state.items_map[
batch,
next_state.robot_pos[:, 0],
next_state.robot_pos[:, 1],
]
item_there_before_transition = state.items_map[
batch,
next_state.robot_pos[:, 0],
next_state.robot_pos[:, 1],
]
return (
(item_below_robot_after_transition == Item.SHARDS)
& (item_there_before_transition == Item.URN)
).float()
Quantifying reward hacking
Having written these reward functions, we can get a quantitative
signal about whether our policy is engaging in these misbehaviours.
We can collect a batch of trajectories and then score them according
to each reward function and plot the result (with utils.plot_return_hists), using the following
convenience function (from part32_goal_misgen/evaluation.py):
```python no-copy def evaluate_behaviour( env: Environment, net: ActorCriticNetwork, reward_fn: RewardFunction, num_steps: int = 64, num_rollouts: int = 1000, discount_rate: float = 0.995, generator: torch.Generator | None = None, ) -> Float[Tensor, "num_rollouts"]
Note that, unlike a reward function that correctly incentivises
the *intended* behaviour, we ideally want these reward functions
*to be minimised.* Moreover, note that to tell if a reward function
is minimised or maximised, we need to consider the range of possible
returns, which depends on the reward function (in these simple
environments it can be derived analytically).
```python
reward_fns = [reward1, reward_drop, reward_break]
return_vecs = [
evaluation.evaluate_behaviour(
env=env,
net=net1,
reward_fn=r,
generator=t.Generator().manual_seed(1),
)
for r in reward_fns
]
utils.plot_return_hists(
reward_fns,
return_vecs,
title="net1 (trained on reward1): is the return real cleanup, or farming?",
)
Potential shaping
Regarding the first problematic behaviour (repeatedly picking up and dropping shards), one solution would be to remove the reward for picking up shards altogether. The intention is for the shards to end up in the bin, not in the inventory for their own sake, anyway.
However, sometimes rewarding instrumental goals can assist the agent's learning process. When we know that something is instrumentally useful for achieving a task, it would be nice if we could incorporate that knowledge into the reward.
The only problem is that incorporating these kinds of hints into the reward is a delicate operation: as we have seen, it can lead to misspecification and reward hacking if it becomes possible to satisfy the hint without completing the original task! We want the shaped reward to provide extra feedback to the agent to help it learn the desired behaviour, rather than changing the behaviour that the reward function selects for.
Fortunately, there is a sure-fire scheme for adding a so-called shaping term to a reward function without introducing such exploits. This is called potential shaping, and it works as follows:
- Formulate the information as a (bounded) function of states
$$ \Phi : \mathcal{S} \to \mathbb{R} $$called a potential function: $\Phi(s)$ is high in states that we know are good progress towards the task.
- When we transition from state $s$ to state $s'$, add
reward $\gamma \Phi(s')$ for entering state $s'$, and subtract
$\Phi(s)$ for leaving state $s$:
$$ R^\Phi(s,a,s') = R(s,a,s') + \gamma\Phi(s') - \Phi(s). $$
- The extra terms reward the agent for making progress, not for being somewhere. Moving into a high-potential state earns its potential, but moving out of it again pays that potential back. So the agent can't farm shaping reward by hanging around in high-potential states: over a whole trajectory, the extra terms cancel out (you'll prove this in the next exercise). What shaping does change is when the agent gets feedback. For example, with a potential that is high while the robot is holding shards, the agent is rewarded as soon as it picks shards up, rather than only later, when it drops them in the bin and earns the true reward. The potential is a temporary advance on that true reward: if the agent goes on to earn it, the potential is handed back as the robot empties its inventory, and if it doesn't (say it drops the shards on the floor), the potential is handed back anyway.
The reason this is useful is that the set of all optimal policies is unchanged under potential shaping (replacing $R$ with $R^\Phi$ for any potential function $\Phi$), but it can still change the training dynamics (hopefully in a way that makes learning faster).
Exercise (optional) - prove that potentials cancel
Let
- $\Phi : \mathcal{S} \to \mathbb{R}$ be a bounded potential function,
- $R : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$ be a bounded reward function, and
- $\gamma \in (0,1)$ a discount rate.
- $\mathcal{H}$ is the set of all trajectories (histories).
- $T = |\tau|$ is the number of timesteps a trajectory runs for (possibly infinite).
Assume throughout that the environmental dynamics $\mu : \mathcal{S} \times \mathcal{A} \to \Delta \mathcal{S}$ is fixed, and trajectories are sampled from the interaction between some policy $\pi$ and the environment $\mu$.
Given a potential function $\Phi$, define the shaped reward function $R^\Phi : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$ as
Given a reward function $R$, define the return function $G_R : \mathcal{H} \to \mathbb{R}$ that maps a trajectory $\tau = s_0, a_0, s_1, a_1, \ldots$ to
Given a trajectory $\tau$, a reward function $R$ and a potential function $\Phi$, prove that $G_R(\tau) = G_{R^\Phi}(\tau) + c$ for some additive constant $c$ that depends only on the initial state $s_0$ and the final state $s_T$.
Solution
So $c = \Phi(s_0) - \gamma^T \Phi(s_T)$, which depends only on the initial and final states. For an infinite trajectory there is no final state: $\Phi$ is bounded and $\gamma < 1$, so $\gamma^T \Phi(s_T) \to 0$ as $T \to \infty$, and $c = \Phi(s_0)$. (Splitting the sum is allowed in the infinite case too, because $R$ and $\Phi$ are bounded and $\gamma < 1$, so every series converges absolutely.)
Given reward function $R$ and policy $\pi$, let $V^R_\pi : \mathcal{S} \to \mathbb{R}$ be the familiar value function:
Conclude that for infinite trajectories, the ordering on policies induced by $R$ is the same as the ordering on policies induced by $R^\Phi$ for any potential function $\Phi$.
Solution
Using the previous result, fix any policy $\pi$ and any starting state $s$, and take expectations over trajectories that start at $s_0 = s$, with $a_t \sim \pi(s_t)$ and $s_{t+1} \sim \mu(s_t, a_t)$:
Since $s_0 = s$ is fixed, $\Phi(s_0) = \Phi(s)$ is a constant and comes straight out of the expectation. For infinite trajectories the last term is zero, so $V^{R^\Phi}_\pi(s) = V^R_\pi(s) - \Phi(s)$: at every state $s$, the two value functions differ by $\Phi(s)$, which does not depend on the policy. For any two policies $\pi_1, \pi_2$ and any state $s$,
and taking this for all $s$ gives $\pi_1 \preceq_R \pi_2 \iff \pi_1 \preceq_{R^\Phi} \pi_2$. So $R$ and $R^\Phi$ induce the same ordering on policies, and in particular have the same optimal policies.
What can go wrong when trajectories are finite?
Solution
In the previous solution, the term $\mathbb{E}_\pi[\gamma^T \Phi(s_T) \mid s_0 = s]$ only vanishes for infinite trajectories. For finite ones it depends on the policy, because the policy influences which final state it ends up in. So the shaping can reward or punish where the episode ends, and change which policy is optimal. For example, take three states: purgatory $p$, heaven $h$ and hell $\ell$. The agent starts in $p$ and makes one move, to $h$ or to $\ell$, and then the episode ends ($T = 1$). Let $R$ give $+1$ for moving to heaven and $-1$ for moving to hell, and let $\Phi(\ell) = 10$ with $\Phi = 0$ elsewhere. Then with $\gamma = 0.9$:
- under $R$, heaven returns $1$ and hell returns $-1$, so the optimal policy goes to heaven;
- under $R^\Phi$, heaven returns $1 + \gamma \cdot 0 - 0 = 1$ and hell returns $-1 + \gamma \cdot 10 - 0 = 8$, so the optimal policy goes to hell.
There are two standard fixes:
- set $\Phi = 0$ at every final state, so the extra term vanishes;
- treat the end of the episode as an absorbing state the agent stays in forever, which makes the trajectory infinite. Each step in hell then earns a shaping reward of $\gamma\Phi(\ell) - \Phi(\ell) = -(1-\gamma) \cdot 10$, and these add up to exactly cancel the $+\gamma \cdot 10$ the agent got for entering hell.
Exercise - implement potential shaping
Using potential shaping, write a reward function reward_shaped that still incentivises the robot to pick up shards but not to put them back down again.
There are many valid choices of potential and bin reward. The tests (here, and for reward2 later) and the later analysis use the ones in the hint below: a potential of $0.5$ while holding shards, and $+1$ for binning them, so use those.
Notes:
I'm stuck and need a hint!
- Consider a potential function that looks at the contents of the inventory.
- Like the original reward function, this reward function should also give a reward for putting the shards into the bin.
- Moreover, to get a clear training signal, make the reward for putting shards into the bin larger than the potential lost from this action: use
+1reward for binning shards.
Our choice of potential function (spoilers!)
We use the potential $\Phi(s) = 0.5$ if the robot is holding shards in state $s$ and $0$ otherwise.
def reward_shaped(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
raise NotImplementedError()
tests.test_reward_shaped(reward_shaped)
Discussion - what does shaping do here?
The effect of shaping is to transform a reward function that only gives reward when the robot drops a shard into the bin into a reward function that gets this reward in advance and then loses small amounts of reward while it is holding the shard until it drops it into the bin, so that after discounting, the total return after dropping the shard into the bin is equal. If the robot drops the shard on the floor, it loses the initial reward that was advanced.
In particular, a pickup-then-drop cycle now yields exactly zero discounted return:
the loophole that reward1 left open is now no longer available for the agent to abuse.
Solution
def inventory_potential(state: State) -> Float[Tensor, "B"]:
return 0.5 * (state.inventory == Item.SHARDS).float()
def reward_bin(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
return (
(state.bin_pos[:, 0] == state.robot_pos[:, 0])
& (state.bin_pos[:, 1] == state.robot_pos[:, 1])
& (state.inventory == Item.SHARDS)
& (action == Action.PUTDOWN)
).float()
def reward_shaped(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
pickup_shaping_term = DISCOUNT_RATE * inventory_potential(next_state) - inventory_potential(state)
bin_reward_term = reward_bin(state, action, next_state)
return bin_reward_term + pickup_shaping_term
Disincentivising specific behaviours
Let's turn to the second problematic behaviour: breaking urns to find more shards. There are a couple of different approaches to disincentivising this behaviour. We'll use a simple approach of directly penalising transitions in which the robot breaks an urn.
The main question is, how much of a negative reward should we assign for breaking an urn?
- We need to set the negative reward large enough so that the agent is better off not breaking the urn rather than breaking the urn and binning the resulting shards.
- We can't set the negative reward to be too large, or it will cause training instability
- During training, while the policy is still stochastic, the agent might learn to be scared of urns, and avoid going anywhere near them lest the wrong action is sampled and the agent accidentally breaks the urn. You can try different values, but a good default value would be $-2$ reward for breaking an urn. The agent can recover $+1$ reward later if it bins the shards created by breaking the urn (the gain in return will be slightly less than 1 due to discounting). Penalising -2 means the agent is clearly better off not breaking the urn.
Exercise (optional) - everyone has a price
If you have time, consider the following two questions:
-
If the reward for breaking an urn is -2, there are still contrived environments in which breaking an urn can lead to higher return than not breaking an urn. Can you think of any?
-
Can you think of any approach to defining a reward function that would make breaking urns always suboptimal?
Hint (question 1)
Think about discounting, and about urns that stand between the robot and something valuable. A penalty received far in the future is discounted; what does that imply about a penalty received now versus rewards unlocked sooner?
Solution (question 1)
Breaking an urn just to bin its own shards never pays. Under reward2, the best case is break at $t=0$, pick up at $t=1$, bin at some $k \ge 2$:
The penalty is twice the most one bin can return, so this is always negative.
Breaking pays only when the urn blocks a lot of value. If an urn (or a wall of urns) forces a long detour, every delivery behind it is delayed and so discounted. Breaking costs $-2$ once, now, and brings all those deliveries forward. Roughly, if breaking saves $D$ steps on deliveries at times $t$, it pays when
On small open grids you can step around any urn in a cell or two, so $D$ is tiny and this never fires. You need to build a layout for it, such as a wall of urns with shards behind it:
env_price = Environment(
init_robot_pos=t.tensor((0, 1), dtype=t.long),
init_items_map=t.tensor(
(
(0, 0, 0, 2, 1, 1, 0, 0), # bin (0,0); urn wall in col 3; shards just behind it
(0, 0, 0, 2, 1, 1, 0, 0),
(0, 0, 0, 2, 1, 0, 0, 0),
(0, 0, 0, 2, 0, 0, 0, 0),
(0, 0, 0, 2, 0, 0, 0, 0),
(0, 0, 0, 2, 0, 0, 0, 0),
(0, 0, 0, 2, 0, 0, 0, 0),
(0, 0, 0, 0, 0, 0, 0, 0), # gap at (7,3): the only way around the wall
),
dtype=t.long,
),
bin_pos=t.tensor((0, 0), dtype=t.long),
)

Checking both strategies: plan each with BFS, roll it through the real environment, and score it with reward2:
from collections import deque
GAMMA, HORIZON = 0.995, 64
WS = env_price.size
MOVES = {Action.UP: (-1, 0), Action.DOWN: (1, 0), Action.LEFT: (0, -1), Action.RIGHT: (0, 1)}
URNS = {(r, c) for r in range(WS) for c in range(WS) if env_price.init_items_map[r, c] == Item.URN}
SHARDS = {(r, c) for r in range(WS) for c in range(WS) if env_price.init_items_map[r, c] == Item.SHARDS}
BIN = tuple(env_price.bin_pos.tolist())
def bfs(start, goal, blocked): # shortest move-action path avoiding `blocked` cells
if start == goal: return []
seen, q = {start}, deque([(start, [])])
while q:
(r, c), path = q.popleft()
for a, (dr, dc) in MOVES.items():
nxt = (max(0, min(WS - 1, r + dr)), max(0, min(WS - 1, c + dc)))
if nxt in seen or nxt in blocked: continue
if nxt == goal: return path + [a]
seen.add(nxt); q.append((nxt, path + [a]))
return None # unreachable without breaking the wall
def plan(blocked): # greedy nearest-shard-first cleaning plan
robot, shards, actions = tuple(env_price.init_robot_pos.tolist()), set(SHARDS), []
while shards:
opts = [(s, bfs(robot, s, blocked)) for s in shards]
opts = [(s, p) for s, p in opts if p is not None]
if not opts: break
s, to_s = min(opts, key=lambda sp: len(sp[1]))
actions += to_s + [Action.PICKUP] + bfs(s, BIN, blocked) + [Action.PUTDOWN]
robot = BIN; shards.discard(s)
return [int(a) for a in actions]
def discounted_return(actions): # roll through the real env, score with reward2
state, G = env_price.reset(num_rollouts=1), 0.0
for ts in range(HORIZON):
a = t.tensor([actions[ts] if ts < len(actions) else int(Action.WAIT)])
nxt = env_price.step(state, a)
G += GAMMA ** ts * float(reward2(state, a, nxt)[0]); state = nxt
return G
detour = discounted_return(plan(blocked=URNS)) # never break: use the bottom gap
smash = max(discounted_return(plan(blocked=URNS - {u})) for u in URNS) # break the single best urn
print(f"detour {detour:+.3f} smash {smash:+.3f}") # -> detour +1.215 smash +2.261
Smashing wins easily, +2.26 vs +1.22. Within the 64-step horizon the detour bins one pile of shards; breaking one urn (−2) lets the agent bin all five. So even with the −2 penalty, breaking the urn is return-maximising here.
Will a trained agent break it? Not necessarily. Training PPO here (ppo.train_agent(env_price, ..., reward_fn=reward2)) for 512–2048 steps across several seeds, the agent learns nothing: it never bins and never breaks. The optimal plan is a long, precise sequence, the −2 lands on its very first step, and the payoff is far away. So a behaviour being optimal under the reward is necessary but not sufficient for an agent to learn it; exploration matters too.
Solution (question 2)
Make the penalty larger than the most return breaking could ever unlock. In a bounded world (finite grid, finite shards) the total discounted return is at most some $R_{\max}$, so a penalty $\ge R_{\max}$ can never be paid back. The cost is the trade-off from above: a huge penalty destabilises training and makes the agent avoid urns, even when walking past them is safe.
Adding a potential for intact urns does not work. Potential shaping doesn't change the ordering over policies (you proved this for infinite trajectories in the optional exercise above), so it can't make breaking suboptimal if it wasn't already. You have to change the reward's preferences (a penalty) or the environment (e.g. impassable urns).
Exercise - implement the penalty
Your task is to implement a reward function that assigns a negative reward (of -2) to transitions in which the robot breaks an urn. Hint: You already implemented a related function earlier in this section. Should be a one-liner solution.
def reward_no_break(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
raise NotImplementedError()
tests.test_reward_no_break(reward_no_break)
Solution
def reward_no_break(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
return -2.0 * reward_break(state, action, next_state)
Exercise - fix the specification
Your next task is to bring together the previous exercises to eliminate specification gaming:
-
Combine the shaped reward and the urn penalty into a single new reward function,
reward2. -
Train a new network, using the same environment, agent architecture, and hyperparameters as last time, but this time using the new reward function
- we recommend
num_train_steps=192this time, since the shaped reward takes a little longer to master.
- we recommend
-
Inspect some rollouts, manually and/or by using your evaluation reward function probes, to confirm that the agent now behaves as intended.
def reward2(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
raise NotImplementedError()
tests.test_reward2(reward2)
# Train a new agent with the fixed reward function
net2 = ActorCriticNetwork.init(
obs_height=env.size,
obs_width=env.size,
net_channels=8,
net_width=16,
num_conv_layers=2,
num_dense_layers=1,
num_actions=len(Action),
generator=t.Generator().manual_seed(42),
)
net2 = ppo.train_agent(
env=env,
net=net2,
reward_fn=reward2,
num_train_steps=192,
)
# Manually inspect a rollout...
rollout = potteryshop.collect_rollout(
env=env,
policy_fn=net2.policy,
num_steps=64,
generator=t.Generator().manual_seed(1),
device=device,
)
utils.display_rollout(env, rollout)
# ...and quantitatively inspect the behaviour with our probes
reward_fns = [reward2, reward_drop, reward_break]
return_vecs = [
evaluation.evaluate_behaviour(
env=env,
net=net2,
reward_fn=r,
generator=t.Generator().manual_seed(1),
)
for r in reward_fns
]
utils.plot_return_hists(
reward_fns,
return_vecs,
title="net2 (trained on reward2): the fix — drop & break probes should collapse to ~0",
)
Solution
def reward2(state: State, action: Int[Tensor, "B"], next_state: State) -> Float[Tensor, "B"]:
shaped = reward_shaped(state, action, next_state)
nobreak = reward_no_break(state, action, next_state)
return shaped + nobreak
If everything goes to plan, you should see the agent learning to actually clean up at least one pile of shards, and the behavioural probes reward_drop and reward_break should now be concentrated at (or very near) zero.