2️⃣ Reward Functions
Learning Objectives
- Interpret a reward function: what behaviour does it incentivise, and what behaviour was it meant to incentivise?
- Train an agent with the provided PPO implementation
- Interpret a trained agent's behaviour, and spot when it diverges from the designer's intent
Reward Functions
Typically for reinforcement learning, the reward function is given, and the goal is to design an algorithm that finds a policy that maximises the return. You already did that by implementing PPO in 2.3, so we take that for granted. Today is about designing reward functions!
The reward hypothesis
Reward functions play a central role in the discipline of reinforcement learning. Their centrality is driven by the following hypothesis, called the reward hypothesis:
"That all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward)."
— Richard Sutton, The reward hypothesis
The reward hypothesis essentially claims that, regardless of what kind of purposes we want a system to fulfil, we can sensibly formulate that behaviour as the maximum expected return for some reward function. Once we find the right reward function, then we just need to apply a reinforcement learning algorithm to find a policy with the behaviour we desire.
While there is some debate in the field as to the extent to which the reward hypothesis is true in its maximal form (that maximisation of scalar returns covers all goals and purposes), it is certainly true that:
-
There are some AI problems where finding the right reward function has allowed us to specify behaviours that were infeasible to specify imperatively (e.g. by programming, as in classical software engineering) or even declaratively (e.g. by generating labelled examples, as in supervised learning), notable examples including playing superhuman Go, and improving capabilities for frontier reasoning models.
-
This possibility has driven a substantial portion of resurgence in the field of reinforcement learning, with some seeing reinforcement learning as a key component of the likely path to human-level and super-human AI in the future.
Of course, it remains to find the right reward function (if one even exists!)
Interpreting a reward function
Here is a reward function. Note that it operates on batched transitions, like everything else today:
stateandnext_stateare batchedStates,actionis an integer tensor of shape(envs,)- The result is a float tensor of shape
(envs,)containing one reward per transition.
What is batch_idx?
The pattern state.items_map[batch_idx, rows, cols] is just standard integer-array indexing, gathering the item under each environment's robot.
This allows us to run several copies of the environment in parallel on the GPU, ideal for training
RL agents at scale efficiently. Written unvectorized for ease of understanding:
python no-copy
item_below_robot = torch.zeros(envs)
for i in range(envs):
x_i = state.robot_pos[i, 0]
y_i = state.robot_pos[i, 1]
item_below_robot[i] = state.items_map[i, x_i, y_i]
def reward1(state: State,
action: Int[Tensor, "envs"],
next_state: State
) -> Float[Tensor, "envs"]:
batch_idx = t.arange(state.inventory.shape[0], device=state.inventory.device)
item_below_robot = state.items_map[
batch_idx,
state.robot_pos[:, 0],
state.robot_pos[:, 1],
]
pickup_reward = (
(item_below_robot == Item.SHARDS)
& (state.inventory == Item.EMPTY)
& (action == Action.PICKUP)
).float()
dispose_reward = (
(state.bin_pos[:, 0] == state.robot_pos[:, 0])
& (state.bin_pos[:, 1] == state.robot_pos[:, 1])
& (state.inventory == Item.SHARDS)
& (action == Action.PUTDOWN)
).float()
total_reward = pickup_reward + dispose_reward
return total_reward
Exercise - interpret this reward function
Study the reward function above and answer the following questions
(recall & represents element-wise 'and' for boolean tensors):
-
Describe the set of all rewards that the reward function can attain, and the kinds of transitions for which the reward function attains each of these rewards.
-
What kinds of qualitative behaviours do you think a reward designer who came up with this reward function is trying to incentivise?
-
What kinds of qualitative behaviours maximise return subject to this reward function? Are they the same as the previous answer?
Answers (1 and 2)
-
The reward function assigns reward 1 to transitions in which the robot picks up a pile of shards, and transitions in which it drops a pile of shards into the bin. (It would assign reward 2 if both of these happen simultaneously in one transition, but that's impossible, since
PICKUPandPUTDOWNare distinct actions.) -
The reward designer is probably trying to incentivise the agent to operate the robot to pick up shards and put them into the bin, 'cleaning up' the pottery shop.
-
No spoilers yet: train the agent below and see for yourself!
Reinforcement learning
Next, let's train the agent to see what behaviours the agent learns given this reward function.
How are we training the agent? (Optional)
The provided module part32_goal_misgen/ppo.py implements a function ppo.ppo_train_step
that collects some rollouts and trains an agent network on these using a
optimized version of the Proximal Policy Optimisation (PPO) algorithm you implemented in 2.3.
All of its hyperparameters have sensible defaults tuned for today's environments,
so the training loops below only pass the network, environment, reward function and optimiser.
The same module also provides ppo.train_agent, which wraps ppo.ppo_train_step into a training loop. While the agent
trains, it shows a live plot of stats like the loss and the average return, and a looping video of a few rollouts
of the current policy. Each video plays through once before it is replaced by one of the latest policy, and a
final video shows the trained policy:
```python no-copy def train_agent( env: Environment, net: ActorCriticNetwork, reward_fn: RewardFunction, num_train_steps: int = 128, seed: int = 42, liveplot: bool = True, # show the live plot (curves and rollout videos) progress: bool = True, # show the progress bar video_every: int | str | None = "loop", # "loop": new video once the last has played; int: every N steps; None: off ... ) -> ActorCriticNetwork:
## Cleaning up shop
Let's use this training loop to train a policy.
1. We'll need a neural network parametrisation of a policy we can train by
gradient descent. The provided `ActorCriticNetwork` (see `part32_goal_misgen/agent.py`)
is a small residual CNN over the observation grid with separate **actor** and **critic** heads:
* the actor provides a logit vector across actions, from which we can recover a policy,
* the critic estimates the value of the current state, which is useful to help stability of training.
The network sees the world as a grid of 4 boolean channels, plus a 2-element inventory vector.
A residual CNN embeds the grid. Its output is flattened, the inventory is appended,
and a residual MLP processes the result. The two heads are single linear layers on top of that shared trunk.
Below, `ws` is `size`, `C` is `net_channels`, and `W` is `net_width`:
```mermaid
flowchart TD
G["obs.grid (B, 4, ws, ws)<br/>channels: robot, bin, shards, urn"] --> C0["initial Conv2d 4 to C<br/>3x3, pad same, then ReLU"]
C0 --> CR["residual conv block<br/>× (num_conv_layers - 1)"]
CR --> F["flatten<br/>(B, ws*ws*C)"]
V["obs.vec (B, 2)<br/>holding shards?, holding urn?"] --> CAT
F --> CAT["concat<br/>(B, ws*ws*C + 2)"]
CAT --> D0["initial Linear to W<br/>then ReLU"]
D0 --> DR["residual dense block<br/>× (num_dense_layers - 1)"]
DR --> CH["critic<br/>Linear W to 1"]
DR --> AH["actor<br/>Linear W to num_actions"]
CH --> VAL["value (B,)<br/>expected discounted return"]
AH --> L["logits (B, num_actions)<br/>one score per action"]
Each residual block adds its input back on, so it only has to learn a correction.
In a conv block the layer is Conv2d C to C (3x3, pad same); in a dense block it's Linear W to W:
flowchart TD
X(["x"]) --> A["Conv2d C to C, 3x3<br/>(dense block: Linear W to W)"]
A --> R["ReLU"]
X -. skip .-> S(("+"))
R --> S
S --> O["out"]
The network below is tiny: C=8, W=16, 2 conv layers (so 1 residual conv),
and 1 dense layer (so no residual dense layers).
- We'll then call the training function with this network, your manually-instantiated environment from section 1, and the above reward function.
Training takes well under a minute on a Colab GPU.
net1 = ActorCriticNetwork.init(
obs_height=env.size,
obs_width=env.size,
net_channels=8,
net_width=16,
num_conv_layers=2,
num_dense_layers=1,
num_actions=len(Action),
generator=t.Generator().manual_seed(42),
)
net1 = ppo.train_agent(
env=env,
net=net1,
reward_fn=reward1,
num_train_steps=60,
)
Exercise - interpret the agent's behaviour
The next task is to study the behaviour of the agent we learned, and see to what extent it matches the behaviour we intended or expected when we designed this reward function. The following code samples and animates a trajectory from the learned network. Your task is to run the code and study the policy, discerning if there are any differences between:
- The behaviour the designer of the reward function likely intended;
- The behaviour you expected after studying the reward function; and
- The actual behaviour observed.
Note: Change the seed below to see trajectories with different random results when sampling actions from the policy.
Questions:
-
What qualitative behaviours do you observe?
-
Are there discrepancies between the behaviour you observe and the intended/expected behaviour? If so, list them.
⚠️ Don't read on to section 3 until you've answered these questions: the next section spoils the answers.
rollout = potteryshop.collect_rollout(
env=env,
policy_fn=net1.policy,
num_steps=64,
generator=t.Generator().manual_seed(1),
device=device,
)
utils.display_rollout(env, rollout)