4️⃣ Generalisation
Learning Objectives
- Understand generalisation in RL as a special case of generalisation in machine learning
- Train an agent on procedurally generated environments
- Probe how a trained policy behaves on layouts it has never seen
So far, we have worked with a specific grid-world layout. In practice, we want our agents to be able to navigate and complete tasks in complex environments, up to and including the real world.
Real-world environments are almost infinitely complex: an agent will almost never see the same situation more than once. This means we could never hope to give our agents direct experience in every possible state they might encounter prior to deployment.
Instead, we need to train agents that will generalise from the situations they encountered during training to the new situations they will face after deployment.
In this section, we'll investigate a more complex version of our pottery shop environment where the same agent might face different shop layouts.
Generalisation in RL
Generalisation is a foundational concept in machine learning. In reinforcement learning, generalisation is essentially no different:
-
We want a function. In this case, it's a function called a 'policy' and it maps from states/observations to action probabilities.
-
We use deep learning to learn that function from data. The details here are a little different in reinforcement learning than in supervised learning, but at the end of the day, we are still using a gradient descent algorithm to find weights that optimise some objective. In this case, the objective is derived from maximising expected return.
-
We haven't explored all possible inputs to the function. There are some states for which the policy's outputs have never been queried and subjected to calibration through the objective.
Generalisation in reinforcement learning refers to how the policy responds to unseen states, particularly whether the action probabilities that it outputs for these states are consistent with the training objective of maximising expected return.
How does the policy generalise?
net2 cleans up its training shop perfectly. But it only ever saw one layout.
What does it do in a new one?
Below is a shop of the same size (the network's input shape depends on it) in
which the shards and urns have swapped places, and the robot starts somewhere else.
We run four rollouts of net2 in it, with no further training.
env2 = Environment(
init_robot_pos=t.tensor((3, 4), dtype=t.long),
init_items_map=t.tensor(
(
(0, 0, 0, 0, 1, 1),
(0, 2, 0, 0, 0, 1),
(0, 0, 0, 0, 0, 0),
(0, 2, 2, 0, 0, 1),
(0, 0, 0, 0, 1, 1),
(1, 0, 2, 0, 1, 1),
),
dtype=t.long,
),
bin_pos=t.tensor((0, 0), dtype=t.long),
)
rollouts = potteryshop.collect_rollout(
env=env2,
policy_fn=net2.policy,
num_steps=64,
num_rollouts=4,
generator=t.Generator().manual_seed(1),
device=device,
)
utils.display_gif(utils.animate_rollouts(env2, rollouts, grid_width=2), upscale=3)
The policy doesn't generalise. It heads left to where the shards were in its training shop, at (3, 1) and (3, 2). Here those cells hold urns, so it smashes both (in every rollout, though the same agent never breaks an urn in its training shop), and carries the resulting shards towards the bin. It ignores the 9 real piles of shards, most of them on the right.
It was trained in a single fixed layout, so nothing pushed it to look at where the shards and urns actually are. It could get full reward by memorising a route around the training shop, and it follows that route here too.
Procedurally generated environments
In most cases, training a policy in a fixed environment will cause the agent to learn a brittle policy that relies on many assumptions about the environment that happen to hold in the fixed environment.
A common approach to learning more robust policies is to train the agent in a broad distribution of procedurally generated environments, all of which share some commonality (e.g. same world size) but do not allow the agent to rely on spurious assumptions (e.g. robot and item positions). Otherwise the agent will not learn to identify where the items are, but just memorise the trajectory that obtains high return.
Improving generalisation via environment diversity
This is a common problem in reinforcement learning. Consider CartPole, a classic RL control problem about teaching the agent to balance an inverted pendulum. The state space is given by $(x, \dot{x}, \theta, \dot{\theta})$, the cart position, velocity, angle, and angular velocity respectively, which fully describes the instantaneous state of the system.
The starting state is $(x, \dot{x}, \theta, \dot{\theta}) = (0,0,0,0)$, i.e. the cart is centred and stationary, and the pole is vertical and stationary. The environment then injects a small amount of uniformly sampled noise $\epsilon \sim \mathcal{U}(-0.05, 0.05)$ to each of the state variables, to prevent the agent from learning a policy that just memorises a sequence of actions that works only for the starting state, but is brittle and fails under any slight perturbation (i.e. the cart can't correct itself back to vertical if you poke it a little).
Another approach to preventing memorisation used for the Arcade Learning Environment (ALE) (a class of environments that are video games for the Atari 2600 console) is to add noise to the actions: when the agent attempts to execute some action $a_{t}$, there is a small probability that the previous action $a_{t-1}$ is executed instead, as if the buttons on the joystick were sticky. This method injects noise without changing the game dynamics, which isn't always so easy: many of the ALE games are deterministic, and are implemented as literal Atari ROM files running on an emulator of the original hardware, so it would be non-trivial to adjust the game mechanics directly without domain knowledge/the original source code.
The first step to such a training approach is to write some code to randomly generate variations of the environment. Take a look at the following example. It generates a whole batch of environments at once, and for each environment in the batch, it places the bin in the top-left corner, then samples distinct cells for the robot and each item by taking the first few entries of a random permutation of the remaining grid cells.
(Don't worry if you can't fully understand the code here, we will just be making use of it to generate environments for training.)
def generate(
size: int,
num_shards: int,
num_urns: int,
num_envs: int,
generator: t.Generator | None = None,
) -> Environment:
num_cells = size**2
# place the bin in the top left corner of the world
bin_pos = t.zeros((num_envs, 2), dtype=t.long)
# sample robot and item positions without replacement from the remaining
# cells (cell 0 is the bin), by taking the first few cells of a random
# permutation
num_positions = 1 + num_shards + num_urns
perm = t.rand(num_envs, num_cells - 1, generator=generator).argsort(dim=1) + 1
positions = perm[:, :num_positions]
rows, cols = positions // size, positions % size
robot_pos = t.stack((rows[:, 0], cols[:, 0]), dim=-1)
# create item map
items_map = t.zeros((num_envs, size, size), dtype=t.long)
batch = t.arange(num_envs)[:, None]
items_map[
batch,
rows[:, 1 : 1 + num_shards],
cols[:, 1 : 1 + num_shards],
] = Item.SHARDS
items_map[
batch,
rows[:, 1 + num_shards :],
cols[:, 1 + num_shards :],
] = Item.URN
return Environment(
init_robot_pos=robot_pos,
init_items_map=items_map,
bin_pos=bin_pos,
)
Let's render a sample of generated environments to see the kind of layouts the agent will be trained on (6x6 shops, the same size we'll train on). Each tile below is a separate randomly-generated layout (not a rollout): together they show the distribution of environments the policy must handle. Notice the robot, shards, and urns land in different places every time, but the bin stays pinned in the top-left corner: that constant is what the agent will later (mis)learn to rely on.
envs = generate(
size=6,
num_shards=4,
num_urns=2,
num_envs=32,
generator=t.Generator().manual_seed(1),
)
utils.display_envs(
envs,
grid_width=8,
title="Training distribution from generate(): robot & items random, bin always top-left",
)
Training on a distribution of environments
Once we have a distribution of environments, we can integrate it into a
reinforcement learning algorithm by, for example, sampling a new batch of
environments for every batch of rollouts we collect. The provided
ppo.ppo_train_step_multienv does exactly this: it's the same PPO step
as before, except that instead of collecting all rollouts from one environment,
it collects one rollout from each environment in a batch. ppo.py also provides
ppo.train_agent_multienv, a version of ppo.train_agent that takes a
procedural environment generator gen rather than a single environment,
and samples a fresh batch of 256 environments from it for every training step.
Its videos show rollouts in freshly generated environments.
Learning to solve a distribution of environments is more challenging than learning an individual environment, so we'll use a larger policy network and more parallel environments per step.
Colab users: if you are not on a GPU runtime yet, now is the time to switch (the runs below take a minute or two on a GPU, but many minutes on CPU).
size = 6
net3 = ActorCriticNetwork.init(
obs_height=size,
obs_width=size,
net_channels=16,
net_width=64,
num_conv_layers=5,
num_dense_layers=2,
num_actions=len(Action),
generator=t.Generator().manual_seed(1),
)
net3 = ppo.train_agent_multienv(
gen=functools.partial(
generate,
size=size,
num_shards=4,
num_urns=2,
),
net=net3,
reward_fn=reward2,
num_train_steps=160,
seed=1,
)
Exercise - how does the policy generalise now?
Once net3 finishes training, your task is to explore its generalisation properties:
- Design some different shop layouts (with
size=6, the sizenet3was trained on) to probe the agent's generalisation properties. - For each shop layout you design, predict how you think the agent will behave.
- Then manually inspect the agent's behaviour using
potteryshop.collect_rolloutandutils.display_rollout. - Qualitatively characterise the kinds of environments where the agent generalises correctly and the ones where it does not.
Think carefully about what was constant across every environment the agent ever trained in. These are the kinds of things that if varied, the agent may fail to generalise to.
⚠️ Don't read on to section 5 until you've found at least one layout where the agent behaves capably but wrongly. That is, the agent should be acting in a goal-directed manner, but not towards the goal that was desired.
Help! I need some ideas for things to vary! (spoilers!)
Some ideas for things to vary:
- the number of shards and urns (the agent saw exactly 4 shards and 2 urns in every training environment),
- the positions of the robot and items,
- the position of the bin itself.
# YOUR CODE HERE - design probe layouts
# env_probe = Environment(...)
raise NotImplementedError()
rollout = potteryshop.collect_rollout(
env=env_probe,
policy_fn=net3.policy,
num_steps=64,
generator=t.Generator().manual_seed(1),
device=device,
)
utils.display_rollout(env_probe, rollout)
What you should expect to see (spoilers!)
The agent generalises well across layouts that look like training layouts: new robot/item positions, and even new item counts, are usually handled correctly (shards get picked up and carried to the top-left corner, and urns are mostly avoided).
The most interesting probe is moving the bin. In every training environment, the bin was in the top-left corner. If you place the bin somewhere else (e.g. the top-right corner, as in the example layout above), you should observe that the agent still capably picks up shards and carries them to the top-left corner: where it drops them on the floor, ignoring the actual bin. The next section is about exactly this phenomenon.
Solution
# same as training (4 shards, 2 urns, 6x6), except the bin is in the top-right corner
env_probe = Environment(
init_robot_pos=t.tensor((3, 3), dtype=t.long),
init_items_map=t.tensor(
(
(0, 0, 0, 0, 0, 0),
(0, 0, 1, 0, 0, 0),
(0, 1, 0, 0, 2, 0),
(0, 0, 0, 0, 0, 0),
(0, 0, 2, 0, 1, 0),
(0, 1, 0, 0, 0, 0),
),
dtype=t.long,
),
bin_pos=t.tensor((0, 5), dtype=t.long),
)
rollouts = potteryshop.collect_rollout(
env=env_probe,
policy_fn=net3.policy,
num_steps=64,
num_rollouts=4,
generator=t.Generator().manual_seed(1),
device=device,
)
utils.display_gif(utils.animate_rollouts(env_probe, rollouts, grid_width=2), upscale=3)