1️⃣ Environments

Learning Objectives
  • Understand how the pottery shop environment is implemented as batched tensor operations
  • Build intuition for the dynamics of the environment

Pottery shop: a simple environment

Our environment, called "pottery shop", features a robot that the agent can control.

This grid world contains various objects:

  • Urns : fragile urns that will get smashed and turned into shards if walked into.
  • Shards : broken pieces of an urn we want cleaned up.
  • A bin : a place for the robot to dispose of shards.
  • A robot : there is a small blue robot who can move around the grid, pick up shards, carry them around, and drop them (e.g., into the bin). If the robot walks into one of the urns, the urn will break, creating a new pile of shards.

The agent can make the robot move in any of the four cardinal directions, pick up shards, and dump held shards onto the floor (or the bin).

Here we see the robot cleaning up the room (with a hard-coded exemplar policy): it picks up each pile of shards in turn and carries it to the bin in the top left corner, without smashing any urns in the process.

The pottery shop environment is implemented in the source file part32_goal_misgen/potteryshop.py. Some relevant snippets of code are as follows.

Environment details

Pottery shop actually refers to a family of environments of different sizes and with different initial configurations of objects. We represent envs many parallel environments as an Environment dataclass, defined in part32_goal_misgen/potteryshop.py along with the types below:

```python no-copy @dataclass(frozen=True) class Environment: init_robot_pos: Int64[Tensor, "envs 2"] init_items_map: Int64[Tensor, "envs size size"] bin_pos: Int64[Tensor, "envs 2"]

In particular, the fields are as follows:

* `init_robot_pos` contains the row `init_robot_pos[:, 0]` and column `init_robot_pos[:, 1]` grid coordinates of the spawn position of the robot.
* `bin_pos` similarly contains the row `bin_pos[:, 0]` and column `bin_pos[:, 1]` grid coordinates of the spawn position of the bin.
* `init_items_map` is an `(envs, N, N)` tensor where `N` is the size of the grid world. The contents of the tensor map to the presence of shards or urns in the respective grid squares.

The following enumeration type explains how to interpret the numbers in `init_items_map`.

```python no-copy
class Item(enum.IntEnum):
    EMPTY = 0
    SHARDS = 1
    URN = 2

The coordinates in init_robot_pos and bin_pos should range from 0 to size-1, and the values in init_items_map should all be 0, 1, or 2.

Environment state

The environment object encodes the initial configuration of the pottery shop. But once we start taking actions, the state will change. At any time, the current state of the grid world is represented by the following dataclass.

```python no-copy @dataclass(frozen=True) class State: robot_pos: Int[Tensor, "envs 2"] bin_pos: Int[Tensor, "envs 2"] items_map: Int[Tensor, "envs size size"] inventory: Int[Tensor, "envs"]

The fields `robot_pos`, `bin_pos`, and `items_map` are the dynamic versions of 
`init_robot_pos`, `bin_pos`, and `init_items_map` from the environment configuration.

What's new is `inventory`, an integer that represents what kind of item 
the robot is carrying. This is initially `Item.EMPTY`, but changes to 
`Item.SHARDS` if the robot picks up a pile of shards 
(and changes back to `Item.EMPTY` if the robot drops the shards).


### Agent actions

The agent is responsible for controlling the robot as it moves around the grid. 
In each interaction, the agent sees the current state, and then chooses what the robot should do from the following options.

```python no-copy
class Action(enum.IntEnum):
    WAIT = 0 # do nothing
    UP = 1 # move up
    LEFT = 2 # move left
    DOWN = 3 # move down
    RIGHT = 4 # move right
    PICKUP = 5 # pick up item
    PUTDOWN = 6 # drop held item

Environment methods

The environment class provides the following methods:

  • def reset(self, num_rollouts: int | None = None) -> State:

    • Initialises a batched State (for a single environment, num_rollouts parallel copies of the initial state).
  • def step(self, state: State, action: Int[Tensor, "B"]) -> State:

    • Takes the current (batched) state and the agent's actions ( one per environment in the batch) and returns the resulting state of the environment (for example, moving the robot, updating its inventory, smashing urns).

We encourage you to skim the implementation in part32_goal_misgen/potteryshop.py: the environment dynamics are about 60 lines of (heavily commented) tensor operations.

Reward Function

We deliberately don't specify a reward function for this environment, as the goal of today is thinking about how to design reward functions to incentivise desired behaviour.

Play it yourself!

You can play the pottery shop in your browser to get a feel for how the world works.

Below we instantiate an instance of the pottery shop environment. We can see the Environment object requires the following arguments:

  • init_robot_pos: the initial position of the robot
  • init_items_map: the map of all the location of objects in the environment
    • 0 represents an empty grid square
    • 1 represents a pile of shards
    • 2 represents an urn
  • bin_pos: the position of the bin
env = Environment(
    init_robot_pos=t.tensor((1, 2), dtype=t.long),
    init_items_map=t.tensor(
        (
            (0, 0, 0, 0, 2, 2),
            (0, 1, 0, 0, 0, 2),
            (0, 0, 0, 0, 0, 0),
            (0, 1, 1, 0, 0, 2),
            (0, 0, 0, 0, 2, 2),
            (2, 0, 1, 0, 2, 2),
        ),
        dtype=t.long,
    ),
    bin_pos=t.tensor((0, 0), dtype=t.long),
)

tests.test_env(env)