5️⃣ Bonus
In this bonus section, we'll modify the wikipedia game itself to make it more difficult so that you can then go on and try further elicitation methods of your own.
Alternatively, if you're tired of the Wikipedia game, and are feeling ambitious, you might want to try designing your own agent task, and quantifying performance on that task.
Exercise - Implement additional rules
yaml Difficulty: 🔴🔴⚪⚪⚪ Importance: 🔵⚪⚪⚪⚪
Allow the game to have additional rules. Some suggestions are a "No country pages" rule, and a "No articles above a given length" rule, but feel free to add more. With all of our elicitation methods, the agent generally only fails if the path is impossible or unreasonably hard. To implement a no country rule, you may want to use the wikipedia API's "categories" attribute for WikipediaPage objects.
First, let's modify the WikiGame task to store the rules for the Wikipedia game. We've modified the class for you to allow for the rules we described above.
class WikiGameRules(WikiGame):
def __init__(
self,
starting_page: str,
goal_page: str,
rules: Optional[list[Literal["no countries", "no pages with length above 30000"]]],
):
super().__init__(starting_page, goal_page)
self.rules = rules
Now let's modify the prompts given to the LLM API in the WikiAgent class so that we inform the agent about any additional rules it will have to abide by. We should have the option to maintain our original prompts (in case we decide to run the WikiAgent without any rules), so the new system_instruction method should first check whether there are any additional rules, and only return the modified system prompt if there are.
class WikiAgentRules(WikiAgentChatHistory):
@property
def system_instruction(self):
"""
Provide improved starting instructions for the game.
Returns:
dict: The starting instructions. "role" is "system" for system messages.
"""
raise NotImplementedError("You need to implement the system_instruction property")
Solution
class WikiAgentRules(WikiAgentChatHistory):
@property
def system_instruction(self):
"""
Provide improved starting instructions for the game.
Returns:
dict: The starting instructions. "role" is "system" for system messages.
"""
tool_descriptions = "\n".join(
[
tool.description["function"]["name"]
+ ":"
+ tool.description["function"]["description"]
for tool in self.tools
]
)
if self.task.rules:
return {
"role": "system",
"content": f"""You are a wikipedia-racing AI. Your goal is to reach {self.task.goal_page.title} by accessing links from wikipedia pages. Your current page is {self.task.current_page.title}. You have access to {str(len(self.tools))} tools, which are:\n{tool_descriptions}\n\nThe additional rules of the game are: {",".join(self.task.rules)}""",
}
else:
return {
"role": "system",
"content": f"""You are a wikipedia-racing AI. Your goal is to reach {self.task.goal_page.title} by accessing links from wikipedia pages. Your current page is {self.task.current_page.title}. You have access to {str(len(self.tools))} tools, which are:\n{tool_descriptions}""",
}
Now let's implement these rules by modifying the MovePageTool class, so that the agent can only move page if it's within the rules of the game. If you're running the agent with the reflexion tool, you may also want to modify the logic of that tool to abide by the rules.
class MovePageTool_rules(MovePageTool):
"""
Inherits from move_page_tool and adds the ability to check the rules of the game.
"""
@staticmethod
def execute(new_page: str, task: WikiGame) -> str:
"""
Changes your current page to a specified new page which is accessible via a link from the
current page. You can only call this function once at a time, as it will take you to a
different page.
Only allow the agent to move if it is permitted by the rules.
Args:
task (BaseWikiGame): The current task object.
new_page (str): The title of the new page to move to.
Returns:
str: A message indicating the result of the move
"""
raise NotImplementedError("You need to implement the execute method for the MovePageTool_rules")
@property
def description(self):
"""
Provides the description of the modified move_page tool
Returns:
dict: The description of the move_page tool for the API
"""
raise NotImplementedError("You need to implement the description property for the MovePageTool_rules")
MovePageTool_rules_inst = MovePageTool_rules()
wiki_game_tools = [GetContentTool_inst, MovePageTool_rules_inst, TestPathTool_inst]
task = WikiGameRules("Drupe", "17th parallel north", ["no countries"])
agent = WikiAgentRules(task, wiki_game_tools)
agent_loop(agent, 30)
Solution
class MovePageTool_rules(MovePageTool):
"""
Inherits from move_page_tool and adds the ability to check the rules of the game.
"""
@staticmethod
def execute(new_page: str, task: WikiGame) -> str:
"""
Changes your current page to a specified new page which is accessible via a link from the
current page. You can only call this function once at a time, as it will take you to a
different page.
Only allow the agent to move if it is permitted by the rules.
Args:
task (BaseWikiGame): The current task object.
new_page (str): The title of the new page to move to.
Returns:
str: A message indicating the result of the move
"""
new_page_normalized = new_page.replace("_", " ")
if task.is_permitted_link(new_page_normalized):
if "no countries" in task.rules and any(
"countries in" in category
for category in [i.lower() for i in task.get_page(new_page_normalized).categories]
):
return (
f"Couldn't move page to {new_page}. This page is in the category of countries."
)
if (
"no pages above length 30000" in task.rules
and len(task.get_page(new_page_normalized).content) > 30000
):
return f"Couldn't move page to {new_page}. This page is above the maximum length of 30000 characters."
task.current_page = task.get_page(new_page_normalized)
task.page_history.append(task.current_page.title)
return f"Moving page to {task.current_page.title}"
else:
return f"Couldn't move page to {new_page}. This is not a valid link."
@property
def description(self):
"""
Provides the description of the modified move_page tool
Returns:
dict: The description of the move_page tool for the API
"""
return {
"type": "function",
"function": {
"name": self.name,
"description": "Changes your current page to a specified new page which is accessible via a link from the current page. You can only call this function once at a time, as it will take you to a different page. If any pages violate the rules of the game then the tool will not move to that page, and will let you know which rule was violated.",
"parameters": {
"type": "object",
"properties": {
"new_page": {
"type": "string",
"description": 'The title of the new page you want to move to. This should be formatted the way the title appears on wikipedia (e.g. to move to the wikipedia page for the United States of America, you should enter "United States"). Underscores are not necessary.',
}
},
"required": ["new_page"],
},
},
}
Try further elicitation methods
Read some further resources on building and eliciting behaviour from LLM agents, and try implementing some of your own methods to elicit improved performance on the task. If you start seeing diminishing returns from elicitation (due to saturating performance on the task), come up with new ways to make the task harder. Alternatively, if you're feeling particularly ambitious, you can try and come up with your own more difficult task and build an agent to try and accomplish this from scratch.