Multi-step Tool Use (Agents)

Tool use is a technique which allows Cohere’s models to invoke external tools: search engines, APIs, functions, databases, and so on.

Multi-step tool use happens when the output of one tool calling step is needed as the input to the another. In other words, tool-calling needs to happen in a sequence.

For example, given the web-search tool, the model can start answering complex questions that require performing internet searches.

Notice that the model learned information from the first search, which it then used to perform a second web search. This behavior is called multi-step because the model tackles the task step by step.

Also, note that multi-step is enabled in the Chat API by default.

Multi-step Tool Use With the Chat API

Step 1: Define the tools

PYTHON
1# define the `web_search` tool.
2
3def web_search(query: str) -> list[dict]:
4  # your code for performing a web search goes here
5  # return [{
6  #		"url": "https://en.wikipedia.org/wiki/Ontario",
7  #		"text": "The capital of Ontario is Toronto, ..."
8  #	}]
9
10web_search_tool = {
11    "type": "function",
12    "function": {
13        "name": "web_search",
14        "description": "performs a web search with the specified query",
15        "parameters": {
16            "type": "object",
17            "properties": {
18                "query": {
19                    "type": "str",
20                    "description": "the query to look up"
21                }
22            },
23            "required": ["query"]
24        }
25    }
26}

Step 2: Run the tool use workflow

PYTHON
1import json
2import cohere
3co = cohere.ClientV2(api_key="<YOUR API KEY>")
4
5# 1 - Add the user message
6message = "Who is the mayor of the capital of Ontario?"
7messages = [{"role": "user", "content": message}]
8
9# 2 - Model generates tool calls, if any
10model = "command-r-plus-08-2024"
11res = co.chat(model=model, messages=messages, tools=[web_search_tool])
12
13# As long as the model sends back tool_calls,
14# keep invoking tools and sending the results back to the model
15while res.message.tool_calls:
16    print("\nTool plan:")
17    print(
18        res.message.tool_plan
19    )  # This will be an observation and a plan with next steps
20
21    print("\nTool calls:")
22    for tc in res.message.tool_calls:
23        print(f"Tool name: {tc.function.name} | Parameters: {tc.function.arguments}")
24
25    messages.append(
26        {
27            "role": "assistant",
28            "tool_calls": res.message.tool_calls,
29            "tool_plan": res.message.tool_plan,
30        }
31    )
32
33    # 3 - Execute tools based on the tool calls generated by the model
34    print("\nTool results:")
35    for tc in res.message.tool_calls:
36        tool_result = web_search(**json.loads(tc.function.arguments))
37        print(tool_result)
38        tool_content = []
39        for data in tool_result:
40            tool_content.append({"type": "document", "document": {"data": json.dumps(data)}})
41            # Optional: add an "id" field in the "document" object, otherwise IDs are auto-generated
42        messages.append(
43            {"role": "tool", "tool_call_id": tc.id, "content": tool_content}
44        )
45
46    # 4 - Model either generates more tool calls or returns a response
47    res = co.chat(model=model, messages=messages, tools=[web_search_tool])
48
49print("\nResponse:")
50print(res.message.content[0].text)
51
52if res.message.citations:
53    print("\nCitations:")
54    for citation in res.message.citations:
55        print(citation, "\n")
# EXAMPLE RESPONSE
Tool plan:
I will search for the capital of Ontario, then search for the mayor of that city.
Tool calls:
Tool name: web_search | Parameters: {"query":"capital of ontario"}
Tool results:
[{'url': 'https://simple.wikipedia.org/wiki/Toronto', 'text': 'Toronto is the capital city of the province of Ontario.It is also the largest city in Ontario. It is found on the north-west side of Lake Ontario behind New York City and Chicago.. The City of Toronto itself has a population of almost 3 million people. Even more people live in the regions around it. All together, the Greater Toronto Area is home to over 6.2 million people.'}]
Tool plan:
I have found that Toronto is the capital of Ontario. I will now search for the mayor of Toronto.
Tool calls:
Tool name: web_search | Parameters: {"query":"mayor of toronto"}
Tool results:
[{'url': 'https://en.wikipedia.org/wiki/Mayor_of_Toronto', 'text': 'The mayor of Toronto is the head of Toronto City Council and chief executive officer of the municipal government.The mayor is elected alongside city council every four years on the fourth Monday of October; there are no term limits. [2] While in office, mayors are styled His/Her Worship. [3]Olivia Chow has served as the 66th and current mayor of Toronto since July 12, 2023, after winning the ...'}]
Response:
The capital of Ontario is Toronto. The current mayor of Toronto is Olivia Chow, who has been in office since July 12, 2023.
Citations:
start=26 end=34 text='Toronto.' sources=[ToolSource(type='tool', id='web_search_p7f8t49dzs8e:0', tool_output={'text': 'Toronto is the capital city of the province of Ontario.It is also the largest city in Ontario. It is found on the north-west side of Lake Ontario behind New York City and Chicago.. The City of Toronto itself has a population of almost 3 million people. Even more people live in the regions around it. All together, the Greater Toronto Area is home to over 6.2 million people.', 'url': 'https://simple.wikipedia.org/wiki/Toronto'})] 
start=67 end=78 text='Olivia Chow' sources=[ToolSource(type='tool', id='web_search_6p4e6ba9e2cv:0', tool_output={'text': 'The mayor of Toronto is the head of Toronto City Council and chief executive officer of the municipal government.The mayor is elected alongside city council every four years on the fourth Monday of October; there are no term limits. [2] While in office, mayors are styled His/Her Worship. [3]Olivia Chow has served as the 66th and current mayor of Toronto since July 12, 2023, after winning the ...', 'url': 'https://en.wikipedia.org/wiki/Mayor_of_Toronto'})] 
start=109 end=123 text='July 12, 2023.' sources=[ToolSource(type='tool', id='web_search_6p4e6ba9e2cv:0', tool_output={'text': 'The mayor of Toronto is the head of Toronto City Council and chief executive officer of the municipal government.The mayor is elected alongside city council every four years on the fourth Monday of October; there are no term limits. [2] While in office, mayors are styled His/Her Worship. [3]Olivia Chow has served as the 66th and current mayor of Toronto since July 12, 2023, after winning the ...', 'url': 'https://en.wikipedia.org/wiki/Mayor_of_Toronto'})]

How Does Multi-step Tool Use Work?

Source

Here’s an outline of the basic steps involved in multi-step tool use:

  • Given a user request, the model comes up with a plan to solve the problem which answers questions such as “Which tools should be used,” and “In what order should they be used.”
  • The model then carries out the plan by repeatedly executing actions (using whatever tools are appropriate), reasoning over the results, and re-evaluating the plan.
  • After each Action -> Observation ->Reflection cycle, the model reflects about what to do next. This reflection involves analyzing what has been figured out so far, determining whether any changes need to be made to the plan, and what to do next. The model can take as many steps as it deems necessary.
  • Once the model decides it knows how to answer the user question, it proceeds to generating the final response.

What is the difference between tool use and Retrieval Augmented Generation (RAG)?

Tool use is a natural extension of retrieval augmented generation (RAG). RAG is about enabling the model to interact with an information retrieval system (like a vector database). Our models are trained to be excellent at RAG use cases.

Tool use pushes this further, allowing Cohere models to go far beyond information retrieval, interact with search engines, APIs, functions, databases, and many other tools.

A Further Example With Multiple Tools

This section provides another example of multi-step tool use, this time with multiple tools. The notebook for this example can be found here.

This example demonstrates an agent that performs analysis on a Spotify tracks dataset (via a Python interpreter tool) while also having access to another tool: web search tool.

Step 1: Define the tools

Here, we define the web search tool, which uses the Tavily Python client to perform web searches.

PYTHON
1# ! pip install tavily-python --q --disable-pip-version-check
2
3from tavily import TavilyClient
4
5tavily_client = TavilyClient(api_key="TAVILY_API_KEY")
6
7# here's a web search engine
8def web_search(query: str) -> list[dict]:
9    results = tavily_client.search(query, max_results=3)["results"]
10    return results
11
12
13# the LLM is equipped with a description of the web search engine
14web_search_tool = {
15    "type": "function",
16    "function": {
17        "name": "web_search",
18        "description": "Returns a list of relevant document snippets for a textual query retrieved from the internet",
19        "parameters": {
20            "type": "object",
21            "properties": {
22                "query": {
23                    "type": "string",
24                    "description": "Query to search the internet with",
25                }
26            },
27            "required": ["query"],
28        },
29    },
30}

Here, we define the Python interpreter tool, which uses the exec function to execute Python code.

PYTHON
1# here's a python console, which can be used to access the spreadsheet, but also more generally to code and plot stuff
2import io, contextlib
3
4
5def python_interpreter(code: str) -> list[dict]:
6    output = io.StringIO()
7    try:
8        # Redirect stdout to capture print statements
9        with contextlib.redirect_stdout(output):
10            exec(code, globals())
11    except Exception as e:
12        return {"error": str(e), "executed_code": code}
13    # Get stdout
14    return [{"console_output": output.getvalue(), "executed_code": code}]
15    
16# the LLM is equipped with a description of a python console
17python_interpreter_tool = {
18    "type": "function",
19    "function": {
20        "name": "python_interpreter",
21        "description": "Executes python code and returns the result. The code runs in a static sandbox without internet access and without interactive mode, so print output or save output to a file.",
22        "parameters": {
23            "type": "object",
24            "properties": {
25                "code": {
26                    "type": "string",
27                    "description": "Python code to execute"
28                }
29            },
30            "required": ["code"]
31        }
32    }
33}
34
35functions_map = {
36    "web_search": web_search,
37    "python_interpreter": python_interpreter,
38}

We’ll also need the spotify_data dataset, which contains information about Spotify tracks such as the track information, release information, popularity metrics, and musical characteristics. You can find the dataset here.

Here is the task that the agent needs to perform:

PYTHON
1message = """What's the age and citizenship of the artists who had the top 3 most streamed songs on Spotify in 2023?
2
3You have access to a dataset with information about Spotify songs from the past 10 years, located at ./spotify_dataset.csv.
4You also have access to the internet to search for information not available in the dataset.
5You must use the dataset when you can, and if stuck you can use the internet.
6Remember to inspect the dataset and get a list of its columnsto understand its structure before trying to query it. Take it step by step.
7"""

Step 2: Run the tool use workflow

Next, we run the tool use workflow involving for steps:

  • Get the user message
  • Model generates tool calls, if any
  • Execute tools based on the tool calls generated by the model
  • Model either generates more tool calls or returns a response with citations
PYTHON
1model = "command-r-plus-08-2024"
2tools = [web_search_tool,python_interpreter_tool]
3
4# Step 1: get user message
5print(f"USER MESSAGE:\n{message}")
6print("="*50)
7
8messages = [{'role': 'user','content': message}]
9
10# 2 - Model generates tool calls, if any
11res = co.chat(model=model,
12        messages=messages,
13        tools=tools,
14        temperature=0.1)
15
16# Keep invoking tools as long as the model generates tool calls
17while res.message.tool_calls:
18    # Tool plan and tool calls
19    print("\nTOOL PLAN:")
20    print(res.message.tool_plan)
21
22    print("\nTOOL CALLS:")
23    for tc in res.message.tool_calls:
24        if tc.function.name == "python_interpreter":
25            print(f"Tool name: {tc.function.name}")
26            tool_call_prettified = print("\n".join(f"  {line}" for line_num, line in enumerate(json.loads(tc.function.arguments)["code"].splitlines())))
27            print(tool_call_prettified)
28        else:
29            print(f"Tool name: {tc.function.name} | Parameters: {tc.function.arguments}")
30
31    messages.append({'role': 'assistant',
32                    'tool_calls': res.message.tool_calls,
33                    'tool_plan': res.message.tool_plan})
34
35    # 3 - Execute tools based on the tool calls generated by the model
36    print("\nTOOL RESULTS:")
37    for tc in res.message.tool_calls:
38        tool_result = functions_map[tc.function.name](**json.loads(tc.function.arguments))
39        print(tool_result, "\n")
40        tool_content = []
41        for data in tool_result:
42            tool_content.append({"type": "document", "document": {"data": json.dumps(data)}})
43            # Optional: add an "id" field in the "document" object, otherwise IDs are auto-generated
44        messages.append(
45            {"role": "tool", "tool_call_id": tc.id, "content": tool_content}
46        )
47
48    # 4 - Model either generates more tool calls or returns a response
49    res = co.chat(model=model,
50                messages=messages,
51                tools=tools,
52                temperature=0.1)
53    
54messages.append({"role": "assistant", "content": res.message.content[0].text})
55
56print("\nRESPONSE:")
57print(res.message.content[0].text)
58
59if res.message.citations:
60    print("\nCITATIONS:")
61    for citation in res.message.citations:
62        print(f"Start: {citation.start} | End: {citation.end} | Text: '{citation.text}'")
63        print("Sources:")
64        if citation.sources:
65            for source in citation.sources:
66                print(source.id)
67        print("-"*50)

And here is an example output. In summary, the agent performs the task in a sequence of 3 steps:

  1. Inspect the dataset and get a list of its columns.
  2. Write and execute Python code to find the top 3 most streamed songs on Spotify in 2023 and their respective artists.
  3. Search for the age and citizenship of each artist on the internet.
USER MESSAGE:
What's the age and citizenship of the artists who had the top 3 most streamed songs on Spotify in 2023?
You have access to a dataset with information about Spotify songs from the past 10 years, located at ./spotify_dataset.csv.
You also have access to the internet to search for information not available in the dataset.
You must use the dataset when you can, and if stuck you can use the internet.
Remember to inspect the dataset and get a list of its columnsto understand its structure before trying to query it. Take it step by step.
==================================================
TOOL PLAN:
I will first inspect the dataset to understand its structure. Then, I will write and execute Python code to find the top 3 most streamed songs on Spotify in 2023. After that, I will search for the age and citizenship of the artists of these songs.
TOOL CALLS:
Tool name: python_interpreter
  import pandas as pd
  
  df = pd.read_csv("spotify_dataset.csv")
  
  # Print the first 5 rows of the dataset
  print(df.head())
  
  # Print the column names
  print(df.columns)
None
TOOL RESULTS:
[{'console_output': "                            track_name    artist(s)_name  artist_count  \\\n0  Seven (feat. Latto) (Explicit Ver.)  Latto, Jung Kook             2   \n1                                 LALA       Myke Towers             1   \n2                              vampire    Olivia Rodrigo             1   \n3                         Cruel Summer      Taylor Swift             1   \n4                       WHERE SHE GOES         Bad Bunny             1   \n\n   released_year  released_month  released_day  in_spotify_playlists  \\\n0           2023               7            14                   553   \n1           2023               3            23                  1474   \n2           2023               6            30                  1397   \n3           2019               8            23                  7858   \n4           2023               5            18                  3133   \n\n   in_spotify_charts      streams  in_apple_playlists  ...  key   mode  \\\n0                147  141381703.0                  43  ...    B  Major   \n1                 48  133716286.0                  48  ...   C#  Major   \n2                113  140003974.0                  94  ...    F  Major   \n3                100  800840817.0                 116  ...    A  Major   \n4                 50  303236322.0                  84  ...    A  Minor   \n\n   danceability valence  energy acousticness instrumentalness  liveness  \\\n0            80      89      83           31                0         8   \n1            71      61      74            7                0        10   \n2            51      32      53           17                0        31   \n3            55      58      72           11                0        11   \n4            65      23      80           14               63        11   \n\n   speechiness  release_date  \n0            4    2023-07-14  \n1            4    2023-03-23  \n2            6    2023-06-30  \n3           15    2019-08-23  \n4            6    2023-05-18  \n\n[5 rows x 25 columns]\nIndex(['track_name', 'artist(s)_name', 'artist_count', 'released_year',\n       'released_month', 'released_day', 'in_spotify_playlists',\n       'in_spotify_charts', 'streams', 'in_apple_playlists', 'in_apple_charts',\n       'in_deezer_playlists', 'in_deezer_charts', 'in_shazam_charts', 'bpm',\n       'key', 'mode', 'danceability', 'valence', 'energy', 'acousticness',\n       'instrumentalness', 'liveness', 'speechiness', 'release_date'],\n      dtype='object')\n", 'executed_code': 'import pandas as pd\r\n\r\ndf = pd.read_csv("spotify_dataset.csv")\r\n\r\n# Print the first 5 rows of the dataset\r\nprint(df.head())\r\n\r\n# Print the column names\r\nprint(df.columns)'}] 
TOOL PLAN:
The dataset contains a column called 'artist(s)_name' which contains the names of the artists. It also contains a column called 'released_year' which contains the year the song was released.
I will now write and execute Python code to find the top 3 most streamed songs on Spotify in 2023.
TOOL CALLS:
Tool name: python_interpreter
  import pandas as pd
  
  df = pd.read_csv("spotify_dataset.csv")
  
  # Filter the dataset to only include songs released in 2023
  df_2023 = df[df["released_year"] == 2023]
  
  # Sort the dataset by the number of streams in descending order
  df_2023_sorted = df_2023.sort_values("streams", ascending=False)
  
  # Print the top 3 most streamed songs
  print(df_2023_sorted.head(3))
  
  # Print the names of the artists of the top 3 most streamed songs
  print(df_2023_sorted.head(3)["artist(s)_name"])
None
TOOL RESULTS:
[{'console_output': '                                track_name              artist(s)_name  \\\n12                                 Flowers                 Miley Cyrus   \n6                          Ella Baila Sola  Eslabon Armado, Peso Pluma   \n133  Shakira: Bzrp Music Sessions, Vol. 53           Shakira, Bizarrap   \n\n     artist_count  released_year  released_month  released_day  \\\n12              1           2023               1            12   \n6               2           2023               3            16   \n133             2           2023               1            11   \n\n     in_spotify_playlists  in_spotify_charts       streams  \\\n12                  12211                115  1.316856e+09   \n6                    3090                 50  7.259801e+08   \n133                  5724                 44  7.219756e+08   \n\n     in_apple_playlists  ...  key   mode  danceability valence  energy  \\\n12                  300  ...  NaN  Major            71      65      68   \n6                    34  ...    F  Minor            67      83      76   \n133                 119  ...    D  Minor            78      50      63   \n\n    acousticness instrumentalness  liveness  speechiness  release_date  \n12             6                0         3            7    2023-01-12  \n6             48                0         8            3    2023-03-16  \n133           27                0         9            5    2023-01-11  \n\n[3 rows x 25 columns]\n12                    Miley Cyrus\n6      Eslabon Armado, Peso Pluma\n133             Shakira, Bizarrap\nName: artist(s)_name, dtype: object\n', 'executed_code': 'import pandas as pd\r\n\r\ndf = pd.read_csv("spotify_dataset.csv")\r\n\r\n# Filter the dataset to only include songs released in 2023\r\ndf_2023 = df[df["released_year"] == 2023]\r\n\r\n# Sort the dataset by the number of streams in descending order\r\ndf_2023_sorted = df_2023.sort_values("streams", ascending=False)\r\n\r\n# Print the top 3 most streamed songs\r\nprint(df_2023_sorted.head(3))\r\n\r\n# Print the names of the artists of the top 3 most streamed songs\r\nprint(df_2023_sorted.head(3)["artist(s)_name"])'}] 
TOOL PLAN:
I have found that the top 3 most streamed songs on Spotify in 2023 are:
1. Flowers by Miley Cyrus
2. Ella Baila Sola by Eslabon Armado, Peso Pluma
3. Shakira: Bzrp Music Sessions, Vol. 53 by Shakira, Bizarrap
I will now search the internet for the age and citizenship of these artists.
TOOL CALLS:
Tool name: web_search | Parameters: {"query":"Miley Cyrus age"}
Tool name: web_search | Parameters: {"query":"Miley Cyrus citizenship"}
Tool name: web_search | Parameters: {"query":"Eslabon Armado age"}
Tool name: web_search | Parameters: {"query":"Eslabon Armado citizenship"}
Tool name: web_search | Parameters: {"query":"Peso Pluma age"}
Tool name: web_search | Parameters: {"query":"Peso Pluma citizenship"}
Tool name: web_search | Parameters: {"query":"Shakira age"}
Tool name: web_search | Parameters: {"query":"Shakira citizenship"}
Tool name: web_search | Parameters: {"query":"Bizarrap age"}
Tool name: web_search | Parameters: {"query":"Bizarrap citizenship"}
TOOL RESULTS:
[{'title': 'Miley Cyrus Height, Weight, Age, Husband, Bio, Net Worth, Facts', 'url': 'https://www.celebsline.com/miley-cyrus/', 'content': 'Miley Cyrus Height, Weight, Age, Husband, Bio, Net Worth, Facts Miley Cyrus ...', 'score': 0.853669, 'raw_content': None}] 
[{'title': "Miley Cyrus Talks Global Citizen's Fight to End Inequality, COVID-19", 'url': 'https://www.rollingstone.com/music/music-news/miley-cyrus-global-citizen-global-goal-interview-1021734/', 'content': "Miley Cyrus Talks Fighting Inequality, ...", 'score': 0.39589298, 'raw_content': None}] 
[{'title': 'Eslabon Armado - Wikipedia', 'url': 'https://en.wikipedia.org/wiki/Eslabon_Armado', 'content': 'Eslabon Armado is an American regional Mexican group from Patterson, California, formed in 2017.The group consists of Pedro Tovar ...', 'score': 0.9923638, 'raw_content': None}] 
... 
...
RESPONSE:
The top 3 most streamed songs on Spotify in 2023 were:
1. 'Flowers' by Miley Cyrus
2. 'Ella Baila Sola' by Eslabon Armado and Peso Pluma
3. 'Shakira: Bzrp Music Sessions, Vol. 53' by Shakira and Bizarrap
Miley Cyrus is 31 years old and is an American citizen.
Eslabon Armado is a group of musicians from California, USA, formed in 2017. The group consists of Pedro Tovar (lead vocals), Brian Tovar (bass), Ulises González (acoustic guitar), and Damián Pacheco (twelve-string guitar). The members of the group are all in their early 20s.
Peso Pluma is a 24-year-old rapper, singer, and songwriter from Mexico.
Shakira is a 46-year-old Colombian singer, songwriter, and philanthropist.
Bizarrap is a 25-year-old Argentine DJ and producer.
CITATIONS:
Start: 58 | End: 67 | Text: ''Flowers''
Sources:
python_interpreter_sbccqz8vt4yj:0
--------------------------------------------------
Start: 71 | End: 82 | Text: 'Miley Cyrus'
Sources:
python_interpreter_sbccqz8vt4yj:0
--------------------------------------------------
Start: 86 | End: 103 | Text: ''Ella Baila Sola''
Sources:
python_interpreter_sbccqz8vt4yj:0
--------------------------------------------------
Start: 107 | End: 136 | Text: 'Eslabon Armado and Peso Pluma'
Sources:
python_interpreter_sbccqz8vt4yj:0
--------------------------------------------------
Start: 140 | End: 179 | Text: ''Shakira: Bzrp Music Sessions, Vol. 53''
Sources:
python_interpreter_sbccqz8vt4yj:0
--------------------------------------------------
Start: 183 | End: 203 | Text: 'Shakira and Bizarrap'
Sources:
python_interpreter_sbccqz8vt4yj:0
--------------------------------------------------
Start: 220 | End: 232 | Text: '31 years old'
Sources:
web_search_kvbq3v3yexkc:1
--------------------------------------------------
Start: 243 | End: 260 | Text: 'American citizen.'
Sources:
web_search_kvbq3v3yexkc:1
web_search_kvbq3v3yexkc:2
...