Multi-step Tool Use (Agents)

Tool use is a technique which allows Cohere's models to invoke external tools: search engines, APIs, functions, databases, and so on. Given a list of tool definitions, the model will generate a plan of action and decide which tools to use, in which order, and with what parameters.

For example, given the web-search tool, the model can start answering complex questions that require performing internet searches.

Notice that the model learned information from the first search, which it then used to perform a second web search. This behavior is called "multi-step" because the model tackles the task over a series of steps rather than all in one go.

Also, note that multi-step is enabled by default.

Using the Chat API and Tools Programmatically

Step 1: Define the tools

# define the `web_search` tool.

def web_search(query: str) -> list[dict]:
  # your code for performing a web search goes here
	  return [{
		  "url": "https://en.wikipedia.org/wiki/Ontario",
		  "text": "The capital of Ontario is Toronto, ..."
		}]

web_search_tool = {
  "name": "web_search",
  "description": "performs a web search with the specified query",
  "parameter_definitions": {
    "query": {
      "description": "the query to look up",
      "type": "str",
      "required": True
    }
  }
}

Step 2: Ask model for tool calls and send back tool results

import cohere
co = cohere.Client(api_key="<YOUR API KEY>")

message = "Who is the mayor of the capital of Ontario?"
model = "command-r-plus"

# STEP 2: Check what tools the model wants to use and how

res = co.chat(model=model, message=message, force_single_step=False, tools=[web_search_tool])

# as long as the model sends back tool_calls,
# keep invoking tools and sending the results back to the model
while res.tool_calls:
  print(res.text) # This will be an observation and a plan with next steps
  tool_results = []
  for call in res.tool_calls:
    # use the `web_search` tool with the search query the model sent back
    web_search_results = {"call": call, "outputs": web_search(call.parameters["query"])}
    tool_results.append(web_search_results)
    
  # call chat again with tool results
  res = co.chat(
    model="command-r-plus",
    chat_history=res.chat_history,
    message="",
    force_single_step=False,
    tools=[web_search_tool],
    tool_results=tool_results
  )
  
 print(res.text) # "The mayor of Toronto, the capital of Ontario is Olivia Chow"

Using Tools Directly Through the Chat API

It's now possible to build multi-step, tool-calling agents directly via the Chat API. Between steps, the model will output reasoning in order to guide itself in calling the next tool or providing a final answer.

To ensure best results, you must include this model plan text in the chat history.

Let's demonstrate by asking the model to answer the following question: "What was the weather where I was yesterday?"

We will provide it with the following tools:

  • get_location: A tool that returns the user’s location based on a timestamp.
  • get_weather: A tool to return the temperature at a given location.

We expect the model to answer this question by taking the following steps:

  • Step 1: Ask for the user’s location yesterday.
  • Step 2: Given the user’s location, ask for the temperature in that location.
  • Step 3: Given the weather in the user’s location yesterday, answer the original question.

The code examples below show how the model will execute this sequence of actions.

Step 1: Ask the Model Our Question

Here's the request:

curl --location 'https://api.cohere.ai/v1/chat' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <YOUR_API_KEY>' \
--data '{
    "model": "command-r-plus",
    "message": "What was the weather where I was yesterday?",
    "tools": [
        {
            "name": "get_weather",
            "description": "Gets the weather for a given location",
            "parameter_definitions": {
                "location": {
                    "description": "The city and state, e.g. San Francisco, CA",
                    "type": "str",
                    "required": true
                }
            }
        },
        {
            "name": "get_location",
            "description": "Gets the users current location",
            "parameter_definitions": {
                "time": {
                    "description": "The date in format YYYY/MM/DD",
                    "type": "str"
                }
            }
        }
    ]
}'

In the response, the model asks for the user’s location yesterday through a reasoning message. It also returns the tool results it needs in tool_calls field.

{
    "response_id": "6d6978b1-fdf7-4802-9b5d-953336466d8b",
    "text": "I will find the user's location and then use that to find the weather there yesterday.",
    "generation_id": "5c38b246-4f58-4ccf-b85d-09b25ad506fa",
    "chat_history": [
        {
            "role": "USER",
            "message": "What was the weather where I was yesterday?"
        },
        {
            "role": "CHATBOT",
            "message": "I will find the user's location and then use that to find the weather there yesterday.",
            "tool_calls": [
                {
                    "name": "get_location",
                    "parameters": {
                        "time": "2024/05/09"
                    }
                }
            ]
        }
    ],
    "finish_reason": "COMPLETE",
    "meta": {
        "api_version": {
            "version": "1"
        },
        "billed_units": {
            "input_tokens": 52,
            "output_tokens": 35
        },
        "tokens": {
            "input_tokens": 980,
            "output_tokens": 35
        }
    },
    "tool_calls": [
        {
            "name": "get_location",
            "parameters": {
                "time": "2024/05/09"
            }
        }
    ]
}

Step 2: Get the Next Tool Step

In the next request, we'll send the model the tool result and the reasoning message through chat_history to get the next tool step. Make sure to include the plan message! Although message is now technically an optional field, completing a successful call requires you to provide the model with either a message, tool_results, or both.

Passing along the message is straightforward, as it's included in the output from the first step and can therefore be reused.

curl --location 'https://api.cohere.ai/v1/chat' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <YOUR_API_KEY>' \
--data '{
		"model": "command-r-plus",
    "chat_history": [
        {
            "role": "USER",
            "message": "What was the weather where I was yesterday?"
        },
        {
            "role": "CHATBOT",
            "message": "I will find the user'\''s location and then use that to find the weather there yesterday.",
            "tool_calls": [
                {
                    "name": "get_location",
                    "parameters": {
                        "time": "2024/04/30"
                    }
                }
            ]
        }
    ],
    "tool_results": [{
          "call": {
              "name": "get_location",
              "parameters": {
                  "time": "2024/04/25"
              }
          },
          "outputs": [
              {
                  "location": "Toronto, Ontario"
              }
          ]
    }],
    "tools": [
        {
            "name": "get_weather",
            "description": "Gets the weather for a given location",
            "parameter_definitions": {
                "location": {
                    "description": "The city and state, e.g. San Francisco, CA",
                    "type": "str",
                    "required": true
                }
            }
        },
        {
            "name": "get_location",
            "description": "Gets the users current location",
            "parameter_definitions": {
                "time": {
                    "description": "The date in format YYYY/MM/DD",
                    "type": "str"
                }
            }
        }
    ]
}'

And here's the response, in which the model outputs another reasoning step and tool call request. Again, make sure to include the model plan in the next request via chat_history.

{
    "response_id": "58897e19-5a63-4781-b9a6-1039e9b76ab4",
    "text": "The user was in Toronto, Ontario yesterday. Now I will find the weather there yesterday.",
    "generation_id": "ca6c183f-43d6-4194-a309-6d61a5ced7be",
    "chat_history": [
        {
            "role": "USER",
            "message": "What was the weather where I was yesterday?"
        },
        {
            "role": "CHATBOT",
            "message": "I will find the user's location and then use that to find the weather there yesterday.",
            "tool_calls": [
                {
                    "name": "get_location",
                    "parameters": {
                        "time": "2024/04/30"
                    }
                }
            ]
        },
        {
            "role": "TOOL",
            "tool_results": [
                {
                    "call": {
                        "name": "get_location",
                        "parameters": {
                            "time": "2024/04/25"
                        }
                    },
                    "outputs": [
                        {
                            "location": "Toronto, Ontario"
                        }
                    ]
                }
            ]
        },
        {
            "role": "CHATBOT",
            "message": "The user was in Toronto, Ontario yesterday. Now I will find the weather there yesterday.",
            "tool_calls": [
                {
                    "name": "get_weather",
                    "parameters": {
                        "location": "Toronto, Ontario"
                    }
                }
            ]
        }
    ],
    "finish_reason": "COMPLETE",
    "meta": {
        "api_version": {
            "version": "1"
        },
        "billed_units": {
            "input_tokens": 83,
            "output_tokens": 28
        },
        "tokens": {
            "input_tokens": 1083,
            "output_tokens": 28
        }
    },
    "tool_calls": [
        {
            "name": "get_weather",
            "parameters": {
                "location": "Toronto, Ontario"
            }
        }
    ]
}

Step 3: Get the Results

In this last step, we'll finally get the results for get_weather

curl --location 'https://api.cohere.ai/v1/chat' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <YOUR_API_KEY>' \
--data '{
    "model": "command-r-plus",
    "chat_history": [
        {
            "role": "USER",
            "message": "What was the weather where I was yesterday?"
        },
        {
            "role": "CHATBOT",
            "message": "I will find the user'\''s location and then use that to find the weather there yesterday",
            "tool_calls": [
                {
                    "name": "get_location",
                    "parameters": {
                        "time": "2024/04/30"
                    }
                }
            ]
        },
        {
            "role": "TOOL",
            "tool_results": [
                {
                    "call": {
                        "name": "get_location",
                        "parameters": {
                            "time": "2024/04/25"
                        }
                    },
                    "outputs": [
                        {
                            "location": "Toronto, Ontario"
                        }
                    ]
                }
            ]
        },
        {
            "role": "CHATBOT",
            "message": "The user was in Toronto, Ontario yesterday. Now I will find the weather there yesterday.",
            "tool_calls": [
                {
                    "name": "get_weather",
                    "parameters": {
                        "location": "Toronto, Ontario"
                    }
                }
            ]
        }
    ],
    "tool_results": [{
          "call": {
              "name": "get_weather",
              "parameters": {
                  "time": "Toronto, Ontario"
              }
          },
          "outputs": [
              {
                  "temperature": "18 celsius", "weather": "cloudy"
              }
          ]
    }],
    "tools": [
        {
            "name": "get_weather",
            "description": "Gets the weather for a given location",
            "parameter_definitions": {
                "location": {
                    "description": "The city and state, e.g. San Francisco, CA",
                    "type": "str",
                    "required": true
                }
            }
        },
        {
            "name": "get_location",
            "description": "Gets the users current location",
            "parameter_definitions": {
                "time": {
                    "description": "The date in format YYYY/MM/DD",
                    "type": "str"
                }
            }
        }
    ]
}'

And here, in the response, the model completes its tool sequence and outputs its final message (note that there is no associated tool_call).

{
    "response_id": "8ea52e5f-f376-4d06-b308-3318296ecebb",
    "text": "Yesterday, in Toronto, Ontario, the weather was cloudy and the temperature was 18°C.",
    "generation_id": "67190a18-3cee-490a-95fc-ddd22b611672",
    "chat_history": [
        {
            "role": "USER",
            "message": "What was the weather where I was yesterday?"
        },
        {
            "role": "CHATBOT",
            "message": "I will find the user's location and then use that to find the weather there yesterday",
            "tool_calls": [
                {
                    "name": "get_location",
                    "parameters": {
                        "time": "2024/04/30"
                    }
                }
            ]
        },
        {
            "role": "TOOL",
            "tool_results": [
                {
                    "call": {
                        "name": "get_location",
                        "parameters": {
                            "time": "2024/04/25"
                        }
                    },
                    "outputs": [
                        {
                            "location": "Toronto, Ontario"
                        }
                    ]
                }
            ]
        },
        {
            "role": "CHATBOT",
            "message": "The user was in Toronto, Ontario yesterday. Now I will find the weather there yesterday.",
            "tool_calls": [
                {
                    "name": "get_weather",
                    "parameters": {
                        "location": "Toronto, Ontario"
                    }
                }
            ]
        },
        {
            "role": "TOOL",
            "tool_results": [
                {
                    "call": {
                        "name": "get_weather",
                        "parameters": {
                            "time": "Toronto, Ontario"
                        }
                    },
                    "outputs": [
                        {
                            "temperature": "18 celsius",
                            "weather": "cloudy"
                        }
                    ]
                }
            ]
        },
        {
            "role": "CHATBOT",
            "message": "Yesterday, in Toronto, Ontario, the weather was cloudy and the temperature was 18°C."
        }
    ],
    "finish_reason": "COMPLETE",
    "meta": {
        "api_version": {
            "version": "1"
        },
        "billed_units": {
            "input_tokens": 120,
            "output_tokens": 21
        },
        "tokens": {
            "input_tokens": 1184,
            "output_tokens": 21
        }
    },
    "citations": [
        {
            "start": 14,
            "end": 30,
            "text": "Toronto, Ontario",
            "document_ids": [
                "get_location:0:2:0"
            ]
        },
        {
            "start": 48,
            "end": 54,
            "text": "cloudy",
            "document_ids": [
                "get_weather:0:4:0"
            ]
        },
        {
            "start": 79,
            "end": 83,
            "text": "18°C",
            "document_ids": [
                "get_weather:0:4:0"
            ]
        }
    ],
    "documents": [
        {
            "id": "get_location:0:2:0",
            "location": "Toronto, Ontario",
            "tool_name": "get_location"
        },
        {
            "id": "get_weather:0:4:0",
            "temperature": "18 celsius",
            "tool_name": "get_weather",
            "weather": "cloudy"
        }
    ]
}

How Does Multi-step Tool Use Work?

A diagram of tool use with langchain.

Source

Here’s an outline of the basic steps involved in multi-step tool use:

  • Given a user request, the model comes up with a plan to solve the problem which answers questions such as "Which tools should be used," and "In what order should they be used."
  • The model then carries out the plan by repeatedly executing actions (using whatever tools are appropriate), reasoning over the results, and re-evaluating the plan.
  • After each Action -> Observation ->Reflection cycle, the model reflects about what to do next. This reflection involves analyzing what has been figured out so far, determining whether any changes need to be made to the plan, and what to do next. The model can take as many steps as it deems necessary.
  • Once the model decides it knows how to answer the user question, it proceeds to generating the final response.

How Does Multi-step Tool Use Differ From Single-step Tool Use?

In single-step tool use, the model is also equipped with access to a bevy of tools to answer a question.

That said single-step tool use only facilitates the model calling multiple tools during a single step. The model cannot execute a sequence of steps, and it cannot use the results from one tool call in a subsequent step.

A diagram of tool use with langchain.

Source

FAQs

When Should I Use Multi-step Tool Use?

For more complex queries, such as those that require multiple steps, it's probably better to operate in multi-step mode. You can do this by setting enable_multistep=True and providing a list of tools through the Chat API. In multi-step mode, the model can reason across steps and select multiple tools to answer a question completely.

To illustrate, we'll continue with our example from earlier, in which we asked the model "What was the weather where I was yesterday" and provide it with a location tool (to return the user’s location given a timestamp) and a weather tool (to return the weather at a given location). Here's what happens:

  • First, the model will make a plan, which consists in first calling the location tool (step 1), and then calling the weather tool (step 2) based on the output of the location tool.
  • Then, the model receives the results of these tool calls and the underlying model's reasoning.
  • In a subsequent call, the model will determine that it still doesn’t have all the information required to answer, and select another tool.
  • Etc.

What is the difference between tool use and Retrieval Augmented Generation (RAG)?

Tool use is a natural extension of retrieval augmented generation (RAG). RAG is about enabling the model to interact with an information retrieval system (like a vector database). Our models are trained to be excellent at RAG use cases.

Tool use pushes this further, allowing Cohere models to go far beyond information retrieval, interact with search engines, APIs, functions, databases, and many other tools.