Tool Use

Tool use is a technique which allows developers to connect Cohere’s Command R family of models to external tools like search engines, APIs, functions, databases, etc.

This opens up a richer set of behaviors by leveraging data stored in tools, taking actions through APIs, interacting with a vector database, querying a search engine, etc., and is particularly valuable for enterprise developers, since a lot of enterprise data lives in external sources.

Check out this notebook for worked-out examples.

What Is Possible with Tool Use?

Tool use (or “function calling”) enables a wide range of new use cases; it allows your chatbot to interact with your CRM to change the status of a deal, for example, to engage with a Python interpreter to conduct data science analysis, or to have an assistant to automatically search across different databases to retrieve information.

More broadly, it’s now possible to reliably ask the model to recommend a tool (or set of tools) along with advice on how to use them, which can be passed back to the model for more flexible workflows.

A popular application is to transform a user message into a search query for a vector database or any search engine. Because the user message can be transformed into one or many search queries, it’s possible to do multiple subtasks based on the content of the message.

The Four Steps of Tool Use (Theory)

Tool use allows developers to tell Command R/R+ which tools it can interact with and how to structure interactions (e.g. API requests, or anything that can be formatted in JSON). The model will then dynamically select the right tools and the right parameters to complete these interactions. Developers can then execute these tool calls, and receive tool results in return. Finally, to generate the final response from the model, developers submit these tool results to the Command R/R+ model.

We want to stress that it’s the developers executing tool calls and submitting final results to Command R/R+.

Here’s a graphic that represents the four steps discussed below:

Feel free to refer back to it as you read on.

Step 1 - Configure the Request to the Model

Before being able to run a tool use workflow, a developer must set up a few things:

  • A list of tools to the model
  • (Optionally) a system message containing instructions about the task and the desired style for the output.

Developers can provide one or many tools to the model. Every tool is described with a schema, indicating the tool name, description, and parameters (code snippets below).

Step 2 - The Model Dynamically Chooses the Right Tool

Once you’ve completed step one, the model will intelligently select the right tool(s) to call — and the right parameters for each tool call — based on the content of the user message.

Given a list of tool definitions, the model will generate a plan of action and decide which tools to use, in which order, and with what parameters.

Step 3 - The Developer Can Then Execute The Tool Calls

With the list of tool(s), the developer can then execute the appropriate calls (e.g. by pinging an API) using the tool parameters generated by the model. These tool calls will return tool results that will be fed to the model in Step 4.

As things stand, the developer is responsible for executing these tool calls, as the tool call executes on the developer’s side.

Step 4 - Command R/R+ Generates an Answer Based on the Tool Results

Finally, the developer calls the Cohere model, providing the tool results, in order to generate the model’s final answer, which includes the response and a list of citations.

The Four Steps of Tool Use (Step-by-Step Example)

For the sake of this illustration, we’ll assume a developer is building a chatbot to assist with sales-related questions. The chatbot has access to two tools to answer user questions: a daily sales report tool which holds data on sales volumes, and a product catalog which contains information about each product being sold.

Here is a walkthrough of what a relevant tool use workflow would look like.

Step 1

The developer provides the sales database and the products database to the model using the tools parameter.

Observe that, for each tool, the developer describes the tool name, description, and inputs. Each input can have a type and can be marked as required.

PYTHON
1# Mock database containing daily sales reports
2sales_database = {
3    "2023-09-28": {
4        "total_sales_amount": 5000,
5        "total_units_sold": 100,
6    },
7    "2023-09-29": {
8        "total_sales_amount": 10000,
9        "total_units_sold": 250,
10    },
11    "2023-09-30": {
12        "total_sales_amount": 8000,
13        "total_units_sold": 200,
14    },
15}
16
17# Mock product catalog
18product_catalog = {
19    "Electronics": [
20        {"product_id": "E1001", "name": "Smartphone", "price": 500, "stock_level": 20},
21        {"product_id": "E1002", "name": "Laptop", "price": 1000, "stock_level": 15},
22        {"product_id": "E1003", "name": "Tablet", "price": 300, "stock_level": 25},
23    ],
24    "Clothing": [
25        {"product_id": "C1001", "name": "T-Shirt", "price": 20, "stock_level": 100},
26        {"product_id": "C1002", "name": "Jeans", "price": 50, "stock_level": 80},
27        {"product_id": "C1003", "name": "Jacket", "price": 100, "stock_level": 40},
28    ],
29}
PYTHON
1# Function definitions
2import json
3import cohere
4co = cohere.ClientV2(api_key="<YOUR API KEY>")
5
6def query_daily_sales_report(day: str) -> dict:
7    """
8    Function to retrieve the sales report for the given day
9    """
10    report = sales_database.get(day, {})
11    if report:
12        return [{
13            "date": day,
14            "summary": f"Total Sales Amount: {report['total_sales_amount']}, Total Units Sold: {report['total_units_sold']}",
15        }]
16    else:
17        return [{"date": day, "summary": "No sales data available for this day."}] 
18    # You can return a list of objects e.g. [{"url": "abc.com", "text": "..."}, {"url": "xyz.com", "text": "..."}]
19
20def query_product_catalog(category: str) -> dict:
21    """
22    Function to retrieve products for the given category
23    """
24    products = product_catalog.get(category, [])
25    return [{"category": category, "products": products}]
26    # You can return a list of objects e.g. [{"url": "abc.com", "text": "..."}, {"url": "xyz.com", "text": "..."}]
27
28
29functions_map = {
30    "query_daily_sales_report": query_daily_sales_report,
31    "query_product_catalog": query_product_catalog,
32}
PYTHON
1# Tool definitions
2tools = [
3    {
4        "type": "function",
5        "function": {
6            "name": "query_daily_sales_report",
7            "description": "Connects to a database to retrieve overall sales volumes and sales information for a given day.",
8            "parameters": {
9                "type": "object",
10                "properties": {
11                    "day": {
12                        "type": "string",
13                        "description": "Retrieves sales data for this day, formatted as YYYY-MM-DD.",
14                    }
15                },
16                "required": ["day"],
17            },
18        },
19    },
20    {
21        "type": "function",
22        "function": {
23            "name": "query_product_catalog",
24            "description": "Connects to a product catalog with information about all the products being sold, including categories, prices, and stock levels.",
25            "parameters": {
26                "type": "object",
27                "properties": {
28                    "category": {
29                        "type": "string",
30                        "description": "Retrieves product information data for all products in this category.",
31                    }
32                },
33                "required": ["category"],
34            },
35        },
36    },
37]
PYTHON
1system_message = """
2## Task & Context
3You help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a wide range of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user's needs as best you can, which will be wide-ranging.
4
5## Style Guide
6Unless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.
7"""
8
9# user request
10message = "Can you provide a sales summary for 29th September 2023, and also give me some details about the products in the 'Electronics' category, for example their prices and stock levels?"
11
12messages = [
13    {"role": "system", "content": system_message},
14    {"role": "user", "content": message},
15]

Step 2

The model’s response contains the tool plan, a list of appropriate tools to call in order to answer the user’s question, as well as the appropriate inputs for each tool call.

PYTHON
1response = co.chat(model="command-r-plus-08-2024",
2                   messages=messages,
3                   tools=tools)
4
5print("The model recommends doing the following tool calls:\n")
6print("Tool plan:")
7print(response.message.tool_plan, "\n")
8print("Tool calls:")
9for tc in response.message.tool_calls:
10    print(f"Tool name: {tc.function.name} | Parameters: {tc.function.arguments}")
11
12# append the chat history
13messages.append(
14    {
15        "role": "assistant",
16        "tool_calls": response.message.tool_calls,
17        "tool_plan": response.message.tool_plan,
18    }
19)
# SAMPLE RESPONSE
The model recommends doing the following tool calls:
Tool plan:
I will answer the user's request in two parts. First, I will find the sales summary for 29th September 2023. Then, I will find the details of the products in the 'Electronics' category. 
Tool calls:
Tool name: query_daily_sales_report | Parameters: {"day":"2023-09-29"}
Tool name: query_product_catalog | Parameters: {"category":"Electronics"}

Step 3

Now, the developer will query the appropriate tools and receive a tool result in return.

PYTHON
1# Iterate over the tool calls generated by the model
2for tc in response.message.tool_calls:
3    # here is where you would call the tool recommended by the model, using the parameters recommended by the model
4    tool_result = functions_map[tc.function.name](**json.loads(tc.function.arguments))
5    # store the output in a list
6    tool_content = []
7    for data in tool_result:
8        tool_content.append({"type": "document", "document": {"data": json.dumps(data)}})
9        # Optional: add an "id" field in the "document" object, otherwise IDs are auto-generated
10    # append the chat history
11    messages.append(
12        {"role": "tool", "tool_call_id": tc.id, "content": tool_content}
13    )
14    
15    print("Tool results that will be fed back to the model in step 4:")
16    for result in tool_content:
17        print(result)
# SAMPLE RESPONSE
Tool results that will be fed back to the model in step 4:
{'type': 'document', 'document': {'id': '0', 'data': '{"date": "2023-09-29", "summary": "Total Sales Amount: 10000, Total Units Sold: 250"}'}}
Tool results that will be fed back to the model in step 4:
{'type': 'document', 'document': {'id': '0', 'data': '{"category": "Electronics", "products": [{"product_id": "E1001", "name": "Smartphone", "price": 500, "stock_level": 20}, {"product_id": "E1002", "name": "Laptop", "price": 1000, "stock_level": 15}, {"product_id": "E1003", "name": "Tablet", "price": 300, "stock_level": 25}]}'}}

Step 4

Call the chat endpoint again with the tool results for the model to generate the response with citations.

PYTHON
1response = co.chat(
2    model="command-r-plus-08-2024",
3    messages=messages,
4    tools=tools
5)
6
7print("Final answer:")
8print(response.message.content[0].text)
# SAMPLE RESPONSE
Final answer:
On 29 September 2023, we had total sales of $10,000 and sold 250 units. 
Here are the details for our products in the 'Electronics' category:
- Smartphone: $500, 20 in stock
- Laptop: $1,000, 15 in stock
- Tablet: $300, 25 in stock

This step comes with a unique differentiator: the language model cites which tool results were used to generate the final model answer! These citations make it easy to check where the model’s generated response claims are coming from.

More on this in the next section.

Built-In Citations in Tool Use

At Cohere, we care about building responsible, useful, and factually-accurate models.

For this reason, Cohere’s tool use comes with a unique differentiator; as part of its generation, the underlying model cites which tool results were used to generate the final model answer. These citations make it easy to check where the model’s generated response claims are coming from.

In other words, the model only generates claims that are verifiable through fine-grained citations.

These citations are optional — you can decide to ignore them. Having said that, citations tend to be valuable in tool use; they help users gain visibility into the model reasoning, as well as sanity check the final model generation.

PYTHON
1print("Citations that support the final answer:")
2for citation in response.message.citations:
3    print(f"Start: {citation.start} | End: {citation.end} | Text: '{citation.text}'")
# SAMPLE RESPONSE
Citations that support the final answer:
Start: 29 | End: 51 | Text: 'total sales of $10,000'
Start: 56 | End: 70 | Text: 'sold 250 units'
Start: 145 | End: 174 | Text: 'Smartphone: $500, 20 in stock'
Start: 177 | End: 204 | Text: 'Laptop: $1,000, 15 in stock'
Start: 207 | End: 232 | Text: 'Tablet: $300, 25 in stock'

Structured Outputs (Tools)

Setting the strict_tools parameter to True will enforce each tool call to follow the specified tool schema. To learn more about this feature, visit the Structured Outputs documentation.

Note that strict_tools is currently an experimental feature.

1response = co.chat(model="command-r-plus-08-2024",
2                   messages=messages,
3                   tools=tools,
4                   strict_tools=True)

How to Get Good Answers With Tool Use

To get good answers with tool use, make sure that the tool name and description as well as the names and descriptions for each parameter are descriptive. If you’re not getting the model to recommend your tool correctly, iterate on those descriptions and names to help the model understand the tool better.

When you pass the tool results back to the model make sure that they are structured in a comprehensive way. For example, if you are passing the results of a add_numbers function:

outputs = [{"number": 2343}] # Not Great
outputs = [{"sum": 2343}] # Better

What’s Next?

Here, we’ll preview some of the functionality we plan on adding in the coming months.

Cohere-hosted Tools

The model can currently handle any tool provided by the developer. That having been said, Cohere has implemented some pre-defined tools that users can leverage out-of-the-box.

Specifically we’re going to roll out a Python interpreter tool and a Web search tool.

Please reach out to join the beta.

Getting started

Check out this notebook for a worked-out examples.

FAQ

What is the difference between tool use and Retrieval Augmented Generation (RAG)?

Tool use is a natural extension of retrieval augmented generation (RAG). RAG is about enabling the model to interact with an information retrieval system (like a vector database). Our models are trained to be excellent at RAG use cases.

Tool use pushes this further, allowing Cohere models to go far beyond information retrieval, interact with search engines, APIs, functions, databases, and many other tools.

If I provide many tools to the model, will the model ignore the tools that aren’t useful for the user message?

  • Yes. The model has the ability of assessing the value of a given tool in answering a given query, and will ignore any (and all) tools that don’t serve that purpose.

If I provide many tools to the model, can the model call each tool multiple times?

  • Yes, the model may call each tool 0-to-many times.

If I provide tools to the model, can the model decide to not call any tool?

  • Yes, the model may return an empty list of tool_calls which indicates that no tool call is required. This is common for user queries like greetings, chitchat, out-of-scope request, or safety violations, which do not require calling tools.
  • The model has a tendency to provide tool suggestions even if they might not be directly relevant to the question. To encourage direct answers to irrelevant questions, we recommend including a sentence in the system message such as: “When a question is irrelevant or unrelated to the available tools, please choose to directly answer it.”

Why is the output of a tool a list of objects?

  • Some tools (such as search for example) might produce many different documents (eg: search results). In order for the model to cite the documents individually when generating the response, the output has to be a list of objects. If your tool returns a single object, wrap it in a list. For example:
outputs=[{"sum": 25}]

Are there any other caveats I should be aware of?

  • Yes. An important one is that the model may return tool parameters that are invalid, so be sure to give everything a thorough once-over.