Changes in Chat API and Tool Use

🚧

Breaking Change

On June 10th, the Chat API will undergo a breaking change that affects users of the tools and tool_results parameters.

This document is relevant to users of the Chat API who use the toolsandtool_results parameters. Starting on June 10th, the behavior of the API will change. This guide contains an explanation along with migration steps.

What is changing

The default value of the force_single_step parameter will now be False, which means that the API is in multi-step tool use mode by default when tools and tool_results are passed in.

How to restore the old behavior

Setting the force_single_step True, will put the API into single-step tool use mode which is what the default mode was before the change.

What is 'single-step' tool use?

When force_single_step is set to True the api will be in single-step mode. That means that tool use is restricted to a single pair of tool_calls and tool_results.

In other words, in single-step mode the model does not recommend additional tool calls as a consequence of receiving tool_results. The flow for single-step tool use is as follows:

  1. Send a message along with tools -> Receive a list of tool_calls to execute
res = co.chat(message="what is 13 to the power of 3", tools=[{"id": "calculator", ...}])
print(res.tool_calls) # [ calculator("13^3") ]
  1. Send backtool_results -> Receive a message from the model with the response to the original query
res = co.chat(
  force_single_step=True,
  message="",
  chat_history=res.chat_history,
  tool_results=[{...}])
print(res.text) # "13 to the power of 3 is 2197"

What is 'multi-step' tool use?

In multi-step tool use, sending back tool_results does not necessarily conclude the sequence. Instead the model may choose to send additional tool_calls. This mode allows the model to build a plan with several stages, where each stage depends on the outcomes of the previous stage.

Example
In the example below, to answer the query "who is the mayor of the capital of Ontario" the model first sends tool calls to look up the capital of Ontario, then uses the information it learned to send another tool call for looking up the mayor of Toronto.

# Step 1: get a plan and tool calls from the model 
res = co.chat(
  message="who is the mayor of the capital of Ontario", 
  tools=[{"id": "web-search", ...}],
  force_single_step=False
)

print(res.text) # "I will first look up the capital of Ontario, and then search for the mayor"
print(res.tool_calls) # [ web_search("ontario capital") ]

# Step 2: Keep invoking tools and sending back tool_results 

res = co.chat(
  message="",
  tool_results=invoke_tools(tool_calls),
  chat_history=res.chat_history,
  force_single_step=False
)

print(res.text) # "I learned that Toronto is the capital of Ontario. I will now look up the mayor"
print(res.tool_calls) # [ web_search("toronto mayor") ]

res = co.chat(
  message="",
  tool_results=invoke_tools(tool_calls),
  chat_history=res.chat_history,
  force_single_step=False,
)

print(res.text) # "The mayor of Toronto, the capital of Ontario, is Olivia Chow"
print(res.tool_calls) # None


📘

Note

  • In the example above, force_single_step is set to **False** so multi-step mode of Tool Use is enabled.
  • when sending tool_results, the messageparameter should be set to ""
  • the chat_history returned in each response (response.chat_history) is used in the consequent request using the chat_history parameter

How to migrate your existing code

You have 2 options:

  1. Transition to using multi-step tool use
  2. Force the API into single-step tool use mode

Option 1: Transition to using multi-step tool use

Update your code to follow this pattern, which involves wrapping the second Chat call in a while loop:

# request tool calls
res = co.chat(model="command-r-plus", message="<your message here>", tools=[...])

# keep invoking tools and sending back tool_results
while res.tool_calls:
  print(res.text) # prints out the intermediate observation or plan of action
  tool_results = invoke_tools(res.tool_calls)
  res = co.chat(
    model="command-r-plus",
    chat_history=res.chat_history, # use the chat history returned in the previous call
    message="", # message should be "" when sending tool_results
    tool_results=tool_results
  )

print(res.text) # prints the final answer to the original message

You can see a more complete example in this notebook.

Option 2: Force the API into single-step tool use mode

Revert to single step behavior by setting theforce_single_step parameter to True

Don't hesitate to contact us for help

If you run into problems in this migration, send an email to [email protected] or tag @michael in the #api-discussions channel of our community discord.