Structured Outputs

Overview

Structured Outputs is a feature for forcing the LLM’s output to follow a specified format 100% of the time. This increases the reliability of LLM in enterprise applications where downstream applications expect the LLM output to be correctly formatted. By forcing the model to follow a structured schema, hallucinated fields and entries in structured data can be reliably eliminated.

Compatible models:

• Command R 08 2024
• Command R
• Command R+ 08 2024
• Command R+

How to Use Structured Outputs

There are two ways to use Structured Outputs:

• Structured Outputs (JSON). This is primarily used in text generation use cases.
• Structured Outputs (Tools). This is primarily used in tool use and agents use cases via function calling.

Structured Outputs (JSON)

Here, you can call the Chat API to generate Structured Outputs in JSON format. JSON is a lightweight format that is easy for humans to read and write and is also easy for machines to parse.

This is particularly useful in text generation use cases, for example, when you want to extract specific information from the responses, perform data analysis, or integrate the responses into your applications seamlessly.

There are two ways of specifying the JSON output:

• JSON mode
• JSON Schema mode

JSON mode

In JSON mode, when making an API request, you can specify the response_format parameter to indicate that you want the response in a JSON object format.

1
import cohere
2
co = cohere.ClientV2(api_key="YOUR API KEY")
3
4
res = co.chat(
5
    model="command-r-plus-08-2024",
6
    messages=[
7
        {
8
            "role": "user",
9
            "content": "Generate a JSON describing a person, with the fields 'name' and 'age'",
10
        }
11
    ],
12
    response_format={"type": "json_object"},
13
)
14
15
print(res.message.content[0].text)

By setting the response_format type to "json_object" in the Chat API, the output of the model is guaranteed to be a valid JSON object.

# Example response
{
  "name": "Emma Johnson",
  "age": 32
}

JSON Schema mode

In JSON Schema mode, you can optionally define a schema as part of the response_format parameter. A JSON Schema is a way to describe the structure of the JSON object you want the LLM to generate.

This forces the LLM to stick to this schema, thus giving you greater control over the output.

For example, let’s say you want the LLM to generate a JSON object with specific keys for a book, such as “title,” “author,” and “publication_year.” Your API request might look like this:

1
import cohere
2
co = cohere.ClientV2(api_key="YOUR API KEY")
3
4
res = co.chat(
5
    model="command-r-plus-08-2024",
6
    messages=[
7
        {
8
            "role": "user",
9
            "content": "Generate a JSON describing a book, with the fields 'title' and 'author' and 'publication_year'",
10
        }
11
    ],
12
    response_format={
13
        "type": "json_object",
14
        "schema": {
15
            "type": "object",
16
            "properties": {
17
                "title": {"type": "string"},
18
                "author": {"type": "string"},
19
                "publication_year": {"type": "integer"},
20
            },
21
            "required": ["title", "author", "publication_year"],
22
        },
23
    },
24
)
25
26
print(res.message.content[0].text)

In this schema, we defined three keys (“title,” “author,” “publication_year”) and their expected data types (“string” and “integer”). The LLM will generate a JSON object that adheres to this structure.

# Example response
{
  "title": "The Great Gatsby",
  "author": "F. Scott Fitzgerald",
  "publication_year": 1925
}

Structured Outputs (Tools)

When you use the Chat API with tools, setting the strict_tools parameter to true will enforce that every generated tool call follows the specified tool schema.

With strict_tools enabled, the API will ensure that the tool names and tool parameters are generated according to the tool definitions. This eliminates tool name and parameter hallucinations, ensures that each parameter matches the specified data type, and that all required parameters are included in the model response.

Additionally, this results in faster development. You don’t need to spend a lot of time prompt engineering the model to avoid hallucinations.

In the example below, we create a tool that can retrieve weather data for a given location. The tool is calledget_weather which contains a parameter called location. We then invoke the Chat API with strict_tools set to true to ensure that the generated tool calls always include the correct function and parameter names.

When the strict_tools parameter is set to true, you can define a maximum of 200 fields across all tools being passed to an API call.

1
tools = [
2
    {
3
        "type": "function",
4
        "function": {
5
            "name": "get_weather",
6
            "description" : "Gets the weather of a given location",
7
            "parameters": {
8
                "type": "object",
9
                "properties": {
10
                    "location": {
11
                        "type" : "string",
12
                        "description": "The location to get weather."
13
                    }
14
                },
15
                "required": ["location"]
16
            }
17
        }
18
    },
19
]
20
21
response = co.chat(model="command-r-plus-08-2024",
22
                   messages=[{"role": "user", "content": "What's the weather in Toronto?"}],
23
                   tools=tools,
24
                   strict_tools=True)
25
26
print(response.message.tool_calls)

When to Use Structured Outputs (JSON) vs. Structured Outputs (Tools)

Structured Outputs (JSON) are ideal for text generation use cases where you want to format the model’s responses to users in a specific way.

For example, when building a travel planner application, you might want the LLM to generate itineraries in a specific JSON format, allowing the application to use the output in the other parts of the application.

Structured Outputs (Tools) are ideal for function calling, tool use or agents use cases where you need the model to interact with external data or services. For instance, you can grant the model access to functions that interact with databases or other APIs.

In summary, opt for:

• Structured Outputs (JSON) when you need the model’s response to follow a specific structure.
• Structured Outputs (Tools) when you need the model to interact with external data or services.

Specifying a schema

Generating nested objects

The model can be configured to output objects with up to 5 levels of nesting. When a schema is specified, there are no limitations on the levels of nesting.

Schema constraints

When constructing a schema keep the following constraints in mind:

• The type in the top level schema must be object
• Every object in the schema must have at least one required field specified

Unsupported schema features

We do not support the entirety of the JSON Schema specification. Below is a list of some unsupported features:

• Schema Composition (anyOf, allOf, oneOf and not)
• Numeric Ranges (maximum and minimum)
• Array Length Ranges (minItems and maxItems)
• String limitations:
  • String Length (maxLength and minLength)
  • The following are not supported in Regular Expressions
    • ^
    • $
    • ?=
    • ?!
  • The following formats are the only supported ones
    • date-time
    • uuid
    • date
    • time
• Others:
  • uniqueItems
  • additionalProperties