How do Structured Outputs Work?

Overview

Structured Outputs is a feature that forces the LLM’s response to strictly follow a schema specified by the user. When Structured Outputs is turned on, the LLM will generate structured data that follows the desired schema, provided by the user, 100% of the time. This increases the reliability of LLMs in enterprise applications where downstream applications expect the LLM output to be correctly formatted. With Structured Outputs, hallucinated fields and entries in structured data can be reliably eliminated.

Compatible models:

• Command R+ 08 2024
• Command R+
• Command R 08 2024
• Command R

How to Use Structured Outputs

There are two ways to use Structured Outputs:

• Structured Outputs (JSON). This is primarily used in text generation use cases.
• Structured Outputs (Tools). Structured Outputs (Tools). This is primarily used in tool use (or function calling) and agents use cases.

Structured Outputs (JSON)

Here, you can call the Chat API to generate Structured Outputs in JSON format. JSON is a lightweight format that is easy for humans to read and write and is also easy for machines to parse.

This is particularly useful in text generation use cases, for example, when you want to extract specific information from the responses, perform data analysis, or integrate the responses into your applications seamlessly.

There are two ways of specifying the JSON output:

• JSON mode
• JSON Schema mode

JSON mode

In JSON mode, when making an API request, you can specify the response_format parameter to indicate that you want the response in a JSON object format.

1
import cohere
2
3
co = cohere.ClientV2(api_key="YOUR API KEY")
4
5
res = co.chat(
6
    model="command-r-plus-08-2024",
7
    messages=[
8
        {
9
            "role": "user",
10
            "content": "Generate a JSON describing a person, with the fields 'name' and 'age'",
11
        }
12
    ],
13
    response_format={"type": "json_object"},
14
)
15
16
print(res.message.content[0].text)

By setting the response_format type to "json_object" in the Chat API, the output of the model is guaranteed to be a valid JSON object.

# Example response
{
  "name": "Emma Johnson",
  "age": 32
}

JSON Schema mode

In JSON Schema mode, you can optionally define a schema as part of the response_format parameter. A JSON Schema is a way to describe the structure of the JSON object you want the LLM to generate.

This forces the LLM to stick to this schema, thus giving you greater control over the output.

For example, let’s say you want the LLM to generate a JSON object with specific keys for a book, such as “title,” “author,” and “publication_year.” Your API request might look like this:

1
import cohere
2
3
co = cohere.ClientV2(api_key="YOUR API KEY")
4
5
res = co.chat(
6
    model="command-r-plus-08-2024",
7
    messages=[
8
        {
9
            "role": "user",
10
            "content": "Generate a JSON describing a book, with the fields 'title' and 'author' and 'publication_year'",
11
        }
12
    ],
13
    response_format={
14
        "type": "json_object",
15
        "schema": {
16
            "type": "object",
17
            "properties": {
18
                "title": {"type": "string"},
19
                "author": {"type": "string"},
20
                "publication_year": {"type": "integer"},
21
            },
22
            "required": ["title", "author", "publication_year"],
23
        },
24
    },
25
)
26
27
print(res.message.content[0].text)

In this schema, we defined three keys (“title,” “author,” “publication_year”) and their expected data types (“string” and “integer”). The LLM will generate a JSON object that adheres to this structure.

# Example response
{
  "title": "The Great Gatsby",
  "author": "F. Scott Fitzgerald",
  "publication_year": 1925
}

Nested Array Schema Json Example

Here’s an example of a nested array. Note that the top level json structure must always be a json object.

1
cohere_api_key = os.getenv("cohere_api_key")
2
co = cohere.ClientV2(cohere_api_key)
3
response = co.chat(
4
    response_format={
5
        "type": "json_object",
6
        "schema": {
7
            "type": "object",
8
            "properties": {
9
                "actions": {
10
                    "type": "array",
11
                    "items": {
12
                        "type": "object",
13
                        "properties": {
14
                            "japanese": {"type": "string"},
15
                            "romaji": {"type": "string"},
16
                            "english": {"type": "string"},
17
                        },
18
                        "required": ["japanese", "romaji", "english"],
19
                    },
20
                }
21
            },
22
            "required": ["actions"],
23
        },
24
    },
25
    model="command-r",
26
    messages=[
27
        {
28
            "role": "user",
29
            "content": "Generate a JSON array of objects with the following fields: japanese, romaji, english. These actions should be japanese verbs provided in the dictionary form.",
30
        },
31
    ],
32
)
33
return json.loads(response.message.content[0].text)

The output for this example would be:

1
{
2
    "actions": [
3
        {"japanese": "いこう", "romaji": "ikou", "english": "onward"},
4
        {"japanese": "探す", "romaji": "sagasu", "english": "search"},
5
        {"japanese": "話す", "romaji": "hanasu", "english": "talk"}
6
    ]
7
}

Structured Outputs (Tools)

When you use the Chat API with tools (see tool use and agents), setting the strict_tools parameter to True will enforce that the tool calls generated by the mode strictly adhere to the tool descriptions you provided.

Concretely, this means:

• No hallucinated tool names
• No hallucinated tool parameters
• Every required parameter is included in the tool call
• All parameters produce the requested data types

With strict_tools enabled, the API will ensure that the tool names and tool parameters are generated according to the tool definitions. This eliminates tool name and parameter hallucinations, ensures that each parameter matches the specified data type, and that all required parameters are included in the model response.

Additionally, this results in faster development. You don’t need to spend a lot of time prompt engineering the model to avoid hallucinations.

In the example below, we create a tool that can retrieve weather data for a given location. The tool is called get_weather which contains a parameter called location. We then invoke the Chat API with strict_tools set to True to ensure that the generated tool calls always include the correct function and parameter names.

When the strict_tools parameter is set to True, you can define a maximum of 200 fields across all tools being passed to an API call.

1
tools = [
2
    {
3
        "type": "function",
4
        "function": {
5
            "name": "get_weather",
6
            "description" : "Gets the weather of a given location",
7
            "parameters": {
8
                "type": "object",
9
                "properties": {
10
                    "location": {
11
                        "type" : "string",
12
                        "description": "The location to get weather."
13
                    }
14
                },
15
                "required": ["location"]
16
            }
17
        }
18
    },
19
]
20
21
response = co.chat(model="command-r-plus-08-2024",
22
                   messages=[{"role": "user", "content": "What's the weather in Toronto?"}],
23
                   tools=tools,
24
                   strict_tools=True)
25
26
print(response.message.tool_calls)

Important notes on using strict_tools

• This parameter is only supported in Chat API V2 via the strict_tools parameter (not API V1).
• You must specify at least one required parameter. Tools with only optional parameters are not supported in this mode.
• You can define a maximum of 200 fields across all tools in a single Chat API call.

When to Use Structured Outputs (JSON) vs. Structured Outputs (Tools)

Structured Outputs (JSON) are ideal for text generation use cases where you want to format the model’s responses to users in a specific way.

For example, when building a travel planner application, you might want the LLM to generate itineraries in a specific JSON format, allowing the application to use the output in the other parts of the application.

Structured Outputs (Tools) are ideal for tool use (or function calling) and agents use cases where you need the model to interact with external data or services. For instance, you can grant the model access to functions that interact with databases or other APIs.

In summary, opt for:

• Structured Outputs (JSON) when you need the model’s response to follow a specific structure.
• Structured Outputs (Tools) when you need the model to interact with external data or services.

Specifying a schema

Generating nested objects

In JSON Schema mode, there are no limitations on the levels of nesting. However, in JSON mode (no schema specified), nesting is limited to 5 levels.

Schema constraints

When constructing a schema keep the following constraints in mind:

• The type in the top level schema must be object
• Every object in the schema must have at least one required field specified

Parameter types support

Supported schema features

The Structured Outputs feature (both JSON and Tools mode) relies on the JSON Schema notation for defining the parameters. JSON Schema allows developers to specify the expected format of JSON objects, ensuring that the data adheres to predefined rules and constraints.

Structured Outputs supports a subset of the JSON Schema specification, detailed in the tables below. This is broken down into three categories:

• Structured Outputs (JSON)
• Structured Outputs (Tools) - When strict_tools is set to True
• Tool Use - When strict_tools is set to False

Basic types

See usage examples for JSON and Tools.

Arrays

See usage examples for JSON and Tools.

Others

¹ Const is supported for these types: int, float, bool, type(None), str.

² Format is supported for these values: date-time, uuid, date, time.

³ In Structured Outputs (JSON), additionalProperties does not enforce required, dependencies, propertyNames, anyOf, allOf, oneOf.

⁴ In Structured Outputs (Tools), additionalProperties does not enforce required, dependencies, propertyNames, any Of, all Of, one Of.

See usage examples for JSON and Tools.

Unsupported schema features

We do not support the entirety of the JSON Schema specification. Below is a list of some unsupported features:

• Schema Composition (allOf, oneOf and not)
• Numeric Ranges (maximum and minimum)
• Array Length Ranges (minItems and maxItems)
• String limitations:
  • String Length (maxLength and minLength)
  • The following are not supported in Regular Expressions
    • ^
    • $
    • ?=
    • ?!
  • The following formats are the only supported ones
    • date-time
    • uuid
    • date
    • time