How do Structured Outputs Work?

Overview

Structured Outputs is a feature that forces the LLM’s response to strictly follow a schema specified by the user. When Structured Outputs is turned on, the LLM will generate structured data that follows the desired schema, provided by the user, 100% of the time. This increases the reliability of LLMs in enterprise applications where downstream applications expect the LLM output to be correctly formatted. With Structured Outputs, hallucinated fields and entries in structured data can be reliably eliminated.

Compatible models:

• Command R 08 2024
• Command R
• Command R+ 08 2024
• Command R+

How to Use Structured Outputs

There are two ways to use Structured Outputs

• Structured Outputs (JSON). This is primarily used in text generation use cases.
• Structured Outputs (Tools). This is primarily used in tool use and agents use cases via function calling.

Structured Outputs (JSON)

Here, you can call the Chat API to generate Structured Outputs in JSON format. JSON is a lightweight format that is easy for humans to read and write and is also easy for machines to parse.

This is particularly useful in text generation use cases, for example, when you want to extract specific information from the responses, perform data analysis, or integrate the responses into your applications seamlessly.

There are two ways of specifying the JSON output:

• JSON mode
• JSON Schema mode

JSON mode

In JSON mode, when making an API request, you can specify the response_format parameter to indicate that you want the response in a JSON object format.

1
import cohere
2
3
co = cohere.Client(api_key="YOUR API KEY")
4
5
res = co.chat(
6
    model="command-r-plus-08-2024",
7
    message="Generate a JSON describing a person, with the fields 'name' and 'age'",
8
    response_format={"type": "json_object"},
9
)
10
11
print(res.text)

By setting the response_format type to "json_object" in the Chat API, the output of the model is guaranteed to be a valid JSON object.

# Example response
{
  "name": "Emma Johnson",
  "age": 32
}

JSON Schema mode

In JSON Schema mode, you can optionally define a schema as part of the response_format parameter. A JSON Schema is a way to describe the structure of the JSON object you want the LLM to generate.

This forces the LLM to stick to this schema, thus giving you greater control over the output.

For example, let’s say you want the LLM to generate a JSON object with specific keys for a book, such as “title,” “author,” and “publication_year.” Your API request might look like this:

1
import cohere
2
3
co = cohere.Client(api_key="YOUR API KEY")
4
5
res = co.chat(
6
    model="command-r-plus-08-2024",
7
    message="Generate a JSON describing a book, with the fields 'title' and 'author' and 'publication_year'",
8
    response_format={
9
        "type": "json_object",
10
        "schema": {
11
            "type": "object",
12
            "required": ["title", "author", "publication_year"],
13
            "properties": {
14
                "title": {"type": "string"},
15
                "author": {"type": "string"},
16
                "publication_year": {"type": "integer"},
17
            },
18
        },
19
    },
20
)
21
22
print(res.text)

In this schema, we defined three keys (“title,” “author,” “publication_year”) and their expected data types (“string” and “number”). The LLM will generate a JSON object that adheres to this structure.

# Example response
{
  "title": "The Great Gatsby",
  "author": "F. Scott Fitzgerald",
  "publication_year": 1925
}

Specifying a schema

Generating nested objects

In JSON Schema mode, there are no limitations on the levels of nesting. However, in JSON mode (no schema specified), nesting is limited to 5 levels.

Schema constraints

When constructing a schema keep the following constraints in mind:

• The type in the top level schema must be object
• Every object in the schema must have at least one required field specified

Unsupported schema features

We do not support the entirety of the JSON Schema specification. Below is a list of some unsupported features:

• Schema Composition (anyOf, allOf, oneOf and not)
• Numeric Ranges (maximum and minimum)
• Array Length Ranges (minItems and maxItems)
• String limitations:
  • String Length (maxLength and minLength)
  • The following are not supported in Regular Expressions
    • ^
    • $
    • ?=
    • ?!
  • The following formats are the only supported ones
    • date-time
    • uuid
    • date
    • time
• Others:
  • uniqueItems
  • additionalProperties