Structured Generations (JSON)

Cohere models such as Command R and Command R+ are great at producing structured outputs in formats such as JSON.

Why generate JSON Objects using an LLM?

JSON is a lightweight format that is easy for humans to read and write and is also easy for machines to parse. By generating JSON objects, you can structure and organize the model’s responses in a way that can be used in downstream applications. This is particularly useful when you want to extract specific information from the responses, perform data analysis, or integrate the responses into your applications seamlessly.

How to use the response_format parameter

When making an API request, you can specify the response_format parameter to indicate that you want the response in a JSON object format.

1import cohere
2co = cohere.ClientV2(api_key="YOUR API KEY")
3
4res = co.chat(
5 model="command-r-plus-08-2024",
6 messages=[{"role": "user", "content": "Generate a JSON describing a person, with the fields 'name' and 'age'"}],
7 response_format={ "type": "json_object" }
8)
9
10print(res.message.content[0].text)

By setting the response_format type to "json_object" in the Chat API, the output of the model is guaranteed to be a valid JSON object.

# Example response
{
"name": "Emma Johnson",
"age": 32
}
Important

When using { "type": "json_object" } your message should always explicitly instruct the model to generate a JSON (eg: “Generate a JSON …”) . Otherwise the model may end up getting stuck generating an infinite stream of characters and eventually run out of context length.

Specifying a schema (beta)

The response_format parameter also allows you to define a schema for the generated JSON object. A JSON Schema is a way to describe the structure of the JSON object you want the LLM to generate. This is optional, but it gives you more control over the response format.

For example, let’s say you want the LLM to generate a JSON object with specific keys for a book, such as “title,” “author,” and “publication_year.” Your API request might look like this:

1import cohere
2co = cohere.ClientV2(api_key="YOUR API KEY")
3
4res = co.chat(
5 model="command-r-plus-08-2024",
6 messages=[
7 {
8 "role": "user",
9 "content": "Generate a JSON describing a book, with the fields 'title' and 'author' and 'publication_year'",
10 }
11 ],
12 response_format={
13 "type": "json_object",
14 "schema": {
15 "type": "object",
16 "required": ["title", "author", "publication_year"],
17 "properties": {
18 "title": {"type": "string"},
19 "author": {"type": "string"},
20 "publication_year": {"type": "integer"},
21 },
22 },
23 },
24)
25
26print(res.message.content[0].text)

In this schema, we defined three keys (“title,” “author,” “publication_year”) and their expected data types (“string” and “number”). The LLM will generate a JSON object that adheres to this structure.

# Example response
{
"title": "The Great Gatsby",
"author": "F. Scott Fitzgerald",
"publication_year": 1925
}
Important

Specifying a json_schema adds even more latency, proportional to the complexity of the schema. This parameter is in beta, and will continue seeing performance improvements.

Generating nested objects

By setting response_format={ "type": "json_object" }the model can be configured to output objects with up to 5 levels of nesting. When a schema is specified, there are no limitations on the levels of nesting.

Schema constraints

When constructing a schema keep the following constraints in mind:

  • The type in the top level schema must be object
  • Every object in the schema must have at least one required field specified

Unsupported schema features

We do not support the entirety of the JSON Schema specification. Below is a list of some unsupported features: