Streaming

Our Chat API is capable of streaming events (such as text generation) as they come. This means that partial results from the model can be displayed within moments, even if the full generation takes longer.

You're likely already familiar with streaming. When you ask Coral a question in the playground, it doesn't output a single block of text, it streams the text out a few words at a time. In many user interfaces enabling streaming improves the user experience by lowering the perceived latency.

Example

import cohere
from cohere.responses.chat import StreamEvent

co = cohere.Client('<YOUR API KEY>')

for event in co.chat("What is an LLM?", stream=True):
    if event.event_type == StreamEvent.TEXT_GENERATION:
      print(event.text)
    elif event.event_type == StreamEvent.STREAM_END:
      print(event.finish_reason)

Stream Events

When streaming is enabled, the API sends events down one by one. Each event has a type. Events of different types need to be handled correctly.

Basic Stream Events

stream-start

The first event in the stream contains metadata for the request such as the generation_id. Only one stream-start event will be emitted.

stream-end

A stream-end event is the final event of the stream, and is returned only when streaming is finished. This event contains aggregated data from all the other events such as the complete text, as well as a finish_reason for why the stream ended (i.e. because of it was finished or there was an error).

Only one stream-end event will be returned.

text-generation

A text-generation event is emitted whenever the next chunk of text comes back from the model. As the model continues generating text, multiple events of this type will be will be emitted.

RAG Stream Events

search-queries-generation

This event will be emitted when search queries are required to complete chat with RAG -- when, for example, the connector id parameter is included in the request.

Search Results Event

A search results event returns the documents relevant to your query. Only one search results event will be returned, and this event is only applicable to RAG with connectors.

Citation Generation Event (RAG with connectors or documents parameter)

This event contains streamed citations and references to the documents being cited (if citations have been generated by the model). Multiple citation generation events will be returned.

For an illustration of a generated citation with document-specific indices, look at the "Example Response" below. As you can see, each document has an id, and when that document is used as part of the response, it's cited by that id.

Getting Started

Install the SDK

If you haven't already, you'll need to install Cohere's SDK:

pip install cohere

Using Streaming

You can use the stream parameter to toggle token streaming in any generative model. The following example uses the co.chat endpoint.

Example Responses

While Streaming

Below, we have a json object which shows the full output you might see during a streaming session:


{
    "is_finished": false,
    "event_type": "stream-start",
    "generation_id": "6789661c-731c-4d83-b0fe-8926f6194811"
}
{
    "is_finished": false,
    "event_type": "search-queries-generation",
    "search_queries": [
        {
            "text": "What is the tallest penguin in the world?",
            "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
        }
    ]
}
{
    "is_finished": false,
    "event_type": "search-results",
    "search_results": [
        {
            "search_query": {
                "text": "What is the tallest penguin in the world?",
                "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
            },
            "document_ids": [
                "web-search_0",
                "web-search_1",
            ],
            "connector": {
                "id": "web-search"
            }
        }
    ],
    "documents": [
        {
            "id": "web-search_0",
            "snippet": "The emperor penguin (Aptenodytes forsteri) is the tallest and heaviest of all living penguin species and is endemic to Antarctica. The male and female are similar in plumage and size, reaching 100 cm (39 in) in length and weighing from 22 to 45 kg (49 to 99 lb). Feathers of the head and back are black and sharply delineated from the white belly, pale-yellow breast and bright-yellow ear patches.\n\nLike all penguins, it is flightless, with a streamlined body, and wings stiffened and flattened into flippers for a marine habitat. Its diet consists primarily of fish, but also includes crustaceans, such as krill, and cephalopods, such as squid.",
            "title": "Emperor penguin - Wikipedia",
            "url": "https://en.wikipedia.org/wiki/Emperor_penguin"
        },
        {
            "id": "web-search_1",
            "snippet": "King penguins can weigh up to 40 pounds, growing to be 33 to 37 inches tall.\n\nThey are also able swimmers, diving to depths of over 200 feet in search of squid and small fish, which are their main food sources.\n\nKing penguins lay unique pear-shaped eggs, which they incubate in a pooch and carry around with their legs.\n\n3. Gentoo Penguin\n\nThe Gentoo Penguin is the world’s third-largest penguin. While its average height is 31 inches, it can grow to a maximum height of 35 inches.\n\nView this post on Instagram\n\nA post shared by Ricardo Peralta Ayala (@ricardo_peralta_ayala)\n\nGentoo Penguins have a white stripe across their black head, making them easily distinguishable from other penguin species.",
            "title": "A Ranking of the 10 Biggest Penguin Species - American Oceans",
            "url": "https://www.americanoceans.org/facts/the-largest-penguins-ranked-by-size/"
        }
    ]
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": "The"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " tallest"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " penguin"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " in"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " the"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " world"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " is"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " the"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " Emperor"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " Penguin"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": "."
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " They"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " have"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " an"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " average"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " height"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " of"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " 45"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " inches"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " and"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " weigh"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " up"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " to"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " 100"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": " pounds"
}
{
    "is_finished": false,
    "event_type": "text-generation",
    "text": "."
}
{
    "is_finished": false,
    "event_type": "citation-generation",
    "citations": [
        {
            "start": 40,
            "end": 56,
            "text": "Emperor Penguin.",
            "document_ids": [
                "web-search_0",
                "web-search_1",
            ]
        }
    ]
}
{
    "is_finished": true,
    "event_type": "stream-end",
    "response": {
        "response_id": "75f0d364-5086-4d38-8afd-b67d19e06bf1",
        "text": "The tallest penguin in the world is the Emperor Penguin. They have an average height of 45 inches and weigh up to 100 pounds.",
        "generation_id": "6789661c-731c-4d83-b0fe-8926f6194811",
        "token_count": {
            "prompt_tokens": 2821,
            "response_tokens": 29,
            "total_tokens": 2850,
            "billed_tokens": 37
        },
        "citations": [
            {
                "start": 40,
                "end": 56,
                "text": "Emperor Penguin.",
                "document_ids": [
                    "web-search_0",
                    "web-search_1",
                ]
            },
        ],
        "documents": [
        {
            "id": "web-search_0",
            "snippet": "The emperor penguin (Aptenodytes forsteri) is the tallest and heaviest of all living penguin species and is endemic to Antarctica. The male and female are similar in plumage and size, reaching 100 cm (39 in) in length and weighing from 22 to 45 kg (49 to 99 lb). Feathers of the head and back are black and sharply delineated from the white belly, pale-yellow breast and bright-yellow ear patches.\n\nLike all penguins, it is flightless, with a streamlined body, and wings stiffened and flattened into flippers for a marine habitat. Its diet consists primarily of fish, but also includes crustaceans, such as krill, and cephalopods, such as squid.",
            "title": "Emperor penguin - Wikipedia",
            "url": "https://en.wikipedia.org/wiki/Emperor_penguin"
        },
        {
            "id": "web-search_1",
            "snippet": "King penguins can weigh up to 40 pounds, growing to be 33 to 37 inches tall.\n\nThey are also able swimmers, diving to depths of over 200 feet in search of squid and small fish, which are their main food sources.\n\nKing penguins lay unique pear-shaped eggs, which they incubate in a pooch and carry around with their legs.\n\n3. Gentoo Penguin\n\nThe Gentoo Penguin is the world’s third-largest penguin. While its average height is 31 inches, it can grow to a maximum height of 35 inches.\n\nView this post on Instagram\n\nA post shared by Ricardo Peralta Ayala (@ricardo_peralta_ayala)\n\nGentoo Penguins have a white stripe across their black head, making them easily distinguishable from other penguin species.",
            "title": "A Ranking of the 10 Biggest Penguin Species - American Oceans",
            "url": "https://www.americanoceans.org/facts/the-largest-penguins-ranked-by-size/"
        }
        ],
        "search_results": [
            {
                "search_query": {
                    "text": "What is the tallest penguin in the world?",
                    "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
                },
                "document_ids": [
                    "web-search_0",
                    "web-search_1",
                ],
                "connector": {
                    "id": "web-search"
                }
            }
        ],
        "search_queries": [
            {
                "text": "What is the tallest penguin in the world?",
                "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
            }
        ]
    },
    "finish_reason": "COMPLETE"

It contains information about whether the streaming session is finished, what type of event is being fired, and the text that was generated by the model.

Of course, the print(event.text) and print(event.finish_reason) lines in the code snippet above peels a lot of the extra information away, so what your output would look more like this:

The
 tallest
 living
 penguins
 in
 the
 world
 are
 Emperor
 penguins
,
 which
 can
 reach
 heights
 of
 approximately
 115
 cm
 (
45
.
3
 inches
)
 tall
.
 Interestingly
,
 they
 are
 only
 found
 in
 Antarctica
.
[{'start': 45, 'end': 61, 'text': 'Emperor penguins', 'document_ids': ['doc_0']}]
[{'start': 104, 'end': 130, 'text': '115 cm (45.3 inches) tall.', 'document_ids': ['doc_0']}]
[{'start': 169, 'end': 180, 'text': 'Antarctica.', 'document_ids': ['doc_1']}]
COMPLETE

It should be (more or less) the same text, but that text is on its own rather than being accompanied by search queries, event types, etc.

Note that the citation objects appear at the end, just before the stream completes. If you're not sure what these mean or how to read them, check out the "Document Mode" section in the linked doc, which furnishes additional context.

When the model has finished generating, it returns the full text, some metadata, citations, and the documents that were used to ground the reply.