Streaming Responses

The Chat API is capable of streaming events (such as text generation) as they come. This means that partial results from the model can be displayed within moments, even if the full generation takes longer.

You’re likely already familiar with streaming. When you ask the model a question using the Coral UI, the interface doesn’t output a single block of text, instead it streams the text out a few words at a time. In many user interfaces enabling streaming improves the user experience by lowering the perceived latency.

Example

PYTHON
1import cohere
2
3co = cohere.Client(api_key='<YOUR API KEY>')
4
5for event in co.chat_stream(message="What is an LLM?"):
6    if event.event_type == "text-generation":
7      print(event.text)
8    elif event.event_type == "stream-end":
9      print(event.finish_reason)

Stream Events

When streaming is enabled, the API sends events down one by one. Each event has an event_type. Events of different types need to be handled correctly.

Basic Stream Events

stream-start

The first event in the stream contains metadata for the request such as the generation_id. Only one stream-start event will be emitted.

stream-end

A stream-end event is the final event of the stream, and is returned only when streaming is finished. This event contains aggregated data from all the other events such as the complete text, as well as a finish_reason to indicate why the stream ended (i.e. either because it finished or due to an error).

Only one stream-end event will be returned.

text-generation

A text-generation event is emitted whenever the next chunk of text comes back from the model. As the model continues generating text, multiple events of this type will be emitted.

Retrieval Augmented Generation Stream Events

These events are generated when using the API with various RAG parameters.

search-queries-generation

Emitted when search queries are generated by the model. Only happens when the Chat API is used with the search_queries_only or connectors parameters .

search-results

Emitted when the specified connectors respond with search results. Only one event of this type will be returned for a given stream.

citation-generation

This event contains streamed citations and references to the documents being cited (if citations have been generated by the model). Multiple citation-generation events will be returned.

For an illustration of a generated citation with document-specific indices, look at the “Example Response” below. As you can see, each document has an id, and when that document is used as part of the response, it’s cited by that id.

Tool Use Stream Events

tool-calls-chunk

Emitted when the next token of the tool plan or the tool call is generated.

JSON
1...
2{
3    "is_finished": false,
4    "event_type": "tool-calls-chunk",
5    "text": "I"
6}
7{
8    "is_finished": false,
9    "event_type": "tool-calls-chunk",
10    "text": " will"
11}
12{
13    "is_finished": false,
14    "event_type": "tool-calls-chunk",
15    "text": " use"
16}
17...
18{
19    "is_finished": false,
20    "event_type": "tool-calls-chunk",
21    "tool_call_delta": {
22      	"index": 0,
23        "name": "reverse_string"
24    }
25}
26{
27    "is_finished": false,
28    "event_type": "tool-calls-chunk",
29    "tool_call_delta": {
30        "index": ,
31        "parameters": "{\n            \""
32    }
33}
34...

tool-calls-generation

Emitted when the model generates tool calls that require actioning upon. The event contains a list of tool_calls.

{
    "is_finished": false,
    "event_type": "tool-calls-generation",
    "text": "I will use the reverse_string tool twice, once for each of the words provided, to find the answers.",
    "tool_calls": [
        {
            "name": "reverse_string",
            "parameters": {
                "s": "hello"
            }
        },
        {
            "name": "reverse_string",
            "parameters": {
                "s": "goodbye"
            }
        }
    ]
}

Example Responses

Below, we have a stream of events which shows the full output you might see during a streaming session:

JSON
1{
2    "is_finished": false,
3    "event_type": "stream-start",
4    "generation_id": "6789661c-731c-4d83-b0fe-8926f6194811"
5}
6{
7    "is_finished": false,
8    "event_type": "search-queries-generation",
9    "search_queries": [
10        {
11            "text": "What is the tallest penguin in the world?",
12            "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
13        }
14    ]
15}
16{
17    "is_finished": false,
18    "event_type": "search-results",
19    "search_results": [
20        {
21            "search_query": {
22                "text": "What is the tallest penguin in the world?",
23                "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
24            },
25            "document_ids": [
26                "web-search_0",
27                "web-search_1",
28            ],
29            "connector": {
30                "id": "web-search"
31            }
32        }
33    ],
34    "documents": [
35        {
36            "id": "web-search_0",
37            "snippet": "The emperor penguin (Aptenodytes forsteri) is the tallest and heaviest of all living penguin species and is endemic to Antarctica. The male and female are similar in plumage and size, reaching 100 cm (39 in) in length and weighing from 22 to 45 kg (49 to 99 lb). Feathers of the head and back are black and sharply delineated from the white belly, pale-yellow breast and bright-yellow ear patches.\n\nLike all penguins, it is flightless, with a streamlined body, and wings stiffened and flattened into flippers for a marine habitat. Its diet consists primarily of fish, but also includes crustaceans, such as krill, and cephalopods, such as squid.",
38            "title": "Emperor penguin - Wikipedia",
39            "url": "https://en.wikipedia.org/wiki/Emperor_penguin"
40        },
41        {
42            "id": "web-search_1",
43            "snippet": "King penguins can weigh up to 40 pounds, growing to be 33 to 37 inches tall.\n\nThey are also able swimmers, diving to depths of over 200 feet in search of squid and small fish, which are their main food sources.\n\nKing penguins lay unique pear-shaped eggs, which they incubate in a pooch and carry around with their legs.\n\n3. Gentoo Penguin\n\nThe Gentoo Penguin is the world’s third-largest penguin. While its average height is 31 inches, it can grow to a maximum height of 35 inches.\n\nView this post on Instagram\n\nA post shared by Ricardo Peralta Ayala (@ricardo_peralta_ayala)\n\nGentoo Penguins have a white stripe across their black head, making them easily distinguishable from other penguin species.",
44            "title": "A Ranking of the 10 Biggest Penguin Species - American Oceans",
45            "url": "https://www.americanoceans.org/facts/the-largest-penguins-ranked-by-size/"
46        }
47    ]
48}
49{
50    "is_finished": false,
51    "event_type": "text-generation",
52    "text": "The"
53}
54{
55    "is_finished": false,
56    "event_type": "text-generation",
57    "text": " tallest"
58}
59{
60    "is_finished": false,
61    "event_type": "text-generation",
62    "text": " penguin"
63}
64{
65    "is_finished": false,
66    "event_type": "text-generation",
67    "text": " in"
68}
69{
70    "is_finished": false,
71    "event_type": "text-generation",
72    "text": " the"
73}
74{
75    "is_finished": false,
76    "event_type": "text-generation",
77    "text": " world"
78}
79{
80    "is_finished": false,
81    "event_type": "text-generation",
82    "text": " is"
83}
84{
85    "is_finished": false,
86    "event_type": "text-generation",
87    "text": " the"
88}
89{
90    "is_finished": false,
91    "event_type": "text-generation",
92    "text": " Emperor"
93}
94{
95    "is_finished": false,
96    "event_type": "text-generation",
97    "text": " Penguin"
98}
99{
100    "is_finished": false,
101    "event_type": "text-generation",
102    "text": "."
103}
104{
105    "is_finished": false,
106    "event_type": "text-generation",
107    "text": " They"
108}
109{
110    "is_finished": false,
111    "event_type": "text-generation",
112    "text": " have"
113}
114{
115    "is_finished": false,
116    "event_type": "text-generation",
117    "text": " an"
118}
119{
120    "is_finished": false,
121    "event_type": "text-generation",
122    "text": " average"
123}
124{
125    "is_finished": false,
126    "event_type": "text-generation",
127    "text": " height"
128}
129{
130    "is_finished": false,
131    "event_type": "text-generation",
132    "text": " of"
133}
134{
135    "is_finished": false,
136    "event_type": "text-generation",
137    "text": " 45"
138}
139{
140    "is_finished": false,
141    "event_type": "text-generation",
142    "text": " inches"
143}
144{
145    "is_finished": false,
146    "event_type": "text-generation",
147    "text": " and"
148}
149{
150    "is_finished": false,
151    "event_type": "text-generation",
152    "text": " weigh"
153}
154{
155    "is_finished": false,
156    "event_type": "text-generation",
157    "text": " up"
158}
159{
160    "is_finished": false,
161    "event_type": "text-generation",
162    "text": " to"
163}
164{
165    "is_finished": false,
166    "event_type": "text-generation",
167    "text": " 100"
168}
169{
170    "is_finished": false,
171    "event_type": "text-generation",
172    "text": " pounds"
173}
174{
175    "is_finished": false,
176    "event_type": "text-generation",
177    "text": "."
178}
179{
180    "is_finished": false,
181    "event_type": "citation-generation",
182    "citations": [
183        {
184            "start": 40,
185            "end": 56,
186            "text": "Emperor Penguin.",
187            "document_ids": [
188                "web-search_0",
189                "web-search_1",
190            ]
191        }
192    ]
193}
194{
195    "is_finished": true,
196    "event_type": "stream-end",
197    "response": {
198        "response_id": "75f0d364-5086-4d38-8afd-b67d19e06bf1",
199        "text": "The tallest penguin in the world is the Emperor Penguin. They have an average height of 45 inches and weigh up to 100 pounds.",
200        "generation_id": "6789661c-731c-4d83-b0fe-8926f6194811",
201        "token_count": {
202            "prompt_tokens": 2821,
203            "response_tokens": 29,
204            "total_tokens": 2850,
205            "billed_tokens": 37
206        },
207        "citations": [
208            {
209                "start": 40,
210                "end": 56,
211                "text": "Emperor Penguin.",
212                "document_ids": [
213                    "web-search_0",
214                    "web-search_1",
215                ]
216            },
217        ],
218        "documents": [
219        {
220            "id": "web-search_0",
221            "snippet": "The emperor penguin (Aptenodytes forsteri) is the tallest and heaviest of all living penguin species and is endemic to Antarctica. The male and female are similar in plumage and size, reaching 100 cm (39 in) in length and weighing from 22 to 45 kg (49 to 99 lb). Feathers of the head and back are black and sharply delineated from the white belly, pale-yellow breast and bright-yellow ear patches.\n\nLike all penguins, it is flightless, with a streamlined body, and wings stiffened and flattened into flippers for a marine habitat. Its diet consists primarily of fish, but also includes crustaceans, such as krill, and cephalopods, such as squid.",
222            "title": "Emperor penguin - Wikipedia",
223            "url": "https://en.wikipedia.org/wiki/Emperor_penguin"
224        },
225        {
226            "id": "web-search_1",
227            "snippet": "King penguins can weigh up to 40 pounds, growing to be 33 to 37 inches tall.\n\nThey are also able swimmers, diving to depths of over 200 feet in search of squid and small fish, which are their main food sources.\n\nKing penguins lay unique pear-shaped eggs, which they incubate in a pooch and carry around with their legs.\n\n3. Gentoo Penguin\n\nThe Gentoo Penguin is the world’s third-largest penguin. While its average height is 31 inches, it can grow to a maximum height of 35 inches.\n\nView this post on Instagram\n\nA post shared by Ricardo Peralta Ayala (@ricardo_peralta_ayala)\n\nGentoo Penguins have a white stripe across their black head, making them easily distinguishable from other penguin species.",
228            "title": "A Ranking of the 10 Biggest Penguin Species - American Oceans",
229            "url": "https://www.americanoceans.org/facts/the-largest-penguins-ranked-by-size/"
230        }
231        ],
232        "search_results": [
233            {
234                "search_query": {
235                    "text": "What is the tallest penguin in the world?",
236                    "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
237                },
238                "document_ids": [
239                    "web-search_0",
240                    "web-search_1",
241                ],
242                "connector": {
243                    "id": "web-search"
244                }
245            }
246        ],
247        "search_queries": [
248            {
249                "text": "What is the tallest penguin in the world?",
250                "generation_id": "2c45db14-85f7-4714-b6c5-19cc9f026165"
251            }
252        ]
253    },
254    "finish_reason": "COMPLETE"

It contains information about whether the streaming session is finished, what type of event is being fired, and the text that was generated by the model.

Of course, the print(event.text) and print(event.finish_reason) lines in the code snippet above peels a lot of the extra information away, so what your output would look more like this:

The
 tallest
 living
 penguins
 in
 the
 world
 are
 Emperor
 penguins
,
 which
 can
 reach
 heights
 of
 approximately
 115
 cm
 (
45
.
3
 inches
)
 tall
.
 Interestingly
,
 they
 are
 only
 found
 in
 Antarctica
.
[{'start': 45, 'end': 61, 'text': 'Emperor penguins', 'document_ids': ['doc_0']}]
[{'start': 104, 'end': 130, 'text': '115 cm (45.3 inches) tall.', 'document_ids': ['doc_0']}]
[{'start': 169, 'end': 180, 'text': 'Antarctica.', 'document_ids': ['doc_1']}]
COMPLETE

It should be (more or less) the same text, but that text is on its own rather than being accompanied by search queries, event types, etc.

Note that the citation objects in the response are returned as part of a RAG request, which you can learn more about in the Retrieval Augmented Generation guide

When the model has finished generating, it returns the full text, some metadata, citations, and the documents that were used to ground the reply.