Tokenize

POST

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Headers

X-Client-NamestringOptional

The name of the project that is making the request.

Request

This endpoint expects an object.

textstringRequired

The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.

modelstringRequired

An optional parameter to provide the model name. This will ensure that the tokenization uses the tokenizer used by that model.

Response

This endpoint returns an object.

tokenslist of integers

An array of tokens, where each token is an integer.

token_stringslist of strings

metaobjectOptional

Detokenize

Up Next

Built with

1	import cohere
2
3	co = cohere.Client()
4
5	response = co.tokenize(
6	text="tokenize me! :D", model="command-r-plus-08-2024"
7	) # optional
8	print(response)

1	{
2	"tokens": [
3	10002,
4	2261,
5	2012,
6	8,
7	2792,
8	43
9	],
10	"token_strings": [
11	"token",
12	"ize",
13	" me",
14	"!",
15	" :",
16	"D"
17	],
18	"meta": {
19	"api_version": {
20	"version": "1"
21	}
22	}
23	}