Tokenize

POST

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Headers

X-Client-NamestringOptional

The name of the project that is making the request.

Request

This endpoint expects an object.

textstringRequired

The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.

modelstringRequired

The input will be tokenized by the tokenizer that is used by this model.

Response

tokenslist of integers

An array of tokens, where each token is an integer.

token_stringslist of strings

metaobjectOptional

Errors

Detokenize

Up Next

Built with

1	import cohere
2
3	co = cohere.Client()
4
5	response = co.tokenize(
6	text="tokenize me! :D", model="command-a-03-2025"
7	) # optional
8	print(response)

1	{
2	"tokens": [
3	10002,
4	2261,
5	2012,
6	8,
7	2792,
8	43
9	],
10	"token_strings": [
11	"token",
12	"ize",
13	" me",
14	"!",
15	" :",
16	"D"
17	],
18	"meta": {
19	"api_version": {
20	"version": "1"
21	}
22	}
23	}