Tokenize

POST

https://api.cohere.com/v1/tokenize

POST

/v1/tokenize

1 import cohere
2 
3 co = cohere.Client()
4 
5 response = co.tokenize(
6     text="tokenize me! :D", model="command-a-03-2025"
7 )  # optional
8 print(response)

Try it

200Successful

1 {
2   "tokens": [
3     10002,
4     2261,
5     2012,
6     8,
7     2792,
8     43
9   ],
10   "token_strings": [
11     "token",
12     "ize",
13     " me",
14     "!",
15     " :",
16     "D"
17   ],
18   "meta": {
19     "api_version": {
20       "version": "1"
21     }
22   }
23 }

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Was this page helpful?

DetokenizeDetokenize

Built with

Detokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Authentication

AuthorizationBearer

Bearer authentication of the form `Bearer <token>`, where token is your auth token.

Headers

X-Client-NamestringOptional

The name of the project that is making the request.

Request

textstringRequired>=1 character<=65536 characters

The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.

modelstringRequired

The input will be tokenized by the tokenizer that is used by this model.

Response

tokenslist of integers

An array of tokens, where each token is an integer.

token_stringslist of strings

metaobject or null

Errors

The name of the project that is making the request.

The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.

Bearer authentication of the form Bearer <token>, where token is your auth token.

1	import cohere
2
3	co = cohere.Client()
4
5	response = co.tokenize(
6	text="tokenize me! :D", model="command-a-03-2025"
7	) # optional
8	print(response)

1	{
2	"tokens": [
3	10002,
4	2261,
5	2012,
6	8,
7	2792,
8	43
9	],
10	"token_strings": [
11	"token",
12	"ize",
13	" me",
14	"!",
15	" :",
16	"D"
17	],
18	"meta": {
19	"api_version": {
20	"version": "1"
21	}
22	}
23	}