For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
DASHBOARDPLAYGROUNDDOCSCOMMUNITYLOG IN
Guides and conceptsAPI ReferenceRelease NotesLLMUCookbooks
Guides and conceptsAPI ReferenceRelease NotesLLMUCookbooks
  • Cohere API
    • About
    • Teams and Roles
    • Errors
    • Migrating From API v1 to API v2
    • Using the OpenAI SDK
  • Endpoints
      • POSTTokenize
  • Deprecated
LogoLogodocs
DASHBOARDPLAYGROUNDDOCSCOMMUNITYLOG IN
Endpointsv1/tokenize

Tokenize

POST
https://api.cohere.com/v1/tokenize
POST
/v1/tokenize
1import cohere
2
3co = cohere.Client()
4
5response = co.tokenize(
6    text="tokenize me! :D", model="command-a-03-2025"
7)  # optional
8print(response)
1{
2  "tokens": [
3    10002,
4    2261,
5    2012,
6    8,
7    2792,
8    43
9  ],
10  "token_strings": [
11    "token",
12    "ize",
13    " me",
14    "!",
15    " :",
16    "D"
17  ],
18  "meta": {
19    "api_version": {
20      "version": "1"
21    }
22  }
23}

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Was this page helpful?
Previous

Detokenize

Next
Built with

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Headers

X-Client-NamestringOptional
The name of the project that is making the request.

Request

textstringRequired1-65536 characters
The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
modelstringRequired
The input will be tokenized by the tokenizer that is used by this model.

Response headers

X-API-Warningstring
The name of the project that is making the request.

Response

OK
tokenslist of integers
An array of tokens, where each token is an integer.
token_stringslist of strings
metaobject

Errors

400
Bad Request Error
401
Unauthorized Error
403
Forbidden Error
404
Not Found Error
422
Unprocessable Entity Error
429
Too Many Requests Error
498
Invalid Token Error
499
Client Closed Request Error
500
Internal Server Error
501
Not Implemented Error
503
Service Unavailable Error
504
Gateway Timeout Error