Endpointsv1/tokenize

Tokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Headers

X-Client-NamestringOptional
The name of the project that is making the request.

Request

textstringRequired1-65536 characters
The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
modelstringRequired
The input will be tokenized by the tokenizer that is used by this model.

Response headers

X-API-Warningstring
The name of the project that is making the request.

Response

OK
tokenslist of integers
An array of tokens, where each token is an integer.
token_stringslist of strings
metaobject

Errors

400
Bad Request Error
401
Unauthorized Error
403
Forbidden Error
404
Not Found Error
422
Unprocessable Entity Error
429
Too Many Requests Error
498
Invalid Token Error
499
Client Closed Request Error
500
Internal Server Error
501
Not Implemented Error
503
Service Unavailable Error
504
Gateway Timeout Error