Endpointsv1/tokenize

Tokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Headers

X-Client-NamestringOptional
The name of the project that is making the request.

Request

textstringRequired1-65536 characters
The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
modelstringRequired
The input will be tokenized by the tokenizer that is used by this model.

Response headers

X-API-Warningstring or null
The name of the project that is making the request.

Response

OK
tokenslist of integers
An array of tokens, where each token is an integer.
token_stringslist of strings
metaobject or null

Errors