Endpointsv1/tokenize

Tokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Headers

AuthorizationstringRequired

Bearer authentication of the form Bearer <token>, where token is your auth token.

X-Client-NamestringOptional
The name of the project that is making the request.

Request

This endpoint expects an object.
textstringRequired
The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
modelstringRequired
The input will be tokenized by the tokenizer that is used by this model.

Response

OK
tokenslist of integers
An array of tokens, where each token is an integer.
token_stringslist of strings
metaobject or null

Errors