Endpointsv1/tokenize

Tokenize

This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.

Headers

AuthorizationstringRequired

Bearer authentication of the form Bearer <token>, where token is your auth token.

X-Client-NamestringOptional
The name of the project that is making the request.

Request

This endpoint expects an object.
textstringRequired
The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
modelstringRequired
The input will be tokenized by the tokenizer that is used by this model.

Response

OK
token_stringslist of strings
tokenslist of integers or null
An array of tokens, where each token is an integer.
metaobject or null

Errors