This endpoint splits input text into smaller units called tokens using byte-pair encoding (BPE). To learn more about tokenization and byte pair encoding, see the tokens page.
The name of the project that is making the request.
The string to be tokenized, the minimum text length is 1 character, and the maximum text length is 65536 characters.
An optional parameter to provide the model name. This will ensure that the tokenization uses the tokenizer used by that model.
An array of tokens, where each token is an integer.