Method tokenize

src/whispercpp/context.cc:221–235

Run the Whisper decoder to obtain the logits and probabilities for the next token. Make sure to call whisper_encode() first. tokens + n_tokens is the provided context for the decoder. n_past is the number of tokens to use from previous decoder calls. Returns vec on success.

// provided context for the decoder. n_past is the number of tokens to use
// from previous decoder calls. Returns vec<whisper_token> on success.
std::vector<whisper_token> Context::tokenize(std::string *text,
                                             size_t max_tokens) {
    std::vector<whisper_token> tokens;
    tokens.reserve(max_tokens);

    int ret = whisper_tokenize(wctx, text->c_str(), tokens.data(), max_tokens);

    if (ret == -1) {
        RAISE_RUNTIME_ERROR("Too many results tokens.");
    } else {
        // ret != -1 then length of the vector is at least ret tokens
        tokens.reserve(ret);
    }
    return tokens;
};

// Returns id of a given language, raise exception if not found
int Context::lang_str_to_id(const char *lang) {