Add one or more special tokens. Returns: int: The new size of the vocabulary.

Add one or more regular tokens. Returns: int: The new size of the vocabulary.

Returns the capcode level of the vocabulary. 0 = disabled 1 = only deleteToken 2 = enabled

Returns one of "UTF-8", "UTF-16", "None"

A decoder object used for decoding token streams. This decoder object is used instead of the vocabulary decode method when y

Returns a new decoder instance used for decoding tokens into text.

Delete one or more regular or special tokens. You can give the token in either its encoded or decoded form. Returns:

Delete one or more regular or special token by specifying the token ID. Returns: int: The new size of the vocabulary.

Disables the UNK token. Without an UNK token, any character for which there is no token is ignored during tokenization Retur

Closes tokenmonsterserver subprocess.

Enables the UNK token. If enabled, the UNK token appears whenever there is a character that is not in the vocabulary. Note th

Get the token string from a single token ID, in it's capcode-encoded form. Parameters: id: int Returns:

Get the token string from a single token ID, in it's capcode-decoded form. Parameters: id: int Returns:

Loads a TokenMonster vocabulary from file, URL or by name. It's safe for multiprocessing, but vocabulary modification is disabled and tokeniz

Returns the optimization mode of the vocabulary.

Modifies the vocabulary using a YAML file. A sample YAML file can be found here: https://github.com/alasdairforsythe/tokenmonster/yam

Returns the normalization of the vocabulary, e.g. "NFD trim"

Resets the token IDs to be sequential beginning from zero. If tokens have been deleted from the vocabulary there will be gaps in the

The default directory for TokenMonster is ~/_tokenmonster Use this function to set the default directory elsewhere, before loading any vocabu

Returns the ID of a single token. This works for both capcode-encoded "raw" tokens, and their decoded form. Parameters:

Same as tokenize, but it returns only the number of tokens. The number of tokens is the same as you would get from `tokenize`. If yo

Returns the ID of the UNK token, or 'None' type if there is no UNK token Parameters: token: string Returns: