* func norm_UTF16_NFD(input []byte) ([]byte, error) { // Assume LittleEndian by default endian := uni.LittleEndian bomPolicy := uni.IgnoreBOM if l

normalizes but avoids double encoding with capcode

Saves the current vocabulary to a file. Parameters: filename (string): The filename to save the vocabulary to.

* Bitwise stuff: Things that I need: 1 ends with a letter 2 begins with a letter 4 begins with a space OR characterToken OR wordToken 8 ends on c

Adds a single special token to the vocabulary. A special token is special because only this token is allowed to tokenize text containing this. If any

Add multiple special tokens and optionally resize. Enter `size` 0 to not resize. Modifying a vocabulary does not change existing token IDs.

Adds a single token to the vocabulary. Modifying a vocabulary does not change existing token IDs. All normalization and capcode is applied automatical

Adds multiple regular and optionally special tokens. You can use `size` to resize the vocabulary to keep it at a specific size. Enter `size` 0 to not

Decodes tokens IDs back into bytes.

Deletes a single token from the vocabulary. Tokens to delete can be capcoded encoded or not, it will look for both. Modifying a vocabulary does not ch

Deletes a single token from the vocabulary by specifying the ID. Modifying a vocabulary does not change existing token IDs.

Delete multiple tokens and optionally resize. Tokens to delete can be capcoded encoded or not, it will look for both. Enter `size` 0 to not resize. Mo

Deserializes tokens encoded in a bytes stream into a slice of uint32 token IDs. `encodingLength` must be one of: 0, 2, 3, 4. If you enter `encodingLen

Flushes the remainder from the Decoder instance These will any trailing incomplete UTF-8 sequences or capcode encoding marks

Returns the encoded token for the token ID, or nil if it does not exist.

Load the vocabulary from a local file.

Add regular & special tokens, delete tokens and resize, all in one. Modifying a vocabulary does not change existing token IDs. Pass resetTokenIds = tr

Add regular & special tokens, delete tokens and resize, all in one. Modifying a vocabulary does not change existing token IDs. Pass resetTokenIds = tr

--------- GENERATE & MODIFY --------- NewVocab makes a fresh vocabulary from a custom list of tokens. If you generated your vocabulary with TokenMonst

NewVocabFromYAML makes a fresh vocabulary from a YAML file.

Returns the number of special tokens in the vocabulary.

Resets all the IDs of the tokens to be assigned alphabetically, starting from 0, with no gaps.

Resize the vocabulary by deleting the worst scoring tokens. You can also resize the vocabulary to be larger if any tokens have previously been deleted

A slice that contains all the single byte tokens in the vocabulary. Note that this is returned as only a slice of bytes, not a slice of slice of bytes

Returns a slice of all tokens in the vocabulary (excluding UNK), in their encoded capcode form.