Serializes tokens from a numpy array into a binary string. The encoding_length needs to be recorded separetely.

Tokenizes but returns the number of tokens instead of the tokens.

Decodes tokens from a serialized bytes slice. `encodingLength` must be one of: 0, 2, 3, 4. If you enter `encodingLength` 0 then it will determine the

Decodes capcode from the bytes.

Disables the UNK token. Without an UNK token, a character that has no token to represent it will be ignored.

Enables the UNK token. Returns true if successful, returns false if an UNK token is not applicable to this vocabulary (all bytes have tokens). If enab

Exports the vocabulary to a human-readable YAML file. It writes to an io.Writer. You can import from YAML with NewVocabFromYAML().

Returns true if the vocabulary is using the UNK token. If used, the UNK token ID is used whenever a character being tokenized doesn't exist in the voc

The number of single byte tokens in the vocabulary.

--------- LOADING AND SAVING --------- Save the vocabulary to local file.

Returns the ID of the token from bytes. This only works for capcode encoded tokens. Apply `Normalize` to the bytes first to use this with decoded toke

Tokenizes directly into serialized bytes with either 16-bit, 24-bit or 32-bit encoded unsigned integers depending on the vocabulary size. Set encoding

Returns a slice of Info struct where the index is the Token ID

Returns the ID of the Unk token. It will return 16777215 if there is no Unk token. You can use HasUnk() to first check if there is an UNK token.

* func norm_UTF16_NFD(input []byte) ([]byte, error) { // Assume LittleEndian by default endian := uni.LittleEndian bomPolicy := uni.IgnoreBOM if l

Deserializes a binary string into a numpy array of tokens IDs. The encoding_length needs to be recorded separetely.

This is a helper function to allow for resuming the progress from a final dictionary It returns the score and true if the filename is score_numbers.wh

Decodes tokens from a serialized bytes slice. `encodingLength` must be one of: 0, 2, 3, 4. If you enter `encodingLength` 0 then it will determine the

The length of the longest (encoded) token in the vocabulary. This can be lower than that chosen during training if none of the longer tokens were chos

--------- DECODER --------- Creates a new Decoder instance. This is for decoding tokens in a sequence when they are to be decoded individually or in b

The type of normalization applied automatically when tokenizing. Returns a string.

The number of tokens deleted from the vocabulary. These can be restored by resizing the vocabulary to be be larger.

Returns the uint8 code corresponding to the training parameters for single byte tokens.

Returns the token IDs and the corresponding tokens of only the. Set `decode` to false to receive the decoded form of the tokens.

Tokenizes text from bytes slice to token IDs. The 2nd returned value (int) is the number of characters for which there were no tokens and were replace