Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current

given split proportions checks if should split Examples: >>> should_split([10,0,0]) False >>> should_split([1,.1,.2]) True

Split a dataset into subsets given proportions of how much to allocate per split. If a split is 0% returns None for that split. Purpose:

Split strings based on string lengths and given start.

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to spli

Break a tensor into equal 1D chunks.

Tokenize a string.

Tokenize a string.

Tokenize a string.

@param text: Text to encode. @param linebreak: Whether to encode newline (\n) in text. @param whitespaces: Whether to encode

This function has been mostly taken from huggingface conversational ai code at https://medium.com/huggingface/how-to-build-a-state-of-t

Utility function for tracking progress

Train the model function.

Provide train and validation datasets.

Truncates a pair of sequences to a maximum sequence length.

Truncate sequence pair according to original BERT implementation: https://github.com/google-research/bert/blob/master/create_pretrain

Convert `url` into a hashed filename in a repeatable way. If `etag` is specified, append its hash to the url's, delimited by a period.

Helper function for the cross entropy.

Convert a list or tokenization object of Ids to a text string

converts ascii ids to tokens before joining them into text

converts ids to a text string

converts ids to wordpiece tokens and joins them as a text string

Convert a list or tokenization object of tokens to a text string

just concatenates ascii tokens into text

converts sentencepiece tokens to a text string

converts wordpiece tokens to a text string

convert text to ascii 256 Ids

convert text to sentencepiece Ids

convert text to wordpiece Ids

Preprocess text and encode as tokens. Return a tokenization object with original text, processed text, and token tokenization.

convert text to ascii 256 characters

convert text to sentencepiece tokens

convert wordpiece token to Id

Convert an Id to Token. Reverse lookup of self.vocab

convert Id to sentencpiece token

logic to set and remove (set to None) tokenizer. combines preprocessing/tokenization into one callable.

convert token to Id accounting for command and type tokens

Convert a Token to Id. Lookup of self.vocab

convert sentencpiece token to Id

convert sentencpiece token to Id