given split proportions checks if should split Examples: >>> should_split([10,0,0]) False >>> should_split([1,.1,.2]) True

Split a dataset into subsets given proportions of how much to allocate per split. If a split is 0% returns None for that split. Purpose:

Split strings based on string lengths and given start.

checks if corpus name is defined in `corpora.py`

Serializes this instance to a Python dictionary.

Serialize this instance to a Python dictionary.

Serializes this instance to a JSON string.

Serialize this instance to a JSON string.

Tokenize a string.

Train the model.

Truncate sequence pair according to original BERT implementation: https://github.com/google-research/bert/blob/master/create_pretrain

@param cmd str @param configs list of dicts

Convert `url` into a hashed filename in a repeatable way. If `etag` is specified, append its hash to the url's, delimited by a period.

Helper function for the cross entropy.

Zero fp32 and fp16 parameter grads.

Convert a list or tokenization object of Ids to a text string

converts ascii ids to tokens before joining them into text

converts ids to a text string

converts ids to wordpiece tokens and joins them as a text string

Convert a list or tokenization object of tokens to a text string

just concatenates ascii tokens into text

converts sentencepiece tokens to a text string

converts wordpiece tokens to a text string

convert text to ascii 256 Ids

convert text to sentencepiece Ids

Preprocess text and encode as tokens. Return a tokenization object with original text, processed text, and token tokenization.

convert text to ascii 256 characters

convert text to sentencepiece tokens

convert wordpiece token to Id

Convert an Id to Token. Reverse lookup of self.vocab

convert Id to sentencpiece token

logic to set and remove (set to None) tokenizer. combines preprocessing/tokenization into one callable.

Convert a Token to Id. Lookup of self.vocab

convert sentencpiece token to Id

convert sentencpiece token to Id

Torch method for processing logits.

run preprocessing and encode text as Ids