Instantiate a PreTrainedBertModel from a pre-trained model file. Download and cache the pre-trained model file if needed.

Build the model.

ADLR autoresume object. It can be None so no need to check if it is initialized.

Get the data parallel group the caller rank belongs to.

gets dataset object based on keyword args and file at `path`

gets text of document corresponding to idx

Calculate the global rank corresponding to a local rank zeor in the model parallel group.

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Returns the sequence length of the cached states. A layer index can be optionally passed.

Get dataset splits from comma or '/' separated string list.

Check if the current word piece is the starting piece (BERT).

Copy master parameters to model parameters. Args: model_params: List of model parameters created by :func:`prep_param_lists`.

helper function to pad sequence pair

LM logits using word embedding weights.

Process batch and produce inputs for the model.

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

split document into sentences

tokenize sentence and get token types

Setup model and optimizer.

Split a full s3 path into the bucket name and path.

Save dict with state dicts of each of the models.

Set lr for all parameters groups.

checks if corpus name is defined in `corpora.py`

Single training step.

Log training information such as losses, timing, ....

Copy the ``.grad`` attribute from stored references to fp16 parameters to the ``.grad`` attribute of the fp32 master parameters that

Returns vocab size

Zero fp32 and fp16 parameter grads.

Utility function for network_to_half(). Retained for legacy purposes.

convert Ids to tokens accounting for command and type tokens, tokens are joined and returned as a string.

convert tokens to a string accounting for command and type tokens.

Preprocess text and encode as ids. Return a tokenization object with original text, processed text, and id tokenization.

encode text as tokens using text tokenizer

convert Id to token accounting for command and type tokens

convert Id to sentencpiece token

ascii character to index

train a tokenizer on a data corpus and save model for future use

train sentencepiece model on corpus using word frequencies