Init method based on N(0, sigma).

Copy the ``.grad`` attribute from stored references to fp16 parameters to the ``.grad`` attribute of the fp32 master parameters that

Utility function for network_to_half(). Retained for legacy purposes.

convert tokens to a string accounting for command and type tokens.

Preprocess text and encode as ids. Return a tokenization object with original text, processed text, and id tokenization.

encode text as tokens using text tokenizer

convert Id to sentencpiece token

convert token to Id accounting for command and type tokens

ascii character to index

train a tokenizer on a data corpus and save model for future use

train sentencepiece model on corpus using word frequencies

extracts samples only pertaining to this worker's batch

Build a looped dataloader with infinite size.

Build the size on rank 0 and broadcast.

Traing and validation dataloaders.

Copied from fairseq for no_repeat_ngram in beam_search

Check that all the keys have the same target data type.

Performs invalid character removal and whitespace cleanup on text.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

Train the model.

Train/valid/test data arguments.

Evaluation arguments.

Mixed precision arguments.

Model arguments

Text generate arguments.

Training arguments.

Backward step.

Build lambada dataset.

Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you

@param configs list of configurations

Checkpoint a model or part of the model. This has been directly copied from torch.utils.checkpoint.

Remove new lines and multiple spaces and adjust end of sentence dot.

Clips fp32 master gradients via ``torch.nn.utils.clip_grad_norm``. Args: max_norm (float or int): max norm of the gradie

add cmdline flags for configuring datasets

Converts a module's immediate parameters and buffers to dtype.

Converts a network's parameters and buffers to dtype.

Converts a sequence of tokens into ids using the vocab.

Mask sequence pair for BERT training according to: https://github.com/google-research/bert/blob/master/create_pretraining_data.py#L33

fetches a random sentencepair corresponding to rng state similar to https://github.com/google-research/bert/blob/master/create_pretra

iterates through data and handles wrap around

Evaluation.

Evaluation.

Evaluate and print results on screen.

Simple forward step with cross-entropy loss.

Convert fp16 `val` to fp32

Convert fp32 `val` to fp16

Constructs a `BertConfig` from a Python dictionary of parameters.

Constructs a `BertConfig` from a json file of parameters.

OpenAI's gelu implementation.