[s, b, num_splits * hp] -->(view) [s, b, num_splits, hp] -->(tranpose) [s, b, hp, num_splits] -->(view) [s, b, num_splits *

Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

Track the rng state.

Allocate a memory buffer.

:attr:`backward` performs the following conceptual steps: 1. fp32_loss = loss.float() (see first Note below) 2. scaled_loss

Data loader. Note that batch-size is the local (per GPU) batch-size.

Apply conversion to val. Recursively apply conversion if `val` is a nested tuple/list structure.

Ensure that numerator is divisible by the denominator.

Fork the cuda rng state, perform operations, and exit with the original state.

Calculate the global rank corresponding to a local rank zeor in the model parallel group.

Initialize the state from pre-trained BERT model

Copy master parameters to model parameters. Args: model_params: List of model parameters created by :func:`prep_param_lists`.

Normalize answer.

LM logits using word embedding weights.

Process batch and produce inputs for the model.

Set to the initial state (no tracker).

Get the top-k blocks by the index distance metric. :param reconstruct: if True: return a [num_queries x k x embed_dim] array of blocks

Setup model and optimizer.

Set lr for all parameters groups.

Single training step.

Log training information such as losses, timing, ....

Copy the ``.grad`` attribute from stored references to fp16 parameters to the ``.grad`` attribute of the fp32 master parameters that

https://discuss.pytorch.org/t/how-can-i-do-the-operation-the-same-as-np-where/1329/9 :param cond: :param x_1: :param x_2: :return

Utility function for network_to_half(). Retained for legacy purposes.

Build a looped dataloader with infinite size.

Build the size on rank 0 and broadcast.

Initialize T0 model.

Initialize T0 tokenizer.

Train dataloader.

Traing and validation dataloaders.

Check that all the keys have the same target data type.

Forward method with activation checkpointing.

Performs invalid character removal and whitespace cleanup on text.

Initialize torch.distributed and mpu.

Initialize manually allocated static memory.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Load the DPR wikipedia evidence file

Parse entire arguments.

Print arguments.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Set random seed for reproducibility.

Set tensorboard writer.

Initialize timers.