Initialize manually allocated static memory.

Set random seed for reproducability.

[s, b, num_splits * hp] -->(view) [s, b, num_splits, hp] -->(tranpose) [s, b, hp, num_splits] -->(view) [s, b, num_splits *

Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

Write arguments to tensorboard.

Track the rng state.

Allocate a memory buffer.

:attr:`backward` performs the following conceptual steps: 1. fp32_loss = loss.float() (see first Note below) 2. scaled_loss

Apply conversion to val. Recursively apply conversion if `val` is a nested tuple/list structure.

Ensure that numerator is divisible by the denominator.

Fork the cuda rng state, perform operations, and exit with the original state.

ADLR autoresume object. It can be None so no need to check if it is initialized.

Calculate the global rank corresponding to a local rank zeor in the model parallel group.

Initialize torch.distributed and mpu.

Copy master parameters to model parameters. Args: model_params: List of model parameters created by :func:`prep_param_lists`.

Normalize answer.

LM logits using word embedding weights.

Process batch and produce inputs for the model.

Set to the initial state (no tracker).

Get the top-k blocks by the index distance metric. :param reconstruct: if True: return a [num_queries x k x embed_dim] array of blocks

Setup model and optimizer.

Set lr for all parameters groups.

Single training step.

Log training information such as losses, timing, ....

Copy the ``.grad`` attribute from stored references to fp16 parameters to the ``.grad`` attribute of the fp32 master parameters that

https://discuss.pytorch.org/t/how-can-i-do-the-operation-the-same-as-np-where/1329/9 :param cond: :param x_1: :param x_2: :return

Utility function for network_to_half(). Retained for legacy purposes.

Build a looped dataloader with infinite size.

Build a looped dataloader with infinite size.

Build the size on rank 0 and broadcast.

Initialize T5 tokenizer.

Traing and validation dataloaders.

Traing and validation dataloaders.

Check that all the keys have the same target data type.

Forward method with activation checkpointing.

Performs invalid character removal and whitespace cleanup on text.

Set autoresume start time.

Initialize torch.distributed and mpu.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.