Return the data currently in use.

Return world size for the data parallel group.

Get the embedding group the caller rank belongs to.

Simple linear layer with weight initialization.

Get the memory buffer.

Get the model parallel group the caller rank belongs to.

get_op_support_info

get_op_support_info

Get a list that maps a sample index to a starting sentence index, end sentence index, and length

Initializ the memory buffer for the checkpointed activations.

Initialize model data parallel groups. Arguments: tensor_model_parallel_size: number of GPUs used to parallelize model tensor.

Dictionary from vocab id token to text token.

Return True if in the first pipeline model-parallel stage, False otherwise.

Return True if in the last pipeline model-parallel stage, False otherwise.

Useful for code segments that may be accessed with or without mpu initialization

layernorm operator interface implementation calculating: x, gamma, beta mean = np.mean(x, reduce_axis, keepdims=True) varian

algorithm: layernorm_grad calculating: gradient of layernorm compute partial derivation of x, gamma and beta pd_xl

selectively load retrieval models for indexing/retrieving from saved checkpoints

r"""load the weights from the checkpoint

r"""load the weights from the checkpoint

Customized load.

Customized load.

Customized load.

Customized load.

Customized load.

Customized load.

Customized load.

Customized load.

Main program.

Main program.

Initialize model parallel cuda seed. This function should be called after the model parallel is initialized. Also, no torch.cuda.manual_seed

Check if model and data parallel groups are initialized.

Build the model.

Build the model.

Build the model.

Return number of elements in use.

function of selecting dynamic format Parameters ---------- input_dy : dict shape and dtype of input dy, only support float16

select format dynamically

Pad sequences and convert them to numpy.

Print memory usage average over time. We would like this value to be as high as possible.

Print min, max, and norm of all parameters.

Returns True if the IO object can be read.

Receive tensor from next rank in pipeline (backward receive).

Receive tensor from previous rank in pipeline (forward receive).

Refreshes any internal state from the current model parameters. Call whenever the parameters are changed outside of the optimizer. For

Reset the buffer start index to the beginning of the buffer.