Make sure the input variable is not None.

Gather tensors and concatinate along the last dimension.

Initialize affine weight for model parallel. Build the master weight on all processes and scatter the relevant chunk.

Initialize affine weight for model parallel on GPU.

All-reduce the the input tensor across model parallel group.

Split the tensor along its last dimension and keep the corresponding slice.

Zero out the gradient for a group of parameters. Note: copied from torch.optim.optimizer.

Reduce a tensor of losses across all GPUs.

yield n sized chunks from list

Create dataset Inputs: batch_size: batch size data_path: path of your MindRecord files device_num: total device numbe

Build filename's path if it does not already exists.

Estimates pass@k of each problem and returns them in an array.

Fork the cuda rng state, perform operations, and exit with the original state.

Tracker file rescords the latest chckpoint during training to restart from.

Build indexed dataset.

FP32 optimizer does not do any scaling.

Get rng states. Copy the dictionary so we have direct pointers to the states, not just a pointer to the dictionary.

Init method based on N(0, sigma/sqrt(2*num_layers).

Set the rng states. For efficiency purposes, we do not check the size of seed for compatibility.

judge the format is fractal NZ Parameters ---------- ori_shape: list or tuple original shape of input ori_axis: list or

Initialize tokenizer.

Build train, valid, and test datasets.

division_sixteen

Use multi-tensor-applier to copy values from one list to another. We don't have a blfoat16 implementation so for now if the overflow_buf is no

update shape_gamma for subsequent calculation Parameters ---------- shape_x: list or tuple shape of x shape_gamma: list

Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

Allocate a memory buffer.

Helper method to run backward step with model split into chunks (run set_virtual_pipeline_model_parallel_rank() before calling backwar

Cleans up the generated code.

Clone the parameters in ParameterTuple element-wisely to generate a new ParameterTuple with float32 data type. Inputs: pr

r""" parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters

Apply conversion to val. Recursively apply conversion if `val` #is a nested tuple/list structure.

Decode the whitespace-encoded strings produced by encode_whitespace. >>> text = 'a\\n b\\n c' >>> s, l = 10, 10 >>> text == decode_wh

Encode whitespaces to extra tokens in GPT-J. >>> encode_whitespaces('a\\n b\\n c', 10, 10) 'a\\n<|extratoken_10|>b\\n<|extratoken_11|>c'

Ensure that numerator is divisible by the denominator.

Evaluation.

Helper method to run forward step with model split into chunks (run set_virtual_pipeline_model_parallel_rank() before calling forward_

ADLR autoresume object. It can be None so no need to check if it is initialized.

Generate batch from context tokens.

Generate a batch

Generate batch from context tokens.

Get the data parallel group the caller rank belongs to.

Learning rate decay functions from: https://openreview.net/pdf?id=BJYwwY9ll pg. 4

Build the model.

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Get the pipeline model parallel group the caller rank belongs to.

Calculate the global rank corresponding to the first local rank in the tensor model parallel group.