Simple scaling.

Set args, tokenizer, tensorboard-writer, adlr-autoresume, and timers.

r"""Set parallel context

r"""Set parallel context

r"""Set parallel context

r"""Set parallel context

Set prarllel context in pipeline training process

Set random seed for reproducability.

Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabula

Set tensor model parallel rank.

Set the tensor model parallel size

Set weight decay coefficient, zero for bias and layernorm, 1e-1 for rest

Optional DeepSpeed Activation Checkpointing features. Gives access to partition activations, contiguous memory optimizations and cpu checkpoin

Setup model and optimizer.

Generate a series of (prompt, code) pairs by sliding the window over the code.

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

Break a tensor into equal 1D chunks.

Clip gradients (if needed) and step the base optimizer. Always return successful since there is no overflow.

Tokenize a string.

tokenize lambada dataset

tokenize openwebtext dataset

tokenize wikitext-2/wikitext-103 dataset

This function has been mostly taken from huggingface conversational ai code at https://medium.com/huggingface/how-to-build-a-state-of-the-

This function has been mostly taken from huggingface conversational ai code at https://medium.com/huggingface/how-to-build-a-state-of-the-

This function has been mostly taken from huggingface conversational ai code at https://medium.com/huggingface/how-to-build-a-state-of-the-

Samples from a multinomial distribution using the top-k sampling strategy. Args: logits: A tensor of shape (batch_size, vocab_size)

Train the model function.

Single training step.

Log training information such as losses, timing, ....

Cleans up the generated code text.

Update list of ppl

function of updating Nz shape

Write arguments to tensorboard.

Reset the buffer to zero.

Copied from torch.optim.optimizer

Set the grad buffer data to zero. Needs to be called at the beginning of each iteration.