Get the path to the index file from the prefix Args: path_prefix (str): The prefix Returns: str: The path to the index file

Build language model and return along with the key to save.

Offset of a local chunk in a global array of chunks. Returns: Tuple[int, ...]: the offset of the whole local chunk in a global ar

Merges dicts and lists recursively.

Replaces the given prefix in *all* sharded keys in a given state dict. Errors out if some key does not begin with a given prefix. Args:

Make a directory, conditional on not being in validation mode. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Symbolic function for tracing.

Codifies the constraints on metadata attributes.

Validate if the ShardedTensors and ShardedObjects from multiple processes define correct sharding. Local ShardedTensors and ShardedObject metadat

Copied from torch.optim.optimizer

Gather tensors and concatinate along the last dimension.

Helper function to get module spec for MLP/MoE

args: y_true : 3-d ndarray in [batch_size, img_rows, img_cols] nb_ch : int return [float] weights

Use multi-tensor-applier to copy values from one list to another. We don't have a bfloat16 implementation so for now if the overflow_buf

We do some checks and return attention mask dimensions for self attention Transformer engine library underwent a lot of change. So we need to

Append given string to progress log.

Given size and type of a tensor on all ranks and the tensor value only on a specific rank, broadcast from that rank to all other ranks. Args:

Config attention-mask for encoder_mask, decoder_mask, encoder_decoder_mask conditioned on transformer-implementation (e.g. TE vs local), TE ve

Crop from ``img``

Calculates the elapsed time and restarts timer. Args: reset (bool, optional): Resets timer before restarting. Defaults to True.

Enable asynchronous grad reductions

Clean up and write the index (.idx) file Args: idx_path (str): The path to the index file

Main finetune function used across all tasks.

Get file path to index that has been populated with vectors. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Get the data-parallel group the caller rank belongs to.

Return my rank for the expert tensor parallel group.

Return world size for the expert tensor parallel group.

Get parameter state (i.e., parameter & optimizer tensors). This method performs two steps: - For each DP rank, copy param & optimizer

Function to get the rotary sequence length. Args: inference_params : Used during Inference time transformer (Transfor

Get pytorch version from __version__; if not available use pip's. Use caching.

Load the state dicts of each of the models

Shift start/end indexes to start at new start index. Both start and end indexes will be shifted by [new start] - [old start]. Args:

Determine if peers are sending, and where in data structure to put received tensors. Return a boolean if the pipeline stage expects to

Ranks without an optimizer will have no grad_norm or num_zeros_in_grad stats. We need to ensure the logging and writer rank has those values.

Adds a given strategy to the registry of default strategies. Args: action (StrategyAction): specifies save/load and sharded/common

Set the rng states. For efficiency purposes, we do not check the size of seed for compatibility.

Provides implementation for sharded_state_dict method for non-MegatronModules. Tries to call `module.sharded_state_dict` when possible, other

Dispatch tokens to experts. Args: tokens (torch.Tensor): Input tokens. probs (torch.Tensor): The routing probability

Restores the expert output to its original ordering. Args: expert_output (torch.Tensor): The output tensor from the expert models

Verifies if checkpoint metadata exists and matches given strategies. If no strategies are passed, they are determined based on the checkpoint met

Initialize self.

Initialize affine weight for model parallel on GPU.

Sets the random number generator state of the current GPU. Argumentss: new_state (torch.ByteTensor): The desired state This function

Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

Zero out the gradient for a group of parameters. Note: copied from torch.optim.optimizer.

Backward step through passed-in output tensor. If last stage, output_tensor_grad is None, otherwise gradient of loss with respect to stage's

Broadcast a tensor from last pipeline stage to all ranks.

Check if it is the first validation step.

Computes the language model loss (Cross entropy across vocabulary) Args: labels (Tensor): The labels of dimension [batch size, se

Returns the inference data given context window This function gets called iteratively in a loop . Given the start and end context pos

Get the path to the data file from the prefix Args: path_prefix (str): The prefix Returns: str: The path to the data file

Get default load sharded strategy.

Get the expert-tensor-parallel group the caller rank belongs to.

Construct an index from an index type string. Args: index_type (str): One of 'faiss-base' (naive Faiss index wrapper) or 'faiss-p

FP32 optimizer does not do any scaling.

Learning rate decay functions from: https://openreview.net/pdf?id=BJYwwY9ll pg. 4 Args: param_group (dict): parameter gro

Build masks and position id for left to right model.

Determine the number of transformer layers to build for the current pipeline stage. Args: config (TransformerConfig): Configuration o

Get list of parameters wrapped in optimizer.

GPT tokens to text.

Initialize the tensor data of this ShardedTensor. Only called if `data` attribute is None. Args: device (Union[

Initialize torch.distributed.

Ascertain whether a path is in S3 Args: path (str): The path Returns: bool: True if the path is in S3, False otherwise

Load the state dicts of each of the models

Customized load.

Returns a 2-dimensional (2-D) attention mask :param source_block: 1-D array :param target_block: 1-D array

Finalizes all available calls. This method must be called on all ranks. Args: blocking (bool, optional): if True, will w

Builds the model. Args: pre_process (bool, optional): Set to true if you need to compute embedings. Defaults to True. post_proces

Record a new sample.