Save the auxiliary loss for logging. Args: name (str): The name of the loss. loss (torch.Tensor): The loss tensor. layer_n

Set the document indices This method is slated for deprecation. Args: document_indices (numpy.ndarray): The document ind

Shard buffer into data_parallel_world_size chunks of equal size.

Provides implementation for sharded_state_dict method for non-MegatronModules. Tries to call `module.sharded_state_dict` when possible, other

Save dict with state dicts of each of the models.

For easy load when model is combined with other heads, add an extra key.

Save dict with state dicts of each of the models.

Clip gradients (if needed) and step the base optimizer. Always return successful since there is no overflow.

Calculate position ids from token ids Args: token_ids (Tensor): input tokens Returns: Tensor: position ids

Tokenize prompts and make them avaiable on all ranks.

Unpermute a tensor of permuted tokens based on sorted indices, and optionally merge the tokens with their corresponding probabilities. Args:

Validate added encodings. Steps: - Randomly sample subset of text dataset blocks. - Encode each block. - Compare against saved encodi

Validate if the ShardedTensors from multiple processes define correct sharding of a global tensor. Local ShardedTensors metadata is exchanged wi

Validate training embeddings. Steps: - Randomly sample subset of text dataset blocks. - Embed each block. - Compare against saved emb

Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C)

Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C)

Copied from torch.optim.optimizer

Initialize Timer. Args: name (str): Name of the timer.

Initialize the Router module. Args: config (TransformerConfig): Configuration object for the Transformer model.

Add to index (rank 0's method). Args: config (RetroPreprocessingConfig): Retro preprocessing config. text_dataset (GP

Add DB chunks to index. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: Path to the populated

Saves the generated key and value tensors to the end of the buffers in inference_params. Returns the full size keys and values from t

All-reduce both word and position embeddings.

All-reduce layernorm grads (for sequence parallelism).

All-reduce position_embeddings grad across first (encoder) and split (decoder) stages to ensure that position embeddings parameters stay in s

All-reduce word embedding grads. Reduce grads across first and last stages to ensure that word_embeddings parameters stay in sync. This

Build all dataset splits according to the provided blend(s) See the BlendedMegatronDatasetBuilder.build alias for more information.

Build an array with length = num epochs * num documents Args: documents (numpy.ndarray): the subset of exposed document indices

Build the document index, the sample index, and the shuffle index The document index: -- 1-D -- An ordered ar

Build mapping between params and their grad buffers. These mappings are partitioned according to data type. Iterate through

Build and optionally cache the dataset index and the dataset sample index The dataset index is a 1-D mapping which determines the dataset to

Build the size on rank 0 and broadcast.

Create main parameter groups needed for the optimizer step. These groups encompass both: 1) groups used by this class, for r

Build mapping from param reference to grad buffer shard ranges. This method builds a mapping from parameter references to grad

Build mapping between params and their grad buffers. This method does the initial setup for the method above. This setup inc

Create a reverse of the gbuf_ranges, for referencing in opposite direction.

Create optimizer groups. Given the set of parameter shard ranges that are owned by the current data-parallel (DP) rank, gath

Check that all the keys have the same target data type.

Forward method with activation checkpointing.

Forward method with selective activation checkpointing.

Forward method with activation checkpointing.

Forward method with activation checkpointing.

Performs invalid character removal and whitespace cleanup on text.