Helper function to get module spec for MoE

Compute seq_length // chunk_length. Args: sample_length (int): Alias of `sequence_length`. chunk_length (int): Retro chunk length

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Get the position embedding on the current context parallel rank. Args: pos_emb (Tensor): Positional embedding tensor seq_dim (int

Number of samples & chunks (e.g., 32*n_samples) in corpus.

Reads MCore data for N-D flattened tensors from checkpoint metadata during ckpt load. Args: sharded_state_dict (ShardedStateDict): sharde

collect rng state across data parallel ranks

Included for backwards compatibility.

Get the tensor- and context-parallel group the caller rank belongs to.

Calculate the global rank corresponding to the first local rank in the tensor model parallel group.

Determine right tensor sizes (based on position of rank with respect to split rank) and model size. Send two tensors if model decoder req

Convert GPT token chunk dataset to a text dataset for passing to the embedder. Args: config (RetroPreprocessingConfig): Retro preproc

Convert GPT token chunk dataset to a text dataset for passing to the embedder. Args: config (RetroPreprocessingConfig): Retro preproc

Train/valid/test num samples.

Return the virtual pipeline-parallel world size.

Returns ViT layer spec with Transformer Engine layers

Get max value across ranks. Args: n (int): Value on this rank. Returns: Max value across all ranks.

Check if grouped_gemm is available.

Checks if a file is an allowed extension. Args: filename (string): path to a file extensions (tuple of strings): extensions to con

Initialize Megatron and build model.

Initialize the dataset This method is called by IndexedDataset.__init__ during object creation and by IndexedDataset.__setstate__ dur

Helper function to initialize the rerun machine instance. Check the RerunStateMachine class for the details.

Create the RNG tracker. 'use_te_rng_tracker' determines whether to use Megatron or TransformerEngine's implementation. In particular, Transfor

Check if an iteration is the first for a model chunk.

Query if currently capturing.

Check if an iteration is the last for a model chunk.

Check if the current word piece is the starting piece (BERT).

selectively load retrieval models for indexing/retrieving from saved checkpoints

Load checkpoint, and forward pass data.

Load common part of the checkpoint.

Loads and parses a yaml file into a JETWorkloadManifest

Populate members from instance saved to file

Loaded indexed datasets into memory-mapped datasets. Args: project_dir (str): Path to Retro project dir. indexed_dataset_infos (L

Preprocesses the given state dictionary by applying factories and extracting non-persistent data, without modifying the original dictionary.

Load Retro's config.json.

Load sharded metadata from the checkpoint. Similar to `load_tensors_metadata`, but includes also ShardedObjects. Returns a dictionary simila

Load just the metadata from the checkpoint.

Load the state dict. As detailed in state_dict(), the state dict contains all non- parameter-related variables. This method is notabl

Load tensors metadata from the checkpoint for ShardedTensors. Returns a dictionary similar to a sharded state dict, but note that the

This function gathers a bool value across the model parallel group

Loss function (with KD Loss support). Args: loss_mask (Tensor): Used to mask out some portions of the loss model (GPTModel): The

Returns a 2-dimensional (2-D) attention mask :param source_block: 1-D array :param target_block: 1-D array

Helper for instantiating a non-sharded ShardedObject (replicated across TP and DP group). Args: obj (object): any object to be sharded

Build a ShardedTensor or ShardedTensorFactory for optimizer param based on model param Args: model_param (Union[ShardedTensor, ShardedTen

Helper for instantiating a non-sharded ShardedTensor (replicated across TP and DP group). Optionally, can provide offsets which prepend new dimen

Raises or logs an error in case missing or unexpected keys are non-empty. Args: missing_keys (Set[str]): missing keys in the state dict

Convert state dict with ShardedTensors and ShardedObjects to state dict compatible with PyT Dist format. Operates in-place and returns the or

Set the logits for none top-k values to -inf.

Set the logits for none top-p values to -inf.

Returns iterator over (nested) tuples (container, key, value) of a given dict or list.

Helper method to get the model chunk ID given the iteration number.

Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, articles and extra whitespace.

Ascertain whether the object at the given S3 path exists in S3 Args: client (S3Client): The S3 client path (str): The S3 path

Returns a list of [start, end) character offsets of each token.

Pad sequences and convert them to numpy.

Pad vocab size based on inference size

Returns true if the passed-in parameter is not a duplicate parameter on another TP rank.

Turn numpy array to torch tensor.

Prepare gradient calculation operands.

Pre-processing gradients before the optimizer step, returns whether inf/nan is found.

Ensure grad_output is stored in a contiguous buffer.

Creates a tensor-aware state dictionary that can be saved using the Local Checkpoint Manager. Args: sharded_state_dict (ShardedStateDict)