Helper method to run forward step with model split into chunks (run set_virtual_pipeline_model_parallel_rank() before calling forward_

Get the context parallel group the caller rank belongs to.

Sub-directory for DB data. Args: project_dir (str): Path to Retro project dir. Returns: Path of the DB sub-directory wit

Retrieves a default strategy for a given action, backend and version.

Get empty index (i.e., trained, but unpopulated). Args: config (RetroPreprocessingConfig): Retro preprocessing config. R

Get the path to the index file from the prefix Args: path_prefix (str): The prefix Returns: str: The path to the index file

Load indexed dataset meta-infos. Args: project_dir (str): Path to Retro project dir. Returns: List of metadata for each data

Path to indexed dataset meta-infos. Args: project_dir (str): Path to Retro project dir. Returns: Path to the `indexed_datase

Get paths of all database blocks of an individual dataset. Args: project_dir (str): Path to Retro project dir. prefix (str): Uniq

Simple linear layer with weight initialization.

Get merged dataset. Args: project_dir (str): Path to Retro project dir. chunk_length (int): GPT chunk length (e.g., 64).

Return one logger. It can be None so no need to check if it is initialized.

Get parameter state (i.e., parameter & optimizer tensors). This method performs two steps: - For each DP rank, copy param & optimizer

Get list of parameters wrapped in optimizer.

Get rng states. Copy the dictionary so we have direct pointers to the states, not just a pointer to the dictionary.

Get directory for of saved embedding blocks. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: P

Get path to merged training embeddings. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: Path t

Get root directory for embeddings (blocks and merged data). Args: config (RetroPreprocessingConfig): Retro preprocessing config. Ret

Return the virtual pipeline-parallel rank.

Return tensorboard writer. It can be None so no need to check if it is initialized.

Import a named object from a module in the context of this function. TODO: make this importer module more robust, at least make sure there ar

Load parameter state (i.e., parameter & optimizer tensors) from DP 0 rank. This method performs the reverse of get_parameter_state_dp_zero():

Customized load.

Load metadata of Zarr arrays. Args: checkpoint_dir (str): checkpoint root directory get_shape_dtype_fn (str -> ((int, ...), np.dt

Merges dicts and lists recursively.

Merge individual DBs into single DB. Args: project_dir (str): Retro project dir. indexed_dataset_infos (List[Dict]): Preprocessin

Do non-exponentiated normalization Args: weights (List[float]): The weights Returns: List[float]: The normalized weights

Opens a Zarr file array with Tensorstore with basic setting. Args: arr_path (Path): path to a Zarr (Tensorstore) array

LM logits using word embedding weights.

Parse the dataset split ratios from a string Args: split (str): The train valid test split string e.g. "99,1,0" Returns: Lis

Reset timer.

Reset metadata in bucket in preparation for the next iteration of training.

Save a model checkpoint.

Init method based on N(0, sigma/sqrt(2*num_layers).

Sets up embedding layer in first stage and output layer in last stage. This function initalizes word embeddings in the final stage when we ar

Initiates grad sync (all-reduce or reduce-scatter) communication operation for this bucket. When overlap_grad_reduce is set

The state dict contains all non-DP-rank-dependent (i.e., non-parameter- related) optimizer variables. The returned state dict can be

Runs basic whitespace cleaning and splitting on a piece of text.

Enter the context introduced by the 'with' keyword Returns: _IndexWriter: The instance

Exit the context introduced by the 'with' keyword Args: exc_type (Optional[Type[BaseException]]): Exception type exc

Initialize the MoE Token Dispatcher.

Allocate memory to store kv cache during inference.

Receive happens between the layers with size [s, b, h].

Build the range [0, size) and shuffle Args: num_samples (int): The size of the first shuffle range [0, num_samples) total_si

Initialize tokenizer.

Creates the predictions for the masked LM objective Args: token_ids (List[int]): The token ids target_sequence_length

Waits on all_gather_handle if necessary, then dispatches the next all-gather as necessary.

Single forward step. Update the allocate memory flag so only the first time the memory is allocated.

Returns elapsed times of timers in names. Assumptions: - All the ranks call this function. - `names` are identical on

Report only min and max times across all ranks.

Build masks and position id for left to right model. Args: data (torch.Tensor): The data tenor that holds the tokens from the dataset

Get Megatron optimizer based on parameter groups. Args: config (OptimizerConfig): optimizer configuration object. param_groups (l

Determine the contribution of the MegatronDataset splits to the BlendedDataset splits Args: blend (List[str]): e.g. ["30", "path/to/d

Check if a tensor is not none and is cuda.

Make a viewless tensor. View tensors have the undesirable side-affect of retaining a reference to the originally-viewed tensor, even after ma

Use multi-tensor-applier to copy values from one list to another. We don't have a bfloat16 implementation so for now if the overflow_buf

Get memory used by current process. Returns (int): memory used by current process

Get the text (token ids) and document ids for a given index Args: idx (int): The index into the dataset Returns:

Group ShardedBase objects by keys and return mappings required for recreating the original dict.

Reset metadata needed to track results of all-gathers.

Helper function to create new bucket, add it to list of buckets, and also update param->bucket mapping.

Verifies if checkpoint metadata exists and matches given strategy. Args: checkpoint_dir (str): checkpoint directory sharded_stra

Add a new hypothesis to the list.

Turn ShardedTensorFactories into ShardedTensors *in-place*. Args: sharded_state_dict (ShardedStateDict): state dict possibly containing

Apply rotary positional embedding to input tensor T. check https://kexue.fm/archives/8265 for detailed formulas Args: t (Tensor): In