If there are enough hypotheses and that none of the hypotheses being generated can become better than the worst one in the heap, then

Useful for code segments that may be accessed with or without mpu initialization

Return True if in the first pipeline model-parallel stage, False otherwise.

Return True if in the last pipeline model-parallel stage, False otherwise.

Return True if pipeline stage executes decoder block for a model with both encoder and decoder.

Return True if pipeline stage executes encoder block for a model with both encoder and decoder.

Loading entrypoint. In the steps below, the following verbs refer to corresponding objects: - load = load from checkpoint - extract = ext

Set required arguments from the checkpoint specified in the arguments. Will overwrite arguments that have a non-None default value, but w

Load the necessary attributes: model, dataloader and empty BlockData

selectively load retrieval models for indexing/retrieving from saved checkpoints

Load a model checkpoint and return the iteration. strict (bool): whether to strictly enforce that the keys in :attr:`state_dict` of the ch

Load common (non-sharded) objects state dict from the checkpoint. Args: checkpoint_dir (Path): checkpoint directory Returns:

Load the dataset & document ids. Load the dataset id & document id of each chunk in the database, to be used for causality filtering

Load the distributed parameter states of all optimizers from a file. Args: filename (str): path to load parameter state from.

Loads the parameter state from an internal representation. Inverse of the `get_parameter_state_internal_repr` method.

Load predefined args from Retro config (if applicable). When using Retro (or GPT for comparison purposes), data arguments are overridden by t

Load Retro's config.json.

Replaces all ShardedObject from a given state dict with values loaded from the checkpoint. Args: sharded_state_dict (ShardedStateDict):

Customized load.

Copies parameters and buffers from state_dict into the wrapped module and its descendants. If strict is True, then the keys of state_

Load the state dict. As detailed in state_dict(), the state dict contains all non- parameter-related variables. This method is notabl

Load tensors metadata from the checkpoint. Returns a dictionary similar to a sharded state dict, but note that the dictionary keys are simply

Load tensors metadata from the checkpoint. Returns a dictionary similar to a sharded state dict, but note that the dictionary keys ar

Loads a vocabulary file into a dictionary.

Returns a 2-dimensional (2-D) attention mask :param source_block: 1-D array :param target_block: 1-D array

Generates a list of samples of a form (path_to_sample, class). Args: directory (str): root dataset directory class_to_idx (Dict[st

Helper for instantiating a non-sharded ShardedObject (replicated across TP and DP group). Args: obj (object): any object to be sharded

Helper for instantiating a non-sharded ShardedTensor (replicated across TP and DP group). Optionally, can provide offsets which prepend new dime

Merge individual embedding blocks into a single binary mmap file. The embeddings are initially stored in block-sized (e.g., ~100k embeddings per

Set the logits for none top-k values to -inf.

Set the logits for none top-p values to -inf.

Split a range (length num_samples) into sequence of block ranges of size block_size. Args: config (RetroPreprocessingConfig): Retro p

Abstract method implementation For GPT, the underlying IndexedDataset should be split by sequence, as opposed to, say, BERT, which sh

Turn optimizer state dict to sharded state dict based on model state dict *in-place*. Can be used to add sharding information to most common opt

Parse all arguments.

First stage of async saving. Copy data to CPU and plan the local saving. Args: plan (SavePlan): save plan generated by t

Preprocess token indices for AlltoAll communication and token permutation. This method computes the number of tokens assigned to each expert

Main training program. This function will run the followings in the order provided: 1) initialize Megatron. 2) setup model, optim

Query neighbors of a dataset block (i.e., range). Args: config (RetroPreprocessingConfig): Retro preprocessing config. db_dataset

Query neighbors of each chunk within a dataset. Args: config (RetroPreprocessingConfig): Retro preprocessing config. db_dataset (

Query a block of embeddings. The block is broken into smaller sub-blocks, for easier tracking of progress. Both the raw neighbor IDs and the

Query neighbors of a block of embeddings. Querying includes: - Query index for neighbor chunk IDs. - Filter chunk IDs that have the s

Receive from previous pipeline stage and update the input buffer inplace.

Remove added codes after adding to index. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Remove embeddings after training. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Simple GPU memory report.

Turn self.write_results into a single results lists. Includes error check. Returns (List[WriteResult]): the list of write results fr

Cross attention for Retro decoder. Notation: ns : Sequence length. bs : Batch size. d : Hidden size.

Cross attention for Retro encoder. Notation: ns : Sequence length. bs : Batch size. d : Hidden size.

Top-k routing function Args: logits (torch.Tensor): Logits tensor. Returns: Tuple[torch.Tensor, torch.Tensor

Performs a consistency check. This function makes sure that tensors across devices are the same during an experiment. This is often n

Sample and generate a token. Note: logits has the dimension [b, v] where b is the batch size and v is the vocabulary size. If vocab

Translates MCore ShardedTensors to PyT ShardedTensors and saves in PyT Distributed format. Args: sharded_state_dict (ShardedStat

Save block of codes to disk. Args: config (RetroPreprocessingConfig): Retro preprocessing config. block (dict): Range

Save block of chunked tokens to disk. These blocks are later used for training and adding to the vector index. Args: block (dict): Ra

Save given config to checkpoint directory. Args: config: checkpoint config checkpoint_dir: checkpoint directory Returns:

Save dataset order & meta-info. Args: project_dir (str): Path to Retro project dir. indexed_dataset_infos (List[Dict]): List of m

Save the distributed parameter states of all optimizers to a file. Args: filename (str): path to save parameter state to.

Save the block data that was created this in this process

Finalization of save_state_dict_async_plan. The input arguments are the same as the save_state_dict_async_plan output, the `write_result

First stage of saving a state dict to storage. This is an async adjustment of torch.distributed.checkpoint.state_dict_saver. In order to

Init method based on N(0, sigma/sqrt(2*num_layers).

Function for just scoring. Args: model: no interleaving is supported. tokens: prompt tokens extended to be of size [b, max_prompt

Send output to the next pipeline stage.

Set args, tokenizer, tensorboard-writer, adlr-autoresume, and timers.