Broadcast a list of float values.

Broadcast a list of values with a given type.

Build pretraining data iterators.

Clear the embedding data structures to save memory. The metadata ends up getting used, and is also much smaller in dimensiona

Compute grad norm.

Apply conversion to val. Recursively apply conversion if `val` #is a nested tuple/list structure.

Converts a sequence of ids in BPE tokens using the vocab.

Build the split matrix from one or optionally two contributing split vectors. Ex. a standard conversion: [0.99, 0.01, 0.0] -> [(0, 0.99), (0

Copy tensor values from last stage into the first stage. Note that the input tensor is updated in place.

Count number of zeros in model's gradients.

Detokenize the generated tokens.

Get the dtype from the code Args: value (int): The code Returns: Type[numpy.number]: The dtype

Embed a range of a text dataset. Args: embedder (Embedder): Embedder used for embedding a text dataset. text_dataset

Encode sub-dataset block, to be later added to index. Encode the data subset, generally in blocks of 1M vectors each. For each block,

Finishes grad sync (all-reduce or reduce-scatter) communication operation for this bucket. When overlap_grad_reduce is set t

Run a forward pass for each of the models and return the respective embeddings.

ADLR autoresume object. It can be None so no need to check if it is initialized.

Get the document indices This method is slated for deprecation. Returns: numpy.ndarray: The document indices

Path of empty index. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: Path to the empty (tr

Retrieves the appropriate forward_backward function given the configuration of parallel_state. Returns a function that will perform all of th

Get project-relative directory of GPT bin/idx datasets.

Individual DB's directory. Args: project_dir (str): Path to Retro project dir. prefix (str): Unique relative path to dataset with

FP32 optimizer does not do any scaling.

Paths to merged datasets. Args: project_dir (str): Path to Retro project dir. Returns: A dict of chunk databases, one for ea

Build the model.

Compute seq_length // chunk_length. Args: sample_length (int): Alias of `sequence_length`. chunk_length (int): Retro chunk length

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Get root directory of all saved query data. Args: project_dir (str): Retro project dir. Returns: Path to query sub-directory

collect rng state across data parallel ranks

Calculate the global rank corresponding to the first local rank in the tensor model parallel group.

Convert GPT token chunk dataset to a text dataset for passing to the embedder. Args: config (RetroPreprocessingConfig): Retro preproc

Convert GPT token chunk dataset to a text dataset for passing to the embedder. Args: config (RetroPreprocessingConfig): Retro preproc

Return the virtual pipeline-parallel world size.

Get max value across ranks. Args: n (int): Value on this rank. Returns: Max value across all ranks.

Checks if a file is an allowed extension. Args: filename (string): path to a file extensions (tuple of strings): extensions to con

Initialize the dataset This method is called by IndexedDataset.__init__ during object creation and by IndexedDataset.__setstate__ dur

Check if an iteration is the first for a model chunk.

Check if an iteration is the last for a model chunk.

Check if the current word piece is the starting piece (BERT).

Populate members from instance saved to file

Loaded indexed datasets into memory-mapped datasets. Args: project_dir (str): Path to Retro project dir. indexed_dataset_infos (L

Loss function. Args: loss_mask (torch.Tensor): Used to mask out some portions of the loss output_tensor (torch.Tensor): The tenso

Returns a 2-dimensional (2-D) attention mask :param source_block: 1-D array :param target_block: 1-D array

Build a ShardedTensor or ShardedTensorFactory for optimizer param based on model param Args: model_param (Union[ShardedTensor, ShardedTe

Simple map-reduce implementation following `more_itertools.map_reduce` interface.

Returns checkpoint config if `checkpoint_dir` is a distributed checkpoint and None otherwise Args: checkpoint_dir: checkpoint directory

Turn state dict with ShardedTensors and ShardedObjects to state dict compatible with PyT Dist format. Operates in-place and returns the original

Returns iterator over (nested) tuples (container, key, value) of a given dict or list.

Permute the tokens based on the indices. Token with the same index will be grouped together. Args: tokens (torch.Tensor): The input token

Registers grads for the passed-in param to be "ready" for grad sync. When the number of microbatches is greater than 1, we only want

Replaces the given prefix in *all* sharded keys in a given state dict. Errors out if some key does not begin with a given prefix. Args:

Set to the initial state (no tracker).

Safely set tensor's '.data' field. Check first that the tensor is viewless (i.e., '._base' not set). If not, raise an exception.

Sample a random subset of all blocks. Args: _blocks (List[Optional[Dict]]): List of all blocks. Returns: