Get the path to the data file from the prefix Args: path_prefix (str): The prefix Returns: str: The path to the data file

Construct an index from an index type string. Args: index_type (str): One of 'faiss-base' (naive Faiss index wrapper) or 'faiss-p

Initialize torch.distributed.

Load the state dicts of each of the models

Customized load.

Make sure the input variable is not None.

Gather tensors and concatinate along the first dimension.

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Reduce-scatter the input tensor across model parallel group.

Reroute to the appropriate apply_rotary_pos_emb function depending on fused/unfused kernels, or bshd (conventional) / thd (packed seq) format

Build dataloader given an input dataset.

Concat with special tokens and pad sequence to self.max_seq_length

Converts a sequence of [tokens|ids] using the vocab.

Embed a text dataset. Args: text_dataset (torch.utils.data.Dataset): Text dataset to embed. Each sample of the text dataset shoul

Fix up query/key/value matrix ordering if checkpoint version is smaller than 2.0

Get directory of saved encodings. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: Path to the

Get the data parallel group the caller rank belongs to.

Get file path to empty index (i.e., trained, but unpopulated). Args: config (RetroPreprocessingConfig): Retro preprocessing confi

Return the global GlobalMemoryBuffer object

Create sub-directory for this index. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: Path to i

Get the pipeline model parallel group the caller rank belongs to.

Function to get the rotary sequence length. Args: inference_params : Used during Inference time transformer (Transfor

Return tensorboard writer. It can be None so no need to check if it is initialized.

Init method based on N(0, sigma).

Make index object verbose. Args: index (faiss.Index): Faiss object to set verbose. verbose (bool): Sets whether index

Entry-point for creating viewless tensors. This method should be used, rather than calling 'MakeViewlessTensor' or '_kernel_make_viewles

Set the rng states. For efficiency purposes, we do not check the size of seed for compatibility.

Gets the emedding weight or output logit weights when share embedding and output weights set to True. Returns: Tensor: During pre

For easy load.

Set lr for all parameters groups.

Train index on a representative set of vectors. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Build the attention mask and postition ids for the input tokens.

All-gather updated model params. When using the distributed optimizer, the params are already laid out in a contiguous buffe

Gather tensors and concatenate along the first dimension.

Return dataset iterator.

Given a model param, get the index sub-range of the param that this data-parallel rank owns.

Check if a tensor is not none, is cuda, and is contiguous.

Load the base state_dict from the given directory If rank0 is true, just loads rank 0 checkpoint, ignoring arguments.

Return a 2-D attention mask Args: source_block (numpy.ndarray): A 1-D array target_block (numpy.ndarray): A 1-D array

All-reduce the input tensor across model parallel group.

Reduce-scatter the input tensor across model parallel group.

Reduce-scatter tensors on the last dimension.

Create a Faiss Flat index with inner product as the metric to search against

Split the tensor along its first dimension and keep the corresponding slice.

Split the tensor along its last dimension and keep the corresponding slice.

Track the rng state.

Helper method to run backward step with model split into chunks (run set_virtual_pipeline_model_parallel_rank() before calling backwar

Given size and type of a tensor on all ranks and the tensor value only on a specific rank, broadcast from that rank to all other ranks.

Build the DistributedDataset Return None if and only if the underlying dataset class is not built on the current rank and torch.distr

Computes the language model loss (Cross entropy across vocabulary) Args: labels (Tensor): The labels of dimension [batch size, se

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

Converts a sequence of tokens into ids using the vocab.

Disable asynchronous grad reductions

Calculates the elapsed time and restarts timer. Args: reset (bool, optional): Resets timer before restarting. Defaults to True.

Embed a batch of tokens using the model

Encode text dataset, to be later added to index. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Build filename's path if it does not already exists.

Helper function to evaluate and dump results on screen.