Converts a sequence of tokens into ids using the vocab.

Helper function to evaluate and dump results on screen.

Generate batch from context tokens.

Gets directory path where lazy files are stored.

Build the model.

Reduce a tensor of losses across all GPUs.

Set to the initial state (no tracker).

For easy load.

Make sure the input variable is not None.

Gather tensors and concatinate along the last dimension.

This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the

Initialize affine weight for model parallel. Build the master weight on all processes and scatter the relevant chunk.

Initialize affine weight for model parallel on GPU.

All-reduce the the input tensor across model parallel group.

Split the tensor along its last dimension and keep the corresponding slice.

Initialize tokenizer.

Clear the embedding data structures to save memory. The metadata ends up getting used, and is also much smaller in dimensionality so i

Compile helper function ar runtime. Make sure this is invoked on a single process.

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

Calculate the elapsed time.

Build masks and position id for left to right model.

Get rng states. Copy the dictionary so we have direct pointers to the states, not just a pointer to the dictionary.

Return tensorboard writer. It can be None so no need to check if it is initialized.

Buld dataloader given an input dataset.

Main training program. This function will run the followings in the order provided: 1) initialize Megatron. 2) setup model, optim

Set the rng states. For efficiency purposes, we do not check the size of seed for compatibility.

If no closure is supplied, :attr:`step` should be called after ``fp16_optimizer_obj.backward(loss)``. :attr:`step` updates th

Runs basic whitespace cleaning and splitting on a piece of text.

Runs basic whitespace cleaning and splitting on a piece of text.

ascii index to character

extracts samples only pertaining to this worker's batch

extracts samples only pertaining to this worker's batch

Initialize tokenizer.

Create a Faiss Flat index with inner product as the metric to search against

Provide function that calculates accuracies.

Copied from `PreTrainedTokenizerBase.add_tokens`

Allocate a memory buffer.

:attr:`backward` performs the following conceptual steps: 1. fp32_loss = loss.float() (see first Note below) 2. scaled_loss

Convert to numpy and return a sample consumed by the batch producer.

Build token types and paddings, trim if needed, and pad if needed.

Check for autoresume signal and exit if it is received.

Apply conversion to val. Recursively apply conversion if `val` is a nested tuple/list structure.

iterates through data and handles wrap around

Embed a batch of tokens using the block model

Ensure that numerator is divisible by the denominator.

read specified portion of file

Main finetune function used across all tasks.

Run a forward pass for each of the models and return the respective embeddings.