Build an array with length = number-of-epochs * number-of-dcuments. Each index is mapped to a corresponding document.

Build doc-idx, sample-idx, and shuffle-idx. doc-idx: is an array (ordered) of documents to be used in training. sample-idx: is the start docum

Build a looped dataloader with infinite size.

Build the size on rank 0 and broadcast.

Build lambada dataset.

Build the range [0, size) and shuffle.

Traing and validation dataloaders.

Check that all the keys have the same target data type.

Forward method with activation checkpointing.

Performs invalid character removal and whitespace cleanup on text.

Performs invalid character removal and whitespace cleanup on text.

Converts an index (integer) in a token (str) using the vocab.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Calculates `cache_position` for the pre-fill stage based on `input_ids` and optionally past length

Set autoresume start time.

Initialize torch.distributed and mpu.

Initialize manually allocated static memory.

Checks whether CP is the codepoint of a CJK character.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Checks whether `chars` is a whitespace character.

Make causal mask used for bi-directional self-attention.

Based on number of samples and sequence lenght, calculate how many epochs will be needed.

Total number of tokens in the dataset.

Prepares the special tokens for generation, overwriting the generation config with their processed versions converted to tensor.

Print arguments.

Splits punctuation on a piece of text.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Strips accents from a piece of text.

Initialize ADLR autoresume.

Set random seed for reproducability.

Set tensorboard writer.

Initialize timers.

Adds whitespace around any CJK character.

Adds whitespace around any CJK character.

Train the model.

Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

Write arguments to tensorboard.

Add data for set of blocks :param block_indices: 1D array of unique int ids for the blocks :param block_embeds: 2D array of embeddings

Add the embedding of each block to the underlying FAISS index

Applies Rotary Position Embedding to the query and key tensors. Args: q (`jt.Var`): The query tensor. k (`jt.Var`): The key tenso

Backward step.

Goes through one epoch of the dataloader and adds all data to this instance's BlockData. The copy of BlockData is saved as a shard, which whe

Helper function to select and build dataset.

Build token types and paddings, trim if needed, and pad if needed.

XXX

Build train, valid, and test datasets.

Biuld training sample. Arguments: sample: A list of sentences in which each sentence is a list token ids. target_seq_length: Desi

Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you

Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you