Communicate tensor shapes between stages. Used to communicate tensor shapes before the actual tensor communication happens. This is required w

Returns list of zarr arrays corresponding to given tensors. For a sharded tensors that: a) is main replica and represents the first chunk (a

Build the dataset and assert any subclass-specific conditions

Perform save finalization. Breakdown into `save` and `save_finalize` cn be useful for async saving.

Finds the class folders in a dataset. Args: dir (string): Root directory path. Returns: tuple: (class

Gather tensors and concatenate along the first dimension.

Report times across all ranks.

Turns ShardedTensor offsets into offsets suitable for ShardedObject.

Report strings for max/minmax times across all ranks.

Get shard views of each of the param buffers. In this nested list, the top level is grouped by the virtual model index and t

Calculate the number of epochs Args: num_tokens_per_epoch (int): The number of tokens in a single epoch Returns:

Compute the number of transformer layers resident on the current rank.

Calculate the number of tokens in a single epoch Returns: int: The number of tokens in a single epoch

Create parameter groups for optimizer. Creates parameter groups based on weight decay condition (regularized vs non regularized), learning ra

Receive happens between the layers.

Return from the dataset by using file pointer Args: idx (Union[int, numpy.integer, slice]): The index or index slice into the dat

Return from the dataset by mmap-ing .bin file Args: idx (Union[int, numpy.integer, slice]): The index or index slice into the dat

Set autoresume start time.

Initialize torch.distributed and core model parallel.

initializing the communicators with user buffers for high-performance tensor-model-parallel communication overlap

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Calculates size (in bytes) of a single write item. Same as torch.distributed.checkpoint.filesystem._item_size, but fixes computing chunk

Return all config attributes which contribute to uniquely identifying the dataset. These attributes will be used to build a uniquely identify

Create a forward pre-hook to wait on all-gather handles when necessary (i.e., when a module uses a parameter in a bucket with a still

Return a 2-D history (lower-left-triangular) mask Args: block (numpy.ndarray): A 1-D array Returns: numpy.nd

Creates the all-reduce / reduce-scatter hook for backprop.

Build io.BytesIO from given sharded objects data.

Build a PyT ShardedTensor from given shards. During loading: - if data is None, initialize it with an empty tensor (will be used to c

If recv_buffer is none, we will allocate one on the fly.

Print arguments.

Print arguments.

Inverse of _replace_state_dict_keys_with_sharded_keys.

Recursively update `x` keys, based on `keys_template`.

Change sign so the last dimension becomes [-odd, +even] Args: x (Tensor): Input tensor Returns: Tensor: Tensor rotated half

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Build the sequence pointers per the sequence lengths and dtype size Args: sequence_lengths (List[int]): The length of each sequen

Initialize ADLR autoresume.

Initialize global buffer

Set random seed for reproducability.

Set tensorboard writer.

Initialize timers.

Splits write items according to item size into close to uniform bins. Same as torch.distributed.checkpoint.filesystem._split_by_size_and_typ

Adds whitespace around any CJK character.

Given a set of prompts and number of tokens to generate: - tokenize prompts - set the sequence length to be the max of length of promp

Train index (rank 0's method). Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Train index on DB chunks. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

Unwrap tensor from PyT ShardedTensor instance. If `prepend_axis_num` was non-zero (which is specific to MCore ShardedTensor) then the tensor

Updates `max_lr` and `min_lr` values in each parameter group, and returns new list. By default, each group will use `lr` / `min_lr` as `max_l