Returns the list of shard files containing only weights offloaded to disk.

r""" Finds the correct formatting function based on the dataset structure. Currently supported datasets are: - `ChatML` with [{"role": str, "c

Computes element-wise mean and variance of the tensor across processes. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b

Prepare the head mask if needed. Args: head_mask (`torch.Tensor` with shape `[num_heads]` or `[num_hidden_layers x num_h

Use the Levenshtein distance (or edit distance) to determine which of the choices is most similar to the given prediction

Calculate the indices of frames to extract from a video. Parameters: total_num_frames (int): Total number of frames in the video. de

Get the rank of this process in the sequence parallel group the caller rank belongs to.

Get the size of the sequence parallel group the caller rank belongs to.

Get the shard and index within the shard corresponding to the given index.

Return the number of cache accesses and misses.

Returns the training [`~torch.utils.data.DataLoader`]. Subclass of transformers.src.transformers.trainer.get_train_dataloader to pre

Returns the trainable parameters of the pipeline

calculate the wups score :param pred: :param truth: :return:

Group the file names by key. Args: names: A list of file names. Returns: A list of lists of indices, where each sublist cont

Meta initialization context manager. A context manager under which models are initialized with all parameters on the meta device, therefore c

Device initialization context manager. A context manager under which models are initialized with all parameters on the specified device.

If needed prunes and maybe initializes weights. If using a custom `PreTrainedModel`, you need to implement any initialization logic i

r""" return a callable function that takes in an "instructions" dataset and returns a formatted dataset, based on the tokenizer apply chat tem

Interpret the transformations argument. This takes care of transformations specified as string shortcuts and returns a list of callables.

Return the intersection of the two half-open integer intervals.

Keep the most recent files in a directory, deleting the rest. The maxsize is the maximum size of the directory in bytes. The maxfiles is the

Convert a list of lengths to a list of ranges.

r""" Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://arxiv.org/abs/

r""" Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://arxiv.org/abs/

Light wrapper around accelerate's register_lad_state_pre_hook which is run before loading state