Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Generate responses for a list of query tensors. args: query_tensors (list[torch.Tensor]): A list of query tensors to gen

Generate samples from the model Args: iterations (int): Number of iterations to generate samples for batch_s

Return my rank for the data parallel group.

Return world size for the data parallel group.

Generate expert parallel and expert data parallel group ranks list. Example - E + M + D parallel world_size = 16 model_degree

Return world size for the expert parallel group.

Get the maximum ep_size from all the created groups.

Get the name of the group with max. ep_size

Get the modules of the model that should not be spit when using device_map. We iterate through the modules to get the underlying `_no

Compute the output size given input size and target long side length.

Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at th

Build a resized Linear Module from a provided old Linear Module. Increasing the size will add newly initialized vectors at the end. R

r""" Initializes the weights of the value head. The default initialization strategy is random. Users can pass a different initializati

r""" We initialize the weights of the value head.

Moves `loaded_state_dict_keys` in model to meta device which frees up the memory taken by those params. `start_prefix` is used for models wh

Validate the `rope_scaling` configuration.

Check if the input data is valid for training. Args: input_ids (List[`torch.LongTensor`]): List of tenso

Check if the input data is valid for training. Args: batch_size (int): Batch size from the config file.

Tie or clone module weights depending of whether we are using TorchScript or not

Train on a batch of samples. Main training segment Args: inner_epoch (int): The current inner epoch epoch (i

r""" Activates the neftune as presented in this code: https://github.com/neelsjain/NEFTune and paper: https://arxiv.org/abs/2310.05914

Adds sentinel tokens and padding token (if missing). Expands the tokenizer vocabulary to include sentinel tokens used in mixture-of-denoiser

r""" Add and load a reward modeling adapter. This method can only be used if the model is a `PeftModel` and if you have initialized th

all-to-all for QKV Args: input (torch.tensor): a tensor sharded along dim scatter dim scatter_idx (int): default 1 g

all-to-all for QKV forward (bs, seqlen/N, 3, hc, hs) -> (bs, seqlen, 3, hc/N, hs) Args: input (torch.tensor): a tensor sharded a

Calculate the loss for a batch of an unpacked sample Args: latents (torch.Tensor): The latents sampled f

Returns canonicalized `text` (puncuation removed). Args: text (`str`): String to be canonicalized. ke

Check if the cache miss rate is too high.

Check if the given string a number.