Get the size of the Ulysses sequence parallel group.

Set a key to a value in a list of dictionaries.

Returns whether this model can generate sequences with `.generate()`. Returns: `bool`: Whether this model can generate s

Flatten dictionary and concatenate nested keys with separator.

Return the communication group with all data-parallel ranks

Returns the model's output embeddings. Returns: `nn.Module`: A torch module mapping hidden states to vocabulary.

Returns the first found floating dtype in parameters if there is one, otherwise returns the last dtype it found.

Get the RingAttn sequence parallel process group.

Checks if `torch_npu` is installed and potentially if a NPU is in the environment

Load a remote or local dataset description in JSON format.

Reads a PyTorch checkpoint file, returning properly formatted errors if they arise.

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Save a model and its configuration file to a directory, so that it can be re-loaded using the [`~PreTrainedModel.from_pretrained`] cl

Helper function for reproducible behavior to set the seed in `random`, `numpy`, and `torch`. Args: seed (`int`): The seed to set.

A context manager to partition the model parameters during the model construction with MiCS partition strategy. Model states are partitioned

Automatically checks and dispatches to a default attention implementation. In order of priority: 1. An implementation specified i

r""" Get the current device. For GPU, we return the local process index using the `accelerate.PartialState` object to handle corner ca

Get the data parallel group the caller rank belongs to.

Get the expert data parallel group the caller rank belongs to.

Get the expert parallel group the caller rank belongs to.

Get the ZeRO parameter partitioning intra parallel group the caller rank belongs to.

Initialize the weights. This method should be overridden by derived class and is the only initialization method that will be called w

As opposed to the default direction of broadcasting (right to left), this function broadcasts from left to right Args: in

Separate the kwargs from the arguments that we support inside `supported_args` and the ones that we don't.

Append a new segment to the history. args: text (`str`): The text of the new segment. tokens (`torch.LongTen

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Calculate model outputs in multiple batches. Args: queries (`torch.LongTensor`): List of tensors contain

Enables the gradients for the input embeddings. This is useful for fine-tuning adapter weights while keeping the model weights fixed.

Expand the given PIL image to a square shape by adding padding. Parameters: - pil_img: The PIL image to be expanded. - background_co

Given a URL, return a local name for the shard.

feature_map: (1, h, w, C) is the feature map of a single image.

Get the index (e.g. 2) from the prediction (e.g. 'C')

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

Compute variance of tensor with masked values.

Normalize the str to lower case and make them float numbers if possible.

Context manager for handling null reference model (that is, peft adapter manipulation).

Parse the prediction from the generated response. Return the predicted index e.g., A, B, C, D.