Whether the task has a test set

Whether the task has a validation set

Checks if `torch_npu` is installed and potentially if a NPU is in the environment

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Draw the first `n` samples in order from the specified split. Used for tasks with "canonical" ordered fewshot examples, such as MMLU

dumps the current config as a dictionary object, as a printable format. null fields will not be printed. Used for dumping results alon

pass a string `regex` to run `re.compile(r"regex")` on. `fallback` defines the output returned if no matches for the regex are locate

r""" Get the current device. For GPU, we return the local process index using the `accelerate.PartialState` object to handle corner ca

As opposed to the default direction of broadcasting (right to left), this function broadcasts from left to right Args: in

Format refs and preds for sacrebleu corpus calculation. It is very particular

Separate the kwargs from the arguments that we support inside `supported_args` and the ones that we don't.

Append a new segment to the history. args: text (`str`): The text of the new segment. tokens (`torch.LongTen

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Calculate model outputs in multiple batches. Args: queries (`torch.LongTensor`): List of tensors contain

Create a filtering pipeline.

Return a list of indices so that each slice of `batch_size` consecutive indices correspond to elements of similar lengths. To do this, the in

Initializes the model with given parameters. Parameters: model_name (str): Name of the model to initialize. projection_path (str, op

Load video frames from a video file. Parameters: vis_path (str): Path to the video file. n_clips (int): Number of clips to extract f

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

Compute variance of tensor with masked values.

Normalize the str to lower case and make them float numbers if possible. https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445

Normalize the str to lower case and make them float numbers if possible. https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445

Normalize the str to lower case and make them float numbers if possible. https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445

Context manager for handling null reference model (that is, peft adapter manipulation).

Set or update the configuration for a given key.

Helper function for reproducible behavior to set the seed in `random`, `numpy`, and `torch`. Args: seed (`int`): The seed to set.