Aggregate the results of the RefCOCO evaluation task using the specified metric. Args: - results (list of dict): List of result dictiona

Aggregate the results of the RefCOCO evaluation task using the specified metric. Args: - results (list of dict): List of result dictiona

Aggregate the results of the RefCOCO evaluation task using the specified metric. Args: - results (list of dict): List of result dictiona

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Aggregate the results of the screenspot evaluation task using the specified metric. Args: - results (list of dict): List of result dicti

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Calculate the instruction level accuracy for given Subject results https://github.com/MMMU-Benchmark/MMMU/blob/51ce7f3e829c16bb44bc5445782686b4c35

Converts a markdown table to the corresponding html string for TEDS computation.

calculate the wups score :param pred: :param truth: :return:

Check if the judge service is available

Generate table of results. Automatically hides columns that are all N/A (e.g., stability metrics when num_samples=1, or baseline comparison w

Normalize facing field to standard direction. Args: facing: The facing value to normalize Returns: Normalized facing directi

Splits the audio into chunks of a given length. Args: audio_arrays: The audio array to split. chunk_lim: The length of each c

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

Compute the accuracy of two bounding boxes based on a specified threshold. Parameters: - box1 (list of float): Bounding box [x_min, y_mi

Compute the accuracy of two bounding boxes based on a specified threshold. Parameters: - box1 (list of float): Bounding box [x_min, y_mi

Compute the accuracy of two bounding boxes based on a specified threshold. Parameters: - box1 (list of float): Bounding box [x_min, y_mi

Compute the accuracy of two bounding boxes based on a specified threshold. Parameters: - box1 (list of float): Bounding box [x_min, y_mi

Parses the 'predict' string and returns a dictionary. Missing or unparseable content is handled gracefully. Parameters: - predict_st

Downloads and returns the task dataset. Override this method to download the dataset from a custom API. :param data_dir: str

Asynchronously evaluate the given request and return a response Args: request: JudgeRequest containing the evaluation co

Extract JSON cognitive map from text response. Returns the JSON object if found, otherwise None. Args: text: Text containing a JSON o

Flatten a nested list.

Sort by value, while iterate over element if data is list

Draw the first `n` samples in order from the specified split. Used for tasks with "canonical" ordered fewshot examples, such as MMLU

Load baseline from local JSONL file.

Add a request and response to the cache

:returns: {str: [metric_score] -> float} A dictionary where keys are the names of submetrics and values are functions

Build the text query by combining the context, question and options. The <image_n> token is still there Return a dictionary with the query and gro

Calculate the IoU between predicted and target bounding boxes.

Compute the F1-score for KIE task between predicted and ground truth dictionaries. Args: preds (dict): The predicted key-value pairs.

Run basic evaluation with answer accuracy only. Args: jsonl_path: Path to JSONL file with model responses output_path

Convert Dictionary into Non-nested Dictionary Example: input(dict) { "menu": [