Take a single document and the LMM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of

Resample audio to 16kHz and convert to mono if needed.

Run a specific tool with the given arguments. :param tool_name: Name of the tool to run. :param tool_args: Arguments for the

Given the model name, returns a sanitized version of it.

Calculate the relevance score.

Stop the evaluation server and clean up.

Add images to the last user message

Add edges to the graph based on relative positions between nodes. Args: G: The directed graph to add edges to

Build the lmms_eval command string.

Given a string for the metric (e.g. 'exact_str_match'), return the actual MetricType or a specialized metric class.

Clamp score to valid 1-5 range.

Create result dict with all metric keys for detailed breakdown. Returns metrics for: - Overall scores (all samples) - English fullse

Compute score for a single field using the given metric.

Generate a dataset from evaluation data. Args: data (List[Dict[str, Any]]): The data to generate a dataset for. confi

Generate a dataset from evaluation data. Args: data (List[Dict[str, Any]]): The data to generate a dataset for. confi

Infer category from document. Note: This fallback method uses question_index thresholds and may not work correctly with HuggingFace datasets

Apply ground truth corrections for verified errors in the dataset. Args: index: The sample index in the dataset original_ans

Extracts the model name from the model arguments.

Check if question is a yes/no question.

Load baseline from HuggingFace Hub.

Load baseline from registry by model and task name.

Convert answer format from '(a)' or '(b)' to 'A' or 'B'.

Normalize mathematical answer for comparison.

Normalize text for comparison.

Args: raw_choices: A Python list literal or list/tuple from the CSV, e.g., ["A. foo", "B. bar"]. Returns: normalized_choices

Process job status and handle terminal states. Returns: Tuple of (should_continue, result_or_none). If should_continue is Fa

Reorders the elements in the array based on the sorting function. Parameters: - arr (Union[List, Tuple[Tuple[int, Any], ...]

Safe-write a JSON file via temp file + replace. Since the results file is long, this avoid breaking the file in case of a crash.

Helper utility. Sorts the task dict at the current level of the hierarchy based on alphabetized task name. Required so that we end up

Return an approximation of the original string.

Convert ASCII text into an image.

for non multiprocessing for CHAIR

Build prompt for binary evaluation