Extract PIL Image from various formats.

Load baseline from local JSONL file.

"Recall" here means: does the model mention the correct label somewhere in its output. This matches the original codebase evaluation style (s

Save audio array to a temporary WAV file and return the path.

Convert a stats dict with acc/caa num/den to output format.

:returns: {str: [metric_score] -> float} A dictionary where keys are the names of submetrics and values are functions

Build the prompt text based on question category.

Build the text query by combining the context, question and options. The <image_n> token is still there Return a dictionary with the query and gro

Weighted average across categories by sample count.

Calculate the IoU between predicted and target bounding boxes.

Compute the F1-score for KIE task between predicted and ground truth dictionaries. Args: preds (dict): The predicted key-value pairs.

Compute the random expected accuracy (1/num_options average). Returns: tuple of (avg_random_acc, total_counted, missing_count)

Compute aggregated statistics from a samples JSONL file. Returns: dict with keys: metric_stats, category_stats, overall

Run basic evaluation with answer accuracy only. Args: jsonl_path: Path to JSONL file with model responses output_path

Check if the prediction string follows the expected format. Now handles both <think><answer> format and \boxed{} format.

Helper function to get value from document with multiple possible column names

Given the results and sample results filenames, extracts and returns the datetime.

Gets the git commit hash of your current repo (if it exists). Source: https://github.com/EleutherAI/gpt-neox/blob/b608043be541602170bfcfb8ec9

Return canonical manifest for a model id or alias.

Calculate various scores based on the given results. Args: scores (dict or list): A dictionary or list containing results where each

Calculate various scores based on the given results. Args: scores (dict or list): A dictionary or list containing results where each

Whether the task has a test set

Whether the task has a validation set

Normalize answer for comparison. Args: answer: Raw answer string Returns: Normalized answer

Normalize mathematical notation for comparison (e.g., n² -> n^2, n³ -> n^3)

Parse the prediction from the generated response. Return the predicted index e.g., A, B, C, D.

Calculate minimum sample size for paired t-test power analysis. For paired samples, the effective variance is: Var(X - Y) = Var(X) +

Prepare messages in the format expected by the API

Resample audio to 16kHz and convert to mono if needed.

Set or update the configuration for a given key.

Parse the prediction string to extract the answer. Args: predict_str (str): The prediction string to be parsed. Returns: str

Converts a string representation of an iterable to an iterable.

Converts a string representation of an iterable to a set.

pass a string `regex` to run `re.compile(r"regex")` on. `fallback` defines the output returned if no matches for the regex are locate