Generates and yields batches from the reordered array. Parameters: - n (int): The size of each batch. Defaults to 1.

Restores the original order of elements from the reordered list. Parameters: - newarr (List): The reordered array.

Return the mean squared error.

Aggregate evaluation results by specified metric and subtask categories.

Import a module or attribute optionally, returning fallback if unavailable. Args: module_name: Full module path (e.g., "decord" or "qwen_

Get a server instance by name. Args: server_name: Name of the server to instantiate. config: Optional configuration for the

Tokenizes table cells

Parse the JSON object, including nested JSON strings.

Check if the judge service is available

Compute all sub-category scores from raw results. Shared logic for all aggregation functions.

Close the HTTP client.

Check cache -> run model on misses -> store results -> return merged list. Maintains positional ordering so the caller can ``zip(resps, reque

Computes TEDS score between the prediction and the ground truth of a given sample

Compute the score for a given solution based on the data source. Args: data_source (str): The source dataset identifier which determines

Compute the evaluation score for keypoint estimation. Args: pred_keypoints (list or np.ndarray): List or array of predic

dumps the current config as a dictionary object, as a printable format. null fields will not be printed. Used for dumping results alon

Record token usage from a single API call. Parameters ---------- model_name: The model identifier (e.g. ``"gpt-4o"``, ``"claude-s

Aggregate results for a category. Sums scores and totals.

Aggregate results for a category. Results are already filtered by lmms-eval.

Log the metrics in a structured format. Args: total_elapsed_time (float): Sum of generation latencies in seconds. total_gen_

Initialize and start the job scheduler.

Read and uniformly sample video frames. Dispatches to the decode backend selected by *backend* (or the ``LMMS_VIDEO_DECODE_BACKEND`` env

Helper function to aggregate scores with filters. Args: results: List of result dicts language: Filter by language ("en" or

Aggregate accuracy for a specific detailed category.

Try to cast an object as a dict.

Evaluate binary correctness

Try to cast an object as a set.

Convert Dictionary into Non-nested Dictionary Example: input(dict) { "menu": [

Splits the audio into chunks of a given length. Args: audio_arrays: The audio array to split. chunk_lim: The length of each c

Converts a string representation of an iterable to a set.

Left-align text in a fixed-width column.

Compute all sub-category scores from raw results. Shared logic for all aggregation functions.

The main entry point to evaluate all tasks in self.data based on the HF dataset’s metric info.

Launch the LLM judge with the given arguments. :param args: Positional arguments for the launch. :param kwargs: Keyword argu

Resolve a model name to one concrete implementation class path.

Prompt the user, returning *default* on empty input.

Make an HTTP request and return JSON response.

Extract the answer tag from the prediction string. This function now handles both <answer> tags and \boxed{} format. Args: predict_s

Extract a multiple-choice answer letter from model output. Searches for choice letters in various common formats and returns the best candida