Close the HTTP client.

Generates and yields batches from the reordered array. Parameters: - n (int): The size of each batch. Defaults to 1.

Aggregate evaluation results by specified metric and subtask categories.

Return the mean squared error.

Restores the original order of elements from the reordered list. Parameters: - newarr (List): The reordered array.

Import a module or attribute optionally, returning fallback if unavailable. Args: module_name: Full module path (e.g., "decord" or "qwen_

Get a server instance by name. Args: server_name: Name of the server to instantiate. config: Optional configuration for the

Parse the JSON object, including nested JSON strings.

Compute the score for a given solution based on the data source. Args: data_source (str): The source dataset identifier which determines

Get the content of the correct answer as target.

Tokenizes table cells

Computes TEDS score between the prediction and the ground truth of a given sample

Compute the evaluation score for keypoint estimation. Args: pred_keypoints (list or np.ndarray): List or array of predic

dumps the current config as a dictionary object, as a printable format. null fields will not be printed. Used for dumping results alon

Read video using the PyAV library. Args: video_path (str): The path to the video file. num_frm (int, optional): The maximum

Helper function to aggregate scores with filters. Args: results: List of result dicts language: Filter by language ("en" or

Try to cast an object as a dict.

Log the metrics in a structured format. Args: e2e_latency (float): The end-to-end latency in seconds. total_tokens (int): Th

Evaluate binary correctness

Defines the operation to perform on a list of the `inst.resps` properties of `Instance` objects. Should return the list of (filtered)

Try to cast an object as a set.

Converts a string representation of an iterable to a set.

The main entry point to evaluate all tasks in self.data based on the HF dataset’s metric info.

Apply FlashVID to the model. Args: model (nn.Module): The model to apply FlashVID to. retention_ratio (float, optional): The rete

Initialize and start the job scheduler.

Make an HTTP request and return JSON response.

Launch the LLM judge with the given arguments. :param args: Positional arguments for the launch. :param kwargs: Keyword argu

Groups elements of an iterable based on a provided function. Parameters: - arr (Iterable): The iterable to be grouped.