Prepare prompt in OpenAI chat format. Args: image_links: Single image or list of PIL Images / image paths te

Print formatted results.

content_data: - 'image': PIL.Image - 'video': List[PIL.Image] or np.ndarray (T, H, W, C) data_type: 'text', 'image' or 'video

Take a single document and the LMM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of

Proportionally sample k total indices across multiple lists. Returns list of index lists for each original list.

Register a manifest and all aliases. When `overwrite=False`, conflicting class paths for an existing model id raise `ValueError`.

Resample audio to 16kHz and convert to mono if needed.

Run a specific tool with the given arguments. :param tool_name: Name of the tool to run. :param tool_args: Arguments for the

Given the model name, returns a sanitized version of it.

Calculate the relevance score.

Set random seeds for reproducibility

Add images to the last user message

Add edges to the graph based on relative positions between nodes. Args: G: The directed graph to add edges to

Convert the text answer to a letter (A/B/C/D) based on its position in choices. Returns "" if the answer is not found in choices.

Answer a question using original image + multiple generated images.

Apply chat template with optional enable_thinking support.

Build the lmms_eval command string.

Build model.generate() kwargs from user gen_kwargs merged with defaults.

Given a string for the metric (e.g. 'exact_str_match'), return the actual MetricType or a specialized metric class.

Build list of (label, text) tuples, skipping empty trailing options.

Build the top-level parser with subcommands.

Build video processing kwargs based on model configuration.

Clamp score to valid 1-5 range.

Create result dict with all metric keys for detailed breakdown. Returns metrics for: - Overall scores (all samples) - English fullse

Decode audio object to standard dict format.

Decode audio object to standard dict format.

json.loads returns list for stored tuples — downstream tuple unpacking still works.

Compute score for a single field using the given metric.

Hash leading text arguments to prevent cache-key collisions. Some flows can issue multiple deterministic requests that share the same ``(task