Compute BLEU score, using SacreBLEU.

Configuration for judge models

Typed wrapper returned by ``generate_until`` implementations. Models that wish to report per-sample token counts should return a list of ``Ge

Standard request format for judge evaluation

Per-request token usage counters. Fields are ``None`` when the backend cannot report them (e.g. cached responses, or backends that only expos

A class that keeps all conversation history.

TaskManager indexes all tasks from the default `lmms_eval/tasks/` and an optional directory if provided.

CARDINAL类

Processes an answer similar to Eval AI copied from https://github.com/facebookresearch/mmf/blob/c46b3b3391275b4181567db80943473a8

Response for get_run_result tool.

Standard response format from judge evaluation

中文数字/数位字符 每个字符除繁简体外还有一个额外的大写字符 e.g. '陆' 和 '陸'

Tree Edit Distance basead Similarity

Comprehensive cognitive map evaluator. This class provides a complete evaluation workflow for cognitive mapping tasks, including answer accur

Response when an evaluation run is submitted.

FRACTION类

Thread-safe job scheduler for managing evaluation jobs. The scheduler maintains a queue of jobs and processes them sequentially using a

Declarative model registration payload for model loading. A single manifest can expose a chat implementation, a simple implementation, or bot

Qwen-Image-Edit 20B Model (Unified: Understanding + Editing + Visual CoT) https://huggingface.co/Qwen/Qwen-Image-Edit Modes: 1. Unde

DIGIT类

Unified OpenAI-compatible backend for VIEScore. Works with: - OpenAI API (GPT-4o, GPT-4V, etc.) - vLLM servers with OpenAI-c

Parse the response.

Arguments for launching the evaluation server.

TELEPHONE类

A GPT-4o judge for assessing if two ASCII art images depict the same thing.

Azure OpenAI compatible endpoint client backed by Azure CLI bearer tokens.

Azure OpenAI compatible endpoint client backed by Azure CLI bearer tokens.

Azure OpenAI compatible endpoint client backed by Azure CLI bearer tokens.

Azure OpenAI compatible endpoint client backed by Azure CLI bearer tokens.

Azure TRAPI client for LLM Judge.

Azure TRAPI client for LLM Judge.

Azure TRAPI client for LLM Judge.

Azure TRAPI client for LLM Judge.

Basic task evaluator for answer accuracy only. This evaluator focuses only on extracting answers and calculating accuracy, without any cognit

Sample from OpenAI's chat completion API

Sample from OpenAI's chat completion API

Sample from OpenAI's chat completion API

中文数字字符

DATE类

Convert any spelled-out numbers into arabic numbers, while handling: - remove any commas - keep the suffixes such as: `1960s`, `274t

Convert any spelled-out numbers into arabic numbers, while handling: - remove any commas - keep the suffixes such as: `1960s`, `274t

Applies British-American spelling mappings as listed in [1]. [1] https://www.tysto.com/uk-us-spelling-list.html

Applies British-American spelling mappings as listed in [1]. [1] https://www.tysto.com/uk-us-spelling-list.html

Response for get_run_status tool.

Request model for evaluation endpoint.

Keeps track and saves relevant information of the evaluation process. Compiles the data from trackers and writes it to files, which can be pu

An example loaded from vibe-eval, stored as jsonl in the repo.

FilterEnsemble creates a pipeline applying multiple filters. Its intended usage is to stack multiple post-processing steps in order. `tas

Tracker for the evaluation parameters. Attributes: model_source (str): Source of the model (e.g. Hugging Face, GGUF, etc.) m

Health check response.