Wrapper around a raw data dict with typed QA pair access. Example:: item = DataItem(raw_data, idx=0) for qa in item.qa_pairs:

Operator execution result. Responsibilities: 1. Store decision lists 2. Apply decisions to DataItem 3. Manage records Can c

Tracks per-file processing progress for resumable execution. Persists a JSON file mapping each data file to its last processed index, allowin

Dataset loader with LMDB image caching and checkpoint resume. Supports automatic format detection, parallel image loading, adaptive batch siz

Execute a sequence of operators on data items. Runs each operator in order, splitting items into kept and rejected after each step. Rejected

Configuration class based on OrderedDict. Supports attribute-style access, hierarchical configuration inheritance, and automatic merging of b

Standard configuration for a single dataset. Attributes: file_path: Path to the data file. source: Data source name/identifier.

Composes sub-pipelines and executes them by priority (lower = earlier). Operators are resolved from the unified ``OPERATORS`` registry, so su

A registry for managing and instantiating registered modules. Example: FILTERS = Registry() @FILTERS.register_module() c

Configurable logger with file, console, and WandB backends. Example config:: logger = dict( type='Logger', level

Mock operator for testing.

Dataset saver that groups output by source and tracks statistics. Output directory structure:: output_dir/ ├── source1/

Collects and aggregates pipeline statistics. Tracks per-operator and per-dataset metrics, and exports to CSV/JSON. Example: >>> coll

Async OpenAI-compatible API wrapper. Uses a shared :class:`aiohttp.ClientSession` with connection pooling and ``asyncio.Semaphore`` for concu

Concrete implementation of Operator for testing.

Filter data items using MLLM quality assessment. Expected response format: ``{"q0": true/false, "q0_reason": "..."}`` where ``true`` means re

Concrete implementation of Filter for testing.

Filter decision for a QA pair or entire item. Attributes: qa_idx: QA pair index (0-based), or -1 for a global decision. rejected:

Statistics for a single operator execution.

MLLM request builder. Responsibilities: 1. Store request_builder configuration 2. Build requests (build_request, build_requests)

Rewrite decision for a single QA pair. Attributes: qa_idx: QA pair index (0-based). new_question: Rewritten question text (None =

Standardized complete configuration containing multiple datasets. Attributes: datasets: List of DatasetConfig objects. extra: Add

Immutable view of a single QA pair. Attributes: idx: QA pair index (0-based). question: Question text. answer: Answer tex

Colored formatter for console output.

Concrete implementation of Rewriter for testing.

Filter data items based on text length anomalies. Detects and filters QA pairs with abnormally short or long text. Supports per-QA-pair

Rewrite data using MLLM. Supports both structured (JSON dict) and plain text responses. Set ``key_templates=None`` in the request builder for

Overall pipeline statistics.

Mock filter operator that rejects some items.

MLLM rewriter that only processes items matching a condition. Example:: rewriter = SelectiveMLLMRewriter( model=model,

Saves processed data grouped by source file. Groups items by their ``_source_file``, removes temporary fields, and delegates to :class:`Forma

Concrete implementation of BaseAPI for testing.

Filter data items based on conversation length (number of QA pairs). Example: filter = ConvLengthFilter(min_length=1, max_length=10)

Checker for detecting already processed datasets. Checks if processed output files exist in the output directory. Used for skipping already c

Generates training dataset YAML configs from processed output. Attributes: output_dir: Output directory path. yaml_name: YAML fil

Statistics for a dataset.

Filter data items based on image aspect ratio. Removes items containing images with extreme aspect ratios.

Filter data items based on image dimensions. Rejects images that are too small or have extreme aspect ratios. Example: filter =

Remove <think>...</think> tags from answers. These tags often contain model reasoning that shouldn't be in the final output. Exampl

Sharded LMDB manager with singleton pattern per cache directory. Distributes images across LMDB shards using hash-based partitioning. Loads s

Filter data items based on image file extensions. Removes items containing images with excluded file extensions.

Mock rewriter operator.

Base class for all operators. Subclasses implement :meth:`process` for single-item logic, or override :meth:`process_batch` for batch-level o

Remove [ANSWER 0] prefix from responses. Some data may have "[ANSWER 0]" prefixes in responses that should be removed for clean output.

Filter data items that have image/video tags in responses. Responses should not contain <image> or <video> tags as these indicate potent

Split multi-turn conversations into single-turn conversations. Each QA pair becomes a separate data item, preserving all metadata and ad

Filter data items with repetitive text patterns. Detects and filters QA pairs containing consecutive repetitive patterns. Supports per-Q

Add empty <think> tags to responses that don't have them. This ensures consistent format for responses that should have think tags but a

Base API class providing common functionality for all API models. This class handles: - Message preprocessing and formatting - Image enco

Abstract base for data format handlers. Subclasses implement :meth:`extensions`, :meth:`load`, and :meth:`save`, then register via ``@FormatR

JSON data format handler. Supports loading and saving data in JSON format. Single objects are wrapped in a list for consistency.

JSONL (JSON Lines) data format handler. Each line contains one JSON object. Invalid lines are skipped with a warning.

Normalize image tag positions in conversations. Moves all <image> tags to the beginning of the first human message, ensuring consistent

Normalize instruction patterns in multi-turn conversations. If the first turn contains instruction patterns (like "Answer concisely"), r

Normalize hint/instruction prompt formatting. Ensures hints like ``"Answer concisely"`` are properly separated with newlines from the main qu

Normalize <think> tag format in answers. Ensures consistent format: - Lowercase tags - Newline after opening tag - Newline befor

Remove "Reasoning:" prefix from responses. Some data may have "Reasoning:" prefixes in responses that should be removed for clean output

Main pipeline runner that orchestrates data loading, processing, and saving. Initializes all components from the global config: - Logger

Operator that returns fewer results than input items.

Configuration loader for loading and saving YAML configs. Provides static methods for loading YAML configurations and converting them to stan

Base class for filter operators. Subclasses implement check(item, qa_idx) for per-QA filtering. Example: class MyFilter(Filter)

Registry for data format handlers with automatic schema conversion. Detects file format by extension, converts between ``conversations`` and

Log levels.

Base class for MLLM-powered operators. Builds requests via :class:`RequestBuilder`, calls ``model.generate()``, and delegates response parsin

Multiprocessing OpenAI-compatible API using fork + COW shared memory. Distributes requests across multiple worker processes, each running an

Base class for rewrite operators. Subclasses implement rewrite(item, qa_idx) for per-QA rewriting. Example: class MyRewriter(Re

Automatic schema detection and conversion. Usage: # Load and normalize data = json.load(f) normalized = SchemaConverter.n

Supported schema types.

Tests for AddNoThinkRewriter.

Tests for async request methods using mocked aiohttp.

Tests for abstract method enforcement.

Tests for allowed_types attribute.

Tests for check_content method.

Edge case tests for BaseAPI.

Tests for encode_image_directly method.

Tests for generate method.

Tests for BaseAPI initialization.

Tests for pre_process method.

Tests for prepare_inputs method.

Tests for preproc_content method.

Tests for process_single_message method.

Tests for shutdown method.

Tests for BaseFormat abstract class.

Tests for build_dataloaders function.

Edge case tests for build_dataloaders.

Tests for build_from_cfg function.