Initialize iteration.

Return the number of batches.

Set attribute as dictionary item.

Add filter decision based on parsed response. Args: result: Result to add decision to. qa_idx: QA pair index

Add rewrite decision based on parsed response. Args: result: Result to add decision to. qa_idx: QA pair inde

Build HTTP headers for OpenAI-compatible API request. Args: api_key: API key for authorization. If empty, no Authorization header

Build requests with only QA pairs needing rewrite.

Decode PNG bytes to PIL Image.

Get resized image, caching by (path, size) to avoid redundant work.

Log text message to wandb.

Open a temporary env with readahead=True for batch reading.

Worker function executed in each forked subprocess. Reads its message slice from the inherited ``_SHARED_MESSAGES`` global (zero-copy via COW

Create a testable API instance.

Create an OpenAIAPI instance with test defaults.

Create an OpenAIAPI instance with return_dict=True.

Create an OpenAIAPI instance using api_base + port pattern.

Assert two JSON-serializable objects are equal.

Base kwargs for creating SelectiveMLLMRewriter.

Create builder with custom settings.

Create builder with default settings.

Not used - item-level filter.

Check for repetitive patterns in a QA pair.

Not used - item-level filter.

Not used - item-level filter.

Not used - this filter operates at item level.

Create ProcessedChecker instance.

Provide a clean registry for testing.

Close all manager instances.

Close a specific shard to release memory. Args: shard_id: Shard ID to close.

Create collector for testing.

Create collector for testing.

Create collector for testing.

Create collector with recorded data.

Create collector with recorded data.

Get the conversations list.

Helper to create mock file content.

Access the underlying data dict.

Create a DataItem instance for testing.

Create DataItem for testing.

Create a multi-turn DataItem for testing.

Create a DataItem with image for testing.

Return supported file extensions. Returns: list: ['.json']

Return supported file extensions. Returns: list: ['.jsonl']

Create filter checking all fields.

Create filter checking all fields.

Create filter with custom settings.

Create filter with custom settings.

Create filter with default settings.

Create filter with default settings.

Create filter with default settings.

Create filter with default settings.

Create filter with default settings.

Create filter with default settings.

Create filter with default settings.

Create MLLMFilter instance.

Create MLLMFilter instance for testing.

Create standard config from dictionary. Args: data: Raw configuration dictionary with 'datasets' key. Returns:

Generate responses using fork-based multiprocessing. Args: messages: List of input messages. **kwargs: Additional gen

Sync version for single request (BaseAPI compatibility). Delegates to a temporary OpenAIAPI instance. Args: inputs: Prep

Get or create ShardedLMDBManager singleton. Args: cache_dir: Cache directory path. num_shards: Number of shards.

Whether any QA pair is marked for filtering.

Whether this item has an image.

Whether any QA pair has rewrite changes.

Get the image (PIL Image or None).

Check if this dataloader has been fully processed.

Whether this item is marked as rejected.

Create DataItem for testing.

Create multi-turn DataItem for testing.

Create DataItem for testing.

Create DataItem for testing.

Create DataItem for testing.

Create JsonFormat instance.

Create JsonlFormat instance.

Load data from JSON file. Args: file_path: Path to JSON file. Returns: list: Data as list of dicts (single o

Load data from file. Args: file_path: Path to data file. add_source_file: Whether to add 'file_path' field to each it

Load data from JSONL file. Args: file_path: Path to JSONL file. Returns: list: List of data dictionaries.

Load image from disk and encode to bytes (no LMDB access).

Read image from disk and resize. Args: img_path: Image path (must be an absolute path). resize_size: Max edge size for resizing.

Create a loader with mocked FormatRegistry for method testing.

Create loader with checkpoint manager.