Get or create OpenAI client. Returns: OpenAI client instance Raises: ValueError: If OPENAI_API_KEY not found in

r"""Get paligemma token type ids for computing loss. It is slightly different with the original token type ids where the prompt part is 0. R

r"""Prepare the tokenized dataset to perform AutoGPTQ. Do not use tensor output for JSON serialization.

Get the run script content.

Hierarchically merge rules by splitting into chunks if needed. Args: rules_list: List of rules to merge llm_args: Diction

r"""Load a single dataset and aligns it to the standard format.

r"""Make nested list of images.

Process files, determine problem ID, and prepare configuration. Args: data: Dictionary containing file information

r"""Helper function to read JSON/JSONL files using fsspec.

Run multiple scripts concurrently with limited parallelism. Args: scripts: List of scripts to execute language: Progr

Function wrapping the `_run_async` function.

Execute a single script in a MorphCloud Sandbox. Args: script: The script to execute language: Programming language

Run tests and evaluate results. Args: instance: The MorphCloud instance data: Dictionary containing runtime

Upload all necessary files to the instance. Args: instance: The MorphCloud instance local_files: Dictionary

r"""Abort the processes recursively in a bottom-up way.

Fix common compilation errors for IOI problems.

r"""Add empty thought to assistant message.

r"""Set module as a leaf module to skip partitioning in deepspeed zero3.

r"""Align the dataset to a specific format. Aligned dataset: _prompt: [{"role": "user", "content": "..."}] * (2T - 1) _response: [{"role"

Rewrite documents using AutoGEO API. Args: num_examples: Number of examples to process data_dir: Directory containing data ch

r"""Cast projector output to half precision for fine-tuning quantized VLMs.

Build ngram lookup for documents.

Calculate citation quality metrics (precision and recall). Args: response: Generated answer text with citations documents: Li

Calculate keypoint coverage metrics (KPR and KPC). Args: keypoint_list: List of keypoint dictionaries with point_number and point_con

Calculate quality dimension scores for a response. Args: query: User query text response: Generated answer text

Call Anthropic (Claude) API with retry logic. Args: user_prompt: User prompt text model_name: Claude model name (default: "cl

Call HuggingFace model for inference. Args: user_prompt: User prompt text model_path: Path to HuggingFace model (can be local

Call OpenAI API with retry logic. Args: user_prompt: User prompt text model_name: OpenAI model name (default: "gpt-4o")

r"""Check the version of the required packages.

Clean rewritten text by removing introductory phrases. Args: text_to_clean: Text that may contain introductory phrases R

r"""Calculate the FLOPs of the device capability per second.

r"""Calculate the FLOPs of model per forward/backward pass.

r"""Compute loss for preference learning.

r"""Compute log probabilities of the reference model.

r"""Compute log probabilities of the reference model.

r"""Priority: PTQ-quantized (train/infer) > AutoGPTQ (export) > On-the-fly quantization (train/infer).

r"""Patch VLMs before loading them.

Creates and configures a FastAPI application instance. Args: args: An object containing configuration parameters for the application.

Creates and configures a FastAPI application instance for the MorphCloud router. Args: args: An object containing configuration

r"""Create deepspeed config in the current directory.

r"""Create reward model for PPO training.

Download JSON files from a GitHub repository folder. Args: repo_owner: GitHub repository owner username repo_name: GitHub

r"""Dump the layernorm parameters in the model. The model is already unwrapped (and gathered).

Evaluate a single example to extract rules. Args: example: Example dictionary with query, documents, and winner llm_args: Dic

Requests to the IOI package return the score as a float in the stdout, as well as optional feedback/errors in stderr. Returns a tuple of (sco

Execute scripts using E2B sandboxes. If e2b_router_url is provided, uses the RoutedSandbox for batch processing. Otherwise, uses dire

r"""Extract a list of tuples from the response message if using tools. Each tuple consists of function name and function arguments.

Filter records that meet quality thresholds. Args: data: Dictionary of records to filter rewrite_method_name: Name of the rew

r"""Find an available port on the local machine.