r"""Add the user input to chatbot. Inputs: infer.chatbot, infer.messages, infer.role, infer.query, infer.escape_html Output: infer.ch

Log info message.

r"""Register a chat template. To add the following chat template: ``` <s><user>user prompt here <model>model response here</s> <u

Log warning message.

r"""Get plugin for multimodal inputs.

r"""Optionally check the package version.

r"""Get element by id. Example: top.lang, train.dataset

Log error message.

r"""Forms a list of slots according to the inputs to encode.

r"""Validate if this model accepts the input modalities.

r"""Get chat template and fixes the tokenizer.

r"""Load pretrained tokenizer and optionally loads processor. Note: including inplace operation of model_args.

Get full path for a dataset configuration key. Args: dataset_name: Name of the dataset path_key: Configuration key (e.g., 'ru

Computes N-gram repetition penalty as described in Appendix C.2 of https://huggingface.co/papers/2502.03373. Reference implementation from: h

r"""Check if the environment variable is enabled.

r"""Get the path to saved model checkpoints.

r"""Process visual inputs. Returns: (llava and paligemma) pixel_values: tensor with shape (B, C, H, W) Returns: (qwen2-v

r"""Register a new composite model. Args: model_type: model type projector_key: multi_modal_projector vision_model_keys:

r"""Load pretrained model.

r"""Validate if the number of images, videos and audios match the number of placeholders in messages.

r"""Pre-process input messages before tokenization for VLMs.

r"""Return a single pair of token ids representing prompt and response respectively.

r"""Get the train dataset and optionally gets the evaluation dataset.

r"""Pre-process token ids after tokenization for VLMs.

Aggregate evaluation results from multiple JSON files. Args: json_folder: Directory containing JSON files dic_name: Dictionar

Log debug message.

r"""Save model checkpoint. Subclass and override to inject custom behavior.

r"""Regularize images to avoid error. Including reading and pre-processing.

Reward function that evaluates code snippets using a code execution provider. Assumes the dataset contains a `verification_info` column with test

r"""Convert slots to jinja template.

Close logger and log final summary.

Load a dataset or a mixture of datasets based on the configuration. Args: args (ScriptArguments): Script arguments containing dataset con

r"""Add elements to manager.

Format reward function specifically for code responses. Args: language: Programming language supported by E2B https://e2b.dev/docs/code-i

r"""Get the number of available devices.

r"""Load user config if exists.

Executes a batch of scripts in the sandbox environment. Args: scripts (list[str]): A list of code scripts to execute.

r"""Collect the device memory.

r"""Check template. Args: model_id: the model id on hugging face hub. template_name: the template name. prompt_str: the s

r"""Optionally concatenate media path to media dir when loading from local disk.

r"""Regularizes videos to avoid error. Including reading, resizing and converting.

r"""Get the current available device.

r"""Return stop token ids.

r"""Compute the real sequence length after truncation by the cutoff_len.

r"""Cast a torch tensor or a numpy array to a numpy array.

r"""Plot loss curves and saves the image.

Registers a LightEval task configuration. - Core tasks can be added from this table: https://github.com/huggingface/lighteval/blob/main/src/light

Reward function that checks if we produce the desired number of think and answer tags associated with `format_reward()`. Adapted from: https://gi

r"""Check token ids and texts. encode(text) == token_ids decode(token_ids) == text

r"""Regularizes audios to avoid error. Including reading and resampling.

r"""Build batched multimodal inputs for VLMs. Arguments: images: a list of image inputs, shape (num_images,) videos:

Compute length-based rewards to discourage overthinking and promote token efficiency. Taken from the Kimi 1.5 tech report: https://huggingface.co

Reconstruct dataset split into a single JSON file. Args: dataset_split: Dataset split from Hugging Face datasets output_f

r"""Get the response token-by-token of the chat model.

r"""Convert slots to ollama template.

r"""Escape HTML characters.

r"""Return a list of names of parameters with weight decay. (weights in non-layernorm layers).

r"""Escape single quotes in content.

Load extracted rules from file if available. Args: dataset: Name of the dataset engine_llm: External generative engine name

Reward function that checks if the completion is the same as the ground truth.

r"""Calculate the number of pixels from the expression.

Call LLM and parse response as Pydantic model. Args: prompt: Prompt text for LLM pydantic_model: Pydantic model class for val