Trainer for Supervised Fine-Tuning (SFT) method. This class is a wrapper around the [`transformers.Trainer`] class and inherits all of its a

r""" Configuration class for the [`SFTTrainer`]. Only the parameters specific to SFT training are listed here. For details on other parameter

r""" Initialize DPOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFor

r""" Configuration class for the [`DPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://

A subclass of [`transformers.HfArgumentParser`] designed for parsing command-line arguments with dataclass-backed configurations, while also

Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the paper [DeepSeekMath: Pushing t

r""" Configuration class for the [`GRPOTrainer`]. Only the parameters specific to GRPO training are listed here. For details on other paramet

A DataProto is a data structure that aims to provide a standard protocol for data exchange between functions. It contains a batch (TensorDict

Data collator used for completion tasks. It ensures that all the tokens of the labels are set to an 'ignore_index' when they do not come from

r""" Initialize BCOTrainer from [BCO](https://huggingface.co/papers/2404.04656) paper. Args: model (`transformers.PreTrainedModel`):

r""" Configuration class for the [`BCOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://

r""" Initialize KTOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFor

The TextHistory class keeps track of the history of an interaction between the language model and the environment.

Iterable dataset that returns constant length chunks of tokens from stream of text files. The dataset also formats the text before tokenizati

r""" Configuration class for the [`KTOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://

r""" Initialize OnlineDPOTrainer. Args: model (`transformers.PreTrainedModel` or `torch.nn.Module`): The model to train,

Sampler that repeats the indices of a dataset in a structured manner. Args: data_source (`Sized`): Dataset to sample fro

Configuration for a dataset in a mixture.

r""" Configuration class for the [`OnlineDPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](ht

r""" Configuration class for the [`RewardTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https

r""" Initialize NashMDTrainer as a subclass of [`OnlineDPOConfig`]. Args: model (`transformers.PreTrainedModel`): The mod

r""" Initialize XPOTrainer as a subclass of [`OnlineDPOConfig`]. Args: model (`transformers.PreTrainedModel`): The model

Samples a length

r""" Configuration class for the [`NashMDTrainer`]. Subclass of [`OnlineDPOConfig`] we can use all its arguments and add the following:

Extended version of ScriptArguments with support for dataset mixtures. Args: dataset_mixture (`dict[str, Any]` or `None`, *optional*

The TextEnvironment enables interaction of a LLM with an environment using tools.

r""" Configuration class for the [`XPOTrainer`]. Subclass of [`OnlineDPOConfig`] we can use all its arguments and add the following: Par

Module class that surfaces all objects but only performs associated imports when the objects are requested.

Configuration for a mixture of datasets.

r""" A [`~transformers.TrainerCallback`] that logs completions to Weights & Biases and/or Comet. Usage: ```python trainer = DPOTraine

Internally used error class for signalling an optional dependency was not found.

A client class to interact with a vLLM server. This class provides methods to generate completions, initialize and manage weight update grou

r""" Initialize CPOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFor

r""" DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch. Args: pad_token_id (`int` defaults to

r""" Initialize ORPOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFo

Context manager under which activation tensors created in the forward pass will be offloaded. Enable the memory efficiency technique of acti

Initialize PRMTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelForToke

r""" Configuration class for the [`CPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://

Data collator used for language modeling data. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same

r""" Configuration class for the [`ORPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https:/

r""" Configuration class for the [`PPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://

r""" Configuration class for the [`PRMTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://

A sandbox environment that routes code execution requests to the E2B Router. This class is designed for batched execution of scripts, primari

ScriptResult is a Pydantic model that represents the result of a script execution. Attributes: text (Optional[str]): The output text

A [`~transformers.TrainerCallback`] that computes the win rate of a model based on a reference. It generates completions using prompts from

The AlignPropTrainer uses Deep Diffusion Policy Optimization to optimise diffusion models. Note, this trainer is heavily inspired by the work

The DDPOTrainer uses Deep Diffusion Policy Optimization to optimise diffusion models. Note, this trainer is heavily inspired by the work here

Sharding manager to support data resharding when using FSDP + Ulysses

r""" Configuration class for merging two models using `mergekit`. This class provides a structured way to configure and generate merge config

Configuration class for the models. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.

r""" Configuration class for the [`RLOOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https:/

Random pairwise judge, for testing purposes.

Client for the MorphCloud router service that mimics the API of MorphCloud's Sandbox. This class provides a simple interface to execute code

r""" Configuration class for the [`DDPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https:/

Output class for the diffusers pipeline to be finetuned with the DDPO trainer Args: images (`torch.Tensor`): The generat

Data collator for ChatML format datasets.

Data collator used for preference data. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length

DataProtoFuture aims to eliminate actual data fetching on driver. By doing so, the driver doesn't have to wait for data so that asynchronous

Used to count mfu during training loop Example: flops_counter = FlopsCounter(config) flops_achieved, flops_promised = flops_

Configuration class for [`GKDTrainer`]. Args: temperature (`float`, *optional*, defaults to `0.9`): Temperature for samp

r""" Geometric Mixture generation wrapper that samples from the logits of two model's geometric mixture. Args: model (`PreTrainedMode

Pairwise judge based on the Hugging Face API with chat completion. This judge is relevant for assessing the quality chat models, where the c

r""" Configuration class for the [`IterativeSFTTrainer`]. Only the parameters specific to iterative SFT training are listed here. For details

r""" A [`~transformers.TrainerCallback`] that merges the policy model (the model being trained) with another model based on a merge configuration.

A `saved_tensors_hook` manager used to disable any other `saved_tensors_hook` manager applied before. This relies on the behavior that only t

We assume the dataset contains a column that contains prompts and other information

Random binary judge, for testing purposes.

The reward manager.

Calculates the running mean and standard deviation of a data stream. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b169

This is an in-memory SFTDataset

ScriptResult is a Pydantic model that represents the result of a script execution. Attributes: execution (Optional[Execution]): An op

Represents the result of a subtask containing multiple test cases. Attributes: problem: Problem identifier subtask: Subtask

Callback to synchronize the model with a reference model.

Represents the result of a single test case execution. Attributes: test_name: Name of the test case score: Score achieved fo

r""" The ValueHead class implements a head for GPT2 that returns a scalar for each output token.

Adaptive KL controller described in the paper: https://arxiv.org/pdf/1909.08593.pdf

This model attempts to predict the aesthetic score of an image. The aesthetic score is a numerical approximation of how much a specific image

This model attempts to predict the aesthetic score of an image. The aesthetic score is a numerical approximation of how much a specific image

r""" Configuration class for the [`AlignPropTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](ht