A subclass of [`transformers.HfArgumentParser`] designed for parsing command-line arguments with dataclass-backed configurations, while also

Trainer for Supervised Fine-Tuning (SFT) method. This class is a wrapper around the [`transformers.Trainer`] class and inherits all of its a

Module class that surfaces all objects but only performs associated imports when the objects are requested.

Internally used error class for signalling an optional dependency was not found.

r""" DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch. Args: pad_token_id (`int` defaults to

Trainer for Direct Preference Optimization (DPO) method. This class is a wrapper around the [`transformers.Trainer`] class and inherits all

r""" Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while keeping a bit of rand

r""" A [`~transformers.TrainerCallback`] that logs completions to Weights & Biases and/or Comet. Usage: ```python trainer = DPOTraine

r""" Configuration class for the [`SFTTrainer`]. This class includes only the parameters that are specific to SFT training. For a full list o

A class that keeps all conversation history.

Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the paper [DeepSeekMath: Pushing t

Samples a length

Output class for the diffusers pipeline to be finetuned with the DDPO trainer Args: images (`torch.Tensor`): The generat

r""" Configuration class for the [`DPOTrainer`]. This class includes only the parameters that are specific to DPO training. For a full list o

Data collator used for language modeling data. Inputs are dynamically padded to the maximum length of a batch. This collator expects each ex

r""" Configuration class for the [`GRPOTrainer`]. This class includes only the parameters that are specific to GRPO training. For a full list

r""" Configuration class for the [`IterativeSFTTrainer`]. This class includes only the parameters that are specific to Iterative SFT training

r""" Initialize KTOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFor

Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance.

r""" Configuration class for the [`PPOTrainer`]. This class includes only the parameters that are specific to PPO training. For a full list o

Sampler that repeats the indices of a dataset in a structured manner. Args: data_source (`Sized`): Dataset to sample fro

Calculates the running mean and standard deviation of a data stream. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b169

Callback to synchronize the model with a reference model.

A client class to interact with a vLLM server. This class provides methods to generate completions, initialize and manage weight update grou

r""" The ValueHead class implements a head for GPT2 that returns a scalar for each output token.

This model attempts to predict the aesthetic score of an image. The aesthetic score is a numerical approximation of how much a specific image

This model attempts to predict the aesthetic score of an image. The aesthetic score is a numerical approximation of how much a specific image

The AlignPropTrainer uses Deep Diffusion Policy Optimization to optimise diffusion models. Note, this trainer is heavily inspired by the work

r""" Initialize BCOTrainer from [BCO](https://huggingface.co/papers/2404.04656) paper. Args: model (`transformers.PreTrainedModel`):

r""" Initialize CPOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFor

Output class for the diffusers scheduler to be finetuned with the DDPO trainer Args: latents (`torch.Tensor`): Predicted

The DDPOTrainer uses Deep Diffusion Policy Optimization to optimise diffusion models. Note, this trainer is heavily inspired by the work here

Data collator for ChatML format datasets.

Data collator used for preference data. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length

Collate examples for supervised fine-tuning.

Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

r""" Geometric Mixture generation wrapper that samples from the logits of two model's geometric mixture. Args: model (`PreTrainedMode

Pairwise judge based on the Hugging Face API with chat completion. This judge is relevant for assessing the quality chat models, where the c

r""" Configuration class for merging two models using `mergekit`. This class provides a structured way to configure and generate merge config

r""" Initialize NashMDTrainer as a subclass of [`OnlineDPOConfig`]. Args: model (`transformers.PreTrainedModel`): The mod

A `saved_tensors_hook` manager used to disable any other `saved_tensors_hook` manager applied before. This relies on the behavior that only t

r""" Initialize ORPOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFo

Context manager under which activation tensors created in the forward pass will be offloaded. Enable the memory efficiency technique of acti

r""" Initialize OnlineDPOTrainer. Args: model (`Union[str, nn.Module, PreTrainedModel]`): Model to be trained. Can be eit

Judge based on the OpenAI API. This judge is relevant for assessing the quality chat models, where the completion is a response to a given p

Initialize PRMTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelForToke

r""" Class for tracking statistics per prompt. Mainly used to calculate advantage for the DPPO algorithm Args: buffer_size (`int`):

r""" Reward DataCollator class that pads the inputs to the maximum length of the batch. Args: tokenizer (`PreTrainedTokenizerBase`):

r""" Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0).

r""" Transformer block introduced in [Sana](https://huggingface.co/papers/2410.10629).

Custom `StoppingCriteria` which checks if all generations in the batch are completed.

The TextHistory class keeps track of the history of an interaction between the language model and the environment.

r""" Initialize XPOTrainer as a subclass of [`OnlineDPOConfig`]. Args: model (`transformers.PreTrainedModel`): The model

Implements the `forward-redirection`. Taken from Pytorch-lightning: https://github.com/Lightning-AI/pytorch-lightning/blob/02311d03fb98256024

A segment tree data structure that, when initialized as `_SegmentTree(maxval)`, efficiently finds the next larger value for a given input wit

r""" Configuration class for the [`AlignPropTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](ht