A class that keeps all conversation history.

r""" Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while keeping a bit of rand

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between

Fake file-like stream object that redirects writes to a logger instance.

r""" The ValueHead class implements a head for GPT2 that returns a scalar for each output token.

Adaptive KL controller described in the paper: https://arxiv.org/pdf/1909.08593.pdf

Construct the embeddings from word and position embeddings.

Iterable dataset that returns constant length chunks of tokens from stream of text files. The dataset also formats the text before tokenizati

Output class for the diffusers pipeline to be finetuned with the DDPO trainer Args: images (`torch.Tensor`): The generat

Output class for the diffusers scheduler to be finetuned with the DDPO trainer Args: latents (`torch.Tensor`): Predicted

Collate examples for DPO fine-tuning.

r""" DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch. Args: pad_token_id (`int` defaults to 0

Dataset for DPODataset fine-tuning.

Collate examples for supervised fine-tuning.

Fixed KL controller.

r""" Class for tracking statistics per prompt. Mainly used to calculate advantage for the DPPO algorithm Args: buffer_size (`int`):

r""" Reward DataCollator class that pads the inputs to the maximum length of the batch. Args: tokenizer (`PreTrainedTokenizerBase`):

Custom `StoppingCriteria` which checks if all generations in the batch are completed.

The TextHistory class keeps track of the history of an interaction between the language model and the environment.

r""" An autoregressive model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper`

r""" A seq2seq model with a value head in addition to the language model head. This class inherits from `~trl.PreTrainedModelWrapper` and wrap

r""" Base class for all trainers - this base class implements the basic functions that we need for a trainer. The trainer needs to have t

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Dataclass for special tokens used in ChatML, including system, user, assistant, bos, eos, and pad tokens.

Configuration class for DDPOTrainer

Main class for the diffusers pipeline to be finetuned with the DDPO trainer

The DDPOTrainer uses Deep Diffusion Policy Optimization to optimise diffusion models. Note, this trainer is heavily inspired by the work here

r""" Initialize DPOTrainer. Args: model (`transformers.PreTrainedModel`): The model to train, preferably an `AutoModelFor

Data collator used for completion tasks. It ensures that all the tokens of the labels are set to an 'ignore_index' when they do not come from

The IterativeSFTTrainer can be used to finetune models with methods that requires some steps between optimization. Attributes: **mod

Samples a length

Arguments which define the model and tokenizer to load.

Configuration class for PPOTrainer

The PPOTrainer uses Proximal Policy Optimization to optimise language models. Note, this trainer is heavily inspired by the original OpenAI l

r""" A wrapper class around a (`transformers.PreTrainedModel`) to be compatible with the (`~transformers.PreTrained`) class in order to keep s