Actor model base class. Args: model (nn.Module): Actor Model. lora_rank (int): LoRA rank. lora_train_bias (str): LoR

Dataset for SFT model Args: dataset: dataset for SFT model tokenizer: tokenizer for SFT model max_length: max length

A group of ray actors Functions start with 'async' should return list of object refs Args: num_nodes (int): Number of nodes for

Dataset for PPO model Args: dataset: dataset for PPO model tokenizer: tokenizer for PPO model max_length: max length

Dataset for reward model Args: dataset: dataset for reward model self.tokenizer: self.tokenizer for reward model sel

Experience is a batch of data. These data should have the the sequence length and number of actions. Left padding for sequences is applied.

GPT Language Model Loss

Samples is a batch of data. There can be 2 formats to store the samples, batched or packed. The batched format means padding is applied to the

Dataset for process reward model Args: dataset: dataset for reward model self.tokenizer: self.tokenizer for reward model

Unpaired preference dataset for algorithm, like KTO Args: dataset: raw dataset self.tokenizer: self.tokenizer for model

Adaptive KL controller described in the paper: https://arxiv.org/pdf/1909.08593.pdf

BufferItem is an item of experience data. Shapes of each tensor: sequences: (S) action_log_probs: (A) values: (1) returns: (1)

DPO Loss

Trainer for Direct Preference Optimization (DPO) training. Args: model (torch.nn.Module): The primary model to be trained. r

The strategy for training with Accelerator.

r"""Sampler that restricts data loading to a subset of the dataset. It is especially useful in conjunction with :class:`torch.nn.parallel.Dis

Fixed KL controller.

Language Model Knowledge Distillation Loss

Trainer for Knowledge Distillation. Args: model (torch.nn.Module): The model to be trained. strategy (Strategy): The trainin

KTO loss for uneven sampling

Trainer for KTO training. Args: model (torch.nn.Module): The primary model to be trained. ref_model (torch.nn.Module): The r

Pairwise Loss for Reward Model Details: https://arxiv.org/abs/2204.05862

Naive experience maker.

Naive replay buffer class. It stores experience. Args: sample_batch_size (int): Batch size when sampling. limit (int, optional):

Adds logging prefix to newlines to align multi-line messages.

Trainer for Proximal Policy Optimization (PPO) algorithm. Args: strategy (Strategy): The training strategy to use. actor (Ac

Process Reward Model Loss

Pairwise Loss for Reward Model

Policy Loss for PPO

Trainer for training a process reward model. Args: model (torch.nn.Module): The model to be trained. strategy (Strategy): Th

Trainer for training a reward model. Args: model (torch.nn.Module): The model to be trained. strategy (Strategy): The traini

Trainer for supervised fine-tuning (SFT). Args: model (torch.nn.Module): The model to be trained. strategy (Strategy): The t

Value Loss for PPO

KTO loss for even sampling