A collection of Example instances, indexed by prompt.

Writes JSON arrays to a file in a streaming fashion.

Class for an example in a preference or SFT dataset. If you want each prompt to be uniquely associated with an Example instance, save it in a dic

A wrapper around the reference model that precomputes the logprobs and saves them in a local dict, after which the reference model and accele

Dataloader for supervised fine-tuning.

r""" The ValueHead class implements a head for autoregressive that returns a scalar for each output token. The weights of the value head need

A wrapper around AutoModelForSequenceClassification that ensures binary classification.

r""" An autoregressive model with a value head in addition to the language model head. Class attributes: - **transformers_parent_clas

Dataloader for token-conditioned SFT, in the style of Korbak et al.'s (2023) "Pretraining Models with Human Feedback." For training, eac

The base data loader class, similar to the one from the DPO repo. Subclass this and overwrite the __iter__ method as needed, since the batch

A variant of UnpairedPreferenceDataLoader that groups examples with the same prompt in the same batch (but distributed across different proce

Dataloader for training on only one output per input. This throws out at least half the data (more than half if there are multiple pairs per

Dataloader for losses that do require pairwise preferences (e.g., DPO).

r""" A wrapper class around a (`transformers.PreTrainedModel`) to be compatible with the (`~transformers.PreTrained`) class in order to keep s

Dataloader for losses that do not require pairwise preferences (e.g., KTO). This assumes that if an example has no pairs, then it is natural