Set prompt ID automatically.

Tensor extenstion to torch.clamp https://github.com/pytorch/pytorch/issues/2793#issuecomment-428784713

Delete all items inside the dict.

Disable dropout in a model.

r""" Applies a forward pass to the wrapped model and returns the logits of the value head. Args: input_ids (`torch.LongTe

Overrides from_pretrained to ensure num_labels=2. Args: pretrained_model_name_or_path: Either: -

Load the AlpacaEval dataset (for evaluation only) and convert it into a Dataset. Args: - split: must be 'test'; otherwise error will

Compute the loss and other metrics for the given batch of inputs. Args: batch: dictionary of inputs for the batch (should

Compute the loss and other metrics for the given batch of inputs.

Compute the loss and other metrics for the given batch of inputs.

Compute the loss and other metrics for the given batch of inputs.

Compute the loss and other metrics for the given batch of inputs.

Compute metrics for a batch of examples. Args: batch: The input batch mode: Either 'train' or 'eval'

Load feedback data created by label.py and convert it into a Dataset. Supports both binary and pairwise feedback formats. Args:

Return a flat list of examples given a list of prompts that index self.full_data. Prepend the examples with the appropriate control t

Return a flat list of examples given a list of prompts that index self.full_data.

Return a flat list of examples given a list of prompts that index self.full_data. Assumes that there are a list of scores.

Return a flat list of examples given a list of prompts that index self.full_data. Only use one preference pair per input.

Return a flat list of examples given a list of prompts.

Get the number of training steps.

Load the Open Assistant dataset from Huggingface and convert it into to a Dataset. For this dataset, the SFT text is the preferred response.

Return the subset of data to be processed in the current process. Examples sharing the same prompt ID will appear consecutively, in

Get reward scores from reward model. Args: batch: Dictionary containing batch data Returns:

Load the Ultrafeedback (binarized) dataset from Huggingface and convert it into to a Dataset. For this dataset, the SFT text is the preferred

Load the PKU-Alignment PKU-SafeRLHF dataset from Huggingface and convert it into to a Dataset. For this dataset, the SFT text is the preferre

Load samples generated by train.sample and convert it into a Dataset.

Load the Stanford Human Preferences dataset from Huggingface and convert it into to a Dataset. We filter preference pairs to only keep pairs

Load the Ultrafeedback (binarized) dataset from Huggingface and convert it into to a Dataset. For this dataset, the SFT text is the preferred

Load the AllenAI WildBench dataset from Huggingface and convert it into to a Dataset. For this dataset, the SFT text is the preferred respons

Args: batch: batch of data, mapping keys to Tensors policy_chosen_logps: Log probabilities of the policy model for th

Compute the CDPO loss for a batch of policy and reference model token-level log probabilities.

Compute the IPO loss for a batch of policy and reference model token-level log probabilities.

Compute the SimPO loss for a batch of policy and reference model token-level log probabilities.

Compute the SLIC loss as defined by Zhao et al. in https://arxiv.org/pdf/2305.10425.pdf Calibration loss defined as: L(x, y) := m

r""" We add the state dictionary of the value head to the state dictionary of the wrapped model by prepending the key with `v_head.`.

Prepare the Accelerator.

Print the amount of GPU memory currently allocated for each GPU.

Save a model and its configuration file to a directory. Ensures the config maintains num_labels=2 and padding token configuration whe

r""" Returns the state dictionary of the model. We add the state dictionary of the value head to the state dictionary of the wrapped m

Begin either SFT or HALO training, with periodic evaluation. This is subclassed when implementing PPO.

Args: batch: the original offline batch completions: the online generations from the updated reference model (after s