r""" Add and load a reward modeling adapter. This method can only be used if the model is a `PeftModel` and if you have initialized th

TODO: optimize this. - We can use async ops - We can use only one allgather

Calculate the loss for a batch of an unpacked sample Args: latents (torch.Tensor): The latents sampled f

Check the consistency of the DataProto. Mainly for batch and non_tensor_batch We expose this function as a public one so that user can call th

Tensor extension to torch.clamp https://github.com/pytorch/pytorch/issues/2793#issuecomment-428784713

Compute the reward for a list of histories.

Concatenate the chosen and rejected inputs into a single tensor. Args: batch: A batch of data. Must contain the keys 'chosen_inpu

Converts the stats from a flattened dict to single scalar dicts

Convert applicable model parameters to low-precision (bf16 or fp16)

This function should return a class instance that delegates the calls to every cls in cls_dict

Creates and saves a model card for a TRL model. Args: path (`str`): The path to save the model card to. model_name (`

Creates and saves a model card for a TRL model. Args: path (`str`): The path to save the model card to. model_name (`

Setup the optimizer. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the

Divides an image into patches of a specified size. Args: image (PIL.Image.Image): The input image. patch_size (int): The siz

Compute the DPO loss for a batch of policy and reference model log probabilities. Args: policy_chosen_logps: Log probabilities of

Calculate entropy from logits.

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Gather stats from all processes. Useful in the context of distributed training. Args: stats (dict[str, Any]):

r""" A simple wrapper around the `generate` method of the wrapped model. Please refer to the [`generate`](https://huggingface.co/docs/