Separate the kwargs from the arguments that we support inside `supported_args` and the ones that we don't.

Build the memory buffer given weight_buffer_meta Args: weight_buffer_meta: contains mapping from name to a dictionary containing shape an

for cuda memory alignment, make sure alignment by 128-bits

Calculate the loss for a batch of an unpacked sample Args: latents (torch.Tensor): The latents sampled f

Computes log probabilities of the reference model for a single padded batch of a DPO specific dataset.

Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens.

r"""Works like shutil.copy() for file, and shutil.copytree for dir, and supports hdfs. Copy data and mode bits ("cp src dst"). Return the file's

Compute the DPO loss for a batch of policy and reference model log probabilities. Args: chosen_logps (`torch.FloatTensor

r""" Extracts the shared prompt from a preference data example, where the prompt is implicit within both the chosen and rejected completions.

Reward function that checks if the reasoning process is enclosed within <think> and </think> tags, while the final answer is enclosed within <answer>

Generate sequences

Get the tokenizer for the model.

Log `logs` on the various objects watching training, including stored metrics. Args: logs (`dict[str, float]`):

Whiten values with masked values.

Context manager for handling null reference model (that is, peft adapter manipulation).

Context manager for handling null reference model (that is, peft adapter manipulation).

r""" Pack sequences in a dataset into chunks of size `seq_length`. Args: dataset (`Dataset` or `DatasetDict`): Dataset to

Print out a sample of model completions to the console with multiple reward metrics. This function creates a nicely formatted table showing

Predict the sample at the previous timestep by reversing the SDE. Core function to propagate the diffusion process from the learned model ou

Set input tensor to be used instead of forward()'s input. When doing pipeline parallelism the input from the previous stage comes fro

Sets the progress bar config for the pipeline

Splits a dictionary of tensors along the first dimension into `num_chunks` equal parts. Example: >>> x = torch.arange(12).reshape(6,

r""" Reward function that checks if the reasoning process is enclosed within `"<think>"` and `"</think>"` tags. The function returns a reward

r""" Unpair a preference dataset. Args: dataset (`Dataset` or `DatasetDict`): Preference dataset to unpair. The dataset m

TODO(zhangchi.usc1992): We may change the implementation later

Returns: entropy: # (bs, response_len) log_probs: # (bs, response_len)

Get or create a snapshot with the necessary dependencies and scripts for evaluation.

Sample instances from dataset and get prompt embeddings. Used for density ratio classifier training.

Initialize or get the current event loop.

Memory-efficient post-order traversal of FSDP modules to extract full parameters and sync with vLLM.

Replaces processing_class.pad_token_id to embedding_tokenizer.pad_token_id and applies self.embedding_func

allgather params of all pp ranks. Return a list of handles

Batch data into lists of length n. The last batch may be shorter.

Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a + b)[len(enc(a)):]`.

Llama tokenizer does satisfy `enc(a + b) = enc(a) + enc(b)`. It does ensure `enc(a + b) = enc(a) + enc(a + b)[len(enc(a)):]`.

Computes log probabilities of the reference model for a single padded batch of a KTO specific dataset.

Computes log probabilities of the reference model for a single padded batch of a BCO specific dataset.

Compute the reward for a list of histories.

r""" return a callable function that takes in a "messages" dataset and returns a formatted dataset, based on the tokenizer apply chat template

r""" Converts a state dict to new diffusers format. The state dict can be from previous diffusers format (`OLD_DIFFUSERS`), or PEFT format (`P

Estimate the FLOPS based on the number of valid tokens in the current batch and the time taken. Args: batch_seqlens (Lis

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the