Turn samples into experience by calculating logprobs, values, rewards, and kl divergence.

Turn samples into experience by calculating logprobs, values, rewards, and kl divergence.

Turn samples into experience by calculating logprobs, values, rewards, and kl divergence.

Turn samples into experience by calculating logprobs, values, rewards, and kl divergence.

Make a list of experience with the micro_rollout_batch_size. This method will first calculate the response sequences and rewards for

Make a list of experience with the micro_rollout_batch_size. This method will first calculate the response sequences and rewards for

Make a list of experience with the micro_rollout_batch_size. This method will first calculate the response sequences and rewards for

r""" Merges multiple datasets to a unified dataset.

r""" Computes ORPO's odds ratio (OR) loss for batched log probabilities of the policy model.

r""" Freezes vision tower for VLM LoRA tuning.

r""" Implements training loop for the PPO stage, like _inner_training_loop() in Huggingface's Trainer.

r""" Expands the attention mask with indices from (batch_size, seq_len) to (batch_size, 1, seq_len, seq_len), while handles packed sequences a

r""" Includes: (1) cast the layernorm in fp32 (2) make output embedding layer require grads (3) add the upcasting of the l

Preprocess data from raw dataset to prompt, response, label Args: data: raw data from dataset

Process experiences, this can be used to filter out some experiences or do some processing on the rewards. Output: - experie

Process experiences, this can be used to filter out some experiences or do some processing on the rewards. Output: - experie

Process experiences, this can be used to filter out some experiences or do some processing on the rewards. Output: - experie

Process experiences, this can be used to filter out some experiences or do some processing on the rewards. Output: - experie

batch_search: original result like: [[bm25-q1-d1, bm25-q1-d2],[bm25-q2-d1, bm25-q2-d2], [e5-q1-d1, e5-q1-d2], [e5-q2-d1, e5-q2-d2]]

Synchronous request API wrapper

Calculate position ids for packed_seq_ids[start:end]. For example, if the packed_seq_lens is [3, 2, 4, 1], start=2, end=8, the position i

r""" Resize token embeddings.

r""" Restores the layernorm parameters in the model. The model is already unwrapped (and gathered).