Append experience to replay buffer.

Get transformer with a sequence classification head on top (linear layer). Args: model_name_or_path (str): Path to pretrained model.

r""" Set the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas use a different random ordering for

Init model from pretrained checkpoint. Returns: List: list of remote object refs.

the front half is matched for spv, the latter half is unmatched for KL

Run the given model on the given batch of inputs, concatenating the chosen and rejected inputs together. We do this to avoid doing two forwar

remote reward model API api_url: RM API, We assume that the API supports two modes: merging query + response and not merging queries: query+re

Compute the log probabilities of the given labels under the given logits. Args: logits: Logits of the model (unnormalized). Shape

Save actor model on rank 0. Returns: List: list of remote object refs.

Compute the approximate KL divergence between two distributions. Schulman blog: http://joschu.net/blog/kl-approx.html Args: log_

Run the given model on the given batch of inputs, concatenating the chosen and rejected inputs together. We do this to avoid doing two forwar

Generate samples and return in batches.

Make a list of experience with the micro_rollout_batch_size. This method will first calculate the response sequences and rewards for

Compute the log probabilities of the given labels under the given logits. Args: logits: Logits of the model (unnormalized). Shape

Train actor model. Args: critic_model_group (PPORayActorGroup): critic model group. initial_model_group (PPORayActorG

Concatenate the chosen and rejected inputs into a single tensor. Args: batch: A batch of data. Must contain the keys 'chosen_inpu

Concatenate the chosen and rejected inputs into a single tensor. Args: batch: A batch of data. Must contain the keys 'chosen_inpu