Compute Categorical entropy loss Args: logits: `(torch.Tensor)` shape: (bs, response_length, vocab_size) eos_mask: `(

Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py Args: token_level_rewards: `(torch.Tensor)`

Compute advantage for GRPO, operating only on Outcome reward (with only one scalar reward for each response). Args: token_level_

Compute the log probability of the responses given input_ids, attention_mask and position_ids Args: data (DataProto): a DataProto

Compute the log probability of the responses given input_ids, attention_mask and position_ids Args: data (DataProto): a DataProto

Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1122 Args: old_log_prob: `(torch.Tensor)`

Compute advantage for Reinforce++, operating on Outcome reward (with only one scalar reward for each response). Args: token_leve

The scoring function for countdown task. Args: solution_str: the solution text ground_truth: dictionary containing target num

Computes comprehensive score for model response. Args: solution_str: Raw model response string ground_truth: Dictionary conta

The scoring function for GSM8k. Reference: Trung, Luong, et al. "Reft: Reasoning with reinforced fine-tuning." Proceedings of the 62nd Annual Mee

The scoring function for GSM8k. Reference: Trung, Luong, et al. "Reft: Reasoning with reinforced fine-tuning." Proceedings of the 62nd Annual Mee

Compute the value loss. Copied from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1151 Args: vpreds (`torch.Fl

Compute values

Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens.

Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-

Args: model_name: override_config_kwargs: Returns:

All the args and kwargs must be DataProto. The batch will be chunked by dp_size and passed to each rank

Calculate entropy from logits.

`Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been set.

r"""Works like os.path.exists() but supports hdfs. Test whether a path exists. Returns False for broken symbolic links. Args: path (

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the PPO dat

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing

Args: input_ids: input ids. shape (1, totol_nnz) position_ids: position ids. shape (batch_size, seq_length)

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing

Args: input_ids: input ids. shape (1, totol_nnz) position_ids: position ids. shape (batch_size, seq_length)

r""" Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing

Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (

Generate dataset for countdown task. Args: num_samples: Number of samples to generate num_operands: Number of numbers provide

Generate sequences

Returns the added tokens in the vocabulary as a dictionary of token to index. Results might be different from the fast call because f