Args: model_name: actor_override_config_kwargs: Returns:

name: name of the parameter param: training parameters infer_params (List[torch.Tensor]): a list of parameters all-gathered f

User passes in dp data. The data is dispatched to all tp/pp ranks with the same dp

treat pp as dp.

Evaluate one subject.

Safely evaluate the arithmetic equation using eval() with precautions.

Extracts the final answer from the model's response string. Args: solution_str: Raw response string from the language model

Extract the equation from the solution string.

Extracts the final answer from the model's response string. Args: solution_str: Raw response string from the language model

Fold a batch dim from [bsz, xxx] into [new_bsz, bsz // new_bsz, xxx]

We assume: - The model takes input: (input_ids, attention_mask, position_ids). No rmpad for the input - The communication sha

Gather the label from data. The value in label should be [0, vocab_size) Args: data: (..., vocab_size) label (torch.IntTensor) :

any score <0.4 is ok, since +- is easy the model have

Generate ground truth response given a prompt.

Return a local temp path that joins cache_dir and basename of hdfs_path Args: hdfs_path: cache_dir: Returns:

Note that here we only asks execute_all and execute_rank_zero to be implemented Leave the choice of how these two functions handle argument '

Get subjects to evaluate.

Get ulysses sequence parallel rank.

Initialize random seeds for reproducibility.

Load evaluation records based on arguments.

Load data from a JSONL file.

Load limited test records based on given arguments.

Load previous accuracy results.

Compute the response log_probs from full logits. Note that logits = model(input_ids) Args: input_ids: [batch_size, seqlen] lo

Compute the log_probs from logits with rmpad logits and pad input. Note that logits_rmpad = model(input_ids_rmpad). For each sentences, there is a