Evaluates the functional correctness of a completion by running the test suite provided in the problem. :param completion_id: an optiona

https://github.com/hkust-nlp/simpleRL-reason/blob/main/eval/grader.py

https://github.com/hkust-nlp/simpleRL-reason/blob/main/eval/grader.py

Clean up the executor and other resources

Estimates pass@k of each problem and returns them in an array.

Calculates 1 - comb(n - c, k) / comb(n, k).

Calculates 1 - comb(n - c, k) / comb(n, k).

Calculates 1 - comb(n - c, k) / comb(n, k).

We take the list of code generations and try to compile them and the run their corresponding unit tests which are retrieved from the APPS dataset

Extract the answer from inside a LaTeX \\boxed{} command

Extract content inside the last \boxed in LaTeX text

extract output between ```output\n...\n```, use regex, there might be multiple outputs, each output may have multiple lines

Filter out tasks that can be solved using less than threshold #instruction.

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the

use the form of "assert func_name(input) == "

Gather all solutions from the folders

Returns all datasets without step information

Extract benchmark name from path