Get HumanEvalPlus locally. Args: err_incomplete (bool, optional): Whether to raise error if HumanEvalPlus is not complete. Defaults to Tru

Start the Docker container using Docker API

Process element types locally before sending to manager

Estimates pass@k of each problem and returns them in an array.

Estimates pass@k of each problem and returns them in an array.

Formats Python code with proper indentation using autopep8.

This disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bomb, killing other p

Parses each jsonl line and yields it as a dictionary

Convert an AST node to a dictionary representation for easier comparison. Args: node: An AST node Returns:

Determine type safely without eval

Get the hash of HumanEvalPlus. Returns: str: The hash of HumanEvalPlus

Get the hash of MbppPlus. Returns: str: The hash of MbppPlus

We accept two formats of inputs. + `sample.jsonl` which is the format from HumanEval, i.e., {task_id, completion or solution}. + A folder whic

Profile the func_code against certain input tests. The function code is assumed to be correct and if a string is returned, it is an error message.

Stop the Docker container

Execute trusted code in place.

The validation loop of PPO. The only difference is logging more metrics.

The validation loop of PPO. The only difference is logging more metrics.