Get sanitized MBPP from Google's Github repo.

Calculate the type counts reward for a Python code snippet. Args: answer: A string containing the answer type_counters: A di

Get statistics about the types and their counts.

Parse the input and output of a code snippet. Args: input_str: A string containing the code snippet parse_input: Whether to

Sort checkpoints numerically with final checkpoint at the end

Check trusted_exec success.

Writes an iterable of dictionaries to jsonl

Normalize answer expressions.

Check correctness of code generation with a global timeout. The global timeout is to catch some extreme/rare cases not handled by the timeouts

Explicitly clean up the process pool

Check if code imports any banned modules using AST parsing.

We use Coefficient of Variation (CV) to as the variance of PE. CV = 100 * standard deviation / mean

Deduplicate solutions

Get HumanEval from OpenAI's github repo and return as a list of parsed dicts. Returns: List[Dict[str, str]]: List of dicts with keys "pro