Extract and normalize model name from path

Parse a Python assert statement and extract the expected output from the right side of the '==' operator as a string. :param statement:

starter_code : str

Post-process the LLM generated text to make it valid.

This disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bomb, killing other pro

This disables various destructive functions and prevents the generated code from interfering with the test (e.g. fork bomb, killing other pro

Remove anything that is not a definition or import. Preserves: - Import/From imports - Class definitions - Function/AsyncFunctio

Remove all comments and docstrings from the code.

Remove all print statements from the code.

Replace all occurrences of `old_name` with `new_name` in the code. Replace the definition and all recursive calls of `old_name` with `new_nam

Run huggingface-cli download and return the model path.

Run the model for a single prompt and return the output Static method to be used in multiprocessing Calls the _run_single met

Args: func: The function to run. The function must accept a single argument. tasks: A list of tasks i.e. arguments to func.

Args: func: The function to run. The function must accept a single argument. tasks: A list of tasks i.e. arguments to func.

if test(generated_code) is not None it'll try to run the code. otherwise it'll just return an input and output pair.

Test solutions, return functionally correct solutions

Validates that the entire string is composed solely of properly nested HTML-like tags, allowing any whitespace (spaces, tabs, newlines) betwe

We will expand this function gradually based on the available datasets

Note that we also return the raw_input_ids so that it can be combined with other chat template