Cleans LaTeX expressions for parsing. Args: latex_expr (str): The LaTeX expression to clean. Returns: str: T

Evaluates the functional correctness of a completion by running the test suite provided in the problem. :param completion_id: an optional com

Check correctness of code generation with a global timeout. The global timeout is to catch some extreme/rare cases not handled by the timeout

Compare target and output answers. Takes only the first sentence from output and filters responses when model tries to generate examples. We

Compare files with specified extensions in two folders.

Compare results according to score when both answers are failed.

Computes core metrics for a given set of items based on the ground truth items. Args: items (list): The list of items generated by th

Computes the minimum edit distance between two strings using dynamic programming.

This is the function what you are probably looking for. a_pred is the answer string your model predicted.

Convert non-standard fraction like \frac\alpha2 to its standard- convertible \frac{\alpha}{2} We support single letter,number or standard form

Converts LaTeX vector syntax to a standardized form. This function processes a given text string and ensures that LaTeX vector notations are

Count style elements for bradley terry + style control. Args: text (str): Text to calculate style features from. suffix (str, opt

Create an API request body for a given text. Args: text: Text to perform evaluation. Returns: dict: Request

Download a file from a url and place it in root. Args: url (str): URL to download file from. root (str): Directory to place downl

Check if the edit distance solution is valid. :param instance: The instance dictionary with 'string_a' and 'string_b'. :param solutio

对字符串进行 Base64 编码. :param s: 字符串 :return: 编码后的字符串

(expression1 is assumed to be Ground_Truth) Function: Check if two equations are mathematically equivalent Step 1: Simplify e

Estimates pass@k of each problem and returns them in an array.

Estimates pass@k of each problem and returns them in an array.

Estimates pass@k of each problem and returns them in an array.

Calculates 1 - comb(n - c, k) / comb(n, k).

We take the list of code generations and try to compile them and the run their corresponding unit tests which are retrieved from the APPS dataset.

Evaluates a single response against the answer. Args: response (str): The model's response. answer (str): The referen

Evaluates a list of responses. Args: data (list): List of records containing responses and answers. question_type (st