Evaluates the functional correctness of a completion by running the test suite provided in the problem. :param completion_id: an optional com

Check correctness of code generation with a global timeout. The global timeout is to catch some extreme/rare cases not handled by the timeout

Compare target and output answers. Takes only the first sentence from output and filters responses when model tries to generate examples. We

Compare files with specified extensions in two folders.

Compare results according to score when both answers are failed.

计算元素命中率

Computes core metrics for a given set of items based on the ground truth items. Args: items (list): The list of items generated by th

Computes the minimum edit distance between two strings using dynamic programming.

Ranks predictions for a single sample across multiple augmentations. Args: prediction_group (list): A 2D list of predictions for one

This is the function what you are probably looking for. a_pred is the answer string your model predicted.

Convert non-standard fraction like \frac\alpha2 to its standard- convertible \frac{\alpha}{2} We support single letter,number or standard form

Converts LaTeX vector syntax to a standardized form. This function processes a given text string and ensures that LaTeX vector notations are

F1 score with the optimal threshold. Handles cases where either predictions or targets are empty. Parameters: pred (Tensor)

F1 score with the optimal threshold. Handles cases where either predictions or targets are empty. Parameters: pred (Tensor)

Count style elements for bradley terry + style control. Args: text (str): Text to calculate style features from. suffix (str, opt

Generate 'previous' (filling with 0's) features for the model Args: num_res: the number of residues Returns:

Create an API request body for a given text. Args: text: Text to perform evaluation. Returns: dict: Request

Run the forward method of the pretrained-language model Args: fasta: the fasta sequence mask: the mask indic

Returns:

Apply torch functionals to the translation and rotations Args: func (): the functional to apply to dim (): t