Get the computational cost of inferring on the dataset. Args: dataset (ConfigDict): The dataset config. get_raw_facto

Get the computational cost of inferring on the dataset. Args: dataset (ConfigDict): The dataset config. get_raw_facto

Load data from a JSON or JSONL file.

Match the config file in workdir recursively given the pattern. Additionally, if the pattern itself points to an existing file, it will be di

Lower text and remove punctuation, storys and extra whitespace.

in Elo we want numpy arrays for matchups and outcomes matchups: int32 (N,2) contains model ids for the competitors in a match outcom

Cilin 詞林 is a thesaurus with semantic information

Calculate scores. Args: predictions (List): List of predictions of each sample. references (List): List of targets fo

Wait till the next query can be sent. Applicable in both single-thread and multi-thread environments.

Runs basic whitespace cleaning and splitting on a peice of text.

Builds the final results dictionary with detailed information.

Cleans up the generated code.

深度优先遍历，获取最小编辑距离相同的所有序列

Evaluate code with retry mechanism. Args: input_data: Can be either: - dict: Dictionary containing code and test

Extract code from markdown-formatted text. Args: text (str): Text that may contain code blocks in markdown format. Retur

NOTE: This is a replacement of ERRANTS lemma cost for Chinese 计算基于字符相似度的编辑距离cost

Get the condition probability of next token. Args: input_texts (List[str]): All the input prompt to be tested. target

Get loglikelihood scores given input string and continuation string. Args: inputs (str): string. conts (str): strings

NOTE: Based on the paper: Improved-Edit-Distance Kernel for Chinese Relation Extraction 获取每个词语的语义类别（基于大词林，有三个级别）

计算基于语义信息的替换操作cost :param a: 单词a的语义类别 :param b: 单词b的语义类别 :return: 替换编辑代价

Process completions with a test case. Args: test_case (dict): A test case containing prompt and stop tokens. completi

Process the response to needed format. Args: datum(dict): inputs. Returns: dict: Processed response data sam

Convert the prompts to a API-style prompts, given an updated role_dict. Args: prompts (Union[List, str]): The prompts to

Convert the prompts to a string, given an updated role_dict. Args: prompts (Union[List, str, dict]): The prompt(s) to be converte

Convert a role prompt to a string, given an updated role_dict. Args: role_prompt (Dict): The role prompt to be converted.

Update the default role dict with the given prompt(s).

Update the default role dict with the given prompts.

Based on ERRANT's alignment 基于改进的动态规划算法，为原句子的每个字打上编辑标签，以便使它能够成功转换为目标句子。 编辑操作类别： 1) M：Match，即KEEP，即当前字保持不变 2)