Replaces all instances of 'src' in the PromptList with 'dst'. Args: src (str): The string to be replaced. dst (Prompt

Replaces all instances of 'src' in the PromptList with 'dst'. Args: src (str): The string to be replaced. dst (Prompt

return dataset id when getting data from ModelScope/HuggingFace repo, otherwise just return local path as is. Args: dataset_id (str):

Update pbar counter when callback.

Encode prompt to tokens. Not necessary for most cases. Args: prompt (str): Input string. Returns: torch.Tens

Generate model abbreviation from the model's confg.

Get the logger for OpenCompass. Args: log_level (str): The log level. Default: 'INFO'. Choices are 'DEBUG', 'INFO', 'WARNING'

Returns dataset abbreviation from the dataset's confg.

Decode tokens to text. Not necessary for most cases. Args: tokens (torch.Tensor): Input tokens. Returns: str

Wait till the next query can be sent. Applicable in both single-thread and multi-thread environments.

Ensure simultaneous emptying of stdout and stderr when concurrent resources are available. When employing multiprocessing with standa

Get lengths given a list of templates. Args: templates (Union[List[str], str]): Input template(s). mode (str): Parsin

Generate the in-context example for one test example. If `ice_template` is an instance of `PromptTemplate`, the `ice_separator` and `i

Get out put path. Args: cfg (ConfigDict): The running config. time_str (str): Current time.

Post a message to Lark. When title is None, message must be a str. otherwise msg can be in rich text format (see https://open

Release concurrent resources if acquired. This behavior will fall back to do nothing if there are no concurrent resources.

Acquire concurrent resources if exists. This behavior will fall back to wait with query_per_second if there are no concurrent resourc

Extract judgements (scores) and references. Args: result (ConfigDict): Dataset config. filename (str): Model path in results dir.

Parse a prompt template, and wrap it with meta template if applicable. Args: prompt_template (List[PromptType]): A prompt

Generate an item based on the provided :obj:`entry` data, as well as optional output field and ice field tokens. Warning:

Generate the prompt for one test example in generative evaluation with `prompt_template`. If `prompt_template` is not provided, the `i

Filter the repeated error message. Args: record (LogRecord): The log record. Returns: bool: Whether to outpu

Extract judgements (scores) and references. Args: dataset (ConfigDict): Dataset config. subdir_path (str): Model path in results

Generate completion from a list of templates. Args: templates (List[PromptType]): A list of templates. max_out_len (i

Get the command template for the task. Args: cfg_path (str): The path to the config file of the task. template (str):

Returns task abbreviation from the task's confg.

Processes a list of dictionaries to compute markdown element counts. Args: data (list): Input data, either a list of dictionaries.

Return a dataloader of the input data list.

Get the hash of the prompt configuration. Args: dataset_cfg (ConfigDict or list[ConfigDict]): The dataset configuration.

Parse the package dependencies listed in a requirements file but strips specific versioning information. Args: fname (str): path to r

Dump the result to a json file.

Calculate toxic scores for each prediction. Args: predictions (List): List of predictions of each sample. Returns:

Normalize a final answer to a quantitative reasoning question.

Find the first available file with given extensions.

Generate the prompt for one test example in perpelxity evaluation with `prompt_template`. If `prompt_template` is not provided, the `i

Get the path to the log file. Args: file_extension (str): The file extension of the log file. Default: 'json'.

Load data from a JSON or JSONL file.

Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, extra whitespace.

The function calculates the numerical results for each group based on the configuration in summary_groups, and updates the contents of

The function reads the numerical results of evaluations from the output folder based on the configuration file, and ultimately returns

>>> edit_distance('abcde', 'avbcude') 2 >>> edit_distance(['至', '刂'], ['亻', '至', '刂']) 1 >>> edit_distance('fang', 'qwe') 4

Extracts text enclosed in double brackets [[ ]]. Args: text (str): The text to extract from. clean_level (str): The l

Generate prompt based on :obj:`entry` data, :obj:`ice` in-context example, and the corresponding :obj:`label`. Args: ent

Parse a prompt template, and wrap it with meta template if applicable. Args: prompt_template (List[PromptType]): A prompt

This function converts ids string to a string that can be used as a tree input to APTED. Any Error raised by this function implies that the i