Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, articles and extra whitespace.

Processes precisions to ensure that precision and recall don't both get worse. Assumes the list precision is sorted in order of recalls

Create a complex radar chart with different scales for each variable Source: https://towardsdatascience.com/how-to-create-and-visualize-complex-ra

Strip accents from string

Path of this evaluation module in cache_dir: Will be: self._data_dir_root/self.name/self.config_name/self.hash (if not none)/

This method defines the common API for all the evaluation module in the library

A utility function enabling the confidence interval calculation for metrics computed by the evaluator based on `scipy`'s `bootstrap`

A utility function computing time performance metrics: - `total_time_in_seconds` - pipeline inference runtime for the evaluation

Copy a script and its required imports to an importable directory Args: name (str): name of the resource to load importable_direc

Create a new cache file. If the default cache file is used, we generated a new hash.

Downloads and prepares resources for the evaluation module. This is the internal implementation to overwrite called when user calls `

Close all the writing process and load/gather the data from all the nodes if main node or all_process is True.

Get a lock on all the cache files in a distributed setup. We wait for timeout second to let all the distributed node finish their tasks (defau

If EVALUATE_VERBOSITY env var is set to one of the valid choices return that as the new default level. If it is not - fall back to ``_default

Construct the EvaluationModuleInfo object. See `EvaluationModuleInfo` for details. Warning: This function is only called once and the result

Scales data[1:] to ranges[0]

Checks an incoming task string, to validate it's correct and returns the default Evaluator class and default metric name. It first performs a

input_column (`str`, defaults to `"path"`): the name of the column containing the input audio path in the dataset specified by `d

input_column (`str`, defaults to `"tokens"`): The name of the column containing the tokens feature in the dataset specified by `d

input_column (`str`, *optional*, defaults to `"text"`): The name of the column containing the text feature in the dataset specifi

input_column (`str`, defaults to `"image"`): The name of the column containing the images as PIL ImageFile in the dataset specifi

Convert a link to a file on a github repo in a link to the raw github object.

Downloads and prepares evaluation module for reading. Args: download_config ([`DownloadConfig`], *optional*): Spe

Estimates pass@k of each problem and returns them in an array.

Calculates 1 - comb(n - c, k) / comb(n, k).

Computes F1 score and Exact Match for MultiRC predictions.

Convert a list of scripts or text files provided in file_paths into a hashed filename in a repeatable way.

Infers which split to use if `None` is given. Args: data (`str`): Name of dataset. subset

Given a URL, look for the corresponding file in the local cache. If it's not there, download it. Then return the path to the cached file.

Import a module at module_path and return its main class, a Metric by default

Import a module at module_path and return the EvaluationSuite class

Update the download count of a dataset or metric.

Maps metric feature types to input types for gradio Dataframes: - float/int -> numbers - string -> strings - any other ->

Add hf_modules_cache to the python path. By default hf_modules_cache='~/.cache/huggingface/modules'. It can also be set with the environm

Calculate intersection and Union. Args: pred_label (`ndarray`): Prediction segmentation map of shape (height, width).

Check if the provided dataset follows the squad v2 data schema, namely possible samples where the answer is not in the context. In th

Maps json input type to str.