Functions — github.com/huggingface/evaluate

hub / github.com/huggingface/evaluate / functions

Functions785 in github.com/huggingface/evaluate

⨍ Functions 785◇ Types & classes 150↳ Endpoints 6

Compute the evaluation module. Usage of positional arguments is not allowed to prevent mistakes. Args: predictions (`lis

src/evaluate/module.py:415

↓ 87 callersMethodload

( path: str, download_mode: Optional[DownloadMode] = None, revision: Optional[Union[st

src/evaluate/evaluation_suite/__init__.py:76

↓ 66 callersFunctionlaunch_gradio_widget

Launches `metric` widget with Gradio.

src/evaluate/utils/gradio.py:92

↓ 23 callersMethodinfo

:class:`evaluate.EvaluationModuleInfo` object containing all the metadata in the evaluation module.

src/evaluate/module.py:92

↓ 21 callersFunctionevaluator

Utility factory method to build an [`Evaluator`]. Evaluators encapsulate a task and a default metric name. They leverage `pipeline` functiona

src/evaluate/evaluator/__init__.py:113

↓ 19 callersMethodexpected_results

(cls)

tests/test_metric.py:45

↓ 19 callersMethodfrom_dict

(cls, metric_info_dict: dict)

src/evaluate/info.py:116

↓ 18 callersMethodadd

Add one prediction and reference for the evaluation module's stack. Args: prediction (`list/array/tensor`, *optional*):

src/evaluate/module.py:548

↓ 17 callersMethodadd_batch

Add a batch of predictions and references for the evaluation module's stack. Args: predictions (`list/array/tensor`, *optional*):

src/evaluate/module.py:488

↓ 15 callersMethodpredictions_and_references

(cls)

tests/test_metric.py:37

↓ 14 callersMethod_feature_names

(self)

src/evaluate/module.py:618

↓ 14 callersFunctionload

Load a [`~evaluate.EvaluationModule`]. Args: path (`str`): Path to the evaluation processing script with the evaluation buil

src/evaluate/loading.py:689

↓ 14 callersMethodrun

( self, model_or_pipeline: Union[str, "Pipeline", Callable, "PreTrainedModel", "TFPreTrainedModel"] #

src/evaluate/evaluation_suite/__init__.py:102

↓ 12 callersMethodload_data

Load dataset with given subset and split. Args: data ([`Dataset`] or `str`, defaults to `None`): Specifie

src/evaluate/evaluator/base.py:350

↓ 11 callersFunctionget_logger

Return a logger with the specified name.

src/evaluate/utils/logging.py:78

↓ 8 callersMethod_infer_device

Helper function to check if GPU or CPU is available for inference.

src/evaluate/evaluator/base.py:182

↓ 8 callersFunctioncombine

Combines several metrics, comparisons, or measurements into a single `CombinedEvaluations` object that can be used like a single evaluation module

src/evaluate/module.py:1008

↓ 7 callersFunctioncached_path

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

src/evaluate/utils/file_utils.py:142

↓ 7 callersFunctionevaluation_module_factory

Download/extract/cache a metric module. Metrics codes are cached inside the the dynamic modules cache to allow easy import (avoid ugly sys.p

src/evaluate/loading.py:559

↓ 7 callersMethodget_module

(self)

src/evaluate/loading.py:396

↓ 7 callersFunctionmerge_eval

(main_eval, new_eval, prefix)

metrics/squad_v2/compute_score.py:158

↓ 7 callersMethodprepare_data

Prepare data. Args: data ([`Dataset`]): Specifies the dataset we will run evaluation on. inp

src/evaluate/evaluator/base.py:390

↓ 7 callersFunctionpush_to_hub

r""" Pushes the result of a metric to the metadata of a model repository in the Hub. Args: model_id (`str`): Model id fro

src/evaluate/hub.py:14

↓ 6 callersFunction_get_library_root_logger

()

src/evaluate/utils/logging.py:63

↓ 6 callersFunctionmake_eval_dict

(exact_scores, f1_scores, qid_list=None)

metrics/squad_v2/compute_score.py:137

↓ 6 callersMethodpredictions_processor

A core method of the `Evaluator` class, which processes the pipeline outputs for compatibility with the metric.

src/evaluate/evaluator/base.py:212

↓ 6 callersFunctionsummarize_if_long_list

(obj)

src/evaluate/module.py:73

↓ 5 callersMethodcheck_required_columns

Ensure the columns required for the evaluation are present in the dataset. Args: data (`str` or [`Dataset`]):

src/evaluate/evaluator/base.py:295

↓ 5 callersMethodformat

(self)

src/evaluate/module.py:141

↓ 5 callersMethodread

(self, *args, **kwargs)

metrics/code_eval/execute.py:125

↓ 4 callersFunctionSARIngram

(sgrams, cgrams, rgramslist, numref)

metrics/wiki_split/wiki_split.py:109

↓ 4 callersFunctionSARIngram

(sgrams, cgrams, rgramslist, numref)

metrics/sari/sari.py:82

↓ 4 callersFunction_raise_if_offline_mode_is_enabled

Raise an OfflineModeIsEnabled error (subclass of ConnectionError) if HF_EVALUATE_OFFLINE is True.

src/evaluate/utils/file_utils.py:240

↓ 4 callersFunctionapply_no_ans_threshold

(scores, na_probs, qid_to_has_ans, na_prob_thresh)

metrics/squad_v2/compute_score.py:126

↓ 4 callersMethodcall_pipeline

(self, pipe, *args, **kwargs)

src/evaluate/evaluator/base.py:511

↓ 4 callersMethodcheck_for_mismatch_in_device_setup

(device, model_or_pipeline)

src/evaluate/evaluator/base.py:282

↓ 4 callersMethodcompute

( self, model_or_pipeline: Union[ str, "Pipeline", Callable, "PreTrainedModel", "T

src/evaluate/evaluator/base.py:218

↓ 4 callersMethodcompute_metric

Compute and return metrics.

src/evaluate/evaluator/base.py:517

↓ 4 callersFunctioncompute_score

(preds, labels, score_type="wer")

metrics/xtreme_s/xtreme_s.py:210

↓ 4 callersFunctionf1_score

(prediction, ground_truth)

metrics/squad/compute_score.py:30

↓ 4 callersFunctionget_jaccard

(prediction, ground_truth)

metrics/cuad/compute_score.py:15

↓ 4 callersFunctionhttp_head

( url, proxies=None, headers=None, cookies=None, allow_redirects=True, timeout=10.0, max_retries=0 )

src/evaluate/utils/file_utils.py:367

↓ 4 callersFunctionnormalize_answer

Lower text and remove punctuation, articles and extra whitespace.

metrics/squad_v2/compute_score.py:60

↓ 4 callersFunctionnormalize_answer

Lower text and remove punctuation, articles and extra whitespace.

metrics/super_glue/record_evaluation.py:15

↓ 4 callersFunctionnormalize_answer

Lower text and remove punctuation, articles and extra whitespace.

metrics/squad/compute_score.py:11

↓ 4 callersFunctionparse_flag_from_env

(key, default=False)

tests/utils.py:14

↓ 4 callersMethodprepare_metric

Prepare metric. Args: metric (`str` or [`EvaluationModule`], defaults to `None`): Specifies the metric w

src/evaluate/evaluator/base.py:480

↓ 4 callersFunctionset_verbosity

Set the level for the Hugging Face Evaluate library's root logger. Args: verbosity: Logging level, e.g., `evaluate.logging.DEB

src/evaluate/utils/logging.py:104

↓ 4 callersMethodtokenize

(self, text)

metrics/rouge/rouge.py:89

↓ 3 callersMethod_enforce_nested_string_type

Recursively checks if there is any Value feature of type string and throws TypeError if corresponding object is not a string. Since a

src/evaluate/module.py:733

↓ 3 callersFunctionf1_score

(prediction, ground_truth)

metrics/super_glue/record_evaluation.py:34

↓ 3 callersFunctionget_hash

Get the hash of a string

measurements/text_duplicates/text_duplicates.py:55

↓ 3 callersMethodget_module

(self)

src/evaluate/loading.py:471

↓ 3 callersFunctioninit_dynamic_modules

Create a module with name `name` in which you can add dynamic modules such as metrics or datasets. The module can be imported using its name.

src/evaluate/loading.py:56

↓ 3 callersMethodinputs_and_targets

(cls)

tests/test_metric.py:637

↓ 3 callersFunctionis_remote_url

(url_or_filename: str)

src/evaluate/utils/file_utils.py:60

↓ 3 callersFunctionmake_precision_recall_eval

(scores, na_probs, num_true_pos, qid_to_has_ans, out_image=None, title=None)

metrics/squad_v2/compute_score.py:175

↓ 3 callersFunctionnormalize

(sentence, lowercase: bool = True, tokenizer: str = "13a", return_str: bool = True)

metrics/wiki_split/wiki_split.py:257

↓ 3 callersFunctionnormalize

(sentence, lowercase: bool = True, tokenizer: str = "13a", return_str: bool = True)

metrics/sari/sari.py:230

↓ 3 callersFunctionoffline

Simulate offline mode. There are three offline simulatiom modes: CONNECTION_FAILS (default mode): a ConnectionError is raised for each

tests/utils.py:225

↓ 3 callersMethodpredict

(self, data, *args, **kwargs)

tests/test_metric_common.py:206

↓ 3 callersMethodprepare_pipeline

Prepare pipeline. Args: model_or_pipeline (`str` or [`~transformers.Pipeline`] or `Callable` or [`~transformers.PreTrain

src/evaluate/evaluator/base.py:422

↓ 3 callersFunctionproperly_del_metric

properly delete a metric on windows if the process is killed during multiprocessing

tests/test_metric.py:89

↓ 3 callersFunctionregard

(group, regard_classifier)

measurements/regard/regard.py:109

↓ 3 callersFunctionrelative_to_absolute_path

Convert relative path to absolute path.

src/evaluate/utils/file_utils.py:76

↓ 3 callersMethodwords_to_offsets

Convert a list of words to a list of offsets, where word are joined by `join_by`. Args: words (`List[str]`):

src/evaluate/evaluator/token_classification.py:144

↓ 2 callersMethod__init__

(self, task="translation", default_metric_name=None)

src/evaluate/evaluator/text2text_generation.py:222

↓ 2 callersFunction_create_importable_file

( local_path: str, local_imports: List[Tuple[str, str]], additional_files: List[Tuple[str, str]],

src/evaluate/loading.py:360

↓ 2 callersFunction_download_additional_modules

Download additional module for a module <name>.py at URL (or local path) <base_path>/<name>.py The imports must have been parsed first using

src/evaluate/loading.py:212

↓ 2 callersFunction_get_library_name

()

src/evaluate/utils/logging.py:59

↓ 2 callersMethod_infer_feature_from_example

(self, example)

src/evaluate/module.py:598

↓ 2 callersMethod_init_writer

(self, timeout=1)

src/evaluate/module.py:625

↓ 2 callersFunction_list_evaluation_modules_type

(module_type, include_community=True, with_details=False)

src/evaluate/inspect.py:74

↓ 2 callersFunction_request_with_retry

Wrapper around requests to retry in case it fails with a ConnectTimeout, with exponential backoff. Note that if the environment variable HF_EVALU

src/evaluate/utils/file_utils.py:274

↓ 2 callersFunctioncamelcase_to_snakecase

Convert camel-case string to snake-case.

src/evaluate/naming.py:32

↓ 2 callersMethodcompute

input_column (`str`, defaults to `"file"`): The name of the column containing either the audio files or a raw waveform, represent

src/evaluate/evaluator/audio_classification.py:102

↓ 2 callersMethodcompute

( self, model_or_pipeline: Union[ str, "Pipeline", Callable, "PreTrainedModel", "T

src/evaluate/evaluator/text2text_generation.py:231

↓ 2 callersMethodcompute

question_column (`str`, defaults to `"question"`): The name of the column containing the question in the dataset specified by `da

src/evaluate/evaluator/question_answering.py:149

↓ 2 callersFunctioncompute_score

(dataset, predictions)

metrics/cuad/compute_score.py:150

↓ 2 callersFunctioncompute_score

(dataset, predictions)

metrics/squad/compute_score.py:55

↓ 2 callersMethoddownload_loading_script

(self, revision)

src/evaluate/loading.py:464

↓ 2 callersFunctionfilename_prefix_for_split

(name, split)

src/evaluate/naming.py:52

↓ 2 callersFunctionfind_all_best_thresh

(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans)

metrics/squad_v2/compute_score.py:270

↓ 2 callersFunctionfind_best_thresh

(preds, scores, na_probs, qid_to_has_ans)

metrics/squad_v2/compute_score.py:247

↓ 2 callersFunctionformat_chunk

(chunk)

src/evaluate/module.py:77

↓ 2 callersFunctionftp_get

(url, temp_file, timeout=10.0)

src/evaluate/utils/file_utils.py:323

↓ 2 callersFunctionftp_head

(url, timeout=10.0)

src/evaluate/utils/file_utils.py:313

↓ 2 callersFunctionget_authentication_headers_for_url

Handle the HF authentication

src/evaluate/utils/file_utils.py:225

↓ 2 callersFunctionget_datasets_user_agent

(user_agent: Optional[Union[str, dict]] = None)

src/evaluate/utils/file_utils.py:209

↓ 2 callersMethodget_hurtlex_category

(self, lemma)

measurements/honest/honest.py:147

↓ 2 callersFunctionget_imports

Find whether we should import or clone additional files for a given processing script. And list the import. We allow: - library depen

src/evaluate/loading.py:139

↓ 2 callersMethodget_module

(self)

src/evaluate/loading.py:528

↓ 2 callersFunctionget_prec_at_recall

Assumes recalls are sorted in increasing order

metrics/cuad/compute_score.py:123

↓ 2 callersFunctionget_raw_scores

(dataset, preds)

metrics/squad_v2/compute_score.py:105

↓ 2 callersFunctionget_supported_tasks

Returns a list of supported task strings.

src/evaluate/evaluator/__init__.py:82

↓ 2 callersFunctionget_tokens

(s)

metrics/squad_v2/compute_score.py:79

↓ 2 callersFunctionhash_url_to_filename

Convert `url` into a hashed filename in a repeatable way. If `etag` is specified, append its hash to the url's, delimited by a period.

src/evaluate/utils/file_utils.py:118

↓ 2 callersFunctionhf_hub_url

(path: str, name: str, revision: Optional[str] = None)

src/evaluate/utils/file_utils.py:99

↓ 2 callersFunctionhistogram_na_prob

(na_probs, qid_list, image_dir, name)

metrics/squad_v2/compute_score.py:234

↓ 2 callersMethodhonest_score_inner

(self, predicted_words)

measurements/honest/honest.py:150

next →1–100 of 785, ranked by callers