Class Metric

src/evaluate/module.py:782–809

A Metric is the base class and common API for all metrics. Args: config_name (`str`): This is used to define a hash specific to a metric computation script and prevents the metric's data to be overridden when the metric loading script is modified. keep_in

class Metric(EvaluationModule):
    """A Metric is the base class and common API for all metrics.

    Args:
        config_name (`str`):
            This is used to define a hash specific to a metric computation script and prevents the metric's data
            to be overridden when the metric loading script is modified.
        keep_in_memory (`bool`):
            Keep all predictions and references in memory. Not possible in distributed settings.
        cache_dir (`str`):
            Path to a directory in which temporary prediction/references data will be stored.
            The data directory should be located on a shared file-system in distributed setups.
        num_process (`int`):
            Specify the total number of nodes in a distributed settings.
            This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
        process_id (`int`):
            Specify the id of the current process in a distributed setup (between 0 and num_process-1)
            This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
        seed (`int`, *optional*):
            If specified, this will temporarily set numpy's random seed when [`~evaluate.Metric.compute`] is run.
        experiment_id (`str`):
            A specific experiment id. This is used if several distributed evaluations share the same file system.
            This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
        max_concurrent_cache_files (`int`):
            Max number of concurrent metric cache files (default `10000`).
        timeout (`Union[int, float]`):
            Timeout in second for distributed setting synchronization.
    """


class Comparison(EvaluationModule):