Base class to store information about an evaluation used for `MetricInfo`, `ComparisonInfo`, and `MeasurementInfo`. `EvaluationModuleInfo` documents an evaluation, including its name, version, and features. See the constructor arguments and properties for a full list. Note: Not all
| 33 | |
| 34 | @dataclass |
| 35 | class EvaluationModuleInfo: |
| 36 | """Base class to store information about an evaluation used for `MetricInfo`, `ComparisonInfo`, |
| 37 | and `MeasurementInfo`. |
| 38 | |
| 39 | `EvaluationModuleInfo` documents an evaluation, including its name, version, and features. |
| 40 | See the constructor arguments and properties for a full list. |
| 41 | |
| 42 | Note: Not all fields are known on construction and may be updated later. |
| 43 | """ |
| 44 | |
| 45 | # Set in the dataset scripts |
| 46 | description: str |
| 47 | citation: str |
| 48 | features: Union[Features, List[Features]] |
| 49 | inputs_description: str = field(default_factory=str) |
| 50 | homepage: str = field(default_factory=str) |
| 51 | license: str = field(default_factory=str) |
| 52 | codebase_urls: List[str] = field(default_factory=list) |
| 53 | reference_urls: List[str] = field(default_factory=list) |
| 54 | streamable: bool = False |
| 55 | format: Optional[str] = None |
| 56 | module_type: str = "metric" # deprecate this in the future |
| 57 | |
| 58 | # Set later by the builder |
| 59 | module_name: Optional[str] = None |
| 60 | config_name: Optional[str] = None |
| 61 | experiment_id: Optional[str] = None |
| 62 | |
| 63 | def __post_init__(self): |
| 64 | if self.format is not None: |
| 65 | for key, value in self.features.items(): |
| 66 | if not isinstance(value, Value): |
| 67 | raise ValueError( |
| 68 | f"When using 'numpy' format, all features should be a `datasets.Value` feature. " |
| 69 | f"Here {key} is an instance of {value.__class__.__name__}" |
| 70 | ) |
| 71 | |
| 72 | def write_to_directory(self, metric_info_dir): |
| 73 | """Write `EvaluationModuleInfo` as JSON to `metric_info_dir`. |
| 74 | Also save the license separately in LICENSE. |
| 75 | |
| 76 | Args: |
| 77 | metric_info_dir (`str`): |
| 78 | The directory to save `metric_info_dir` to. |
| 79 | |
| 80 | Example: |
| 81 | |
| 82 | ```py |
| 83 | >>> my_metric.info.write_to_directory("/path/to/directory/") |
| 84 | ``` |
| 85 | """ |
| 86 | with open(os.path.join(metric_info_dir, config.METRIC_INFO_FILENAME), "w", encoding="utf-8") as f: |
| 87 | json.dump(asdict(self), f) |
| 88 | |
| 89 | with open(os.path.join(metric_info_dir, config.LICENSE_FILENAME), "w", encoding="utf-8") as f: |
| 90 | f.write(self.license) |
| 91 | |
| 92 | @classmethod |