Takes a predicted answer and a gold answer (that are both either a string or a list of strings), and returns exact match and the DROP F1 metric for the prediction. If you are writing a script for evaluating objects in memory (say, the output of predictions during va
(self, predicted, gold)
| 158 | return {"em": max_em, "f1": max_f1} |
| 159 | |
| 160 | def get_metrics(self, predicted, gold): |
| 161 | """ |
| 162 | Takes a predicted answer and a gold answer (that are both either a string or a list of |
| 163 | strings), and returns exact match and the DROP F1 metric for the prediction. If you are |
| 164 | writing a script for evaluating objects in memory (say, the output of predictions during |
| 165 | validation, or while training), this is the function you want to call, after using |
| 166 | :func:`answer_json_to_strings` when reading the gold answer from the released data file. |
| 167 | """ |
| 168 | predicted_bags = self._answer_to_bags(predicted) |
| 169 | gold_bags = self._answer_to_bags(gold) |
| 170 | |
| 171 | if set(predicted_bags[0]) == set(gold_bags[0]) and len( |
| 172 | predicted_bags[0] |
| 173 | ) == len(gold_bags[0]): |
| 174 | exact_match = 1.0 |
| 175 | else: |
| 176 | exact_match = 0.0 |
| 177 | |
| 178 | f1_per_bag = self._align_bags(predicted_bags[1], gold_bags[1]) |
| 179 | f1 = np.mean(f1_per_bag) |
| 180 | f1 = round(f1, 2) |
| 181 | return exact_match, f1 |
| 182 | |
| 183 | def _answer_to_bags(self, answer): |
| 184 | if isinstance(answer, (list, tuple)): |
no test coverage detected