Metric for comparing predictions to references given tables. Args: predictions: An iterator over tokenized predictions. Each prediction is a list. references: An iterator over lists of tokenized references. Each prediction can have multiple references. tables: An iterator
(predictions,
references,
tables,
lambda_weight=0.5,
smoothing=0.00001,
max_order=4,
entailment_fn=overlap_probability,
mention_fn=_mention_probability)
| 302 | |
| 303 | |
| 304 | def parent(predictions, |
| 305 | references, |
| 306 | tables, |
| 307 | lambda_weight=0.5, |
| 308 | smoothing=0.00001, |
| 309 | max_order=4, |
| 310 | entailment_fn=overlap_probability, |
| 311 | mention_fn=_mention_probability): |
| 312 | """Metric for comparing predictions to references given tables. |
| 313 | |
| 314 | Args: |
| 315 | predictions: An iterator over tokenized predictions. |
| 316 | Each prediction is a list. |
| 317 | references: An iterator over lists of tokenized references. |
| 318 | Each prediction can have multiple references. |
| 319 | tables: An iterator over the tables. Each table is a list of tuples, where a |
| 320 | tuple can either be (attribute, value) pair or (head, relation, tail) |
| 321 | triple. The members of the tuples are assumed to be themselves tokenized |
| 322 | lists of strings. E.g. |
| 323 | `[(["name"], ["michael", "dahlquist"]), |
| 324 | (["birth", "date"], ["december", "22", "1965"])]` |
| 325 | is one table in the (attribute, value) format with two entries. |
| 326 | lambda_weight: Float weight in [0, 1] to multiply table recall. |
| 327 | smoothing: Float value for replace zero values of precision and recall. |
| 328 | max_order: Maximum order of the ngrams to use. |
| 329 | entailment_fn: A python function for computing the probability that an |
| 330 | ngram is entailed by the table. Its signature should match that of |
| 331 | `overlap_probability` above. |
| 332 | mention_fn: A python function for computing the probability that a |
| 333 | table entry is mentioned in the text. Its signature should |
| 334 | match that of `_mention_probability` above. |
| 335 | |
| 336 | Returns: |
| 337 | precision: Average precision of all predictions. |
| 338 | recall: Average recall of all predictions. |
| 339 | f1: Average F-scores of all predictions. |
| 340 | all_f_scores: List of all F-scores for each item. |
| 341 | """ |
| 342 | precisions, recalls, all_f_scores = [], [], [] |
| 343 | reference_recalls, table_recalls = [], [] |
| 344 | all_lambdas = [] |
| 345 | for prediction, list_of_references, table in zip( |
| 346 | predictions, references, tables): |
| 347 | c_prec, c_rec, c_f = [], [], [] |
| 348 | ref_rec, table_rec = [], [] |
| 349 | for reference in list_of_references: |
| 350 | # Weighted ngram precisions and recalls for each order. |
| 351 | ngram_prec, ngram_rec = [], [] |
| 352 | for order in range(1, max_order + 1): |
| 353 | # Collect n-grams and their entailment probabilities. |
| 354 | pred_ngram_counts = _ngram_counts(prediction, order) |
| 355 | pred_ngram_weights = {ngram: entailment_fn(ngram, table) |
| 356 | for ngram in pred_ngram_counts} |
| 357 | ref_ngram_counts = _ngram_counts(reference, order) |
| 358 | ref_ngram_weights = {ngram: entailment_fn(ngram, table) |
| 359 | for ngram in ref_ngram_counts} |
| 360 | |
| 361 | # Precision. |
no test coverage detected
searching dependent graphs…