MCPcopy Create free account
hub / github.com/google-research/language / parent

Function parent

language/table_text_eval/table_text_eval.py:304–447  ·  view source on GitHub ↗

Metric for comparing predictions to references given tables. Args: predictions: An iterator over tokenized predictions. Each prediction is a list. references: An iterator over lists of tokenized references. Each prediction can have multiple references. tables: An iterator

(predictions,
           references,
           tables,
           lambda_weight=0.5,
           smoothing=0.00001,
           max_order=4,
           entailment_fn=overlap_probability,
           mention_fn=_mention_probability)

Source from the content-addressed store, hash-verified

302
303
304def parent(predictions,
305 references,
306 tables,
307 lambda_weight=0.5,
308 smoothing=0.00001,
309 max_order=4,
310 entailment_fn=overlap_probability,
311 mention_fn=_mention_probability):
312 """Metric for comparing predictions to references given tables.
313
314 Args:
315 predictions: An iterator over tokenized predictions.
316 Each prediction is a list.
317 references: An iterator over lists of tokenized references.
318 Each prediction can have multiple references.
319 tables: An iterator over the tables. Each table is a list of tuples, where a
320 tuple can either be (attribute, value) pair or (head, relation, tail)
321 triple. The members of the tuples are assumed to be themselves tokenized
322 lists of strings. E.g.
323 `[(["name"], ["michael", "dahlquist"]),
324 (["birth", "date"], ["december", "22", "1965"])]`
325 is one table in the (attribute, value) format with two entries.
326 lambda_weight: Float weight in [0, 1] to multiply table recall.
327 smoothing: Float value for replace zero values of precision and recall.
328 max_order: Maximum order of the ngrams to use.
329 entailment_fn: A python function for computing the probability that an
330 ngram is entailed by the table. Its signature should match that of
331 `overlap_probability` above.
332 mention_fn: A python function for computing the probability that a
333 table entry is mentioned in the text. Its signature should
334 match that of `_mention_probability` above.
335
336 Returns:
337 precision: Average precision of all predictions.
338 recall: Average recall of all predictions.
339 f1: Average F-scores of all predictions.
340 all_f_scores: List of all F-scores for each item.
341 """
342 precisions, recalls, all_f_scores = [], [], []
343 reference_recalls, table_recalls = [], []
344 all_lambdas = []
345 for prediction, list_of_references, table in zip(
346 predictions, references, tables):
347 c_prec, c_rec, c_f = [], [], []
348 ref_rec, table_rec = [], []
349 for reference in list_of_references:
350 # Weighted ngram precisions and recalls for each order.
351 ngram_prec, ngram_rec = [], []
352 for order in range(1, max_order + 1):
353 # Collect n-grams and their entailment probabilities.
354 pred_ngram_counts = _ngram_counts(prediction, order)
355 pred_ngram_weights = {ngram: entailment_fn(ngram, table)
356 for ngram in pred_ngram_counts}
357 ref_ngram_counts = _ngram_counts(reference, order)
358 ref_ngram_weights = {ngram: entailment_fn(ngram, table)
359 for ngram in ref_ngram_counts}
360
361 # Precision.

Callers 1

mainFunction · 0.70

Calls 2

_ngram_countsFunction · 0.70
getMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…