MCPcopy Create free account
hub / github.com/google-research/language / overlap_probability

Function overlap_probability

language/table_text_eval/table_text_eval.py:176–212  ·  view source on GitHub ↗

Returns the probability that the given n-gram overlaps with the table. A simple implementation which checks how many tokens in the n-gram are also among the values in the table. For tables with (attribute, value) pairs on the `value` field is condidered. For tables with (head, relation, tail)

(ngram, table, smoothing=0.0, stopwords=None)

Source from the content-addressed store, hash-verified

174
175
176def overlap_probability(ngram, table, smoothing=0.0, stopwords=None):
177 """Returns the probability that the given n-gram overlaps with the table.
178
179 A simple implementation which checks how many tokens in the n-gram are also
180 among the values in the table. For tables with (attribute, value) pairs on the
181 `value` field is condidered. For tables with (head, relation, tail) triples a
182 concatenation of `head` and `tail` are considered.
183
184 E.g.:
185 >>> overlap_probability(["michael", "dahlquist"],
186 [(["name"], ["michael", "dahlquist"])])
187 >>> 1.0
188
189 Args:
190 ngram: List of tokens.
191 table: List of either (attribute, value) pairs or (head, relation, tail)
192 triples. Each member of the pair / triple is assumed to already be
193 tokenized into a list of strings.
194 smoothing: (Optional) Float parameter for laplace smoothing.
195 stopwords: (Optional) List of stopwords to ignore (assign P = 1).
196
197 Returns:
198 prob: Float probability of ngram being entailed by the table.
199 """
200 # pylint: disable=g-complex-comprehension
201 if len(table[0]) == 2:
202 table_values = set([tok for _, value in table for tok in value])
203 else:
204 table_values = set([tok for head, _, tail in table for tok in head + tail])
205 overlap = 0
206 for token in ngram:
207 if stopwords is not None and token in stopwords:
208 overlap += 1
209 continue
210 if token in table_values:
211 overlap += 1
212 return float(overlap + smoothing) / float(len(ngram) + smoothing)
213
214
215def _mention_probability(table_entry, sentence, smoothing=0.0):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…