Returns the probability that the given n-gram overlaps with the table. A simple implementation which checks how many tokens in the n-gram are also among the values in the table. For tables with (attribute, value) pairs on the `value` field is condidered. For tables with (head, relation, tail)
(ngram, table, smoothing=0.0, stopwords=None)
| 174 | |
| 175 | |
| 176 | def overlap_probability(ngram, table, smoothing=0.0, stopwords=None): |
| 177 | """Returns the probability that the given n-gram overlaps with the table. |
| 178 | |
| 179 | A simple implementation which checks how many tokens in the n-gram are also |
| 180 | among the values in the table. For tables with (attribute, value) pairs on the |
| 181 | `value` field is condidered. For tables with (head, relation, tail) triples a |
| 182 | concatenation of `head` and `tail` are considered. |
| 183 | |
| 184 | E.g.: |
| 185 | >>> overlap_probability(["michael", "dahlquist"], |
| 186 | [(["name"], ["michael", "dahlquist"])]) |
| 187 | >>> 1.0 |
| 188 | |
| 189 | Args: |
| 190 | ngram: List of tokens. |
| 191 | table: List of either (attribute, value) pairs or (head, relation, tail) |
| 192 | triples. Each member of the pair / triple is assumed to already be |
| 193 | tokenized into a list of strings. |
| 194 | smoothing: (Optional) Float parameter for laplace smoothing. |
| 195 | stopwords: (Optional) List of stopwords to ignore (assign P = 1). |
| 196 | |
| 197 | Returns: |
| 198 | prob: Float probability of ngram being entailed by the table. |
| 199 | """ |
| 200 | # pylint: disable=g-complex-comprehension |
| 201 | if len(table[0]) == 2: |
| 202 | table_values = set([tok for _, value in table for tok in value]) |
| 203 | else: |
| 204 | table_values = set([tok for head, _, tail in table for tok in head + tail]) |
| 205 | overlap = 0 |
| 206 | for token in ngram: |
| 207 | if stopwords is not None and token in stopwords: |
| 208 | overlap += 1 |
| 209 | continue |
| 210 | if token in table_values: |
| 211 | overlap += 1 |
| 212 | return float(overlap + smoothing) / float(len(ngram) + smoothing) |
| 213 | |
| 214 | |
| 215 | def _mention_probability(table_entry, sentence, smoothing=0.0): |
nothing calls this directly
no outgoing calls
no test coverage detected
searching dependent graphs…