Yield tuples of automaton matching positions as (qstart, qend) from matching the ``tokens`` sequence of query token ids starting at the `qbegin` absolute query start position position using the `automaton`.
(
tokens,
qbegin,
automaton,
unknown_ngram_length=UNKNOWN_NGRAM_LENGTH,
)
| 240 | |
| 241 | |
| 242 | def get_matched_ngrams( |
| 243 | tokens, |
| 244 | qbegin, |
| 245 | automaton, |
| 246 | unknown_ngram_length=UNKNOWN_NGRAM_LENGTH, |
| 247 | ): |
| 248 | """ |
| 249 | Yield tuples of automaton matching positions as (qstart, qend) |
| 250 | from matching the ``tokens`` sequence of query token ids starting at the |
| 251 | `qbegin` absolute query start position position using the `automaton`. |
| 252 | """ |
| 253 | # iterate over matched strings: the matched value is the matching ngram |
| 254 | # which is an n-tuple of token ids |
| 255 | qtokens = tuple(tokens) |
| 256 | offset = unknown_ngram_length - 1 |
| 257 | for qend, _ in automaton.iter(qtokens): |
| 258 | qend = qbegin + qend |
| 259 | qstart = qend - offset |
| 260 | yield qstart, qend |