A simple, unsmoothed N-gram model. Parameters ---------- N : int The maximum length (in words) of the context-window to use in the langauge model. Model will compute all n-grams from 1, ..., N. unk : bool Whether to includ
(self, N, unk=True, filter_stopwords=True, filter_punctuation=True)
| 312 | |
| 313 | class MLENGram(NGramBase): |
| 314 | def __init__(self, N, unk=True, filter_stopwords=True, filter_punctuation=True): |
| 315 | """ |
| 316 | A simple, unsmoothed N-gram model. |
| 317 | |
| 318 | Parameters |
| 319 | ---------- |
| 320 | N : int |
| 321 | The maximum length (in words) of the context-window to use in the |
| 322 | langauge model. Model will compute all n-grams from 1, ..., N. |
| 323 | unk : bool |
| 324 | Whether to include the ``<unk>`` (unknown) token in the LM. Default |
| 325 | is True. |
| 326 | filter_stopwords : bool |
| 327 | Whether to remove stopwords before training. Default is True. |
| 328 | filter_punctuation : bool |
| 329 | Whether to remove punctuation before training. Default is True. |
| 330 | """ |
| 331 | super().__init__(N, unk, filter_stopwords, filter_punctuation) |
| 332 | |
| 333 | self.hyperparameters["id"] = "MLENGram" |
| 334 | |
| 335 | def log_prob(self, words, N): |
| 336 | """ |