Get the encoded token span corresponding to a word in the sequence of the batch. Token spans are returned as a TokenSpan NamedTuple with: - start: index of the first token - end: index of the token following the last token Can be called as: - ``se
(self, batch_or_word_index: int, word_index: Optional[int] = None)
| 270 | return self._encodings[batch_index].token_to_word(token_index) |
| 271 | |
| 272 | def word_to_tokens(self, batch_or_word_index: int, word_index: Optional[int] = None) -> TokenSpan: |
| 273 | """ |
| 274 | Get the encoded token span corresponding to a word in the sequence of the batch. |
| 275 | |
| 276 | Token spans are returned as a TokenSpan NamedTuple with: |
| 277 | |
| 278 | - start: index of the first token |
| 279 | - end: index of the token following the last token |
| 280 | |
| 281 | Can be called as: |
| 282 | |
| 283 | - ``self.word_to_tokens(word_index)`` if batch size is 1 |
| 284 | - ``self.word_to_tokens(batch_index, word_index)`` if batch size is greater or equal to 1 |
| 285 | |
| 286 | This method is particularly suited when the input sequences are provided as |
| 287 | pre-tokenized sequences (i.e. words are defined by the user). In this case it allows |
| 288 | to easily associate encoded tokens with provided tokenized words. |
| 289 | |
| 290 | Args: |
| 291 | batch_or_word_index (:obj:`int`): |
| 292 | Index of the sequence in the batch. If the batch only comprises one sequence, |
| 293 | this can be the index of the word in the sequence |
| 294 | word_index (:obj:`int`, `optional`): |
| 295 | If a batch index is provided in `batch_or_token_index`, this can be the index |
| 296 | of the word in the sequence. |
| 297 | |
| 298 | Returns: |
| 299 | :obj:`TokenSpan`: |
| 300 | Span of tokens in the encoded sequence. |
| 301 | |
| 302 | :obj:`TokenSpan` are NamedTuple with: |
| 303 | |
| 304 | - start: index of the first token |
| 305 | - end: index of the token following the last token |
| 306 | """ |
| 307 | |
| 308 | if not self._encodings: |
| 309 | raise ValueError("word_to_tokens() is not available when using Python based tokenizers") |
| 310 | if word_index is not None: |
| 311 | batch_index = batch_or_word_index |
| 312 | else: |
| 313 | batch_index = 0 |
| 314 | word_index = batch_or_word_index |
| 315 | if batch_index < 0: |
| 316 | batch_index = self._batch_size + batch_index |
| 317 | if word_index < 0: |
| 318 | word_index = self._seq_len + word_index |
| 319 | return TokenSpan(*(self._encodings[batch_index].word_to_tokens(word_index))) |
| 320 | |
| 321 | def token_to_chars(self, batch_or_token_index: int, token_index: Optional[int] = None) -> CharSpan: |
| 322 | """ |