Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character foll
(self, batch_or_word_index: int, word_index: Optional[int] = None)
| 396 | return self._encodings[batch_index].char_to_token(char_index) |
| 397 | |
| 398 | def word_to_chars(self, batch_or_word_index: int, word_index: Optional[int] = None) -> CharSpan: |
| 399 | """ |
| 400 | Get the character span in the original string corresponding to given word in a sequence |
| 401 | of the batch. |
| 402 | |
| 403 | Character spans are returned as a CharSpan NamedTuple with: |
| 404 | |
| 405 | - start: index of the first character in the original string |
| 406 | - end: index of the character following the last character in the original string |
| 407 | |
| 408 | Can be called as: |
| 409 | |
| 410 | - ``self.word_to_chars(word_index)`` if batch size is 1 |
| 411 | - ``self.word_to_chars(batch_index, word_index)`` if batch size is greater or equal to 1 |
| 412 | |
| 413 | Args: |
| 414 | batch_or_word_index (:obj:`int`): |
| 415 | Index of the sequence in the batch. If the batch only comprise one sequence, |
| 416 | this can be the index of the word in the sequence |
| 417 | word_index (:obj:`int`, `optional`): |
| 418 | If a batch index is provided in `batch_or_token_index`, this can be the index |
| 419 | of the word in the sequence. |
| 420 | |
| 421 | Returns: |
| 422 | :obj:`CharSpan` or :obj:`List[CharSpan]`: |
| 423 | Span(s) of the associated character or characters in the string. |
| 424 | CharSpan are NamedTuple with: |
| 425 | |
| 426 | - start: index of the first character associated to the token in the original string |
| 427 | - end: index of the character following the last character associated to the token in the original string |
| 428 | """ |
| 429 | |
| 430 | if not self._encodings: |
| 431 | raise ValueError("word_to_chars() is not available when using Python based tokenizers") |
| 432 | if word_index is not None: |
| 433 | batch_index = batch_or_word_index |
| 434 | else: |
| 435 | batch_index = 0 |
| 436 | word_index = batch_or_word_index |
| 437 | return CharSpan(*(self._encodings[batch_index].word_to_chars(word_index))) |
| 438 | |
| 439 | def char_to_word(self, batch_or_char_index: int, char_index: Optional[int] = None) -> int: |
| 440 | """ |