Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string associated to the token - end: index of the character follo
(self, batch_or_token_index: int, token_index: Optional[int] = None)
| 319 | return TokenSpan(*(self._encodings[batch_index].word_to_tokens(word_index))) |
| 320 | |
| 321 | def token_to_chars(self, batch_or_token_index: int, token_index: Optional[int] = None) -> CharSpan: |
| 322 | """ |
| 323 | Get the character span corresponding to an encoded token in a sequence of the batch. |
| 324 | |
| 325 | Character spans are returned as a CharSpan NamedTuple with: |
| 326 | |
| 327 | - start: index of the first character in the original string associated to the token |
| 328 | - end: index of the character following the last character in the original string associated to the token |
| 329 | |
| 330 | Can be called as: |
| 331 | |
| 332 | - ``self.token_to_chars(token_index)`` if batch size is 1 |
| 333 | - ``self.token_to_chars(batch_index, token_index)`` if batch size is greater or equal to 1 |
| 334 | |
| 335 | Args: |
| 336 | batch_or_token_index (:obj:`int`): |
| 337 | Index of the sequence in the batch. If the batch only comprise one sequence, |
| 338 | this can be the index of the token in the sequence |
| 339 | token_index (:obj:`int`, `optional`): |
| 340 | If a batch index is provided in `batch_or_token_index`, this can be the index |
| 341 | of the token or tokens in the sequence. |
| 342 | |
| 343 | Returns: |
| 344 | :obj:`CharSpan`: |
| 345 | Span of characters in the original string. |
| 346 | |
| 347 | :obj:`CharSpan` are NamedTuple with: |
| 348 | |
| 349 | - start: index of the first character in the original string |
| 350 | - end: index of the character following the last character in the original string |
| 351 | """ |
| 352 | |
| 353 | if not self._encodings: |
| 354 | raise ValueError("token_to_chars() is not available when using Python based tokenizers") |
| 355 | if token_index is not None: |
| 356 | batch_index = batch_or_token_index |
| 357 | else: |
| 358 | batch_index = 0 |
| 359 | token_index = batch_or_token_index |
| 360 | return CharSpan(*(self._encodings[batch_index].token_to_chars(token_index))) |
| 361 | |
| 362 | def char_to_token(self, batch_or_char_index: int, char_index: Optional[int] = None) -> int: |
| 363 | """ |