Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing ``self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))``. Args:
(
self, token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True
)
| 2254 | return [self.decode(seq, **kwargs) for seq in sequences] |
| 2255 | |
| 2256 | def decode( |
| 2257 | self, token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True |
| 2258 | ) -> str: |
| 2259 | """ |
| 2260 | Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary |
| 2261 | with options to remove special tokens and clean up tokenization spaces. |
| 2262 | Similar to doing ``self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))``. |
| 2263 | |
| 2264 | Args: |
| 2265 | token_ids: list of tokenized input ids. Can be obtained using the `encode` or `encode_plus` methods. |
| 2266 | skip_special_tokens: if set to True, will replace special tokens. |
| 2267 | clean_up_tokenization_spaces: if set to True, will clean up the tokenization spaces. |
| 2268 | """ |
| 2269 | raise NotImplementedError |
| 2270 | |
| 2271 | def get_special_tokens_mask( |
| 2272 | self, token_ids_0: List, token_ids_1: Optional[List] = None, already_has_special_tokens: bool = False |