just concatenates ascii tokens into text
(self, Tokens)
| 605 | return ''.join([self.IdToToken(tok) for tok in Ids]) |
| 606 | |
| 607 | def DecodeTokens(self, Tokens): |
| 608 | """just concatenates ascii tokens into text""" |
| 609 | if isinstance(Tokens, Tokenization): |
| 610 | Tokens = Tokens.tokenization |
| 611 | return ''.join(Tokens) |
| 612 | |
| 613 | |
| 614 | MAX_SENTENCEPIECE_SENTENCES = 100000000 |
nothing calls this directly
no outgoing calls
no test coverage detected