Method decode

utils/tokenizer.py:43–44

(self, tokens: List[int])

        return self._encoding.encode(text)

    def decode(self, tokens: List[int]) -> str:
        return self._encoding.decode(tokens)


class CharEstimationTokenizer: