Method detokenize

tokenizer.py:37–38

(self, token_ids: List[int])

        return self.tokenizer.encode(text)

    def detokenize(self, token_ids: List[int]) -> str:
        return self.tokenizer.decode(token_ids, skip_special_tokens=False)