Method tokenize

codegeex/megatron/tokenizer/tokenizer.py:206–207

(self, text)

        return self.tokenizer.decoder

    def tokenize(self, text):
        return self.tokenizer.encode(text)

    def detokenize(self, token_ids):
        return self.tokenizer.decode(token_ids)