Method tokenize

linear_moe/tokenizer/__init__.py:263–264

(self, text)

                return self.tokenizer.decoder

            def tokenize(self, text):
                return self.tokenizer.encode(text)

            def detokenize(self, token_ids):
                return self.tokenizer.decode(token_ids)