Method encode

linear_moe/tokenizer/__init__.py:208–209

(self, text)

                return self.tokenizer.decode(token_ids)

            def encode(self, text):
                return self.tokenizer.encode(text)

            def decode(self, token_ids):
                return self.tokenizer.decode(token_ids)