Method encode

tools/preprocess_data_nmt.py:26–30

(self, text)

        Encoder.tokenizer = build_tokenizer(self.args)

    def encode(self, text):
        ids = {}
        ids = Encoder.tokenizer.tokenize(text)
        assert len(ids) > 0
        return ids, len(text)


def get_args():