Method exists

Megatron-LM/data_utils/tokenization.py:416–418

check if the filepath for a text tokenizer exists

(model_path)

    @staticmethod
    def exists(model_path):
        """check if the filepath for a text tokenizer exists"""
        raise NotImplementedError('TextTokenizer exists method not implemented')

    def Train(self, corpus):
        """train a tokenizer on a data corpus and save model for future use"""