Method get_vocab_size

nanochat/tokenizer.py:209–210

(self)

        return cls(enc, "<|endoftext|>")

    def get_vocab_size(self):
        return self.enc.n_vocab

    def get_special_tokens(self):
        return self.enc.special_tokens_set