Method encode

tokenizations/bpe_tokenizer.py:79–80 · view source on GitHub ↗

(self, text)

Source from the content-addressed store, hash-verified

77	return word
78
79	def encode(self, text):
80	return [self.encoder.get(token, 1) for token in self.tokenize(text)]
81
82	def decode(self, tokens):
83	text = ''.join([self.decoder[token] for token in tokens])

nothing calls this directly

tokenizeMethod · 0.95

no test coverage detected