Method encode

tokenizations/bpe_tokenizer.py:100–104 · view source on GitHub ↗

text="...."

(self, text)

Source from the content-addressed store, hash-verified

98
99
100	def encode(self, text):
101	"""
102	text="...."
103	"""
104	return self.sp.EncodeAsIds(text)
105
106
107	def decode(self, tokens):

nothing calls this directly

no outgoing calls

no test coverage detected