Method tokenize

gpt_utils_systematicity.py:18–19 · view source on GitHub ↗

(self, text)

Source from the content-addressed store, hash-verified

16	self.token_pattern = re.compile(r'<[^>]+>')
17
18	def tokenize(self, text):
19	return self.token_pattern.findall(text)
20
21	def tokens_to_ids(self, tokens):
22	return [self.vocab[token] for token in tokens]

__getitem__Method · 0.95

no outgoing calls

no test coverage detected