MCPcopy Create free account
hub / github.com/OSU-NLP-Group/Loop-Think-Generalize / tokenize

Method tokenize

gpt_utils_extrapolation.py:21–22  ·  view source on GitHub ↗
(self, text)

Source from the content-addressed store, hash-verified

19 self.token_pattern = re.compile(r'<[^>]+>')
20
21 def tokenize(self, text):
22 return self.token_pattern.findall(text)
23
24 def tokens_to_ids(self, tokens):
25 return [self.vocab[token] for token in tokens]

Callers 1

__getitem__Method · 0.95

Calls

no outgoing calls

Tested by

no test coverage detected