MCPcopy Create free account
hub / github.com/THUDM/GLM / SetTokenizer

Method SetTokenizer

data_utils/datasets.py:170–171  ·  view source on GitHub ↗
(self, tokenizer)

Source from the content-addressed store, hash-verified

168 return self.wrapped_data[self.split_inds[index]]
169
170 def SetTokenizer(self, tokenizer):
171 self.wrapped_data.SetTokenizer(tokenizer)
172
173 def GetTokenizer(self):
174 return self.wrapped_data.GetTokenizer()

Callers

nothing calls this directly

Calls 1

SetTokenizerMethod · 0.45

Tested by

no test coverage detected