MCPcopy Create free account
hub / github.com/THUDM/GLM / GetTokenizer

Method GetTokenizer

data_utils/lazy_loader.py:196–197  ·  view source on GitHub ↗
(self)

Source from the content-addressed store, hash-verified

194 self.map_fn = ProcessorTokenizer(tokenizer, self.process_fn)
195
196 def GetTokenizer(self):
197 return self._tokenizer
198
199 def __getitem__(self, index):
200 """

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected