MCPcopy Index your code
hub / github.com/deepspeedai/DeepSpeedExamples / SetTokenizer

Method SetTokenizer

Megatron-LM/data_utils/datasets.py:144–145  ·  view source on GitHub ↗
(self, tokenizer)

Source from the content-addressed store, hash-verified

142 return self.wrapped_data[self.split_inds[index]]
143
144 def SetTokenizer(self, tokenizer):
145 self.wrapped_data.SetTokenizer(tokenizer)
146
147 def GetTokenizer(self):
148 return self.wrapped_data.GetTokenizer()

Callers

nothing calls this directly

Calls 1

SetTokenizerMethod · 0.45

Tested by

no test coverage detected