MCPcopy Create free account
hub / github.com/zai-org/CodeGeeX / num_tokens

Method num_tokens

codegeex/megatron/data/indexed_dataset.py:216–217  ·  view source on GitHub ↗
(self, index)

Source from the content-addressed store, hash-verified

214 return self._len
215
216 def num_tokens(self, index):
217 return self.sizes[index]
218
219 def size(self, index):
220 return self.sizes[index]

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected