MCPcopy Create free account
hub / github.com/zai-org/CodeGeeX / read_data

Method read_data

codegeex/megatron/data/indexed_dataset.py:177–178  ·  view source on GitHub ↗
(self, path)

Source from the content-addressed store, hash-verified

175 self.doc_idx = read_longs(f, self.doc_count)
176
177 def read_data(self, path):
178 self.data_file = open(data_file_path(path), "rb", buffering=0)
179
180 def check_index(self, i):
181 if i < 0 or i >= self._len:

Callers 2

__getitem__Method · 0.95
prefetchMethod · 0.80

Calls 1

data_file_pathFunction · 0.85

Tested by

no test coverage detected