MCPcopy Create free account
hub / github.com/zai-org/CodeGeeX / __init__

Method __init__

codegeex/megatron/data/indexed_dataset.py:296–303  ·  view source on GitHub ↗
(self, out_file, dtype=np.int32)

Source from the content-addressed store, hash-verified

294 }
295
296 def __init__(self, out_file, dtype=np.int32):
297 self.out_file = open(out_file, "wb")
298 self.dtype = dtype
299 self.data_offsets = [0]
300 self.dim_offsets = [0]
301 self.sizes = []
302 self.element_size = self.element_sizes[self.dtype]
303 self.doc_idx = [0]
304
305 def add_item(self, tensor):
306 bytes = self.out_file.write(np.array(tensor.numpy(), dtype=self.dtype))

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected