MCPcopy Index your code
hub / github.com/togethercomputer/RedPajama-Data / write

Method write

app/src/utilities/io/writer.py:50–56  ·  view source on GitHub ↗
(self, data_obj: Dict[str, Any], flush: bool = False)

Source from the content-addressed store, hash-verified

48 )
49
50 def write(self, data_obj: Dict[str, Any], flush: bool = False):
51 self._encoder.encode_into(self._record(**data_obj), self._buffer)
52 self._buffer.extend(b"\n")
53 self._filehandle.write(self._buffer)
54
55 if flush:
56 self.flush()
57
58 def close(self):
59 self.flush()

Callers 9

__process_uriMethod · 0.95
runMethod · 0.95
runMethod · 0.95
runMethod · 0.95
_writer_workerMethod · 0.95
runMethod · 0.80
__write_train_chunkMethod · 0.80
create_bad_urls_indexFunction · 0.80
create_bad_words_listFunction · 0.80

Calls 1

flushMethod · 0.95

Tested by

no test coverage detected