MCPcopy
hub / github.com/huggingface/datasets / write_chunk

Function write_chunk

src/datasets/packaged_modules/webdataset/_tenbin.py:223–231  ·  view source on GitHub ↗

Write a byte chunk to the stream with magics, length, and padding.

(stream, buf)

Source from the content-addressed store, hash-verified

221
222
223def write_chunk(stream, buf):
224 """Write a byte chunk to the stream with magics, length, and padding."""
225 nbytes = bytelen(buf)
226 stream.write(magic_bytes)
227 stream.write(struct.pack("@q", nbytes))
228 stream.write(bytedata(buf))
229 padding = roundup(nbytes) - nbytes
230 if padding > 0:
231 stream.write(b"\0" * padding)
232
233
234def read_chunk(stream):

Callers 1

writeFunction · 0.85

Calls 4

bytelenFunction · 0.85
bytedataFunction · 0.85
roundupFunction · 0.85
writeMethod · 0.45

Tested by

no test coverage detected