MCPcopy Create free account
hub / github.com/dask/dask / auto_blocksize

Function auto_blocksize

dask/dataframe/io/csv.py:394–397  ·  view source on GitHub ↗
(total_memory, cpu_count)

Source from the content-addressed store, hash-verified

392
393
394def auto_blocksize(total_memory, cpu_count):
395 memory_factor = 10
396 blocksize = int(total_memory // cpu_count / memory_factor)
397 return min(blocksize, int(64e6))
398
399
400def _infer_block_size():

Callers 4

test_auto_blocksizeFunction · 0.90
test_auto_blocksize_max64mbFunction · 0.90
test_auto_blocksize_csvFunction · 0.90
_infer_block_sizeFunction · 0.85

Calls 1

minFunction · 0.85

Tested by 3

test_auto_blocksizeFunction · 0.72
test_auto_blocksize_max64mbFunction · 0.72
test_auto_blocksize_csvFunction · 0.72