MCPcopy Create free account
hub / github.com/dask/dask / iter_chunks

Function iter_chunks

dask/utils.py:1913–1938  ·  view source on GitHub ↗

Split sizes into chunks of total max_size each Parameters ---------- sizes : iterable of numbers The sizes to be chunked max_size : number Maximum total size per chunk. It must be greater or equal than each size in sizes

(sizes, max_size)

Source from the content-addressed store, hash-verified

1911
1912
1913def iter_chunks(sizes, max_size):
1914 """Split sizes into chunks of total max_size each
1915
1916 Parameters
1917 ----------
1918 sizes : iterable of numbers
1919 The sizes to be chunked
1920 max_size : number
1921 Maximum total size per chunk.
1922 It must be greater or equal than each size in sizes
1923 """
1924 chunk, chunk_sum = [], 0
1925 iter_sizes = iter(sizes)
1926 size = next(iter_sizes, None)
1927 while size is not None:
1928 assert size <= max_size
1929 if chunk_sum + size <= max_size:
1930 chunk.append(size)
1931 chunk_sum += size
1932 size = next(iter_sizes, None)
1933 else:
1934 assert chunk
1935 yield chunk
1936 chunk, chunk_sum = [], 0
1937 if chunk:
1938 yield chunk
1939
1940
1941hex_pattern = re.compile("[a-f]+")

Callers 3

repartition_sizeFunction · 0.90
_partition_boundariesMethod · 0.90
test_iter_chunksFunction · 0.90

Calls

no outgoing calls

Tested by 1

test_iter_chunksFunction · 0.72