MCPcopy Create free account
hub / github.com/dask/dask / create_merge_tree

Function create_merge_tree

dask/dataframe/partitionquantiles.py:202–238  ·  view source on GitHub ↗

Create a task tree that merges all the keys with a reduction function. Parameters ---------- func: callable Reduction function that accepts a single list of values to reduce. keys: iterable Keys to reduce from the source dask graph. token: object Included

(func, keys, token, level=0)

Source from the content-addressed store, hash-verified

200
201
202def create_merge_tree(func, keys, token, level=0):
203 """Create a task tree that merges all the keys with a reduction function.
204
205 Parameters
206 ----------
207 func: callable
208 Reduction function that accepts a single list of values to reduce.
209 keys: iterable
210 Keys to reduce from the source dask graph.
211 token: object
212 Included in each key of the returned dict.
213 level: int, default 0
214 The token-level to begin with.
215
216 This creates a k-ary tree where k depends on the current level and is
217 greater the further away a node is from the root node. This reduces the
218 total number of nodes (thereby reducing scheduler overhead), but still
219 has beneficial properties of trees.
220
221 For reasonable numbers of keys, N < 1e5, the total number of nodes in the
222 tree is roughly ``N**0.78``. For 1e5 < N < 2e5, is it roughly ``N**0.8``.
223 """
224 prev_width = len(keys)
225 prev_keys = iter(keys)
226 rv = {}
227 while prev_width > 1:
228 width = tree_width(prev_width)
229 groups = tree_groups(prev_width, width)
230 keys = [(token, level, i) for i in range(width)]
231
232 for num, key in zip(groups, keys):
233 rv[key] = (func, list(take(num, prev_keys)))
234
235 prev_width = width
236 prev_keys = iter(keys)
237 level += 1
238 return rv
239
240
241def percentiles_to_weights(qs, vals, length):

Callers 1

_layerMethod · 0.90

Calls 3

tree_widthFunction · 0.85
tree_groupsFunction · 0.85
takeFunction · 0.50

Tested by

no test coverage detected