Create a task tree that merges all the keys with a reduction function. Parameters ---------- func: callable Reduction function that accepts a single list of values to reduce. keys: iterable Keys to reduce from the source dask graph. token: object Included
(func, keys, token, level=0)
| 200 | |
| 201 | |
| 202 | def create_merge_tree(func, keys, token, level=0): |
| 203 | """Create a task tree that merges all the keys with a reduction function. |
| 204 | |
| 205 | Parameters |
| 206 | ---------- |
| 207 | func: callable |
| 208 | Reduction function that accepts a single list of values to reduce. |
| 209 | keys: iterable |
| 210 | Keys to reduce from the source dask graph. |
| 211 | token: object |
| 212 | Included in each key of the returned dict. |
| 213 | level: int, default 0 |
| 214 | The token-level to begin with. |
| 215 | |
| 216 | This creates a k-ary tree where k depends on the current level and is |
| 217 | greater the further away a node is from the root node. This reduces the |
| 218 | total number of nodes (thereby reducing scheduler overhead), but still |
| 219 | has beneficial properties of trees. |
| 220 | |
| 221 | For reasonable numbers of keys, N < 1e5, the total number of nodes in the |
| 222 | tree is roughly ``N**0.78``. For 1e5 < N < 2e5, is it roughly ``N**0.8``. |
| 223 | """ |
| 224 | prev_width = len(keys) |
| 225 | prev_keys = iter(keys) |
| 226 | rv = {} |
| 227 | while prev_width > 1: |
| 228 | width = tree_width(prev_width) |
| 229 | groups = tree_groups(prev_width, width) |
| 230 | keys = [(token, level, i) for i in range(width)] |
| 231 | |
| 232 | for num, key in zip(groups, keys): |
| 233 | rv[key] = (func, list(take(num, prev_keys))) |
| 234 | |
| 235 | prev_width = width |
| 236 | prev_keys = iter(keys) |
| 237 | level += 1 |
| 238 | return rv |
| 239 | |
| 240 | |
| 241 | def percentiles_to_weights(qs, vals, length): |
no test coverage detected