Normalize user provided arguments to blockwise or map_blocks We do a few things: 1. If they are string literals that might collide with blockwise_token then we quote them 2. IF they are large (as defined by sizeof) then we put them into the graph on their own by using
(x)
| 514 | |
| 515 | |
| 516 | def normalize_arg(x): |
| 517 | """Normalize user provided arguments to blockwise or map_blocks |
| 518 | |
| 519 | We do a few things: |
| 520 | |
| 521 | 1. If they are string literals that might collide with blockwise_token then we |
| 522 | quote them |
| 523 | 2. IF they are large (as defined by sizeof) then we put them into the |
| 524 | graph on their own by using dask.delayed |
| 525 | """ |
| 526 | if is_dask_collection(x): |
| 527 | return x |
| 528 | elif isinstance(x, str) and re.match(r"_\d+", x): |
| 529 | return delayed(x) |
| 530 | elif isinstance(x, list) and len(x) >= 10: |
| 531 | return delayed(x) |
| 532 | elif sizeof(x) > 1e6: |
| 533 | return delayed(x) |
| 534 | else: |
| 535 | return x |
| 536 | |
| 537 | |
| 538 | def _pass_extra_kwargs(func, keys, *args, **kwargs): |
no test coverage detected