Function blockwise

dask/blockwise.py:227–462

Create a Blockwise symbolic mutable mapping Applies a function, ``func``, across blocks from many different input collections. We arrange the pattern with which those blocks interact with sets of matching indices. E.g.:: blockwise(func, 'z', 'i', 'x', 'i', 'y', 'i') yiel

(
    func,
    output,
    output_indices,
    *arrind_pairs,
    numblocks,
    concatenate=None,
    new_axes=None,
    dependencies=(),
    _data_producer=False,
    **kwargs,
)

def blockwise(
    func,
    output,
    output_indices,
    *arrind_pairs,
    numblocks,
    concatenate=None,
    new_axes=None,
    dependencies=(),
    _data_producer=False,
    **kwargs,
):
    """Create a Blockwise symbolic mutable mapping

    Applies a function, ``func``, across blocks from many different input
    collections.  We arrange the pattern with which those blocks interact with
    sets of matching indices.  E.g.::

        blockwise(func, 'z', 'i', 'x', 'i', 'y', 'i')

    yield an embarrassingly parallel communication pattern and is read as

        $$ z_i = func(x_i, y_i) $$

    More complex patterns may emerge, including multiple indices::

        blockwise(func, 'z', 'ij', 'x', 'ij', 'y', 'ji')

        $$ z_{ij} = func(x_{ij}, y_{ji}) $$

    Indices missing in the output but present in the inputs results in many
    inputs being sent to one function (see examples).

    Examples
    --------
    Simple embarrassing map operation

    >>> def inc(x):
    ...    return x + 1
    >>> dict(
    ...     blockwise(
    ...         inc,
    ...         'z', 'ij',
    ...         'x', 'ij',
    ...         numblocks={'x': (2, 2)}
    ...     )
    ... )  # doctest: +NORMALIZE_WHITESPACE
    {('z', 0, 0): <Task ('z', 0, 0) inc(TaskRef(('x', 0, 0)))>,
    ('z', 0, 1): <Task ('z', 0, 1) inc(TaskRef(('x', 0, 1)))>,
    ('z', 1, 0): <Task ('z', 1, 0) inc(TaskRef(('x', 1, 0)))>,
    ('z', 1, 1): <Task ('z', 1, 1) inc(TaskRef(('x', 1, 1)))>}

    Simple operation on two datasets x and y

    >>> def add(x, y):
    ...     return x + y
    >>> dict(
    ...     blockwise(