Initialize Layer object. Parameters ---------- annotations : Mapping[str, Any], optional By default, None.

Utility to aggregate the statistics for N row-groups into a single dictionary. Used by `Engine._construct_parts`

Consolidate list of file-paths into parquet relative paths Note: This function was mostly copied from dask/fastparquet to use in ArrowEngine`

Assign value into a nested configuration dictionary Parameters ---------- keys : Sequence[str] The nested path of

bind operator to this class

Build the list of partition inputs and divisions for read_hdf

Return True if _map_blocks can be sped up via blockwise operations; False otherwise. FIXME this returns False for collections that wrap aroun

Treat the special annotation keys, as fusable since we can apply simple rules to capture their intent in a fused layer.

Categorize a dataframe with given categories df: DataFrame categories: dict mapping column name to iterable of categories

CHeck that _partitions is set to the expected value

Conditionally check if a warning is raised

pyarrow.dataset version of _collect_dataset_info Use pyarrow.dataset API to construct a dictionary of all general information needed t

Collect Parquet statistic for dataset paths

Collect statistics for a list of files and their corresponding fragments

Convert (r, g, b) triple to "#RRGGBB" string For use with ``visualize(color=...)`` Examples -------- >>> _colorize((255, 255, 255))

Utility function for auto_chunk, to fin how much larger or smaller the ideal chunk size is relative to what we have now.

For a given column, compute the min, max, and len of each partition. And make sure that the partitions are sorted relative to each other. NOT

Compute the rechunk of *x* to the given *chunks*.

Compute the rechunk of *x* to the given *chunks*.

pyarrow.dataset version of _construct_collection_plan Use dataset_info to construct the general plan for generating the output DataFra

Construct graph for a simple overlap operation.

Test whether ``columns_or_index`` contains a reference to the index of ``df This is the local (non-collection) version of ``dask.cor

Use parquet schema and hive-partition information (stored in dataset_info) to construct DataFrame metadata.

Determine the necessary dependencies to produce `output_blocks`. This method does not require graph materialization.

Helper function for derived_from to ease testing