Method repartition

python/ray/data/dataset.py:1756–1893

Repartition the :class:`Dataset` into exactly this number of :ref:`blocks `. This method can be useful to tune the performance of your pipeline. To learn more, see :ref:`Advanced: Performance Tips and Tuning `. If you're writi

(
        self,
        num_blocks: Optional[int] = None,
        target_num_rows_per_block: Optional[int] = None,
        *,
        strict: bool = False,
        shuffle: bool = False,
        keys: Optional[List[str]] = None,
        sort: bool = False,
    )

    @PublicAPI(api_group=SSR_API_GROUP)
    def repartition(
        self,
        num_blocks: Optional[int] = None,
        target_num_rows_per_block: Optional[int] = None,
        *,
        strict: bool = False,
        shuffle: bool = False,
        keys: Optional[List[str]] = None,
        sort: bool = False,
    ) -> "Dataset":
        """Repartition the :class:`Dataset` into exactly this number of
        :ref:`blocks <dataset_concept>`.

        This method can be useful to tune the performance of your pipeline. To learn
        more, see :ref:`Advanced: Performance Tips and Tuning <data_performance_tips>`.

        If you&#x27;re writing data to files, you can also use this method to change the
        number of output files. To learn more, see
        :ref:`Changing the number of output files <changing-number-output-files>`.

        .. note::

            Repartition has three modes:

             * When ``num_blocks`` and ``shuffle=True`` are specified Ray Data performs a full distributed shuffle producing exactly ``num_blocks`` blocks.
             * When ``num_blocks`` and ``shuffle=False`` are specified, Ray Data does NOT perform full shuffle, instead opting in for splitting and combining of the blocks attempting to minimize the necessary data movement (relative to full-blown shuffle). Exactly ``num_blocks`` will be produced.
             * If ``target_num_rows_per_block`` is set (exclusive with ``num_blocks`` and ``shuffle``), streaming repartitioning will be executed, where blocks will be made to carry no more than ``target_num_rows_per_block`` rows. Smaller blocks will be combined into bigger ones up to ``target_num_rows_per_block`` as well.

            .. image:: /data/images/dataset-shuffle.svg
                :align: center

            ..
                https://docs.google.com/drawings/d/132jhE3KXZsf29ho1yUdPrCHB9uheHBWHJhDQMXqIVPA/edit

        Examples:
            >>> import ray
            >>> ds = ray.data.range(100).repartition(10).materialize()
            >>> ds.num_blocks()
            10

        Time complexity: O(dataset size / parallelism)

        Args:
            num_blocks: Number of blocks after repartitioning.
            target_num_rows_per_block: [Experimental] The target number of rows per block to
                repartition. Performs streaming repartitioning of the dataset (no shuffling).
                Note that either `num_blocks` or
                `target_num_rows_per_block` must be set, but not both. When
                `target_num_rows_per_block` is set, it only repartitions
                :class:`Dataset` :ref:`blocks <dataset_concept>` that are larger than
                `target_num_rows_per_block`. Note that the system will internally
                figure out the number of rows per :ref:`blocks <dataset_concept>` for
                optimal execution, based on the `target_num_rows_per_block`. This is
                the current behavior because of the implementation and may change in
                the future.
            strict: If ``True``, ``repartition`` guarantees that all output blocks,
                except for the last one, will have exactly ``target_num_rows_per_block`` rows.
                If ``False``, ``repartition`` uses best-effort bundling and may produce at most