Method deduplicate

python/pathway/internals/table.py:1315–1413

Deduplicates rows in `self` on `value` column using acceptor function. It keeps rows which where accepted by the acceptor function. Acceptor operates on two arguments - *CURRENT* value and *PREVIOUS* value. Args: value: column expression used for deduplication.

(
        self,
        *,
        value: expr.ColumnExpression | Value,
        instance: expr.ColumnExpression | None = None,
        acceptor: Callable[[T, T], bool],
        name: str | None = None,
    )

    @check_arg_types
    @contextualized_operator
    def deduplicate(
        self,
        *,
        value: expr.ColumnExpression | Value,
        instance: expr.ColumnExpression | None = None,
        acceptor: Callable[[T, T], bool],
        name: str | None = None,
    ) -> Table:
        """Deduplicates rows in `self` on `value` column using acceptor function.

        It keeps rows which where accepted by the acceptor function.
        Acceptor operates on two arguments - *CURRENT* value and *PREVIOUS* value.

        Args:
            value: column expression used for deduplication.
            instance: Grouping column. For rows with different
                values in this column, deduplication will be performed separately.
                Defaults to None.
            acceptor: callback telling whether two values are different.
            name:  An identifier, under which the state of the table
                will be persisted or ``None``, if there is no need to persist the state of this table.
                When a program restarts, it restores the state for all input tables according to what
                was saved for their ``name``. This way it's possible to configure the start of
                computations from the moment they were terminated last time.

        Returns:
            Table: the result of deduplication.

        Example:

        >>> import pathway as pw
        >>> table = pw.debug.table_from_markdown(
        ...     '''
        ...     val | __time__
        ...      1  |     2
        ...      2  |     4
        ...      3  |     6
        ...      4  |     8
        ... '''
        ... )
        >>>
        >>> def acceptor(new_value, old_value) -> bool:
        ...     return new_value >= old_value + 2
        ...
        >>>
        >>> result = table.deduplicate(value=pw.this.val, acceptor=acceptor)
        >>> pw.debug.compute_and_print_update_stream(result, include_id=False)
        val | __time__ | __diff__
        1   | 2        | 1
        1   | 6        | -1
        3   | 6        | 1
        >>>
        >>> table = pw.debug.table_from_markdown(
        ...     '''
        ...     val | instance | __time__
        ...      1  |     1    |     2
        ...      2  |     1    |     4
        ...      3  |     2    |     6