Method forget

python/pathway/internals/table.py:671–755

Remove old entries when they start to satisfy ``time_column <= max(time_column) - threshold``. This operator is useful for removing old entries from the stateful operators downstream (like joins, groupbys etc.). It stores the entries and when the current time (defined as max

(
        self,
        time_column: expr.ColumnExpression,
        threshold: IntervalType,
        mark_forgetting_records: bool = False,
    )

    @trace_user_frame
    @desugar
    def forget(
        self,
        time_column: expr.ColumnExpression,
        threshold: IntervalType,
        mark_forgetting_records: bool = False,
    ) -> Table[TSchema]:
        """Remove old entries when they start to satisfy ``time_column <= max(time_column) - threshold``.

        This operator is useful for removing old entries from the stateful operators
        downstream (like joins, groupbys etc.). It stores the entries and when the
        current time (defined as max over all ``time_column`` values so far) reaches
        their time plus ``threshold``, a deletion of entries is emitted.

        Args:
            time_column: ``ColumnExpression`` that specifies the event time.
            threshold: value used to determine which entries are old enough to be removed.
                Should match the type of the ``time_column`` (``int -> int``,
                ``float -> float``, ``datetime -> timedelta``).
            mark_forgetting_records : If set to ``True``, Pathway Live Data Framework marks records
                corresponding to the deletion of expired entries in a special way,
                without changing their visible representation.
                This flag is useful when combined with ``filter_out_results_of_forgetting``,
                which can later remove those marked deletion records. In other words, it
                allows you to revert the effects of forgetting at a later stage.

        Example:

        >>> import pathway as pw
        >>> t = pw.debug.table_from_markdown(
        ...     ''&#x27;
        ...     t | v | __time__
        ...     1 | 1 |     2
        ...     2 | 1 |     2
        ...     4 | 2 |     4
        ...     3 | 3 |     6
        ... ''&#x27;
        ... )
        >>> t_with_forgetting = t.forget(pw.this.t, 3)
        >>> s = pw.debug.table_from_markdown(
        ...     ''&#x27;
        ...   v | a |  __time__
        ...   1 | 1 |      2
        ...   2 | 2 |      4
        ...   1 | 3 |      8
        ... ''&#x27;
        ... )
        >>> res = t_with_forgetting.join(s, pw.left.v == pw.right.v).select(
        ...     pw.left.t, pw.left.v, pw.right.a
        ... )
        >>> pw.debug.compute_and_print_update_stream(res)
                    | t | v | a | __time__ | __diff__
        ^YYYD8ZW... | 1 | 1 | 1 | 2        | 1
        ^YYY47FZ... | 2 | 1 | 1 | 2        | 1
        ^Z3QTSKY... | 4 | 2 | 2 | 4        | 1
        ^YYYD8ZW... | 1 | 1 | 1 | 6        | -1
        ^YYY822X... | 2 | 1 | 3 | 8        | 1

        The entry ``t=1,v=1`` is forgotten at the processing time 6. It gets removed from the