Method embed

python/zvec/extension/bm25_embedding_function.py:287–375

Generate BM25 sparse embedding for the input text. This method computes BM25 scores for the input text using DashText's SparseVectorEncoder. The encoding behavior depends on the encoding_type: - ``encoding_type="query"``: Uses ``encode_queries()`` for search queries

(self, input: TEXT)

    @lru_cache(maxsize=10)
    def embed(self, input: TEXT) -> SparseVectorType:
        """Generate BM25 sparse embedding for the input text.

        This method computes BM25 scores for the input text using DashText's
        SparseVectorEncoder. The encoding behavior depends on the encoding_type:

        - ``encoding_type="query"``: Uses ``encode_queries()`` for search queries
        - ``encoding_type="document"``: Uses ``encode_documents()`` for documents

        The result is a sparse vector where keys are term indices in the
        vocabulary and values are BM25 scores.

        Args:
            input (TEXT): Input text string to embed. Must be non-empty after
                stripping whitespace.

        Returns:
            SparseVectorType: A dictionary mapping vocabulary term index to BM25 score.
                Only non-zero scores are included. The dictionary is sorted by indices
                (keys) in ascending order for consistent output.
                Example: ``{1169440797: 0.29, 2045788977: 0.70, ...}``

        Raises:
            TypeError: If ``input`` is not a string.
            ValueError: If input is empty or whitespace-only.
            RuntimeError: If BM25 encoding fails.

        Examples:
            >>> bm25 = BM25EmbeddingFunction(language="zh", encoding_type="query")
            >>> sparse_vec = bm25.embed("query text")
            >>> isinstance(sparse_vec, dict)
            True
            >>> all(isinstance(k, int) and isinstance(v, float) for k, v in sparse_vec.items())
            True

            >>> # Verify sorted output
            >>> keys = list(sparse_vec.keys())
            >>> keys == sorted(keys)
            True

            >>> # Error: empty input
            >>> bm25.embed("   ")
            ValueError: Input text cannot be empty or whitespace only

            >>> # Error: non-string input
            >>> bm25.embed(123)
            TypeError: Expected 'input' to be str, got int

        Note:
            - BM25 scores are relative to the vocabulary statistics
            - Output dictionary is always sorted by indices for consistency
            - Terms not in the vocabulary will have zero scores (not included)
            - This method is cached (maxsize=10) for performance
            - DashText automatically handles Chinese/English text segmentation
        """
        if not isinstance(input, str):
            raise TypeError(f"Expected 'input' to be str, got {type(input).__name__}")