Class BM25EmbeddingFunction

python/zvec/extension/bm25_embedding_function.py:24–375

BM25-based sparse embedding function using DashText SDK. This class provides text-to-sparse-vector embedding capabilities using the DashText library with BM25 algorithm. BM25 (Best Matching 25) is a probabilistic retrieval function used for lexical search and document ranking based

class BM25EmbeddingFunction(SparseEmbeddingFunction[TEXT]):
    """BM25-based sparse embedding function using DashText SDK.

    This class provides text-to-sparse-vector embedding capabilities using
    the DashText library with BM25 algorithm. BM25 (Best Matching 25) is a
    probabilistic retrieval function used for lexical search and document
    ranking based on term frequency and inverse document frequency.

    BM25 generates sparse vectors where each dimension corresponds to a term in
    the vocabulary, and the value represents the BM25 score for that term. It's
    particularly effective for:

    - Lexical search and keyword matching
    - Document ranking and information retrieval
    - Combining with dense embeddings for hybrid search
    - Traditional IR tasks where exact term matching is important

    This implementation uses DashText's SparseVectorEncoder, which provides
    efficient BM25 computation for Chinese and English text using either a
    built-in encoder or custom corpus training.

    Args:
        corpus (Optional[list[str]], optional): List of documents to train the
            BM25 encoder. If provided, creates a custom encoder trained on this
            corpus for better domain-specific accuracy. If ``None``, uses the
            built-in encoder. Defaults to ``None``.
        encoding_type (Literal["query", "document"], optional): Encoding mode
            for text processing. Use ``"query"`` for search queries (default) and
            ``"document"`` for document indexing. This distinction optimizes the
            BM25 scoring for asymmetric retrieval tasks. Defaults to ``"query"``.
        language (Literal["zh", "en"], optional): Language for built-in encoder.
            Only used when corpus is None. ``"zh"`` for Chinese (trained on Chinese
            Wikipedia), ``"en"`` for English. Defaults to ``"zh"``.
        b (float, optional): Document length normalization parameter for BM25.
            Range [0, 1]. 0 means no normalization, 1 means full normalization.
            Only used with custom corpus. Defaults to ``0.75``.
        k1 (float, optional): Term frequency saturation parameter for BM25.
            Higher values give more weight to term frequency. Only used with
            custom corpus. Defaults to ``1.2``.
        **kwargs: Additional parameters for DashText encoder customization.

    Attributes:
        corpus_size (int): Number of documents in the training corpus (0 if using built-in encoder).
        encoding_type (str): The encoding type being used ("query" or "document").
        language (str): The language of the built-in encoder ("zh" or "en").

    Raises:
        ValueError: If corpus is provided but empty or contains non-string elements.
        TypeError: If input to ``embed()`` is not a string.
        RuntimeError: If DashText encoder initialization or training fails.

    Note:
        - Requires Python 3.10, 3.11, or 3.12
        - Requires the ``dashtext`` package: ``pip install dashtext``
        - Two encoder options available:

          1. **Built-in encoder** (no corpus needed): Pre-trained models for
             Chinese (zh) and English (en), good generalization, works out-of-the-box