Function _tokenize

mempalace/searcher.py:63–72

Lowercase + strip to alphanumeric tokens of length ≥ 2. Tolerates ``None`` documents — Chroma can return ``None`` in the ``documents`` field for drawers without text content, which would otherwise raise ``AttributeError`` mid-rerank.

(text: str)

def _tokenize(text: str) -> list:
    """Lowercase + strip to alphanumeric tokens of length ≥ 2.

    Tolerates ``None`` documents — Chroma can return ``None`` in the
    ``documents`` field for drawers without text content, which would
    otherwise raise ``AttributeError`` mid-rerank.
    """
    if not text:
        return []
    return _TOKEN_RE.findall(text.lower())


def _bm25_scores(