MCPcopy Create free account
hub / github.com/cosdata/cosdata / remove_non_alphanumeric

Function remove_non_alphanumeric

tests/test-tf-idf-bm25.py:100–101  ·  view source on GitHub ↗
(text: str)

Source from the content-addressed store, hash-verified

98
99
100def remove_non_alphanumeric(text: str) -> str:
101 return re.sub(r"[^\w\s]", " ", text, flags=re.UNICODE)
102
103
104class SimpleTokenizer:

Callers 1

transform_sentence_to_vectorFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected