MCPcopy Create free account
hub / github.com/YunwenTechnology/QueryGeneration / _clean_text

Method _clean_text

tokenization/tokenization.py:273–284  ·  view source on GitHub ↗

Performs invalid character removal and whitespace cleanup on text.

(self, text)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 1

tokenizeMethod · 0.95

Calls 2

_is_controlFunction · 0.85
_is_whitespaceFunction · 0.85

Tested by

no test coverage detected