MCPcopy Create free account
hub / github.com/MiniMax-AI/VTP / basic_clean

Function basic_clean

vtp/tokenizers/text_tokenizer.py:110–114  ·  view source on GitHub ↗

Basic text cleaning.

(text)

Source from the content-addressed store, hash-verified

108
109
110def basic_clean(text):
111 """Basic text cleaning."""
112 text = ftfy.fix_text(text)
113 text = html.unescape(html.unescape(text))
114 return text.strip()
115
116
117def whitespace_clean(text):

Callers 2

_clean_lowerFunction · 0.85
_clean_whitespaceFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected