Function basic_clean

utils/simple_tokenizer.py:50–53 · view source on GitHub ↗

(text)

Source from the content-addressed store, hash-verified

48
49
50	def basic_clean(text):
51	text = ftfy.fix_text(text)
52	text = html.unescape(html.unescape(text))
53	return text.strip()
54
55
56	def whitespace_clean(text):

encodeMethod · 0.85

no outgoing calls

no test coverage detected