Function whitespace_clean

utils/simple_tokenizer.py:56–59 · view source on GitHub ↗

(text)

Source from the content-addressed store, hash-verified

54
55
56	def whitespace_clean(text):
57	text = re.sub(r'\s+', ' ', text)
58	text = text.strip()
59	return text
60
61
62	class SimpleTokenizer(object):

encodeMethod · 0.85

no outgoing calls

no test coverage detected