Function whitespace_clean

src/mini_clip/tokenizer.py:64–67 · view source on GitHub ↗

(text)

Source from the content-addressed store, hash-verified

62
63
64	def whitespace_clean(text):
65	text = re.sub(r'\s+', ' ', text)
66	text = text.strip()
67	return text
68
69
70	class SimpleTokenizer(object):

encodeMethod · 0.85

no outgoing calls

no test coverage detected