MCPcopy Create free account
hub / github.com/OpenBitSys/BitDistiller / _remove_punc

Method _remove_punc

test/general/lm_eval/tasks/drop.py:258–263  ·  view source on GitHub ↗
(self, text)

Source from the content-addressed store, hash-verified

256 return " ".join(text.split())
257
258 def _remove_punc(self, text):
259 exclude = set(string.punctuation)
260 if not self._is_number(text):
261 return "".join(ch for ch in text if ch not in exclude)
262 else:
263 return text
264
265 def _fix_number(self, text):
266 return str(float(text)) if self._is_number(text) else text

Callers 1

_normalizeMethod · 0.95

Calls 1

_is_numberMethod · 0.95

Tested by

no test coverage detected