MCPcopy Create free account
hub / github.com/OpenSparseLLMs/MoM / clean_python

Method clean_python

lm-eval-harness/lm_eval/decontamination/janitor.py:218–224  ·  view source on GitHub ↗
(self, dirty_string: str)

Source from the content-addressed store, hash-verified

216 )
217
218 def clean_python(self, dirty_string: str) -> List[str]:
219 contamination_indices = (
220 (None, *idx_pair)
221 for dirty_ngram, idx_pair in word_ngrams_indices(dirty_string, self.ngram_n)
222 if self.normalize_string(dirty_ngram) in self.dirt_ngrams
223 )
224 return self._split_chunks(dirty_string, contamination_indices)
225
226
227##################################################################

Callers 1

cleanMethod · 0.95

Calls 3

normalize_stringMethod · 0.95
_split_chunksMethod · 0.95
word_ngrams_indicesFunction · 0.85

Tested by

no test coverage detected