MCPcopy Create free account
hub / github.com/OpenBitSys/BitDistiller / detokenize

Method detokenize

test/general/lm_eval/tasks/mutual.py:70–84  ·  view source on GitHub ↗
(self, text)

Source from the content-addressed store, hash-verified

68 return lls
69
70 def detokenize(self, text):
71 text = text.replace(" '", "'")
72 text = text.replace(" \n", "\n")
73 text = text.replace("\n ", "\n")
74 text = text.replace(" n't", "n't")
75 text = text.replace("`` ", '"')
76 text = text.replace("''", '"')
77 # punctuation
78 text = text.replace(" :", ":")
79 text = text.replace(" ;", ";")
80 text = text.replace(" !", "!")
81 text = text.replace(" ?", "?")
82 text = text.replace(" ,", ",")
83 text = text.replace(" .", ".")
84 return text
85
86 def process_results(self, doc, results):
87 gold = self.CHOICES.index(doc["answers"])

Callers 3

doc_to_textMethod · 0.95
doc_to_targetMethod · 0.95
construct_requestsMethod · 0.95

Calls

no outgoing calls

Tested by

no test coverage detected