Function tokenize

evaluations.py:13–16 · view source on GitHub ↗

(text)

Source from the content-addressed store, hash-verified

11	## BLEU
12	########################
13	def tokenize(text):
14	tokens = re.split(r'\s\|\.', text)
15	tokens = [t for t in tokens if len(t) > 0]
16	return tokens
17
18
19	def bleu_score(reference, hypothesis, gram):

bleu_scoreFunction · 0.85

no outgoing calls

no test coverage detected