MCPcopy Create free account
hub / github.com/google-research/language / tokenizer

Function tokenizer

language/decontext/eval.py:77–83  ·  view source on GitHub ↗
(text)

Source from the content-addressed store, hash-verified

75 sp_tokenizer = get_sentencepiece_tokenizer(vocab_path)
76
77 def tokenizer(text):
78 split_on_punctuation = re.split(
79 r"\s+|\s*([%s])\s*" % re.escape(string.punctuation), text)
80 spaced = " ".join(x for x in split_on_punctuation if x)
81 tokens = sp_tokenizer.EncodeAsPieces(spaced)
82 tokens = [t.lstrip("▁") for t in tokens if t and t != "▁"]
83 return tokens
84
85 return tokenizer
86

Callers 1

_compute_raw_sari_scoreFunction · 0.85

Calls 1

splitMethod · 0.80

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…