MCPcopy Create free account
hub / github.com/google-research/language / get_tokenizer

Function get_tokenizer

language/multivec/utils/data_processor.py:100–104  ·  view source on GitHub ↗
(module_handle)

Source from the content-addressed store, hash-verified

98
99
100def get_tokenizer(module_handle):
101 tokenization_info = get_tokenization_info(module_handle)
102 return tokenization.FullTokenizer(
103 vocab_file=tokenization_info["vocab_file"],
104 do_lower_case=tokenization_info["do_lower_case"])
105
106
107def main(_):

Callers 1

mainFunction · 0.70

Calls 1

get_tokenization_infoFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…