MCPcopy Create free account
hub / github.com/google-research/language / get_tokenizer

Function get_tokenizer

language/orqa/utils/bert_utils.py:30–34  ·  view source on GitHub ↗
(module_handle)

Source from the content-addressed store, hash-verified

28
29
30def get_tokenizer(module_handle):
31 tokenization_info = get_tokenization_info(module_handle)
32 return tokenization.FullTokenizer(
33 vocab_file=tokenization_info["vocab_file"],
34 do_lower_case=tokenization_info["do_lower_case"])
35
36
37def get_tf_tokenizer(module_handle):

Callers

nothing calls this directly

Calls 1

get_tokenization_infoFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…