MCPcopy Create free account
hub / github.com/google-research/language / get_tokenization_info

Function get_tokenization_info

language/orqa/utils/bert_utils.py:23–27  ·  view source on GitHub ↗
(module_handle)

Source from the content-addressed store, hash-verified

21
22
23def get_tokenization_info(module_handle):
24 with tf.Graph().as_default():
25 bert_module = hub.Module(module_handle)
26 with tf.Session() as sess:
27 return sess.run(bert_module(signature="tokenization_info", as_dict=True))
28
29
30def get_tokenizer(module_handle):

Callers 2

get_tokenizerFunction · 0.70
get_tf_tokenizerFunction · 0.70

Calls 1

runMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…