MCPcopy Create free account
hub / github.com/google-research/language / get_tokenization_info

Function get_tokenization_info

language/multivec/utils/data_processor.py:93–97  ·  view source on GitHub ↗
(module_handle)

Source from the content-addressed store, hash-verified

91
92
93def get_tokenization_info(module_handle):
94 with tf.Graph().as_default():
95 bert_module = hub.Module(module_handle)
96 with tf.Session() as sess:
97 return sess.run(bert_module(signature="tokenization_info", as_dict=True))
98
99
100def get_tokenizer(module_handle):

Callers 1

get_tokenizerFunction · 0.70

Calls 1

runMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…