MCPcopy Create free account
hub / github.com/google-research/language / create_block_info

Function create_block_info

language/multivec/utils/data_processor.py:81–90  ·  view source on GitHub ↗

Create block info.

(input_path, preprocessor)

Source from the content-addressed store, hash-verified

79
80
81def create_block_info(input_path, preprocessor):
82 """Create block info."""
83 results = []
84 with tf.io.gfile.GFile(input_path) as fid:
85 input_file = fid.read()
86 data = json.loads(input_file)
87 for key in data:
88 text = data[key]
89 results.append(preprocessor.example_from_json_line(key, text))
90 return results
91
92
93def get_tokenization_info(module_handle):

Callers

nothing calls this directly

Calls 1

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…