MCPcopy Create free account
hub / github.com/google-research/language / wrap_numpy_function

Function wrap_numpy_function

language/mentionmemory/utils/data_utils.py:127–160  ·  view source on GitHub ↗

Wraps python function into Tensorflow op. Args: preprocess_input_names: names of all input features to the `preprocess_fn` function. preprocess_fn: a function that preprocesses a dictionary of features using numpy library. preprocess_output_types: names of all output featu

(
    preprocess_input_names, preprocess_fn, preprocess_output_types
)

Source from the content-addressed store, hash-verified

125
126
127def wrap_numpy_function(
128 preprocess_input_names, preprocess_fn, preprocess_output_types
129) -> Callable[[Dict[str, tf.Tensor]], Dict[str, tf.Tensor]]:
130 """Wraps python function into Tensorflow op.
131
132 Args:
133 preprocess_input_names: names of all input features to the `preprocess_fn`
134 function.
135 preprocess_fn: a function that preprocesses a dictionary of features using
136 numpy library.
137 preprocess_output_types: names of all output features and their types
138 returned to the `preprocess_fn` by function.
139
140 Returns:
141 A function that preprocesses features using tensorflow ops.
142 """
143
144 def preprocess_with_list_input_fn(*args) -> Tuple[tf.Tensor]:
145 input_dict = {k: v for k, v in zip(preprocess_input_names, args)}
146 output_dict = preprocess_fn(input_dict)
147 output_list = tuple(output_dict[k] for k in preprocess_output_types)
148 return output_list
149
150 def f(example: Dict[str, tf.Tensor]) -> Dict[str, tf.Tensor]:
151 input_list = [example[k] for k in preprocess_input_names]
152 output_types = [v for k, v in preprocess_output_types.items()]
153 output_list = tf.numpy_function(preprocess_with_list_input_fn, input_list,
154 output_types)
155 output_dict = {
156 k: v for k, v in zip(preprocess_output_types.keys(), output_list)
157 }
158 return output_dict
159
160 return f
161
162
163def load_dataset(

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…