MCPcopy Create free account
hub / github.com/google-research/language / input_fn

Function input_fn

language/multivec/predict/encode_blocks.py:128–144  ·  view source on GitHub ↗

An input function satisfying the tf.estimator API.

(params)

Source from the content-addressed store, hash-verified

126
127
128def input_fn(params):
129 """An input function satisfying the tf.estimator API."""
130 compression_type = "GZIP" if FLAGS.examples_path.endswith(".gz") else ""
131 examples_paths = sorted(tf.io.gfile.glob(FLAGS.examples_path))
132 dataset = tf.data.TFRecordDataset(
133 examples_paths,
134 compression_type=compression_type,
135 buffer_size=16 * 1024 * 1024)
136 dataset = dataset.map(parse_examples, num_parallel_calls=FLAGS.num_threads)
137
138 # Repeat just one extra batch to make sure we don't lose any remainder
139 # blocks that might be dropped.
140 dataset = dataset.concatenate(dataset.take(params["batch_size"]))
141
142 dataset = dataset.batch(params["batch_size"], drop_remainder=True)
143 dataset = dataset.prefetch(10)
144 return dataset
145
146
147def save_to_h5py(outfn, output_array, field_name):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…