MCPcopy Create free account
hub / github.com/google-research/language / input_fn

Function input_fn

language/orqa/predict/encode_blocks.py:93–109  ·  view source on GitHub ↗

An input function satisfying the tf.estimator API.

(params)

Source from the content-addressed store, hash-verified

91
92
93def input_fn(params):
94 """An input function satisfying the tf.estimator API."""
95 compression_type = "GZIP" if FLAGS.examples_path.endswith(".gz") else ""
96 examples_paths = sorted(tf.io.gfile.glob(FLAGS.examples_path))
97 dataset = tf.data.TFRecordDataset(
98 examples_paths,
99 compression_type=compression_type,
100 buffer_size=16 * 1024 * 1024)
101 dataset = dataset.map(parse_examples, num_parallel_calls=FLAGS.num_threads)
102
103 # Repeat just one extra batch to make sure we don't lose any remainder
104 # blocks that might be dropped.
105 dataset = dataset.concatenate(dataset.take(params["batch_size"]))
106
107 dataset = dataset.batch(params["batch_size"], drop_remainder=True)
108 dataset = dataset.prefetch(10)
109 return dataset
110
111
112def main(_):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…