MCPcopy Create free account
hub / github.com/google-research/language / input_fn

Function input_fn

language/conpono/binary_order/run_binary_coherence.py:177–197  ·  view source on GitHub ↗

The actual input function.

(params)

Source from the content-addressed store, hash-verified

175 return example
176
177 def input_fn(params):
178 """The actual input function."""
179 batch_size = params["batch_size"]
180
181 # For training, we want a lot of parallel reading and shuffling.
182 # For eval, we want no shuffling and parallel reading doesn't matter.
183 d = tf.data.TFRecordDataset(input_file)
184 if is_training:
185 d = d.repeat()
186 d = d.shuffle(buffer_size=100)
187 # Every iteration the input_fn() gets called so the dataset starts from 0
188 # by skipping we won't repeat the same data
189 d = d.skip(skip)
190
191 d = d.apply(
192 tf.data.experimental.map_and_batch(
193 lambda record: _decode_record(record, name_to_features),
194 batch_size=batch_size,
195 drop_remainder=drop_remainder))
196
197 return d
198
199 return input_fn
200

Callers

nothing calls this directly

Calls 1

_decode_recordFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…