MCPcopy Create free account
hub / github.com/tensorflow/datasets / file_instructions

Method file_instructions

tensorflow_datasets/core/splits.py:202–235  ·  view source on GitHub ↗

Returns the list of dict(filename, take, skip). This allows for creating your own `tf.data.Dataset` using the low-level TFDS values. Example: ``` file_instructions = info.splits['train[75%:]'].file_instructions instruction_ds = tf.data.Dataset.from_generator( lambd

(self)

Source from the content-addressed store, hash-verified

200
201 @property
202 def file_instructions(self) -> list[shard_utils.FileInstruction]:
203 """Returns the list of dict(filename, take, skip).
204
205 This allows for creating your own `tf.data.Dataset` using the low-level
206 TFDS values.
207
208 Example:
209
210 ```
211 file_instructions = info.splits['train[75%:]'].file_instructions
212 instruction_ds = tf.data.Dataset.from_generator(
213 lambda: file_instructions,
214 output_types={
215 'filename': tf.string,
216 'take': tf.int64,
217 'skip': tf.int64,
218 },
219 )
220 ds = instruction_ds.interleave(
221 lambda f: tf.data.TFRecordDataset(
222 f['filename']).skip(f['skip']).take(f['take'])
223 )
224 ```
225
226 When `skip=0` and `take=-1`, the full shard will be read, so the `ds.skip`
227 and `ds.take` could be skipped.
228
229 Returns:
230 A `dict(filename, take, skip)`
231 """
232 return _make_file_instructions(
233 split_infos=[self],
234 instruction=str(self.name),
235 )
236
237 @property
238 def filenames(self) -> list[str]:

Callers 2

__post_init__Method · 0.45
__post_init__Method · 0.45

Calls 1

_make_file_instructionsFunction · 0.85

Tested by

no test coverage detected