MCPcopy Create free account
hub / github.com/alibaba/nann / StreamingFilesDataset

Function StreamingFilesDataset

tensorflow/tensorflow/python/tpu/datasets.py:51–199  ·  view source on GitHub ↗

StreamingFilesDataset constructs a dataset to stream from workers (GCE VM). Because Cloud TPUs are allocated over the network, a Cloud TPU cannot read files local to your GCE VM. In order to train using files stored on your local VM (e.g. on local SSD for extreme performance), use the Streami

(files,
                          filetype=None,
                          file_reader_job=None,
                          worker_job=None,
                          num_epochs=None,
                          filename_shuffle_buffer_size=None,
                          num_parallel_reads=None,
                          batch_transfer_size=None,
                          sloppy=None)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers

nothing calls this directly

Calls 12

string_handleMethod · 0.80
unbatchMethod · 0.80
deviceMethod · 0.45
list_filesMethod · 0.45
shuffleMethod · 0.45
applyMethod · 0.45
repeatMethod · 0.45
batchMethod · 0.45
prefetchMethod · 0.45
mapMethod · 0.45
rangeMethod · 0.45

Tested by

no test coverage detected