MCPcopy Create free account
hub / github.com/apache/arrow / DiscoverDirectoryFiles

Method DiscoverDirectoryFiles

cpp/src/arrow/filesystem/localfs.cc:374–386  ·  view source on GitHub ↗

The main procedure to start async streaming discovery using a given `FileSelector`. The result is a two-level generator, i.e. "generator of FileInfoGenerator:s", where each individual generator represents an FileInfo item stream from coming an individual sub-directory under the selector's `base_dir`.

Source from the content-addressed store, hash-verified

372 /// where each individual generator represents an FileInfo item stream from coming an
373 /// individual sub-directory under the selector's `base_dir`.
374 static Result<AsyncGenerator<FileInfoGenerator>> DiscoverDirectoryFiles(
375 FileSelector selector, LocalFileSystemOptions fs_opts,
376 const io::IOContext& io_context) {
377 PushGenerator<FileInfoGenerator> file_gen;
378
379 ARROW_ASSIGN_OR_RAISE(
380 auto base_dir, arrow::internal::PlatformFilename::FromString(selector.base_dir));
381 ARROW_RETURN_NOT_OK(DoDiscovery(std::move(base_dir), 0, std::move(selector),
382 std::make_shared<DiscoveryState>(file_gen.producer()),
383 io_context, fs_opts.file_info_batch_size));
384
385 return file_gen;
386 }
387
388 /// Version of `DiscoverDirectoryFiles` which flattens the stream of generators
389 /// into a single FileInfoGenerator stream.

Callers

nothing calls this directly

Calls 1

producerMethod · 0.45

Tested by

no test coverage detected