The main procedure to start async streaming discovery using a given `FileSelector`. The result is a two-level generator, i.e. "generator of FileInfoGenerator:s", where each individual generator represents an FileInfo item stream from coming an individual sub-directory under the selector's `base_dir`.
| 372 | /// where each individual generator represents an FileInfo item stream from coming an |
| 373 | /// individual sub-directory under the selector's `base_dir`. |
| 374 | static Result<AsyncGenerator<FileInfoGenerator>> DiscoverDirectoryFiles( |
| 375 | FileSelector selector, LocalFileSystemOptions fs_opts, |
| 376 | const io::IOContext& io_context) { |
| 377 | PushGenerator<FileInfoGenerator> file_gen; |
| 378 | |
| 379 | ARROW_ASSIGN_OR_RAISE( |
| 380 | auto base_dir, arrow::internal::PlatformFilename::FromString(selector.base_dir)); |
| 381 | ARROW_RETURN_NOT_OK(DoDiscovery(std::move(base_dir), 0, std::move(selector), |
| 382 | std::make_shared<DiscoveryState>(file_gen.producer()), |
| 383 | io_context, fs_opts.file_info_batch_size)); |
| 384 | |
| 385 | return file_gen; |
| 386 | } |
| 387 | |
| 388 | /// Version of `DiscoverDirectoryFiles` which flattens the stream of generators |
| 389 | /// into a single FileInfoGenerator stream. |