MCPcopy Create free account
hub / github.com/apache/arrow / ListAsync

Method ListAsync

cpp/src/arrow/filesystem/s3fs.cc:2752–2767  ·  view source on GitHub ↗

Lists all file, potentially recursively, in a bucket include_implicit_dirs controls whether or not implicit directories should be included. These are directories that are not actually file objects but instead are inferred from other objects. For example, if a file exists with path A/B/C then implicit directories A/ and A/B/ will exist even if there are no file objects with these paths.

Source from the content-addressed store, hash-verified

2750 // For example, if a file exists with path A/B/C then implicit directories A/ and A/B/
2751 // will exist even if there are no file objects with these paths.
2752 void ListAsync(const FileSelector& select, const std::string& bucket,
2753 const std::string& key, bool include_implicit_dirs,
2754 util::AsyncTaskScheduler* scheduler, FileInfoSink sink) {
2755 // We can only fetch kListObjectsMaxKeys files at a time and so we create a
2756 // scheduler and schedule a task to grab the first batch. Once that's done we
2757 // schedule a new task for the next batch. All of these tasks share the same
2758 // FileListerState object but none of these tasks run in parallel so there is
2759 // no need to worry about mutexes
2760 auto state = std::make_shared<FileListerState>(sink, select, bucket, key,
2761 include_implicit_dirs, io_context_,
2762 this->holder_.get());
2763
2764 // Create the first file lister task (it may spawn more)
2765 auto file_lister_task = std::make_unique<FileListerTask>(state, scheduler);
2766 scheduler->AddTask(std::move(file_lister_task));
2767 }
2768
2769 // Fully list all files from all buckets
2770 void FullListAsync(bool include_implicit_dirs, util::AsyncTaskScheduler* scheduler,

Callers 1

GetFileInfoGeneratorMethod · 0.95

Calls 2

getMethod · 0.45
AddTaskMethod · 0.45

Tested by

no test coverage detected