MCPcopy Create free account
hub / github.com/apache/arrow / GetFragmentsImpl

Method GetFragmentsImpl

cpp/src/arrow/dataset/dataset_internal.h:109–130  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

107
108 protected:
109 Result<FragmentIterator> GetFragmentsImpl(compute::Expression predicate) override {
110 if (fragment_gen_) {
111 // TODO(ARROW-8163): Async fragment scanning can be forwarded rather than waiting
112 // for the whole generator here. For now, all Dataset impls have a vector of
113 // Fragments anyway
114 auto fragments_fut = CollectAsyncGenerator(std::move(fragment_gen_));
115 ARROW_ASSIGN_OR_RAISE(fragments_, fragments_fut.result());
116 }
117
118 // TODO(ARROW-12891) Provide subtree pruning for any vector of fragments
119 FragmentVector fragments;
120 for (const auto& fragment : fragments_) {
121 ARROW_ASSIGN_OR_RAISE(
122 auto simplified_filter,
123 compute::SimplifyWithGuarantee(predicate, fragment->partition_expression()));
124
125 if (simplified_filter.IsSatisfiable()) {
126 fragments.push_back(fragment);
127 }
128 }
129 return MakeVectorIterator(std::move(fragments));
130 }
131
132 FragmentVector fragments_;
133 AsyncGenerator<std::shared_ptr<Fragment>> fragment_gen_;

Callers

nothing calls this directly

Calls 6

CollectAsyncGeneratorFunction · 0.85
MakeVectorIteratorFunction · 0.85
IsSatisfiableMethod · 0.80
push_backMethod · 0.80
ARROW_ASSIGN_OR_RAISEFunction · 0.50
resultMethod · 0.45

Tested by

no test coverage detected