MCPcopy Create free account
hub / github.com/apache/arrow / InferColumnProjection

Function InferColumnProjection

cpp/src/arrow/dataset/file_parquet.cc:288–322  ·  view source on GitHub ↗

Compute the column projection based on the scan options

Source from the content-addressed store, hash-verified

286
287// Compute the column projection based on the scan options
288Result<std::vector<int>> InferColumnProjection(const parquet::arrow::FileReader& reader,
289 const ScanOptions& options) {
290 auto manifest = reader.manifest();
291 // Checks if the field is needed in either the projection or the filter.
292 auto field_refs = options.MaterializedFields();
293
294 // Build a lookup table from top level field name to field metadata.
295 // This is to avoid quadratic-time mapping of projected fields to
296 // column indices, in the common case of selecting top level
297 // columns. For nested fields, we will pay the cost of a linear scan
298 // assuming for now that this is relatively rare, but this can be
299 // optimized. (Also, we don't want to pay the cost of building all
300 // the lookup tables up front if they're rarely used.)
301 std::unordered_map<std::string, const SchemaField*> field_lookup;
302 std::unordered_set<std::string> duplicate_fields;
303 for (const auto& schema_field : manifest.schema_fields) {
304 const auto it = field_lookup.emplace(schema_field.field->name(), &schema_field);
305 if (!it.second) {
306 duplicate_fields.emplace(schema_field.field->name());
307 }
308 }
309
310 std::vector<int> columns_selection;
311 for (auto& ref : field_refs) {
312 // In the (unlikely) absence of a known dataset schema, we require that all
313 // materialized refs are named.
314 if (options.dataset_schema) {
315 ARROW_ASSIGN_OR_RAISE(
316 ref, MaybeConvertFieldRef(std::move(ref), *options.dataset_schema));
317 }
318 RETURN_NOT_OK(ResolveOneFieldRef(manifest, ref, field_lookup, duplicate_fields,
319 &columns_selection));
320 }
321 return columns_selection;
322}
323
324Status WrapSourceError(const Status& status, const std::string& path) {
325 return status.WithMessage("Could not open Parquet input source '", path,

Callers

nothing calls this directly

Calls 5

MaybeConvertFieldRefFunction · 0.85
ResolveOneFieldRefFunction · 0.85
MaterializedFieldsMethod · 0.80
ARROW_ASSIGN_OR_RAISEFunction · 0.50
nameMethod · 0.45

Tested by

no test coverage detected