MCPcopy Create free account
hub / github.com/apache/arrow / _dataset_to_decl

Function _dataset_to_decl

python/pyarrow/acero.py:60–80  ·  view source on GitHub ↗
(dataset, use_threads=True, implicit_ordering=False)

Source from the content-addressed store, hash-verified

58
59
60def _dataset_to_decl(dataset, use_threads=True, implicit_ordering=False):
61 decl = Declaration("scan", ScanNodeOptions(
62 dataset, use_threads=use_threads,
63 implicit_ordering=implicit_ordering))
64
65 # Get rid of special dataset columns
66 # "__fragment_index", "__batch_index", "__last_in_fragment", "__filename"
67 projections = [field(f) for f in dataset.schema.names]
68 decl = Declaration.from_sequence(
69 [decl, Declaration("project", ProjectNodeOptions(projections))]
70 )
71
72 filter_expr = dataset._scan_options.get("filter")
73 if filter_expr is not None:
74 # Filters applied in CScanNodeOptions are "best effort" for the scan node itself
75 # so we always need to inject an additional Filter node to apply them for real.
76 decl = Declaration.from_sequence(
77 [decl, Declaration("filter", FilterNodeOptions(filter_expr))]
78 )
79
80 return decl
81
82
83def _perform_join(join_type, left_operand, left_keys,

Callers 3

_perform_joinFunction · 0.85
_perform_join_asofFunction · 0.85
_sort_sourceFunction · 0.85

Calls 4

fieldFunction · 0.90
ScanNodeOptionsClass · 0.85
DeclarationClass · 0.50
getMethod · 0.45

Tested by

no test coverage detected