MCPcopy Create free account
hub / github.com/apache/arrow / write_to_dataset

Function write_to_dataset

python/pyarrow/parquet/core.py:2144–2322  ·  view source on GitHub ↗

Wrapper around dataset.write_dataset for writing a Table to Parquet format by partitions. For each combination of partition columns and values, a subdirectories are created in the following manner: root_dir/ group1=value1 group2=value1 .parquet

(table, root_path, partition_cols=None,
                     filesystem=None, schema=None, partitioning=None,
                     basename_template=None, use_threads=None,
                     file_visitor=None, existing_data_behavior=None,
                     **kwargs)

Source from the content-addressed store, hash-verified

2142
2143
2144def write_to_dataset(table, root_path, partition_cols=None,
2145 filesystem=None, schema=None, partitioning=None,
2146 basename_template=None, use_threads=None,
2147 file_visitor=None, existing_data_behavior=None,
2148 **kwargs):
2149 """Wrapper around dataset.write_dataset for writing a Table to
2150 Parquet format by partitions.
2151 For each combination of partition columns and values,
2152 a subdirectories are created in the following
2153 manner:
2154
2155 root_dir/
2156 group1=value1
2157 group2=value1
2158 <uuid>.parquet
2159 group2=value2
2160 <uuid>.parquet
2161 group1=valueN
2162 group2=value1
2163 <uuid>.parquet
2164 group2=valueN
2165 <uuid>.parquet
2166
2167 Parameters
2168 ----------
2169 table : pyarrow.Table
2170 root_path : str, pathlib.Path
2171 The root directory of the dataset.
2172 partition_cols : list,
2173 Column names by which to partition the dataset.
2174 Columns are partitioned in the order they are given.
2175 filesystem : FileSystem, default None
2176 If nothing passed, will be inferred based on path.
2177 Path will try to be found in the local on-disk filesystem otherwise
2178 it will be parsed as an URI to determine the filesystem.
2179 schema : Schema, optional
2180 This Schema of the dataset.
2181 partitioning : Partitioning or list[str], optional
2182 The partitioning scheme specified with the
2183 ``pyarrow.dataset.partitioning()`` function or a list of field names.
2184 When providing a list of field names, you can use
2185 ``partitioning_flavor`` to drive which partitioning type should be
2186 used.
2187 basename_template : str, optional
2188 A template string used to generate basenames of written data files.
2189 The token '{i}' will be replaced with an automatically incremented
2190 integer. If not specified, it defaults to "guid-{i}.parquet".
2191 use_threads : bool, default True
2192 Write files in parallel. If enabled, then maximum parallelism will be
2193 used determined by the number of available CPU cores.
2194 file_visitor : function
2195 If set, this function will be called with a WrittenFile instance
2196 for each file created during the call. This object will have both
2197 a path attribute and a metadata attribute.
2198
2199 The path attribute will be a string containing the path to
2200 the created file.
2201

Callers

nothing calls this directly

Calls 7

_ensure_filesystemFunction · 0.90
guidFunction · 0.90
ParquetFileFormatMethod · 0.80
partitioningMethod · 0.80
formatMethod · 0.45
signatureMethod · 0.45
selectMethod · 0.45

Tested by

no test coverage detected