Wrapper around dataset.write_dataset for writing a Table to Parquet format by partitions. For each combination of partition columns and values, a subdirectories are created in the following manner: root_dir/ group1=value1 group2=value1 .parquet
(table, root_path, partition_cols=None,
filesystem=None, schema=None, partitioning=None,
basename_template=None, use_threads=None,
file_visitor=None, existing_data_behavior=None,
**kwargs)
| 2142 | |
| 2143 | |
| 2144 | def write_to_dataset(table, root_path, partition_cols=None, |
| 2145 | filesystem=None, schema=None, partitioning=None, |
| 2146 | basename_template=None, use_threads=None, |
| 2147 | file_visitor=None, existing_data_behavior=None, |
| 2148 | **kwargs): |
| 2149 | """Wrapper around dataset.write_dataset for writing a Table to |
| 2150 | Parquet format by partitions. |
| 2151 | For each combination of partition columns and values, |
| 2152 | a subdirectories are created in the following |
| 2153 | manner: |
| 2154 | |
| 2155 | root_dir/ |
| 2156 | group1=value1 |
| 2157 | group2=value1 |
| 2158 | <uuid>.parquet |
| 2159 | group2=value2 |
| 2160 | <uuid>.parquet |
| 2161 | group1=valueN |
| 2162 | group2=value1 |
| 2163 | <uuid>.parquet |
| 2164 | group2=valueN |
| 2165 | <uuid>.parquet |
| 2166 | |
| 2167 | Parameters |
| 2168 | ---------- |
| 2169 | table : pyarrow.Table |
| 2170 | root_path : str, pathlib.Path |
| 2171 | The root directory of the dataset. |
| 2172 | partition_cols : list, |
| 2173 | Column names by which to partition the dataset. |
| 2174 | Columns are partitioned in the order they are given. |
| 2175 | filesystem : FileSystem, default None |
| 2176 | If nothing passed, will be inferred based on path. |
| 2177 | Path will try to be found in the local on-disk filesystem otherwise |
| 2178 | it will be parsed as an URI to determine the filesystem. |
| 2179 | schema : Schema, optional |
| 2180 | This Schema of the dataset. |
| 2181 | partitioning : Partitioning or list[str], optional |
| 2182 | The partitioning scheme specified with the |
| 2183 | ``pyarrow.dataset.partitioning()`` function or a list of field names. |
| 2184 | When providing a list of field names, you can use |
| 2185 | ``partitioning_flavor`` to drive which partitioning type should be |
| 2186 | used. |
| 2187 | basename_template : str, optional |
| 2188 | A template string used to generate basenames of written data files. |
| 2189 | The token '{i}' will be replaced with an automatically incremented |
| 2190 | integer. If not specified, it defaults to "guid-{i}.parquet". |
| 2191 | use_threads : bool, default True |
| 2192 | Write files in parallel. If enabled, then maximum parallelism will be |
| 2193 | used determined by the number of available CPU cores. |
| 2194 | file_visitor : function |
| 2195 | If set, this function will be called with a WrittenFile instance |
| 2196 | for each file created during the call. This object will have both |
| 2197 | a path attribute and a metadata attribute. |
| 2198 | |
| 2199 | The path attribute will be a string containing the path to |
| 2200 | the created file. |
| 2201 |
nothing calls this directly
no test coverage detected