Wrapper around dataset.write_dataset for writing a Table to Parquet format by partitions. For each combination of partition columns and values, a subdirectories are created in the following manner: root_dir/ group1=value1 group2=value1 .parquet
(table, root_path, partition_cols=None,
filesystem=None, schema=None, partitioning=None,
basename_template=None, use_threads=None,
file_visitor=None, existing_data_behavior=None,
**kwargs)
| 2114 | |
| 2115 | |
| 2116 | def write_to_dataset(table, root_path, partition_cols=None, |
| 2117 | filesystem=None, schema=None, partitioning=None, |
| 2118 | basename_template=None, use_threads=None, |
| 2119 | file_visitor=None, existing_data_behavior=None, |
| 2120 | **kwargs): |
| 2121 | """Wrapper around dataset.write_dataset for writing a Table to |
| 2122 | Parquet format by partitions. |
| 2123 | For each combination of partition columns and values, |
| 2124 | a subdirectories are created in the following |
| 2125 | manner: |
| 2126 | |
| 2127 | root_dir/ |
| 2128 | group1=value1 |
| 2129 | group2=value1 |
| 2130 | <uuid>.parquet |
| 2131 | group2=value2 |
| 2132 | <uuid>.parquet |
| 2133 | group1=valueN |
| 2134 | group2=value1 |
| 2135 | <uuid>.parquet |
| 2136 | group2=valueN |
| 2137 | <uuid>.parquet |
| 2138 | |
| 2139 | Parameters |
| 2140 | ---------- |
| 2141 | table : pyarrow.Table |
| 2142 | root_path : str, pathlib.Path |
| 2143 | The root directory of the dataset. |
| 2144 | partition_cols : list, |
| 2145 | Column names by which to partition the dataset. |
| 2146 | Columns are partitioned in the order they are given. |
| 2147 | filesystem : FileSystem, default None |
| 2148 | If nothing passed, will be inferred based on path. |
| 2149 | Path will try to be found in the local on-disk filesystem otherwise |
| 2150 | it will be parsed as an URI to determine the filesystem. |
| 2151 | schema : Schema, optional |
| 2152 | This Schema of the dataset. |
| 2153 | partitioning : Partitioning or list[str], optional |
| 2154 | The partitioning scheme specified with the |
| 2155 | ``pyarrow.dataset.partitioning()`` function or a list of field names. |
| 2156 | When providing a list of field names, you can use |
| 2157 | ``partitioning_flavor`` to drive which partitioning type should be |
| 2158 | used. |
| 2159 | basename_template : str, optional |
| 2160 | A template string used to generate basenames of written data files. |
| 2161 | The token '{i}' will be replaced with an automatically incremented |
| 2162 | integer. If not specified, it defaults to "guid-{i}.parquet". |
| 2163 | use_threads : bool, default True |
| 2164 | Write files in parallel. If enabled, then maximum parallelism will be |
| 2165 | used determined by the number of available CPU cores. |
| 2166 | file_visitor : function |
| 2167 | If set, this function will be called with a WrittenFile instance |
| 2168 | for each file created during the call. This object will have both |
| 2169 | a path attribute and a metadata attribute. |
| 2170 | |
| 2171 | The path attribute will be a string containing the path to |
| 2172 | the created file. |
| 2173 |
nothing calls this directly
no test coverage detected