Wrapper around dataset.write_dataset for writing a Table to Parquet format by partitions. For each combination of partition columns and values, a subdirectories are created in the following manner: root_dir/ group1=value1 group2=value1 .parquet
(table, root_path, partition_cols=None,
filesystem=None, schema=None, partitioning=None,
basename_template=None, use_threads=None,
file_visitor=None, existing_data_behavior=None,
**kwargs)
| 2119 | |
| 2120 | |
| 2121 | def write_to_dataset(table, root_path, partition_cols=None, |
| 2122 | filesystem=None, schema=None, partitioning=None, |
| 2123 | basename_template=None, use_threads=None, |
| 2124 | file_visitor=None, existing_data_behavior=None, |
| 2125 | **kwargs): |
| 2126 | """Wrapper around dataset.write_dataset for writing a Table to |
| 2127 | Parquet format by partitions. |
| 2128 | For each combination of partition columns and values, |
| 2129 | a subdirectories are created in the following |
| 2130 | manner: |
| 2131 | |
| 2132 | root_dir/ |
| 2133 | group1=value1 |
| 2134 | group2=value1 |
| 2135 | <uuid>.parquet |
| 2136 | group2=value2 |
| 2137 | <uuid>.parquet |
| 2138 | group1=valueN |
| 2139 | group2=value1 |
| 2140 | <uuid>.parquet |
| 2141 | group2=valueN |
| 2142 | <uuid>.parquet |
| 2143 | |
| 2144 | Parameters |
| 2145 | ---------- |
| 2146 | table : pyarrow.Table |
| 2147 | root_path : str, pathlib.Path |
| 2148 | The root directory of the dataset. |
| 2149 | partition_cols : list, |
| 2150 | Column names by which to partition the dataset. |
| 2151 | Columns are partitioned in the order they are given. |
| 2152 | filesystem : FileSystem, default None |
| 2153 | If nothing passed, will be inferred based on path. |
| 2154 | Path will try to be found in the local on-disk filesystem otherwise |
| 2155 | it will be parsed as an URI to determine the filesystem. |
| 2156 | schema : Schema, optional |
| 2157 | This Schema of the dataset. |
| 2158 | partitioning : Partitioning or list[str], optional |
| 2159 | The partitioning scheme specified with the |
| 2160 | ``pyarrow.dataset.partitioning()`` function or a list of field names. |
| 2161 | When providing a list of field names, you can use |
| 2162 | ``partitioning_flavor`` to drive which partitioning type should be |
| 2163 | used. |
| 2164 | basename_template : str, optional |
| 2165 | A template string used to generate basenames of written data files. |
| 2166 | The token '{i}' will be replaced with an automatically incremented |
| 2167 | integer. If not specified, it defaults to "guid-{i}.parquet". |
| 2168 | use_threads : bool, default True |
| 2169 | Write files in parallel. If enabled, then maximum parallelism will be |
| 2170 | used determined by the number of available CPU cores. |
| 2171 | file_visitor : function |
| 2172 | If set, this function will be called with a WrittenFile instance |
| 2173 | for each file created during the call. This object will have both |
| 2174 | a path attribute and a metadata attribute. |
| 2175 | |
| 2176 | The path attribute will be a string containing the path to |
| 2177 | the created file. |
| 2178 |
nothing calls this directly
no test coverage detected