| 1320 | |
| 1321 | |
| 1322 | class ParquetDataset: |
| 1323 | __doc__ = f""" |
| 1324 | Encapsulates details of reading a complete Parquet dataset possibly |
| 1325 | consisting of multiple files and partitions in subdirectories. |
| 1326 | |
| 1327 | Parameters |
| 1328 | ---------- |
| 1329 | path_or_paths : str or List[str] |
| 1330 | A directory name, single file name, or list of file names. |
| 1331 | filesystem : FileSystem, default None |
| 1332 | If nothing passed, will be inferred based on path. |
| 1333 | Path will try to be found in the local on-disk filesystem otherwise |
| 1334 | it will be parsed as an URI to determine the filesystem. |
| 1335 | schema : pyarrow.parquet.Schema |
| 1336 | Optionally provide the Schema for the Dataset, in which case it will |
| 1337 | not be inferred from the source. |
| 1338 | filters : pyarrow.compute.Expression or List[Tuple] or List[List[Tuple]], default None |
| 1339 | Rows which do not match the filter predicate will be removed from scanned |
| 1340 | data. Partition keys embedded in a nested directory structure will be |
| 1341 | exploited to avoid loading files at all if they contain no matching rows. |
| 1342 | Within-file level filtering and different partitioning schemes are supported. |
| 1343 | |
| 1344 | {_DNF_filter_doc} |
| 1345 | {_read_docstring_common} |
| 1346 | ignore_prefixes : list, optional |
| 1347 | Files matching any of these prefixes will be ignored by the |
| 1348 | discovery process. |
| 1349 | This is matched to the basename of a path. |
| 1350 | By default this is ['.', '_']. |
| 1351 | Note that discovery happens only if a directory is passed as source. |
| 1352 | pre_buffer : bool, default True |
| 1353 | Coalesce and issue file reads in parallel to improve performance on |
| 1354 | high-latency filesystems (e.g. S3, GCS). If True, Arrow will use a |
| 1355 | background I/O thread pool. If using a filesystem layer that itself |
| 1356 | performs readahead (e.g. fsspec's S3FS), disable readahead for best |
| 1357 | results. Set to False if you want to prioritize minimal memory usage |
| 1358 | over maximum speed. |
| 1359 | coerce_int96_timestamp_unit : str, default None |
| 1360 | Cast timestamps that are stored in INT96 format to a particular resolution |
| 1361 | (e.g. 'ms'). Setting to None is equivalent to 'ns' and therefore INT96 |
| 1362 | timestamps will be inferred as timestamps in nanoseconds. |
| 1363 | decryption_properties : FileDecryptionProperties or None |
| 1364 | File-level decryption properties. |
| 1365 | The decryption properties can be created using |
| 1366 | ``CryptoFactory.file_decryption_properties()``. |
| 1367 | thrift_string_size_limit : int, default None |
| 1368 | If not None, override the maximum total string size allocated |
| 1369 | when decoding Thrift structures. The default limit should be |
| 1370 | sufficient for most Parquet files. |
| 1371 | thrift_container_size_limit : int, default None |
| 1372 | If not None, override the maximum total size of containers allocated |
| 1373 | when decoding Thrift structures. The default limit should be |
| 1374 | sufficient for most Parquet files. |
| 1375 | page_checksum_verification : bool, default False |
| 1376 | If True, verify the page checksum for each page read from the file. |
| 1377 | arrow_extensions_enabled : bool, default True |
| 1378 | If True, read Parquet logical types as Arrow extension types where possible, |
| 1379 | (e.g., read JSON as the canonical `arrow.json` extension type or UUID as |