| 1317 | |
| 1318 | |
| 1319 | class ParquetDataset: |
| 1320 | __doc__ = f""" |
| 1321 | Encapsulates details of reading a complete Parquet dataset possibly |
| 1322 | consisting of multiple files and partitions in subdirectories. |
| 1323 | |
| 1324 | Parameters |
| 1325 | ---------- |
| 1326 | path_or_paths : str or List[str] |
| 1327 | A directory name, single file name, or list of file names. |
| 1328 | filesystem : FileSystem, default None |
| 1329 | If nothing passed, will be inferred based on path. |
| 1330 | Path will try to be found in the local on-disk filesystem otherwise |
| 1331 | it will be parsed as an URI to determine the filesystem. |
| 1332 | schema : pyarrow.parquet.Schema |
| 1333 | Optionally provide the Schema for the Dataset, in which case it will |
| 1334 | not be inferred from the source. |
| 1335 | filters : pyarrow.compute.Expression or List[Tuple] or List[List[Tuple]], default None |
| 1336 | Rows which do not match the filter predicate will be removed from scanned |
| 1337 | data. Partition keys embedded in a nested directory structure will be |
| 1338 | exploited to avoid loading files at all if they contain no matching rows. |
| 1339 | Within-file level filtering and different partitioning schemes are supported. |
| 1340 | |
| 1341 | {_DNF_filter_doc} |
| 1342 | {_read_docstring_common} |
| 1343 | ignore_prefixes : list, optional |
| 1344 | Files matching any of these prefixes will be ignored by the |
| 1345 | discovery process. |
| 1346 | This is matched to the basename of a path. |
| 1347 | By default this is ['.', '_']. |
| 1348 | Note that discovery happens only if a directory is passed as source. |
| 1349 | pre_buffer : bool, default True |
| 1350 | Coalesce and issue file reads in parallel to improve performance on |
| 1351 | high-latency filesystems (e.g. S3, GCS). If True, Arrow will use a |
| 1352 | background I/O thread pool. If using a filesystem layer that itself |
| 1353 | performs readahead (e.g. fsspec's S3FS), disable readahead for best |
| 1354 | results. Set to False if you want to prioritize minimal memory usage |
| 1355 | over maximum speed. |
| 1356 | coerce_int96_timestamp_unit : str, default None |
| 1357 | Cast timestamps that are stored in INT96 format to a particular resolution |
| 1358 | (e.g. 'ms'). Setting to None is equivalent to 'ns' and therefore INT96 |
| 1359 | timestamps will be inferred as timestamps in nanoseconds. |
| 1360 | decryption_properties : FileDecryptionProperties or None |
| 1361 | File-level decryption properties. |
| 1362 | The decryption properties can be created using |
| 1363 | ``CryptoFactory.file_decryption_properties()``. |
| 1364 | thrift_string_size_limit : int, default None |
| 1365 | If not None, override the maximum total string size allocated |
| 1366 | when decoding Thrift structures. The default limit should be |
| 1367 | sufficient for most Parquet files. |
| 1368 | thrift_container_size_limit : int, default None |
| 1369 | If not None, override the maximum total size of containers allocated |
| 1370 | when decoding Thrift structures. The default limit should be |
| 1371 | sufficient for most Parquet files. |
| 1372 | page_checksum_verification : bool, default False |
| 1373 | If True, verify the page checksum for each page read from the file. |
| 1374 | arrow_extensions_enabled : bool, default True |
| 1375 | If True, read Parquet logical types as Arrow extension types where possible, |
| 1376 | (e.g., read JSON as the canonical `arrow.json` extension type or UUID as |