MCPcopy Create free account
hub / github.com/apache/arrow / ParquetDataset

Class ParquetDataset

python/pyarrow/parquet/core.py:1332–1741  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1330
1331
1332class ParquetDataset:
1333 __doc__ = f"""
1334Encapsulates details of reading a complete Parquet dataset possibly
1335consisting of multiple files and partitions in subdirectories.
1336
1337Parameters
1338----------
1339path_or_paths : str or List[str]
1340 A directory name, single file name, or list of file names.
1341filesystem : FileSystem, default None
1342 If nothing passed, will be inferred based on path.
1343 Path will try to be found in the local on-disk filesystem otherwise
1344 it will be parsed as an URI to determine the filesystem.
1345schema : pyarrow.parquet.Schema
1346 Optionally provide the Schema for the Dataset, in which case it will
1347 not be inferred from the source.
1348filters : pyarrow.compute.Expression or List[Tuple] or List[List[Tuple]], default None
1349 Rows which do not match the filter predicate will be removed from scanned
1350 data. Partition keys embedded in a nested directory structure will be
1351 exploited to avoid loading files at all if they contain no matching rows.
1352 Within-file level filtering and different partitioning schemes are supported.
1353
1354 {_DNF_filter_doc}
1355{_read_docstring_common}
1356ignore_prefixes : list, optional
1357 Files matching any of these prefixes will be ignored by the
1358 discovery process.
1359 This is matched to the basename of a path.
1360 By default this is ['.', '_'].
1361 Note that discovery happens only if a directory is passed as source.
1362pre_buffer : bool, default True
1363 Coalesce and issue file reads in parallel to improve performance on
1364 high-latency filesystems (e.g. S3, GCS). If True, Arrow will use a
1365 background I/O thread pool. If using a filesystem layer that itself
1366 performs readahead (e.g. fsspec's S3FS), disable readahead for best
1367 results. Set to False if you want to prioritize minimal memory usage
1368 over maximum speed.
1369coerce_int96_timestamp_unit : str, default None
1370 Cast timestamps that are stored in INT96 format to a particular resolution
1371 (e.g. 'ms'). Setting to None is equivalent to 'ns' and therefore INT96
1372 timestamps will be inferred as timestamps in nanoseconds.
1373decryption_properties : FileDecryptionProperties or None
1374 File-level decryption properties.
1375 The decryption properties can be created using
1376 ``CryptoFactory.file_decryption_properties()``.
1377thrift_string_size_limit : int, default None
1378 If not None, override the maximum total string size allocated
1379 when decoding Thrift structures. The default limit should be
1380 sufficient for most Parquet files.
1381thrift_container_size_limit : int, default None
1382 If not None, override the maximum total size of containers allocated
1383 when decoding Thrift structures. The default limit should be
1384 sufficient for most Parquet files.
1385schema_depth_limit : int, default None
1386 If not None, override the maximum nesting depth of the Parquet file schema.
1387 This guards against recursion overflow on invalid schemas.
1388 The default limit should be sufficient for most Parquet files.
1389page_checksum_verification : bool, default False

Callers 1

read_tableFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected