MCPcopy Create free account
hub / github.com/apache/arrow / scan_contents

Method scan_contents

python/pyarrow/parquet/core.py:651–687  ·  view source on GitHub ↗

Read contents of file for the given columns and batch size. Notes ----- This function's primary purpose is benchmarking. The scan is executed on a single thread. Parameters ---------- columns : list of integers, default None

(self, columns=None, batch_size=65536)

Source from the content-addressed store, hash-verified

649 use_threads=use_threads)
650
651 def scan_contents(self, columns=None, batch_size=65536):
652 """
653 Read contents of file for the given columns and batch size.
654
655 Notes
656 -----
657 This function's primary purpose is benchmarking.
658 The scan is executed on a single thread.
659
660 Parameters
661 ----------
662 columns : list of integers, default None
663 Select columns to read, if None scan all columns.
664 batch_size : int, default 64K
665 Number of rows to read at a time internally.
666
667 Returns
668 -------
669 num_rows : int
670 Number of rows in file
671
672 Examples
673 --------
674 >>> import pyarrow as pa
675 >>> table = pa.table({'n_legs': [2, 2, 4, 4, 5, 100],
676 ... 'animal': ["Flamingo", "Parrot", "Dog", "Horse",
677 ... "Brittle stars", "Centipede"]})
678 >>> import pyarrow.parquet as pq
679 >>> pq.write_table(table, 'example.parquet')
680 >>> parquet_file = pq.ParquetFile('example.parquet')
681
682 >>> parquet_file.scan_contents()
683 6
684 """
685 column_indices = self._get_column_indices(columns)
686 return self.reader.scan_contents(column_indices,
687 batch_size=batch_size)
688
689 def _get_column_indices(self, column_names, use_pandas_metadata=False):
690 if column_names is None:

Callers 1

test_scan_contentsFunction · 0.95

Calls 1

_get_column_indicesMethod · 0.95

Tested by 1

test_scan_contentsFunction · 0.76