MCPcopy Create free account
hub / github.com/apache/arrow / test_write_dataset_write_page_index

Function test_write_dataset_write_page_index

python/pyarrow/tests/test_dataset.py:5783–5812  ·  view source on GitHub ↗
(tempdir)

Source from the content-addressed store, hash-verified

5781
5782
5783def test_write_dataset_write_page_index(tempdir):
5784 for write_statistics in [True, False]:
5785 for write_page_index in [True, False]:
5786 schema = pa.schema([
5787 pa.field("x", pa.int64()),
5788 pa.field("y", pa.int64())])
5789
5790 arrays = [[1, 2, 3], [None, 5, None]]
5791 table = pa.Table.from_arrays(arrays, schema=schema)
5792
5793 file_format = ds.ParquetFileFormat()
5794 base_dir = tempdir / f"write_page_index_{write_page_index}"
5795 ds.write_dataset(
5796 table,
5797 base_dir,
5798 format="parquet",
5799 file_options=file_format.make_write_options(
5800 write_statistics=write_statistics,
5801 write_page_index=write_page_index,
5802 ),
5803 existing_data_behavior='overwrite_or_ignore',
5804 )
5805 ds1 = ds.dataset(base_dir, format="parquet")
5806
5807 for file in ds1.files:
5808 # Can retrieve sorting columns from metadata
5809 metadata = pq.read_metadata(file)
5810 cc = metadata.row_group(0).column(0)
5811 assert cc.has_offset_index is write_page_index
5812 assert cc.has_column_index is write_page_index & write_statistics
5813
5814
5815@pytest.mark.parametrize('dstype', [

Callers

nothing calls this directly

Calls 5

ParquetFileFormatMethod · 0.80
read_metadataMethod · 0.80
schemaMethod · 0.45
fieldMethod · 0.45
columnMethod · 0.45

Tested by

no test coverage detected