MCPcopy Create free account
hub / github.com/apache/arrow / test_read_partition_keys_only

Function test_read_partition_keys_only

python/pyarrow/tests/test_dataset.py:2245–2267  ·  view source on GitHub ↗
(tempdir)

Source from the content-addressed store, hash-verified

2243@pytest.mark.parquet
2244@pytest.mark.pandas
2245def test_read_partition_keys_only(tempdir):
2246 BATCH_SIZE = 2 ** 15
2247 # This is a regression test for ARROW-15318 which saw issues
2248 # reading only the partition keys from files with batches larger
2249 # than the default batch size (e.g. so we need to return two chunks)
2250 table = pa.table({
2251 'key': pa.repeat(0, BATCH_SIZE + 1),
2252 'value': np.arange(BATCH_SIZE + 1)})
2253 pq.write_to_dataset(
2254 table[:BATCH_SIZE],
2255 tempdir / 'one', partition_cols=['key'])
2256 pq.write_to_dataset(
2257 table[:BATCH_SIZE + 1],
2258 tempdir / 'two', partition_cols=['key'])
2259
2260 table = pq.read_table(tempdir / 'one', columns=['key'])
2261 assert table['key'].num_chunks == 1
2262
2263 table = pq.read_table(tempdir / 'two', columns=['key', 'value'])
2264 assert table['key'].num_chunks == 2
2265
2266 table = pq.read_table(tempdir / 'two', columns=['key'])
2267 assert table['key'].num_chunks == 2
2268
2269
2270def _has_subdirs(basedir):

Callers

nothing calls this directly

Calls 1

read_tableMethod · 0.45

Tested by

no test coverage detected