MCPcopy Create free account
hub / github.com/apache/arrow / test_hive_partitioning_dictionary_key

Function test_hive_partitioning_dictionary_key

python/pyarrow/tests/test_dataset.py:2371–2396  ·  view source on GitHub ↗
(multisourcefs)

Source from the content-addressed store, hash-verified

2369
2370
2371def test_hive_partitioning_dictionary_key(multisourcefs):
2372 # ARROW-8088 specifying partition key as dictionary type
2373 schema = pa.schema([
2374 pa.field('year', pa.dictionary(pa.int8(), pa.int16())),
2375 pa.field('month', pa.dictionary(pa.int8(), pa.int16()))
2376 ])
2377 part = ds.HivePartitioning.discover(schema=schema)
2378
2379 dataset = ds.dataset(
2380 "hive", format="parquet", filesystem=multisourcefs, partitioning=part
2381 )
2382 assert dataset.partitioning.schema == schema
2383 table = dataset.to_table()
2384
2385 year_dictionary = list(range(2006, 2011))
2386 month_dictionary = list(range(1, 13))
2387 assert table.column('year').type.equals(schema.types[0])
2388 for chunk in table.column('year').chunks:
2389 actual = chunk.dictionary.to_pylist()
2390 actual.sort()
2391 assert actual == year_dictionary
2392 assert table.column('month').type.equals(schema.types[1])
2393 for chunk in table.column('month').chunks:
2394 actual = chunk.dictionary.to_pylist()
2395 actual.sort()
2396 assert actual == month_dictionary
2397
2398
2399def _create_single_file(base_dir, table=None, row_group_size=None):

Callers

nothing calls this directly

Calls 7

listFunction · 0.85
equalsMethod · 0.80
schemaMethod · 0.45
fieldMethod · 0.45
dictionaryMethod · 0.45
to_tableMethod · 0.45
columnMethod · 0.45

Tested by

no test coverage detected