MCPcopy Create free account
hub / github.com/apache/arrow / _check_dataset_roundtrip

Function _check_dataset_roundtrip

python/pyarrow/tests/test_dataset.py:4205–4221  ·  view source on GitHub ↗
(dataset, base_dir, expected_files, sort_col,
                             base_dir_path=None, partitioning=None)

Source from the content-addressed store, hash-verified

4203
4204
4205def _check_dataset_roundtrip(dataset, base_dir, expected_files, sort_col,
4206 base_dir_path=None, partitioning=None):
4207 base_dir_path = base_dir_path or base_dir
4208
4209 ds.write_dataset(dataset, base_dir, format="arrow",
4210 partitioning=partitioning, use_threads=False)
4211
4212 # check that all files are present
4213 file_paths = list(base_dir_path.rglob("*"))
4214 assert set(file_paths) == set(expected_files)
4215
4216 # check that reading back in as dataset gives the same result
4217 dataset2 = ds.dataset(
4218 base_dir_path, format="arrow", partitioning=partitioning)
4219
4220 assert _sort_table(dataset2.to_table(), sort_col).equals(
4221 _sort_table(dataset.to_table(), sort_col))
4222
4223
4224@pytest.mark.parquet

Callers 3

test_write_datasetFunction · 0.85

Calls 4

listFunction · 0.85
_sort_tableFunction · 0.85
equalsMethod · 0.80
to_tableMethod · 0.45

Tested by

no test coverage detected