MCPcopy Create free account
hub / github.com/apache/arrow / test_csv_format_compressed

Function test_csv_format_compressed

python/pyarrow/tests/test_dataset.py:3513–3530  ·  view source on GitHub ↗
(tempdir, compression, dataset_reader)

Source from the content-addressed store, hash-verified

3511 "zstd",
3512])
3513def test_csv_format_compressed(tempdir, compression, dataset_reader):
3514 if not pyarrow.Codec.is_available(compression):
3515 pytest.skip(f"{compression} support is not built")
3516 table = pa.table({'a': pa.array([1, 2, 3], type="int64"),
3517 'b': pa.array([.1, .2, .3], type="float64")})
3518 filesystem = fs.LocalFileSystem()
3519 suffix = compression if compression != 'gzip' else 'gz'
3520 path = str(tempdir / f'test.csv.{suffix}')
3521 with filesystem.open_output_stream(path, compression=compression) as sink:
3522 # https://github.com/pandas-dev/pandas/issues/23854
3523 # With CI version of Pandas (anything < 1.2), Pandas tries to write
3524 # str to the sink
3525 csv_str = table.to_pandas().to_csv(index=False)
3526 sink.write(csv_str.encode('utf-8'))
3527
3528 dataset = ds.dataset(path, format=ds.CsvFileFormat())
3529 result = dataset_reader.to_table(dataset)
3530 assert result.equals(table)
3531
3532
3533def test_csv_format_options(tempdir, dataset_reader):

Callers

nothing calls this directly

Calls 8

LocalFileSystemMethod · 0.80
CsvFileFormatMethod · 0.80
equalsMethod · 0.80
arrayMethod · 0.45
open_output_streamMethod · 0.45
writeMethod · 0.45
encodeMethod · 0.45
to_tableMethod · 0.45

Tested by

no test coverage detected