MCPcopy Create free account
hub / github.com/apache/arrow / test_csv_fragment_options

Function test_csv_fragment_options

python/pyarrow/tests/test_dataset.py:3570–3591  ·  view source on GitHub ↗
(tempdir, dataset_reader)

Source from the content-addressed store, hash-verified

3568
3569
3570def test_csv_fragment_options(tempdir, dataset_reader):
3571 path = str(tempdir / 'test.csv')
3572 with open(path, 'w') as sink:
3573 sink.write('col0\nfoo\nspam\nMYNULL\n')
3574 dataset = ds.dataset(path, format='csv')
3575 convert_options = pyarrow.csv.ConvertOptions(null_values=['MYNULL'],
3576 strings_can_be_null=True)
3577 options = ds.CsvFragmentScanOptions(
3578 convert_options=convert_options,
3579 read_options=pa.csv.ReadOptions(block_size=2**16))
3580 result = dataset_reader.to_table(dataset, fragment_scan_options=options)
3581 assert result.equals(pa.table({'col0': pa.array(['foo', 'spam', None])}))
3582
3583 csv_format = ds.CsvFileFormat(convert_options=convert_options)
3584 dataset = ds.dataset(path, format=csv_format)
3585 result = dataset_reader.to_table(dataset)
3586 assert result.equals(pa.table({'col0': pa.array(['foo', 'spam', None])}))
3587
3588 options = ds.CsvFragmentScanOptions()
3589 result = dataset_reader.to_table(dataset, fragment_scan_options=options)
3590 assert result.equals(
3591 pa.table({'col0': pa.array(['foo', 'spam', 'MYNULL'])}))
3592
3593
3594@pytest.mark.pandas

Callers

nothing calls this directly

Calls 5

equalsMethod · 0.80
CsvFileFormatMethod · 0.80
writeMethod · 0.45
to_tableMethod · 0.45
arrayMethod · 0.45

Tested by

no test coverage detected