MCPcopy Create free account
hub / github.com/apache/arrow / test_json_fragment_options

Function test_json_fragment_options

python/pyarrow/tests/test_dataset.py:3637–3656  ·  view source on GitHub ↗
(tempdir, dataset_reader)

Source from the content-addressed store, hash-verified

3635
3636@pytest.mark.pandas
3637def test_json_fragment_options(tempdir, dataset_reader):
3638 table = pa.table({'a': pa.array([1, 2, 3], type="int64"),
3639 'b': pa.array([.1, .2, .3], type="float64")})
3640
3641 path = str(tempdir / 'test.json')
3642 out = table.to_pandas().to_json(orient='records')[1:-1].replace('},{', '}\n{')
3643 with open(path, 'w') as f:
3644 f.write(out)
3645
3646 with pytest.raises(ValueError,
3647 match="try to increase block size"):
3648 options = ds.JsonFragmentScanOptions(
3649 read_options=pa.json.ReadOptions(block_size=4))
3650 dataset = ds.dataset(path, format=ds.JsonFileFormat(options))
3651
3652 options = ds.JsonFragmentScanOptions(
3653 read_options=pa.json.ReadOptions(block_size=64))
3654 dataset = ds.dataset(path, format=ds.JsonFileFormat(options))
3655 result = dataset_reader.to_table(dataset)
3656 assert result.equals(table)
3657
3658
3659def test_encoding(tempdir, dataset_reader):

Callers

nothing calls this directly

Calls 5

JsonFileFormatMethod · 0.80
equalsMethod · 0.80
arrayMethod · 0.45
writeMethod · 0.45
to_tableMethod · 0.45

Tested by

no test coverage detected