MCPcopy Create free account
hub / github.com/apache/arrow / test_column_names_encoding

Function test_column_names_encoding

python/pyarrow/tests/test_dataset.py:3684–3705  ·  view source on GitHub ↗
(tempdir, dataset_reader)

Source from the content-addressed store, hash-verified

3682
3683# Test if a dataset with non-utf8 chars in the column names is properly handled
3684def test_column_names_encoding(tempdir, dataset_reader):
3685 path = str(tempdir / 'test.csv')
3686
3687 with open(path, 'wb') as sink:
3688 sink.write(b"\xe9,b\nun,\xe9l\xe9phant")
3689
3690 # Interpret as utf8:
3691 expected_schema = pa.schema([("é", pa.string()), ("b", pa.string())])
3692 expected_table = pa.table({'é': ["un"],
3693 'b': ["éléphant"]}, schema=expected_schema)
3694
3695 # Reading as string without specifying encoding should produce an error
3696 dataset = ds.dataset(path, format='csv', schema=expected_schema)
3697 with pytest.raises(pyarrow.lib.ArrowInvalid, match="invalid UTF8"):
3698 dataset_reader.to_table(dataset)
3699
3700 # Setting the encoding in the read_options should transcode the data
3701 read_options = pa.csv.ReadOptions(encoding='latin-1')
3702 file_format = ds.CsvFileFormat(read_options=read_options)
3703 dataset_transcoded = ds.dataset(path, format=file_format)
3704 assert dataset_transcoded.schema.equals(expected_schema)
3705 assert dataset_transcoded.to_table().equals(expected_table)
3706
3707
3708@pytest.mark.filterwarnings("ignore:Feather V1:DeprecationWarning")

Callers

nothing calls this directly

Calls 6

CsvFileFormatMethod · 0.80
equalsMethod · 0.80
writeMethod · 0.45
schemaMethod · 0.45
stringMethod · 0.45
to_tableMethod · 0.45

Tested by

no test coverage detected