MCPcopy Create free account
hub / github.com/apache/arrow / test_encoding

Method test_encoding

python/pyarrow/tests/test_csv.py:1859–1887  ·  view source on GitHub ↗
(self)

Source from the content-addressed store, hash-verified

1857 'f': [3, 6]}])
1858
1859 def test_encoding(self):
1860 # latin-1 (invalid utf-8)
1861 rows = b"a,b\nun,\xe9l\xe9phant"
1862 read_options = ReadOptions()
1863 reader = self.open_bytes(rows, read_options=read_options)
1864 expected_schema = pa.schema([('a', pa.string()),
1865 ('b', pa.binary())])
1866 self.check_reader(reader, expected_schema,
1867 [{'a': ["un"],
1868 'b': [b"\xe9l\xe9phant"]}])
1869
1870 read_options.encoding = 'latin1'
1871 reader = self.open_bytes(rows, read_options=read_options)
1872 expected_schema = pa.schema([('a', pa.string()),
1873 ('b', pa.string())])
1874 self.check_reader(reader, expected_schema,
1875 [{'a': ["un"],
1876 'b': ["éléphant"]}])
1877
1878 # utf-16
1879 rows = (b'\xff\xfea\x00,\x00b\x00\n\x00u\x00n\x00,'
1880 b'\x00\xe9\x00l\x00\xe9\x00p\x00h\x00a\x00n\x00t\x00')
1881 read_options.encoding = 'utf16'
1882 reader = self.open_bytes(rows, read_options=read_options)
1883 expected_schema = pa.schema([('a', pa.string()),
1884 ('b', pa.string())])
1885 self.check_reader(reader, expected_schema,
1886 [{'a': ["un"],
1887 'b': ["éléphant"]}])
1888
1889 def test_small_random_csv(self):
1890 csv, expected = make_random_csv(num_cols=2, num_rows=10)

Callers

nothing calls this directly

Calls 5

open_bytesMethod · 0.95
check_readerMethod · 0.95
ReadOptionsClass · 0.90
schemaMethod · 0.45
stringMethod · 0.45

Tested by

no test coverage detected