MCPcopy Create free account
hub / github.com/apache/arrow / test_file_format_pickling

Function test_file_format_pickling

python/pyarrow/tests/test_dataset.py:1073–1110  ·  view source on GitHub ↗
(pickle_module)

Source from the content-addressed store, hash-verified

1071
1072
1073def test_file_format_pickling(pickle_module):
1074 formats = [
1075 ds.IpcFileFormat(),
1076 ds.CsvFileFormat(),
1077 ds.CsvFileFormat(pa.csv.ParseOptions(delimiter='\t',
1078 ignore_empty_lines=True)),
1079 ds.CsvFileFormat(read_options=pa.csv.ReadOptions(
1080 skip_rows=3, column_names=['foo'])),
1081 ds.CsvFileFormat(read_options=pa.csv.ReadOptions(
1082 skip_rows=3, block_size=2**20)),
1083 ds.JsonFileFormat(),
1084 ds.JsonFileFormat(
1085 parse_options=pa.json.ParseOptions(newlines_in_values=True,
1086 unexpected_field_behavior="ignore")),
1087 ds.JsonFileFormat(read_options=pa.json.ReadOptions(
1088 use_threads=False, block_size=14)),
1089 ]
1090 try:
1091 formats.append(ds.OrcFileFormat())
1092 except ImportError:
1093 pass
1094
1095 if pq is not None:
1096 formats.extend([
1097 ds.ParquetFileFormat(),
1098 ds.ParquetFileFormat(dictionary_columns={'a'}),
1099 ds.ParquetFileFormat(use_buffered_stream=True),
1100 ds.ParquetFileFormat(
1101 use_buffered_stream=True,
1102 buffer_size=4096,
1103 thrift_string_size_limit=123,
1104 thrift_container_size_limit=456,
1105 schema_depth_limit=42,
1106 ),
1107 ])
1108
1109 for file_format in formats:
1110 assert pickle_module.loads(pickle_module.dumps(file_format)) == file_format
1111
1112
1113def test_fragment_scan_options_pickling(pickle_module):

Callers

nothing calls this directly

Calls 9

IpcFileFormatMethod · 0.80
CsvFileFormatMethod · 0.80
JsonFileFormatMethod · 0.80
OrcFileFormatMethod · 0.80
extendMethod · 0.80
ParquetFileFormatMethod · 0.80
loadsMethod · 0.80
dumpsMethod · 0.80
appendMethod · 0.45

Tested by

no test coverage detected