MCPcopy Create free account
hub / github.com/apache/arrow / test_file_format_inspect_fsspec

Function test_file_format_inspect_fsspec

python/pyarrow/tests/test_dataset.py:3081–3103  ·  view source on GitHub ↗
(tempdir)

Source from the content-addressed store, hash-verified

3079
3080@pytest.mark.parquet
3081def test_file_format_inspect_fsspec(tempdir):
3082 # https://issues.apache.org/jira/browse/ARROW-16413
3083 fsspec = pytest.importorskip("fsspec")
3084
3085 # create bucket + file with pyarrow
3086 table = pa.table({'a': [1, 2, 3]})
3087 path = tempdir / "data.parquet"
3088 pq.write_table(table, path)
3089
3090 # read using fsspec filesystem
3091 fsspec_fs = fsspec.filesystem("file")
3092 assert fsspec_fs.ls(tempdir)[0].endswith("data.parquet")
3093
3094 # inspect using dataset file format
3095 format = ds.ParquetFileFormat()
3096 # manually creating a PyFileSystem instead of using fs._ensure_filesystem
3097 # which would convert an fsspec local filesystem to a native one
3098 filesystem = fs.PyFileSystem(fs.FSSpecHandler(fsspec_fs))
3099 schema = format.inspect(path, filesystem)
3100 assert schema.equals(table.schema)
3101
3102 fragment = format.make_fragment(path, filesystem)
3103 assert fragment.physical_schema.equals(table.schema)
3104
3105
3106@pytest.mark.pandas

Callers

nothing calls this directly

Calls 6

filesystemMethod · 0.80
ParquetFileFormatMethod · 0.80
PyFileSystemMethod · 0.80
equalsMethod · 0.80
write_tableMethod · 0.45
inspectMethod · 0.45

Tested by

no test coverage detected