MCPcopy Create free account
hub / github.com/apache/arrow / write_metadata

Function write_metadata

python/pyarrow/parquet/core.py:2325–2394  ·  view source on GitHub ↗

Write metadata-only Parquet file from schema. This can be used with `write_to_dataset` to generate `_common_metadata` and `_metadata` sidecar files. Parameters ---------- schema : pyarrow.Schema where : string or pyarrow.NativeFile metadata_collector : list

(schema, where, metadata_collector=None, filesystem=None,
                   **kwargs)

Source from the content-addressed store, hash-verified

2323
2324
2325def write_metadata(schema, where, metadata_collector=None, filesystem=None,
2326 **kwargs):
2327 """
2328 Write metadata-only Parquet file from schema. This can be used with
2329 `write_to_dataset` to generate `_common_metadata` and `_metadata` sidecar
2330 files.
2331
2332 Parameters
2333 ----------
2334 schema : pyarrow.Schema
2335 where : string or pyarrow.NativeFile
2336 metadata_collector : list
2337 where to collect metadata information.
2338 filesystem : FileSystem, default None
2339 If nothing passed, will be inferred from `where` if path-like, else
2340 `where` is already a file-like object so no filesystem is needed.
2341 **kwargs : dict,
2342 Additional kwargs for ParquetWriter class. See docstring for
2343 `ParquetWriter` for more information.
2344
2345 Examples
2346 --------
2347 Generate example data:
2348
2349 >>> import pyarrow as pa
2350 >>> table = pa.table({'n_legs': [2, 2, 4, 4, 5, 100],
2351 ... 'animal': ["Flamingo", "Parrot", "Dog", "Horse",
2352 ... "Brittle stars", "Centipede"]})
2353
2354 Write a dataset and collect metadata information.
2355
2356 >>> metadata_collector = []
2357 >>> import pyarrow.parquet as pq
2358 >>> pq.write_to_dataset(
2359 ... table, 'dataset_metadata',
2360 ... metadata_collector=metadata_collector)
2361
2362 Write the `_common_metadata` parquet file without row groups statistics.
2363
2364 >>> pq.write_metadata(
2365 ... table.schema, 'dataset_metadata/_common_metadata')
2366
2367 Write the `_metadata` parquet file with row groups statistics.
2368
2369 >>> pq.write_metadata(
2370 ... table.schema, 'dataset_metadata/_metadata',
2371 ... metadata_collector=metadata_collector)
2372 """
2373 filesystem, where = _resolve_filesystem_and_path(where, filesystem)
2374
2375 if hasattr(where, "seek"): # file-like
2376 cursor_position = where.tell()
2377
2378 writer = ParquetWriter(where, schema, filesystem, **kwargs)
2379 writer.close()
2380
2381 if metadata_collector is not None:
2382 # ParquetWriter doesn't expose the metadata until it's written. Write

Callers

nothing calls this directly

Calls 6

closeMethod · 0.95
ParquetWriterClass · 0.85
read_metadataFunction · 0.85
seekMethod · 0.80
open_output_streamMethod · 0.45

Tested by

no test coverage detected