Open a dataset. Datasets provides functionality to efficiently work with tabular, potentially larger than memory and multi-file dataset. - A unified interface for different sources, like Parquet and Feather - Discovery of sources (crawling directories, handle directory-based
(source, schema=None, format=None, filesystem=None,
partitioning=None, partition_base_dir=None,
exclude_invalid_files=None, ignore_prefixes=None)
| 578 | |
| 579 | |
| 580 | def dataset(source, schema=None, format=None, filesystem=None, |
| 581 | partitioning=None, partition_base_dir=None, |
| 582 | exclude_invalid_files=None, ignore_prefixes=None): |
| 583 | """ |
| 584 | Open a dataset. |
| 585 | |
| 586 | Datasets provides functionality to efficiently work with tabular, |
| 587 | potentially larger than memory and multi-file dataset. |
| 588 | |
| 589 | - A unified interface for different sources, like Parquet and Feather |
| 590 | - Discovery of sources (crawling directories, handle directory-based |
| 591 | partitioned datasets, basic schema normalization) |
| 592 | - Optimized reading with predicate pushdown (filtering rows), projection |
| 593 | (selecting columns), parallel reading or fine-grained managing of tasks. |
| 594 | |
| 595 | Note that this is the high-level API, to have more control over the dataset |
| 596 | construction use the low-level API classes (FileSystemDataset, |
| 597 | FilesystemDatasetFactory, etc.) |
| 598 | |
| 599 | Parameters |
| 600 | ---------- |
| 601 | source : path, list of paths, dataset, list of datasets, (list of) \ |
| 602 | RecordBatch or Table, iterable of RecordBatch, RecordBatchReader, or URI |
| 603 | Path pointing to a single file: |
| 604 | Open a FileSystemDataset from a single file. |
| 605 | Path pointing to a directory: |
| 606 | The directory gets discovered recursively according to a |
| 607 | partitioning scheme if given. |
| 608 | List of file paths: |
| 609 | Create a FileSystemDataset from explicitly given files. The files |
| 610 | must be located on the same filesystem given by the filesystem |
| 611 | parameter. |
| 612 | Note that in contrary of construction from a single file, passing |
| 613 | URIs as paths is not allowed. |
| 614 | List of datasets: |
| 615 | A nested UnionDataset gets constructed, it allows arbitrary |
| 616 | composition of other datasets. |
| 617 | Note that additional keyword arguments are not allowed. |
| 618 | (List of) batches or tables, iterable of batches, or RecordBatchReader: |
| 619 | Create an InMemoryDataset. If an iterable or empty list is given, |
| 620 | a schema must also be given. If an iterable or RecordBatchReader |
| 621 | is given, the resulting dataset can only be scanned once; further |
| 622 | attempts will raise an error. |
| 623 | schema : Schema, optional |
| 624 | Optionally provide the Schema for the Dataset, in which case it will |
| 625 | not be inferred from the source. |
| 626 | format : FileFormat or str |
| 627 | Currently "parquet", "ipc"/"arrow"/"feather", "csv", "json", and "orc" are |
| 628 | supported. For Feather, only version 2 files are supported. |
| 629 | filesystem : FileSystem or URI string, default None |
| 630 | If a single path is given as source and filesystem is None, then the |
| 631 | filesystem will be inferred from the path. |
| 632 | If an URI string is passed, then a filesystem object is constructed |
| 633 | using the URI's optional path component as a directory prefix. See the |
| 634 | examples below. |
| 635 | Note that the URIs on Windows must follow 'file:///C:...' or |
| 636 | 'file:/C:...' patterns. |
| 637 | partitioning : Partitioning, PartitioningFactory, str, list of str |
nothing calls this directly
no test coverage detected