Specify a partitioning scheme. The supported schemes include: - "DirectoryPartitioning": this scheme expects one segment in the file path for each field in the specified schema (all fields are required to be present). For example given schema the pa
(schema=None, field_names=None, flavor=None,
dictionaries=None)
| 119 | |
| 120 | |
| 121 | def partitioning(schema=None, field_names=None, flavor=None, |
| 122 | dictionaries=None): |
| 123 | """ |
| 124 | Specify a partitioning scheme. |
| 125 | |
| 126 | The supported schemes include: |
| 127 | |
| 128 | - "DirectoryPartitioning": this scheme expects one segment in the file path |
| 129 | for each field in the specified schema (all fields are required to be |
| 130 | present). For example given schema<year:int16, month:int8> the path |
| 131 | "/2009/11" would be parsed to ("year"_ == 2009 and "month"_ == 11). |
| 132 | - "HivePartitioning": a scheme for "/$key=$value/" nested directories as |
| 133 | found in Apache Hive. This is a multi-level, directory based partitioning |
| 134 | scheme. Data is partitioned by static values of a particular column in |
| 135 | the schema. Partition keys are represented in the form $key=$value in |
| 136 | directory names. Field order is ignored, as are missing or unrecognized |
| 137 | field names. |
| 138 | For example, given schema<year:int16, month:int8, day:int8>, a possible |
| 139 | path would be "/year=2009/month=11/day=15" (but the field order does not |
| 140 | need to match). |
| 141 | - "FilenamePartitioning": this scheme expects the partitions will have |
| 142 | filenames containing the field values separated by "_". |
| 143 | For example, given schema<year:int16, month:int8, day:int8>, a possible |
| 144 | partition filename "2009_11_part-0.parquet" would be parsed |
| 145 | to ("year"_ == 2009 and "month"_ == 11). |
| 146 | |
| 147 | Parameters |
| 148 | ---------- |
| 149 | schema : pyarrow.Schema, default None |
| 150 | The schema that describes the partitions present in the file path. |
| 151 | If not specified, and `field_names` and/or `flavor` are specified, |
| 152 | the schema will be inferred from the file path (and a |
| 153 | PartitioningFactory is returned). |
| 154 | field_names : list of str, default None |
| 155 | A list of strings (field names). If specified, the schema's types are |
| 156 | inferred from the file paths (only valid for DirectoryPartitioning). |
| 157 | flavor : str, default None |
| 158 | The default is DirectoryPartitioning. Specify ``flavor="hive"`` for |
| 159 | a HivePartitioning, and ``flavor="filename"`` for a |
| 160 | FilenamePartitioning. |
| 161 | dictionaries : dict[str, Array] |
| 162 | If the type of any field of `schema` is a dictionary type, the |
| 163 | corresponding entry of `dictionaries` must be an array containing |
| 164 | every value which may be taken by the corresponding column or an |
| 165 | error will be raised in parsing. Alternatively, pass `infer` to have |
| 166 | Arrow discover the dictionary values, in which case a |
| 167 | PartitioningFactory is returned. |
| 168 | |
| 169 | Returns |
| 170 | ------- |
| 171 | Partitioning or PartitioningFactory |
| 172 | The partitioning scheme |
| 173 | |
| 174 | Examples |
| 175 | -------- |
| 176 | |
| 177 | Specify the Schema for paths like "/2009/June": |
| 178 |
no test coverage detected