MCPcopy Create free account
hub / github.com/apache/arrow / _reconstruct_columns_from_metadata

Function _reconstruct_columns_from_metadata

python/pyarrow/pandas_compat.py:1126–1211  ·  view source on GitHub ↗

Construct a pandas MultiIndex from `columns` and column index metadata in `column_indexes`. Parameters ---------- columns : List[pd.Index] The columns coming from a pyarrow.Table column_indexes : List[Dict[str, str]] The column index metadata deserialized from th

(columns, column_indexes)

Source from the content-addressed store, hash-verified

1124
1125
1126def _reconstruct_columns_from_metadata(columns, column_indexes):
1127 """Construct a pandas MultiIndex from `columns` and column index metadata
1128 in `column_indexes`.
1129
1130 Parameters
1131 ----------
1132 columns : List[pd.Index]
1133 The columns coming from a pyarrow.Table
1134 column_indexes : List[Dict[str, str]]
1135 The column index metadata deserialized from the JSON schema metadata
1136 in a :class:`~pyarrow.Table`.
1137
1138 Returns
1139 -------
1140 result : MultiIndex
1141 The index reconstructed using `column_indexes` metadata with levels of
1142 the correct type.
1143
1144 Notes
1145 -----
1146 * Part of :func:`~pyarrow.pandas_compat.table_to_blockmanager`
1147 """
1148 pd = _pandas_api.pd
1149 # Get levels and labels, and provide sane defaults if the index has a
1150 # single level to avoid if/else spaghetti.
1151 levels = getattr(columns, 'levels', None) or [columns]
1152 labels = getattr(columns, 'codes', None) or [None]
1153
1154 # Convert each level to the dtype provided in the metadata
1155 levels_dtypes = [
1156 (level, col_index.get('pandas_type', str(level.dtype)),
1157 col_index.get('numpy_type', None))
1158 for level, col_index in zip_longest(
1159 levels, column_indexes, fillvalue={}
1160 )
1161 ]
1162
1163 new_levels = []
1164 encoder = operator.methodcaller('encode', 'UTF-8')
1165
1166 for level, pandas_dtype, numpy_dtype in levels_dtypes:
1167 dtype = _pandas_type_to_numpy_type(pandas_dtype)
1168 # Since our metadata is UTF-8 encoded, Python turns things that were
1169 # bytes into unicode strings when json.loads-ing them. We need to
1170 # convert them back to bytes to preserve metadata.
1171 if dtype == np.bytes_:
1172 level = level.map(encoder)
1173 # ARROW-13756: if index is timezone aware DataTimeIndex
1174 elif pandas_dtype == "datetimetz":
1175 tz = pa.lib.string_to_tzinfo(
1176 column_indexes[0]['metadata']['timezone'],
1177 prefer_zoneinfo=_pandas_api.is_ge_v3())
1178 level = pd.to_datetime(level, utc=True).tz_convert(tz)
1179 if _pandas_api.is_ge_v3():
1180 # with pandas 3+, to_datetime returns a unit depending on the string
1181 # data, so we restore it to the original unit from the metadata
1182 level = level.as_unit(np.datetime_data(numpy_dtype)[0])
1183 # GH-41503: if the column index was decimal, restore to decimal

Callers 1

Calls 7

lenFunction · 0.85
mapMethod · 0.80
to_datetimeMethod · 0.80
getMethod · 0.45
DecimalMethod · 0.45
appendMethod · 0.45

Tested by

no test coverage detected