MCPcopy Create free account
hub / github.com/apache/arrow / _resolve_filesystem_and_path

Function _resolve_filesystem_and_path

python/pyarrow/fs.py:130–187  ·  view source on GitHub ↗

Return filesystem/path from path which could be an URI or a plain filesystem path or a combination of fsspec protocol and URI.

(path, filesystem=None, *, memory_map=False)

Source from the content-addressed store, hash-verified

128
129
130def _resolve_filesystem_and_path(path, filesystem=None, *, memory_map=False):
131 """
132 Return filesystem/path from path which could be an URI or a plain
133 filesystem path or a combination of fsspec protocol and URI.
134 """
135 if not _is_path_like(path):
136 if filesystem is not None:
137 raise ValueError(
138 "'filesystem' passed but the specified path is file-like, so"
139 " there is nothing to open with 'filesystem'."
140 )
141 return filesystem, path
142
143 if filesystem is not None:
144 filesystem = _ensure_filesystem(filesystem, use_mmap=memory_map)
145 if isinstance(filesystem, LocalFileSystem):
146 path = _stringify_path(path)
147 elif not isinstance(path, str):
148 raise TypeError(
149 "Expected string path; path-like objects are only allowed "
150 "with a local filesystem"
151 )
152 path = filesystem.normalize_path(path)
153 return filesystem, path
154
155 path = _stringify_path(path)
156
157 # if filesystem is not given, try to automatically determine one
158 # first check if the file exists as a local (relative) file path
159 # if not then try to parse the path as an URI
160 filesystem = LocalFileSystem(use_mmap=memory_map)
161
162 try:
163 file_info = filesystem.get_file_info(path)
164 except ValueError: # ValueError means path is likely an URI
165 file_info = None
166 exists_locally = False
167 else:
168 exists_locally = (file_info.type != FileType.NotFound)
169
170 # if the file or directory doesn't exists locally, then assume that
171 # the path is an URI describing the file system as well
172 if not exists_locally:
173 try:
174 filesystem, path = FileSystem.from_uri(path)
175 except ValueError as e:
176 msg = str(e)
177 if "empty scheme" in msg or "Cannot parse URI" in msg:
178 # neither an URI nor a locally existing path, so assume that
179 # local path was given and propagate a nicer file not found
180 # error instead of a more confusing scheme parsing error
181 pass
182 else:
183 raise e
184 else:
185 path = filesystem.normalize_path(path)
186
187 return filesystem, path

Callers 11

_ensure_single_sourceFunction · 0.90
write_datasetFunction · 0.90
read_tableFunction · 0.90
__init__Method · 0.90
__init__Method · 0.90
__init__Method · 0.90
read_tableFunction · 0.90
write_metadataFunction · 0.90
read_metadataFunction · 0.90
read_schemaFunction · 0.90
copy_filesFunction · 0.85

Calls 7

_is_path_likeFunction · 0.90
_stringify_pathFunction · 0.90
_ensure_filesystemFunction · 0.85
LocalFileSystemClass · 0.85
TypeErrorFunction · 0.50
normalize_pathMethod · 0.45
get_file_infoMethod · 0.45

Tested by

no test coverage detected