A pure-python interface to the Azure Data-lake Storage Gen 1 system, providing pythonic file-system and file objects, seamless transition between Windows and POSIX remote paths, high-performance up- and down-loader.
This software is under active development and not yet recommended for general use.
Note: This library supports ADLS Gen 1. For Gen 2, please see
azure-storage-file-datalake, documented
here
Using pip:
pip install azure-datalake-store
Manually (bleeding edge):
Download the repo from https://github.com/Azure/azure-data-lake-store-python
install the requirements (pip install -r dev_requirements.txt)
install in develop mode (python setup.py develop)
Although users can generate and supply their own tokens to the base file-system
class, and there is a password-based function in the lib module for
generating tokens, the most convenient way to supply credentials is via
environment parameters. This latter method is the one used by default in
library. The following variables are required:
azure_tenant_id
azure_username
azure_password
azure_store_name
azure_url_suffix (optional)
The AzureDLFileSystem object is the main API for library usage of this
package. It provides typical file-system operations on the remote azure
store
token = lib.auth(tenant_id, username, password)
adl = core.AzureDLFileSystem(store_name, token)
# alternatively, adl = core.AzureDLFileSystem()
# uses environment variables
print(adl.ls()) # list files in the root directory
for item in adl.ls(detail=True):
print(item) # same, but with file details as dictionaries
print(adl.walk('')) # list all files at any directory depth
print('Usage:', adl.du('', deep=True, total=True)) # total bytes usage
adl.mkdir('newdir') # create directory
adl.touch('newdir/newfile') # create empty file
adl.put('remotefile', '/home/myuser/localfile') # upload a local file
In addition, the file-system generates file objects that are compatible with
the python file interface, ensuring compatibility with libraries that work on
python files. The recommended way to use this is with a context manager
(otherwise, be sure to call close() on the file object).
with adl.open('newfile', 'wb') as f:
f.write(b'index,a,b\n')
f.tell() # now at position 9
f.flush() # forces data upstream
f.write(b'0,1,True')
with adl.open('newfile', 'rb') as f:
print(f.readlines())
with adl.open('newfile', 'rb') as f:
df = pd.read_csv(f) # read into pandas.
To seamlessly handle remote path representations across all supported platforms, the main API will take in numerous path types: string, Path/PurePath, and AzureDLPath. On Windows in particular, you can pass in paths separated by either forward slashes or backslashes.
import pathlib # only >= Python 3.4
from pathlib2 import pathlib # only <= Python 3.3
from azure.datalake.store.core import AzureDLPath
# possible remote paths to use on API
p1 = '\\foo\\bar'
p2 = '/foo/bar'
p3 = pathlib.PurePath('\\foo\\bar')
p4 = pathlib.PureWindowsPath('\\foo\\bar')
p5 = pathlib.PurePath('/foo/bar')
p6 = AzureDLPath('\\foo\\bar')
p7 = AzureDLPath('/foo/bar')
# p1, p3, and p6 only work on Windows
for p in [p1, p2, p3, p4, p5, p6, p7]:
with adl.open(p, 'rb') as f:
print(f.readlines())
Classes ADLUploader and ADLDownloader will chunk large files and send
many files to/from azure using multiple threads. A whole directory tree can
be transferred, files matching a specific glob-pattern or any particular file.
# download the whole directory structure using 5 threads, 16MB chunks
ADLDownloader(adl, '', 'my_temp_dir', 5, 2**24)
Access Azure DataLake Store as if it were a file-system
Parameters
store_name: str (“”)
Store name to connect to.
token: credentials object
When setting up a new connection, this contains the authorization
credentials (see lib.auth()).
url_suffix: str (None)
Domain to send REST requests to. The end-point URL is constructed
using this and the store_name. If None, use default.
api_version: str (2018-09-01)
The API version to target with requests. Changing this value will
change the behavior of the requests, and can cause unexpected behavior or
breaking changes. Changes to this value should be undergone with caution.
per_call_timeout_seconds: float(60)
This is the timeout for each requests library call.
kwargs: optional key/values
See `lib.auth()`; full list: tenant_id, username, password, client_id,
client_secret, resource
Does such a file/directory exist?
Parameters
path: str or AzureDLPath
Path to query
invalidate_cache: bool
Whether to invalidate cache
Returns
True or false depending on whether the path exists.
Return contents of file
Parameters
path: str or AzureDLPath
Path to query
Returns
Contents of file
Change access mode of path
Note this is not recursive.
Parameters
path: str
Location to change
mod: str
Octal representation of access, e.g., “0777” for public read/write.
See [docs]([http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Permission](http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Permission))
Change owner and/or owning group
Note this is not recursive.
Parameters
path: str
Location to change
owner: str
UUID of owning entity
group: str
UUID of group
Concatenate a list of files into one new file
Parameters
outfile: path
The file which will be concatenated to. If it already exists,
the extra pieces will be appended.
filelist: list of paths
Existing adl files to concatenate, in order
delete_source: bool (False)
If True, assume that the paths to concatenate exist alone in a
directory, and delete that whole directory when done.
Returns
None
Establish connection object.
Not implemented. Copy file between locations on ADL
Return the most recently created AzureDLFileSystem
Resource summary of path
Parameters
path: str
Path to query
Bytes in keys at path
Parameters
path: str or AzureDLPath
Path to query
total: bool
Return the sum on list
deep: bool
Recursively enumerate or just use files under current dir
invalidate_cache: bool
Whether to invalidate cache
Returns
List of dict of name:size pairs or total size.
Does such a file/directory exist?
Parameters
path: str or AzureDLPath
Path to query
invalidate_cache: bool
Whether to invalidate cache
Returns
True or false depending on whether the path exists.
Stream data from file at path to local filename
Parameters
path: str or AzureDLPath
ADL Path to read
filename: str or Path
Local file path to write to
Returns
None
Gets Access Control List (ACL) entries for the specified file or directory.
Parameters
path: str
Location to get the ACL.
Find files (not directories) by glob-matching.
Parameters
path: str or AzureDLPath
Path to query
details: bool
Whether to include file details
invalidate_cache: bool
Whether to invalidate cache
Returns
List of files
Return first bytes of file
Parameters
path: str or AzureDLPath
Path to query
size: int
How many bytes to return
Returns
First(size) bytes of file
File information for path
Parameters
path: str or AzureDLPath
Path to query
invalidate_cache: bool
Whether to invalidate cache or not
expected_error_code: int
Optionally indicates a specific, expected error code, if any.
Returns
File information
Remove entry from object file-cache
Parameters
path: str or AzureDLPath
Remove the path from object file-cache
Returns
None
List all elements under directory specified with path
Parameters
path: str or AzureDLPath
Path to query
detail: bool
Detailed info or not.
invalidate_cache: bool
Whether to invalidate cache or not
Returns
List of elements under directory specified with path
List all elements under directory specified with path
Parameters
path: str or AzureDLPath
Path to query
detail: bool
Detailed info or not.
invalidate_cache: bool
Whether to invalidate cache or not
Returns
List of elements under directory specified with path
Concatenate a list of files into one new file
Parameters
outfile: path
The file which will be concatenated to. If it already exists,
the extra pieces will be appended.
filelist: list of paths
Existing adl files to concatenate, in order
delete_source: bool (False)
If True, assume that the paths to concatenate exist alone in a
directory, and delete that whole directory when done.
Returns
None
Make new directory
Parameters
path: str or AzureDLPath
Path to create directory
Returns
None
Modify existing Access Control List (ACL) entries on a file or folder. If the entry does not exist it is added, otherwise it is updated based on the spec passed in. No entries are removed by this process (unlike set_acl).
Note: this is by default not recursive, and applies only to the file or folder specified.
Parameters
path: str
Location to set the ACL entries on.
acl_spec: str
The ACL specification to use in modifying the ACL at the path in the format
‘[default:]user|group|other:[entity id or UPN]:r|-w|-x|-,[default:]user|group|other:[entity id or UPN]:r|-w|-x|-,…’
recursive: bool
Specifies whether to modify ACLs recursively or not
Move file between locations on ADL
Parameters
path1:
Source Path
path2:
Destination path
Returns
None
Open a file for reading or writing
Parameters
path: string
Path of file on ADL
mode: string
One of ‘rb’, ‘ab’ or ‘wb’
blocksize: int
Size of data-node blocks if reading
delimiter: byte(s) or None
For writing delimiter-ended blocks
Stream data from local filename to file at path
Parameters
filename: str or Path
Local file path to read from
path: str or AzureDLPath
ADL Path to write to
delimiter:
Optional delimeter for delimiter-ended blocks
Returns
None
Read a block of bytes from an ADL file
Starting at offset of the file, read length bytes. If
delimiter is set then we ensure that the read starts and stops at
delimiter boundaries that follow the locations offset and `off
$ claude mcp add azure-data-lake-store-python \
-- python -m otcore.mcp_server <graph>