s4cmd

Super S3 command line tool

---

Author: Chou-han Yang (@chouhanyang)

Current Maintainers: Naveen Vardhi (@rozuur) | Navin Pai (@navinpai)

---

What's New in s4cmd 2.x

• Fully migrated from old boto 2.x to new boto3 library, which provides more reliable and up-to-date S3 backend.
• Support S3 --API-ServerSideEncryption along with 36 new API pass-through options. See API pass-through options section for complete list.
• Support batch delete (with delete_objects API) to delete up to 1000 files with single call. 100+ times faster than sequential deletion.
• Support S4CMD_OPTS environment variable for commonly used options such as --API-ServerSideEncryption across all your s4cmd operations.
• Support moving files larger than 5GB with multipart upload. 20+ times faster then sequential move operation when moving large files.
• Support timestamp filtering with --last-modified-before and --last-modified-after options for all operations. Human friendly timestamps are supported, e.g. --last-modified-before='2 months ago'
• Faster upload with lazy evaluation of md5 hash.
• Listing large number of files with S3 pagination, with memory is the limit.
• New directory to directory dsync command is better and standalone implementation to replace old sync command, which is implemented based on top of get/put/mv commands. --delete-removed work for all cases including local to s3, s3 to local, and s3 to s3. sync command preserves the old behavior in this version for compatibility.
• Tested on both python 2 and 3.
• Special thanks to onera.com for supporting s4cmd.

Motivation

S4cmd is a command-line utility for accessing Amazon S3, inspired by s3cmd.

We have used s3cmd heavily for a number of scripted, data-intensive applications. However as the need for a variety of small improvements arose, we created our own implementation, s4cmd. It is intended as an alternative to s3cmd for enhanced performance and for large files, and with a number of additional features and fixes that we have found useful.

It strives to be compatible with the most common usage scenarios for s3cmd. It does not offer exact drop-in compatibility, due to a number of corner cases where different behavior seems preferable, or for bugfixes.

Features

S4cmd supports the regular commands you might expect for fetching and storing files in S3: ls, put, get, cp, mv, sync, del, du.

The main features that distinguish s4cmd are:

• Simple (less than 1500 lines of code) and implemented in pure Python, based on the widely used Boto3 library.
• Multi-threaded/multi-connection implementation for enhanced performance on all commands. As with many network-intensive applications (like web browsers), accessing S3 in a single-threaded way is often significantly less efficient than having multiple connections actively transferring data at once. In general, we get a 2X boost to upload/download speeds from this.
• Path handling: S3 is not a traditional filesystem with built-in support for directory structure: internally, there are only objects, not directories or folders. However, most people use S3 in a hierarchical structure, with paths separated by slashes, to emulate traditional filesystems. S4cmd follows conventions to more closely replicate the behavior of traditional filesystems in certain corner cases. For example, "ls" and "cp" work much like in Unix shells, to avoid odd surprises. (For examples see compatibility notes below.)
• Wildcard support: Wildcards, including multiple levels of wildcards, like in Unix shells, are handled. For example: s3://my-bucket/my-folder/20120512//chunk00?1?
• Automatic retry: Failure tasks will be executed again after a delay.
• Multi-part upload support for files larger than 5GB.
• Handling of MD5s properly with respect to multi-part uploads (for the sordid details of this, see below).
• Miscellaneous enhancements and bugfixes:
• Partial file creation: Avoid creating empty target files if source does not exist. Avoid creating partial output files when commands are interrupted.
• General thread safety: Tool can be interrupted or killed at any time without being blocked by child threads or leaving incomplete or corrupt files in place.
• Ensure exit code is nonzero on all failure scenarios (a very important feature in scripts).
• Expected handling of symlinks (they are followed).
• Support both s3:// and s3n:// prefixes (the latter is common with Amazon Elastic Mapreduce).

Limitations:

• No CloudFront or other feature support.
• Currently, we simulate sync with get and put with --recursive --force --sync-check.

Installation and Setup

You can install s4cmd PyPI.

pip install s4cmd

• Copy or create a symbolic link so you can run s4cmd.py as s4cmd. (It is just a single file!)
• If you already have a ~/.s3cfg file from configuring s3cmd, credentials from this file will be used. Otherwise, set the S3_ACCESS_KEY and S3_SECRET_KEY environment variables to contain your S3 credentials.
• If no keys are provided, but an IAM role is associated with the EC2 instance, it will be used transparently.

s4cmd Commands

s4cmd ls [path]

List all contents of a directory.

• -r/--recursive: recursively display all contents including subdirectories under the given path.
• -d/--show-directory: show the directory entry instead of its content.

s4cmd put [source] [target]

Upload local files up to S3.

• -r/--recursive: also upload directories recursively.
• -s/--sync-check: check md5 hash to avoid uploading the same content.
• -f/--force: override existing file instead of showing error message.
• -n/--dry-run: emulate the operation without real upload.

s4cmd get [source] [target]

Download files from S3 to local filesystem.

• -r/--recursive: also download directories recursively.
• -s/--sync-check: check md5 hash to avoid downloading the same content.
• -f/--force: override existing file instead of showing error message.
• -n/--dry-run: emulate the operation without real download.

s4cmd dsync [source dir] [target dir]

Synchronize the contents of two directories. The directory can either be local or remote, but currently, it doesn't support two local directories.

• -r/--recursive: also sync directories recursively.
• -s/--sync-check: check md5 hash to avoid syncing the same content.
• -f/--force: override existing file instead of showing error message.
• -n/--dry-run: emulate the operation without real sync.
• --delete-removed: delete files not in source directory.

s4cmd sync [source] [target]

(Obsolete, use dsync instead) Synchronize the contents of two directories. The directory can either be local or remote, but currently, it doesn't support two local directories. This command simply invoke get/put/mv commands.

• -r/--recursive: also sync directories recursively.
• -s/--sync-check: check md5 hash to avoid syncing the same content.
• -f/--force: override existing file instead of showing error message.
• -n/--dry-run: emulate the operation without real sync.
• --delete-removed: delete files not in source directory. Only works when syncing local directory to s3 directory.

s4cmd cp [source] [target]

Copy a file or a directory from a S3 location to another.

• -r/--recursive: also copy directories recursively.
• -s/--sync-check: check md5 hash to avoid copying the same content.
• -f/--force: override existing file instead of showing error message.
• -n/--dry-run: emulate the operation without real copy.

s4cmd mv [source] [target]

Move a file or a directory from a S3 location to another.

• -r/--recursive: also move directories recursively.
• -s/--sync-check: check md5 hash to avoid moving the same content.
• -f/--force: override existing file instead of showing error message.
• -n/--dry-run: emulate the operation without real move.

s4cmd del [path]

Delete files or directories on S3.

• -r/--recursive: also delete directories recursively.
• -n/--dry-run: emulate the operation without real delete.

s4cmd du [path]

Get the size of the given directory.

Available parameters:

• -r/--recursive: also add sizes of sub-directories recursively.

s4cmd Control Options

-p S3CFG, --config=[filename]

path to s3cfg config file

-f, --force

force overwrite files when download or upload

-r, --recursive

recursively checking subdirectories

-s, --sync-check

check file md5 before download or upload

-n, --dry-run

trial run without actual download or upload

-t RETRY, --retry=[integer]

number of retries before giving up

--retry-delay=[integer]

seconds to sleep between retries

-c NUM_THREADS, --num-threads=NUM_THREADS

number of concurrent threads

--endpoint-url

endpoint url used in boto3 client

-d, --show-directory

show directory instead of its content

--ignore-empty-source

ignore empty source from s3

--use-ssl

(obsolete) use SSL connection to S3

--verbose

verbose output

--debug

debug output

--validate

(obsolete) validate lookup operation

-D, --delete-removed

delete remote files that do not exist in source after sync

--multipart-split-size=[integer]

size in bytes to split multipart transfers

--max-singlepart-download-size=[integer]

files with size (in bytes) greater than this will be downloaded in multipart transfers

--max-singlepart-upload-size=[integer]

files with size (in bytes) greater than this will be uploaded in multipart transfers

--max-singlepart-copy-size=[integer]

files with size (in bytes) greater than this will be copied in multipart transfers

--batch-delete-size=[integer]

Number of files (<1000) to be combined in batch delete.

--last-modified-before=[datetime]

Condition on files where their last modified dates are before given parameter.

--last-modified-after=[datetime]

Condition on files where their last modified dates are after given parameter.

S3 API Pass-through Options

Those options are directly translated to boto3 API commands. The options provided will be filtered by the APIs that are taking parameters. For example, --API-ServerSideEncryption is only needed for put_object, create_multipart_upload but not for list_buckets and get_objects for example. Therefore, providing --API-ServerSideEncryption for s4cmd ls has no effect.

For more information, please see boto3 s3 documentations http://boto3.readthedocs.io/en/latest/reference/services/s3.html

--API-ACL=[string]

The canned ACL to apply to the object.

--API-CacheControl=[string]

Specifies caching behavior along the request/reply chain.

--API-ContentDisposition=[string]

Specifies presentational information for the object.

--API-ContentEncoding=[string]

Specifies what content encodings have been applied to the object and thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field.

--API-ContentLanguage=[string]

The language the content is in.

--API-ContentMD5=[string]

The base64-encoded 128-bit MD5 digest of the part data.

--API-ContentType=[string]

A standard MIME type describing the format of the object data.

--API-CopySourceIfMatch=[string]

Copies the object if its entity tag (ETag) matches the specified tag.

--API-CopySourceIfModifiedSince=[datetime]

Copies the object if it has been modified since the specified time.

--API-CopySourceIfNoneMatch=[string]

Copies the object if its entity tag (ETag) is different than the specified ETag.

--API-CopySourceIfUnmodifiedSince=[datetime]

Copies the object if it hasn't been modified since the specified time.

--API-CopySourceRange=[string]

The range of bytes to copy from the source object. The range value must use the form bytes=first-last, where the first and last are the zero-based byte offsets to copy. For example, bytes=0-9 indicates that you want to copy the first ten bytes of the source. You can copy a range only if the source object is greater than 5 GB.

--API-CopySourceSSECustomerAlgorithm=[string]

Specifies the algorithm to use when decrypting the source object (e.g., AES256).

--API-CopySourceSSECustomerKeyMD5=[string]

Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. Amazon S3 uses this header for a message integrity check to ensure the encryption key was transmitted without error. Please note that this parameter is automatically populated if it is not provided. Including this parameter is not required

--API-CopySourceSSECustomerKey=[string]

Specifies the customer-provided encryption key for Amazon S3 to use to decrypt the source object. The encryption key provided in this header must be one that was used when the source object was created.

--API-ETag=[string]

Entity tag returned when the part was uploaded.

--API-Expires=[datetime]

The date and time at which the object is no longer cacheable.

--API-GrantFullControl=[string]

Gives the grantee READ,