MCPcopy
hub / github.com/deepspeedai/DeepSpeedExamples / s3_get

Function s3_get

Megatron-LM/data_utils/file_utils.py:159–163  ·  view source on GitHub ↗

Pull a file directly from S3.

(url, temp_file)

Source from the content-addressed store, hash-verified

157
158@s3_request
159def s3_get(url, temp_file):
160 """Pull a file directly from S3."""
161 s3_resource = boto3.resource("s3")
162 bucket_name, s3_path = split_s3_path(url)
163 s3_resource.Bucket(bucket_name).download_fileobj(s3_path, temp_file)
164
165
166def http_get(url, temp_file):

Callers 1

get_from_cacheFunction · 0.70

Calls 1

split_s3_pathFunction · 0.70

Tested by

no test coverage detected