Question

What is the fastest way to move data over long distances using the internet, for instance across countries or continents to your Amazon S3 bucket?

Answer 1

Assuming you are using the AWS CLI to upload data to s3, there are two optimizations to increase the speed:

1) Parallel chunked uploading, which uploads your (large) file in chunks in parallel.

2) use Amazon S3 Transfer Acceleration

This will upload your file to a nearby CloudFront instance

You can enable this in your CLI configuration file, named config in the .aws folder. Add the following lines:

s3 =

max_concurrent_requests = 20

max_queue_size = 10000

multipart_threshold = 64MB

multipart_chunksize = 16MB

max_bandwidth = 100MB/s

use_accelerate_endpoint = true

addressing_style = path