docs: add section for improving performance for s3
This commit is contained in:
parent
1bc9b94cf2
commit
da70088b11
1 changed files with 32 additions and 0 deletions
|
@ -401,6 +401,38 @@ there for more details.
|
||||||
|
|
||||||
Setting this flag increases the chance for undetected upload failures.
|
Setting this flag increases the chance for undetected upload failures.
|
||||||
|
|
||||||
|
### Increasing performance
|
||||||
|
|
||||||
|
#### Using server-side copy
|
||||||
|
|
||||||
|
If you are copying objects between S3 buckets in the same region, you should
|
||||||
|
use server-side copy.
|
||||||
|
This is much faster than downloading and re-uploading the objects, as no data is transferred.
|
||||||
|
|
||||||
|
For rclone to use server-side copy, you must use the same remote for the source and destination.
|
||||||
|
|
||||||
|
rclone copy s3:source-bucket s3:destination-bucket
|
||||||
|
|
||||||
|
When using server-side copy, the performance is limited by the rate at which rclone issues
|
||||||
|
API requests to S3.
|
||||||
|
See below for how to increase the number of API requests rclone makes.
|
||||||
|
|
||||||
|
#### Increasing the rate of API requests
|
||||||
|
|
||||||
|
You can increase the rate of API requests to S3 by increasing the parallelism using `--transfers` and `--checkers`
|
||||||
|
options.
|
||||||
|
|
||||||
|
Rclone uses a very conservative defaults for these settings, as not all providers support high rates of requests.
|
||||||
|
Depending on your provider, you can increase significantly the number of transfers and checkers.
|
||||||
|
|
||||||
|
For example, with AWS S3, if you can increase the number of checkers to values like 200.
|
||||||
|
If you are doing a server-side copy, you can also increase the number of transfers to 200.
|
||||||
|
|
||||||
|
rclone sync --transfers 200 --checkers 200 --checksum s3:source-bucket s3:destination-bucket
|
||||||
|
|
||||||
|
You will need to experiment with these values to find the optimal settings for your setup.
|
||||||
|
|
||||||
|
|
||||||
### Versions
|
### Versions
|
||||||
|
|
||||||
When bucket versioning is enabled (this can be done with rclone with
|
When bucket versioning is enabled (this can be done with rclone with
|
||||||
|
|
Loading…
Reference in a new issue