forked from TrueCloudLab/rclone
s3: add note about Storj provider bug and workaround
See: https://github.com/storj/gateway-mt/issues/39
This commit is contained in:
parent
bcb07a67f6
commit
9cc50a614b
1 changed files with 25 additions and 1 deletions
|
@ -2873,7 +2873,6 @@ y/n> n
|
||||||
Note that s3 credentials are generated when you [create an access
|
Note that s3 credentials are generated when you [create an access
|
||||||
grant](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#usage).
|
grant](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#usage).
|
||||||
|
|
||||||
|
|
||||||
#### Backend quirks
|
#### Backend quirks
|
||||||
|
|
||||||
- `--chunk-size` is forced to be 64 MiB or greater. This will use more
|
- `--chunk-size` is forced to be 64 MiB or greater. This will use more
|
||||||
|
@ -2882,6 +2881,31 @@ grant](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#usage).
|
||||||
gateway.
|
gateway.
|
||||||
- GetTier and SetTier are not supported.
|
- GetTier and SetTier are not supported.
|
||||||
|
|
||||||
|
#### Backend bugs
|
||||||
|
|
||||||
|
Due to [issue #39](https://github.com/storj/gateway-mt/issues/39)
|
||||||
|
uploading multipart files via the S3 gateway causes them to lose their
|
||||||
|
metadata. For rclone's purpose this means that the modification time
|
||||||
|
is not stored, nor is any MD5SUM (if one is available from the
|
||||||
|
source).
|
||||||
|
|
||||||
|
This has the following consequences:
|
||||||
|
|
||||||
|
- Using `rclone rcat` will fail as the medatada doesn't match after upload
|
||||||
|
- Uploading files with `rclone mount` will fail for the same reason
|
||||||
|
- This can worked around by using `--vfs-cache-mode writes` or `--vfs-cache-mode full` or setting `--s3-upload-cutoff` large
|
||||||
|
- Files uploaded via a multipart upload won't have their modtimes
|
||||||
|
- This will mean that `rclone sync` will likely keep trying to upload files bigger than `--s3-upload-cutoff`
|
||||||
|
- This can be worked around with `--checksum` or `--size-only` or setting `--s3-upload-cutoff` large
|
||||||
|
- The maximum value for `--s3-upload-cutoff` is 5GiB though
|
||||||
|
|
||||||
|
One general purpose workaround is to set `--s3-upload-cutoff 5G`. This
|
||||||
|
means that rclone will upload files smaller than 5GiB as single parts.
|
||||||
|
Note that this can be set in the config file with `upload_cutoff = 5G`
|
||||||
|
or configured in the advanced settings. If you regularly transfer
|
||||||
|
files larger than 5G then using `--checksum` or `--size-only` in
|
||||||
|
`rclone sync` is the recommended workaround.
|
||||||
|
|
||||||
#### Comparison with the native protocol
|
#### Comparison with the native protocol
|
||||||
|
|
||||||
Use the [the native protocol](/tardigrade) to take advantage of
|
Use the [the native protocol](/tardigrade) to take advantage of
|
||||||
|
|
Loading…
Reference in a new issue