TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
rclone/backend
History
Michał Matczuk 19a4d74ee7 backend/s3: Fail fast multipart upload 
When a part upload request fails error is returned and gCtx is cancelled.
This does not prevent from other parts being tried.
They immediately fail due to a canceled context, but are retried by rclone anyway...

Example AWS debug output

```
-----------------------------------------------------
2020/02/11 14:12:17 DEBUG: Retrying Request s3/UploadPart, attempt 4
2020/02/11 14:12:17 DEBUG: Request s3/UploadPart Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /backuptest-rclone/huge/file.db?partNumber=11&uploadId=190939b4-3c43-4b98-ac11-92303e3f11b0 HTTP/1.1
Host: 192.168.100.99:9000
User-Agent: aws-sdk-go/1.23.8 (go1.13.1; linux; amd64)
Content-Length: 5242880
Authorization: AWS4-HMAC-SHA256 Credential=miniouser/20200211/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-md5;expect;host;x-amz-content-sha256;x-amz-date, Signature=3fc03a01f651cec09b05290459e9ceb26db9a8aa00c4e1b16e8cf5617eb81da8
Content-Md5: XzY+DlipXwbL6bvGYsXftg==
Expect: 100-Continue
X-Amz-Content-Sha256: c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29
X-Amz-Date: 20200211T131217Z
Accept-Encoding: gzip

-----------------------------------------------------
http://192.168.100.99:9000/backuptest-rclone/huge/file.db?partNumber=11&uploadId=190939b4-3c43-4b98-ac11-92303e3f11b0
2020/02/11 14:12:17 DEBUG: Response s3/UploadPart Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 500 InternalServerError
Content-Length: 0
-----------------------------------------------------
UploadPartWithContext() error InternalError: We encountered an internal error. Please try again
	status code: 500, request id: , host id:

2020/02/11 14:12:18 DEBUG ERROR: Request s3/UploadPart:
---[ REQUEST DUMP ERROR ]-----------------------------
context canceled
------------------------------------------------------
UploadPartWithContext() error RequestCanceled: request context canceled
caused by: context canceled
2020/02/11 14:12:20 DEBUG ERROR: Request s3/UploadPart:
---[ REQUEST DUMP ERROR ]-----------------------------
context canceled
------------------------------------------------------
UploadPartWithContext() error RequestCanceled: request context canceled
caused by: context canceled
2020/02/11 14:12:22 DEBUG ERROR: Request s3/UploadPart:
---[ REQUEST DUMP ERROR ]-----------------------------
context canceled
------------------------------------------------------
UploadPartWithContext() error RequestCanceled: request context canceled
caused by: context canceled
```

This adds a fail fast behaviour in case the context was cancelled.
2020-02-12 11:40:34 +00:00
..
alias build: fix up package paths after repo move 2019-07-28 18:47:38 +01:00
all The memory backend 2020-01-18 10:41:08 +00:00
amazonclouddrive backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
azureblob backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
b2 backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
box backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
cache backend/cache: Remove Unused Functions 2020-02-12 11:23:57 +00:00
chunker chunker: reduce length of temporary suffix 2019-12-09 16:56:32 +00:00
crypt crypt: correctly handle trailing dot 2020-01-22 01:40:04 +01:00
drive drive: hide dangerous config from the configurator 2020-01-31 10:09:33 +00:00
dropbox dropbox: treat insufficient_space errors as non retriable errors 2020-01-18 11:10:18 +00:00
fichier backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
ftp build: implement a framework for starting test servers during tests 2020-01-18 16:47:37 +00:00
googlecloudstorage backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
googlephotos operations: make NewOverrideObjectInfo public and factor uses 2020-01-18 11:41:33 +00:00
http http: fix race introduced in 7982aaf151 2019-09-14 08:48:13 +01:00
hubic Add context to all http.NewRequest #3257 2019-09-09 23:27:07 +01:00
jottacloud backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
koofr backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
local backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
mailru backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
mega backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
memory The memory backend 2020-01-18 10:41:08 +00:00
onedrive onedrive: fix occasional 416 errors on multipart uploads 2020-02-01 21:15:07 +00:00
opendrive opendrive: implement --opendrive-chunk-size #3707 2020-01-18 11:56:01 +00:00
pcloud backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
premiumizeme backend/premiumizeme: prune unused functions 2020-02-11 12:17:35 +00:00
putio backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
qingstor backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
s3 backend/s3: Fail fast multipart upload 2020-02-12 11:40:34 +00:00
sftp build: implement a framework for starting test servers during tests 2020-01-18 16:47:37 +00:00
sharefile backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
sugarsync sugarsync: new backend - fixes #622 2020-01-17 17:39:34 +00:00
swift swift: Update OVH API endpoint - Fixes #3890 2020-01-24 17:43:00 +00:00
union docs: make the config walkthroughs consistent for each backend 2019-08-26 10:47:17 +01:00
webdav webdav: Fix X-OC-Mtime header for Transip compatibility - fixes #3126 2020-02-10 11:57:12 +00:00
yandex backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00