From e5a1bcb1ce771bc80fee0072565bb4bfa1e86dca Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Thu, 16 Mar 2023 17:49:11 +0000 Subject: [PATCH] s3: fix InvalidRequest copying to a locked bucket from a source with no MD5SUM Before this change, we would upload files as single part uploads even if the source MD5SUM was not available. AWS won't let you upload a file to a locket bucket without some sort of hash protection of the upload which we don't have with no MD5SUM. So we switch to multipart upload when the source does not have an MD5SUM. This means that if --s3-disable-checksum is set or we are copying from a source with no MD5SUMs we will copy with multipart uploads. This patch changes all uploads, not just those to locked buckets because having no MD5SUM protection on uploads is undesirable. Fixes #6846 --- backend/s3/s3.go | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/backend/s3/s3.go b/backend/s3/s3.go index 4d4994302..785969bd7 100644 --- a/backend/s3/s3.go +++ b/backend/s3/s3.go @@ -1912,6 +1912,9 @@ size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google photos or google docs) they will be uploaded as multipart uploads using this chunk size. +Files with no source MD5 will also be uploaded with multipart uploads +as will all files if --s3-disable-checksum is set. + Note that "--s3-upload-concurrency" chunks of this size are buffered in memory per transfer. @@ -1967,7 +1970,11 @@ The minimum is 0 and the maximum is 5 GiB.`, Normally rclone will calculate the MD5 checksum of the input before uploading it so it can add it to metadata on the object. This is great for data integrity checking but can cause long delays for large files -to start uploading.`, +to start uploading. + +Note that setting this flag forces all uploads to be multipart uploads +as we can't protect the body of the transfer unless we have an MD5. +`, Default: false, Advanced: true, }, { @@ -5500,6 +5507,12 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op } } } + // If source MD5SUM not available then do multipart upload + // otherwise uploads are not hash protected and locked buckets + // will complain #6846 + if !multipart && md5sumHex == "" { + multipart = true + } // Set the content type it it isn't set already if req.ContentType == nil {