TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions

s3: fix hashes on small files with aws:kms and sse-c

Browse source 
If rclone is configured for server side encryption - either aws:kms or
sse-c (but not sse-s3) then don't treat the ETags returned on objects
as MD5 hashes.

This fixes being able to upload small files.

Fixes #1824
This commit is contained in:
Nick Craig-Wood 2020-11-20 12:15:56 +00:00
parent 53aa03cc44
commit a06f4c2514
2 changed files with 25 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
backend/s3/s3.go
View file

@ -1294,6 +1294,7 @@ type Fs struct {
pacer *fs.Pacer // To pace the API calls pacer *fs.Pacer // To pace the API calls
srv *http.Client // a plain http client srv *http.Client // a plain http client
pool *pool.Pool // memory pool pool *pool.Pool // memory pool
etagIsNotMD5 bool // if set ETags are not MD5s
} }
// Object describes a s3 object // Object describes a s3 object
@ -1618,7 +1619,16 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
opt.MemoryPoolUseMmap, opt.MemoryPoolUseMmap,
), ),
} }
if opt.ServerSideEncryption == "aws:kms" || opt.SSECustomerAlgorithm != "" {
// From: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
//
// Objects encrypted by SSE-S3 or plaintext have ETags that are an MD5
// digest of their data.
//
// Objects encrypted by SSE-C or SSE-KMS have ETags that are not an
// MD5 digest of their object data.
f.etagIsNotMD5 = true
}
f.setRoot(root) f.setRoot(root)
f.features = (&fs.Features{ f.features = (&fs.Features{
ReadMimeType: true, ReadMimeType: true,
@ -1666,7 +1676,11 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *s3.Obje
} else { } else {
o.lastModified = *info.LastModified o.lastModified = *info.LastModified
} }
o.etag = aws.StringValue(info.ETag) if o.fs.etagIsNotMD5 {
o.etag = ""
} else {
o.etag = aws.StringValue(info.ETag)
}
o.bytes = aws.Int64Value(info.Size) o.bytes = aws.Int64Value(info.Size)
o.storageClass = aws.StringValue(info.StorageClass) o.storageClass = aws.StringValue(info.StorageClass)
} else { } else {
@ -2759,7 +2773,11 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
if resp.ContentLength != nil { if resp.ContentLength != nil {
size = *resp.ContentLength size = *resp.ContentLength
} }
o.etag = aws.StringValue(resp.ETag) if o.fs.etagIsNotMD5 {
o.etag = ""
} else {
o.etag = aws.StringValue(resp.ETag)
}
o.bytes = size o.bytes = size
o.meta = resp.Metadata o.meta = resp.Metadata
if o.meta == nil { if o.meta == nil {

9
docs/content/s3.md
View file

@ -433,11 +433,10 @@ that will generate one or more buckets that will work with `rclone sync`.
### Key Management System (KMS) ### ### Key Management System (KMS) ###
If you are using server-side encryption with KMS then you will find If you are using server-side encryption with KMS then you must make
you can't transfer small objects. As a work-around you can use the sure rclone is configured with `server_side_encryption = aws:kms`
`--ignore-checksum` flag. otherwise you will find you can't transfer small objects - these will
create checksum errors.
A proper fix is being worked on in [issue #1824](https://github.com/rclone/rclone/issues/1824).
### Glacier and Glacier Deep Archive ### ### Glacier and Glacier Deep Archive ###