Commit graph

85 commits

Author SHA1 Message Date
James Hewitt
7622d0a453
Don't return the from of a walk
Other storage drivers will only return children and below, s3 should do
the same. The only reason it was returning was because of the addition
of a / to ensure we treat the from as a directory.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:26:37 +01:00
James Hewitt
f7bdd9127b
Don't test the OUTPOSTS storage class
This test will only work on an s3 bucket on an s3 outpost. Most
developers won't have access to one of these.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:52 +01:00
James Hewitt
6ceb904c3e
Don't check returned storage class if we use NONE
If we haven't set a storage class there's no point in checking the
storage class applied to the object - s3 will choose one.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
2d316a12d3
We don't use gocheck in these tests
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
f78d81e78a
Remove test as S3 does not support empty directories
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:48 +01:00
Lucas França de Oliveira
035a8ec52a
Fix panic in the s3 backend walk logic
Signed-off-by: Lucas França de Oliveira <lucasfdo@palantir.com>
2023-05-25 14:56:05 -07:00
Milos Gajdos
2fb8dbdeca
Merge pull request #3839 from kirat-singh/feature.azure-sdk-update
Update Azure SDK and support additional authentication schemes
2023-04-25 19:35:34 +01:00
Kirat Singh
ba4a6bbe02 Update Azure SDK and support additional authentication schemes
Microsoft has updated the golang Azure SDK significantly.  Update the
azure storage driver to use the new SDK.  Add support for client
secret and MSI authentication schemes in addition to shared key
authentication.

Implement rootDirectory support for the azure storage driver to mirror
the S3 driver.

Signed-off-by: Kirat Singh <kirat.singh@beacon.io>

Co-authored-by: Cory Snider <corhere@gmail.com>
2023-04-25 17:23:20 +00:00
Milos Gajdos
0c958010ac
Merge pull request #3763 from distribution/multipart-upload-empty-files
Enable pushing empty blobs
2023-03-27 10:18:44 +01:00
Milos Gajdos
5fa926a609
Enable pushing empty blobs
This is an edge case when we are trying to upload an empty chunk of data using
a MultiPart upload. As a result we are trying to complete the MultipartUpload
with an empty slice of `completedUploadedParts` which will always lead to 400
being returned from S3 See: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#CompletedMultipartUpload
Solution: we upload an empty i.e. 0 byte part as a single part and then append it
to the completedUploadedParts slice used to complete the Multipart upload.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-03-27 10:11:07 +01:00
Aaron Lehmann
2074688be9 Fix S3 multipart upload pagination loop condition
The loop that iterates over paginated lists of S3 multipart upload parts
appears to be using the wrong variable in its loop condition. Nothing
inside the loop affects the value of `resp.IsTruncated`, so this loop
will either be wrongly skipped or loop forever.

It looks like this is a regression caused by commit
7736319f2e. The return value of
`ListMultipartUploads` used to be assigned to a variable named `resp`,
but it was renamed to `partsList` without updating the for loop
condition.

I believe this is causing an error we're seeing with large layer uploads
at commit time:

    upload resumed at wrong offset: 5242880000 != 5815706782

Missing parts of the multipart S3 upload would cause an incorrect size
calculation in `newWriter`.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-02-21 20:57:50 -08:00
Kirat Singh
3117e2eb2f
Use default http.Transport for AWS S3 session
Previously we used a custom Transport in order to modify the user agent header.
This prevented the AWS SDK from being able to customize SSL and other client TLS
parameters since it could not understand the Transport type.

Instead we can simply use the SDK function MakeAddToUserAgentFreeFormHandler to
customize the UserAgent if necessary and leave all the TLS configuration to the
AWS SDK.

The only exception being SkipVerify which we have to handle, but we can set it
onto the standard http.Transport which does not interfere with the SDKs ability
to set other options.

Signed-off-by: Kirat Singh <kirat.singh@gmail.com>
2023-02-15 13:37:01 -05:00
Sebastiaan van Stijn
f8b3af78fc
replace deprecated io/ioutil
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 23:47:52 +01:00
Hayley Swimelar
e3509fc1de
Merge pull request #3635 from milosgajdos/make-s3-driver-delete-faster
Delete S3 keys incrementally in batches
2022-11-04 16:56:41 +01:00
Hayley Swimelar
52d948a9f5
Merge pull request #3766 from thaJeztah/gofumpt
format code with gofumpt
2022-11-04 12:19:53 +01:00
Sebastiaan van Stijn
e0281dc609
format code with gofumpt
gofumpt (https://github.com/mvdan/gofumpt) provides a supserset of `gofmt` / `go fmt`,
and addresses various formatting issues that linters may be checking for.

We can consider enabling the `gofumpt` linter to verify the formatting in CI, although
not every developer may have it installed, so for now this runs it once to get formatting
in shape.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-03 22:48:20 +01:00
Sebastiaan van Stijn
f9ccd2c6ea
use http consts for request methods
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 23:31:47 +01:00
Milos Gajdos
ebc4234fd5
Delete S3 keys incrementally in batches
Instead of first collecting all keys and then batch deleting them,
we will do the incremental delete _online_ per max allowed batch.
Doing this prevents frequent allocations for large S3 keyspaces
and OOM-kills that might happen as a result of those.

This commit introduces storagedriver.Errors type that allows to return
multierrors as a single error from any storage driver implementation.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-10-30 19:10:24 +00:00
João Pereira
a7fc49b067
Merge pull request #3622 from ddelange/patch-1
Support all S3 instant retrieval storage classes
2022-04-26 10:23:14 +01:00
Milos Gajdos
27b5563245
Merge pull request #3624 from milosgajdos/aws-s3-listv2
Update s3 ListObjects to V2 API
2022-04-22 13:34:13 +01:00
duanhongyi
15de9e21ba Add forcepathstyle parameter for s3
Signed-off-by: duanhongyi <duanhongyi@doopai.com>
2022-04-20 08:44:12 +08:00
Milos Gajdos
48f3d9ad29
Fix typo
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-04-09 12:31:27 +01:00
Milos Gajdos
8eab5d1bd6
Update s3 ListObjects to V2 API
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-04-09 12:16:46 +01:00
Simone Locci
80952c9e2b
Rename s3accelerate parameter to accelerate
Signed-off-by: Simone Locci <simonelocci88@gmail.com>
2022-04-04 19:35:21 +02:00
Simone Locci
ea27621d4a
Fix review
Signed-off-by: Simone Locci <simonelocci88@gmail.com>
2022-04-04 19:35:09 +02:00
Kirat Singh
51c0c8148a
Add new parameter s3accelerate to S3 storage driver.
If s3accelerate is set to true then we turn on S3 Transfer
Acceleration via the AWS SDK.  It defaults to false since this is an
opt-in feature on the S3 bucket.

Signed-off-by: Kirat Singh <kirat.singh@wsq.io>
Signed-off-by: Simone Locci <simonelocci88@gmail.com>
2022-04-04 19:34:57 +02:00
ddelange
966fae5463
Add tests for all supported storage classes
Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>
2022-04-04 10:54:18 +02:00
ddelange
fb937deabf
Support all S3 instant retrieval storage classes
Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>
2022-04-01 11:55:35 +02:00
João Pereira
514cbd71be
Merge pull request #3519 from jtherin/mpu-paginate
fix: paginate through s3 multipart uploads
2022-03-11 16:06:46 +00:00
libo.huang
117757a5cb feat: add option to disable combining the pending part
Signed-off-by: Libo Huang <huanglibo2010@gmail.com>
2022-01-07 18:20:31 +08:00
Adam Kaplan
e2caaf9cba Add dualstack option to S3 storage driver
Allow the storage driver to optionally use AWS SDK's dualstack mode.
This allows the registry to communicate with S3 in IPv6 environments.

Signed-off-by: Adam Kaplan <adam.kaplan@redhat.com>
2022-01-04 17:19:05 -05:00
Jeremy THERIN
7736319f2e fix: paginate through s3 multipart uploads
Signed-off-by: Jeremy THERIN <jtherin@scaleway.com>
2021-10-26 17:55:03 +02:00
Collin Shoop
cf81f67a16 storagedriver/s3: Optimized Walk implementation + bugfix
Optimized S3 Walk impl by no longer listing files recursively. Overall gives a huge performance increase both in terms of runtime and S3 calls (up to ~500x).

Fixed a bug in WalkFallback where ErrSkipDir for was not handled as documented for non-directory.

Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-16 16:07:25 -04:00
Collin Shoop
9e873f31ec storagedriver/s3: Adding back missing import.
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
e625bc7160 storagedriver/s3: Removed temporary S3 test
(cherry picked from commit ce80e98cea1d15aa2a2ab931c8b9a1161fc6e218)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
dc5b77101d storagedriver/s3: Cleaning up tests
(cherry picked from commit 483ba26165ca66bcf18a1eaadf41ebe4d3bd5f85)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
6da7217b99 storagedriver/s3: Optimize s3 Delete test cleanup.
(cherry picked from commit e4af4dc3a6da6da724e7cff18cf5b6da6ef2a3fd)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
03f9eb3a18 storagedriver/s3: Fixed a Delete noop edgecase
Delete was not working when the subpath immediately followed the given path started with an ascii lower than "/" such as dash "-" and underscore "_" and requests no files to be deleted.

(cherry picked from commit 5d8fa0ce94b68cce70237805db92cdd8d40de282)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
05a258e711 storagedriver/s3: Added Delete tests to s3_test
(cherry picked from commit 1e3b6b67a8e6d7f01307518370f0731212935d05)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 10:57:24 -04:00
Derek McGowan
a01c71e247
Merge pull request #2815 from bainsy88/issue_2814
Add code to handle pagination of parts. Fixes max layer size of 10GB bug
2021-03-16 09:12:03 -07:00
Sebastiaan van Stijn
1d33874951
go.mod: change imports to github.com/distribution/distribution/v3
Go 1.13 and up enforce import paths to be versioned if a project
contains a go.mod and has released v2 or up.

The current v2.x branches (and releases) do not yet have a go.mod,
and therefore are still allowed to be imported with a non-versioned
import path (go modules add a `+incompatible` annotation in that case).

However, now that this project has a `go.mod` file, incompatible
import paths will not be accepted by go modules, and attempting
to use code from this repository will fail.

This patch uses `v3` for the import-paths (not `v2`), because changing
import paths itself is a breaking change, which means that  the
next release should increment the "major" version to comply with
SemVer (as go modules dictate).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-08 18:30:46 +01:00
Ricardo Maraschini
87cbd09fa7 Ignore self reference object on empty prefix
When a given prefix is empty and we attempt to list its content AWS
returns that the prefix contains one object with key defined as the
prefix with an extra "/" at the end.

e.g.

If we call ListObjects() passing to it an existing but empty prefix,
say "my/empty/prefix", AWS will return that "my/empty/prefix/" is an
object inside "my/empty/prefix" (ListObjectsOutput.Contents).

This extra "/" causes the upload purging process to panic. On normal
circunstances we never find empty prefixes on S3 but users may touch
it.

Signed-off-by: Ricardo Maraschini <rmarasch@redhat.com>
2020-11-30 13:04:14 +01:00
olegburov
34f1322664 Fix hardcoded credential provides.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2020-09-02 17:32:34 -07:00
Andrew Bulford
9690d843fa Support ECS TaskRole in S3 storage driver
Instead of constructing the list of credential providers manually, if we
use the default list we can take advantage of the AWS SDK checking the
environment and returning either the EC2RoleProvider or the generic HTTP
credentials provider, configured to use the ECS credentials endpoint.

Also, use the `defaults.Config()` function instead of `aws.NewConfig()`,
as this results in an initialised HTTP client which prevents a fatal
error when retrieving credentials from the ECS credentials endpoint.

Fixes #2960

Signed-off-by: Andrew Bulford <andrew.bulford@redmatter.com>
2020-07-01 08:42:56 +01:00
sayboras
66809646d9 Migrate to golangci-lint
Signed-off-by: Tam Mach <sayboras@yahoo.com>
2020-02-14 08:11:16 +11:00
Thomas Berger
c18c6c33b2 S3 Driver: added comment for missing KeyCount workaround
Signed-off-by: Thomas Berger <loki@lokis-chaos.de>
2019-03-15 21:05:21 +01:00
Eohyung Lee
f877726503 Fix s3 driver for supporting ceph radosgw
Radosgw does not support S3 `GET Bucket` API v2 API but v1.
This API has backward compatibility, so most of this API is working
correctly but we can not get `KeyCount` in v1 API and which is only
for v2 API.

Signed-off-by: Eohyung Lee <liquidnuker@gmail.com>
2019-03-08 12:45:20 +01:00
Manish Tomar
da8db4666b Fix gometalint errors
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:01:04 -08:00
Jack Baines
bda79219b2 Add code to handle pagination of parts. Fixes max layer size of 10GB bug
Signed-off-by: Jack Baines <jack.baines@uk.ibm.com>
2019-01-14 14:54:55 +00:00
David Wu
166874ade9 fix gofmt and goimports
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 16:14:10 -07:00