Commit graph

376 commits

Author SHA1 Message Date
Milos Gajdos
1089800643
Preallocate created slice in S3 tests
In case drvr.PutContent fails and returns error we'd have
some extra memory allocated, though in this case
(test with known size of the slice being iterated), that's fine.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:26:32 +01:00
Milos Gajdos
a9d31ec7b9
Avoid unnecessary type assertion in mfs driver
We already make sure the node in *dir

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:23:25 +01:00
Milos Gajdos
59fd8656ac
Enable prealloc linter
This will give us nice little performance gains in some code paths.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 22:41:51 +01:00
James Hewitt
e22f7cbc73
Pass the last paging flag to storage drivers
Storage drivers may be able to take advantage of the hint to start
their walk more efficiently.

For S3: The API takes a start-after parameter. Registries with many
repositories can drastically reduce calls to s3 by telling s3 to only
list results lexographically after the last parameter.

For the fallback: We can start deeper in the tree and avoid statting
the files and directories before the hint in a walk. For a filesystem
this improves performance a little, but many of the API based drivers
are currently treated like a filesystem, so this drastically improves
the performance of GCP and Azure blob.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-29 11:27:42 +01:00
James Hewitt
1a3e73cb84
Handle rand deprecations in go 1.20
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-28 09:33:12 +01:00
Milos Gajdos
59dd684cc8
Merge pull request #3713 from Jamstah/s3-tests 2023-08-21 13:48:43 +01:00
Sebastiaan van Stijn
5b3be39870
s3: add interface assertion
This was added for the other drivers in 6b388b1ba6,
but it missed the s3 storage driver.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 13:54:13 +02:00
Milos Gajdos
3dbfbc7255
Enable bodyclose linter
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-19 09:45:44 +01:00
Milos Gajdos
3f1859af26
Remove oss storage driver and alicdn storage driver middleware
This commit removes `oss` storage driver from distribution as well as
`alicdn` storage middleware which only works with the `oss` driver.

There are several reasons for it:
* no real-life expertise among the maintainers
* oss is compatible with S3 API operations required by S3 storage driver

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-16 08:39:20 +01:00
Milos Gajdos
65b57464f9
Merge pull request #3982 from milosgajdos/remove-swift-storage-driver
Remove SWIFT storage driver
2023-08-16 07:47:42 +01:00
James Hewitt
46ff5f8528
Fix Azure tests
The Azure tests fail if there is no Azure configuration available,
instead they should be skipped.

Also, one of the Azure tests is wrong and doesn't match the code.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:46:36 +01:00
James Hewitt
7622d0a453
Don't return the from of a walk
Other storage drivers will only return children and below, s3 should do
the same. The only reason it was returning was because of the addition
of a / to ensure we treat the from as a directory.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:26:37 +01:00
James Hewitt
f7bdd9127b
Don't test the OUTPOSTS storage class
This test will only work on an s3 bucket on an s3 outpost. Most
developers won't have access to one of these.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:52 +01:00
James Hewitt
6ceb904c3e
Don't check returned storage class if we use NONE
If we haven't set a storage class there's no point in checking the
storage class applied to the object - s3 will choose one.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
2d316a12d3
We don't use gocheck in these tests
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
f78d81e78a
Remove test as S3 does not support empty directories
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:48 +01:00
Milos Gajdos
c6b9944ab1
Remove SWIFT storage driver
This commit removes swift storage driver from distribution.
There are several reasons for it:
* no real life expertise among the maintainers
* swift is compatible with S3 API operations required by S3 storage driver

This will also remove depedencies that are also hard to keep up with.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-15 09:14:11 +01:00
Milos Gajdos
d5c1b39b8b
Merge pull request #3206 from takmatsu/suppurt-path-in-middleware
Make redirect middleware can use path
2023-07-14 10:50:29 +01:00
Milos Gajdos
316e1c6b82
Get rid of unnecessary import alias
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-07-14 10:37:42 +01:00
MATSUMOTO TAKEAKI
a3eb956464
use path.Join() for building path
Signed-off-by: MATSUMOTO TAKEAKI <takeaki.matsumoto@linecorp.com>
2023-07-14 10:37:21 +01:00
MATSUMOTO TAKEAKI
a1cfd267c8
Make redirect middleware can use path
Signed-off-by: MATSUMOTO TAKEAKI <takeaki.matsumoto@linecorp.com>
2023-07-14 10:36:23 +01:00
Milos Gajdos
6b388b1ba6
Enable Go build tags
This enables go build tags so the GCS and OSS driver support is
available in the binary distributed via the image build by Dockerfile.

This led to quite a few fixes in the GCS and OSS packages raised as
warning by golang-ci linter.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-06-28 11:41:22 +01:00
Milos Gajdos
22725209e3
Merge pull request #3936 from flavianmissi/azure-path-not-found
Fix path not found error in Azure
2023-06-26 13:21:00 +01:00
Flavian Missi
2b72c4d1ca registry/storage/driver/azure: fix Move method
Something seems broken on azure/azure sdk side - it is currently not
possible to copy a blob of type AppendBlob using `CopyFromURL`.
Using the AppendBlob client via NewAppendBlobClient does not work
either.

According to Azure the correct way to do this is by using
StartCopyFromURL. Because this is an async operation, we need to do
polling ourselves. A simple backoff mechanism is used, where during each
iteration, the configured delay is multiplied by the retry number.

Also introduces two new config options for the Azure driver:
copy_status_poll_max_retry, and copy_status_poll_delay.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-06-26 13:47:30 +02:00
Milos Gajdos
5902a24fd0
Merge pull request #3929 from flavianmissi/fix-gcs-oss-file-writer
Fix gcs storage driver

Thanks to @jmontleon who laid the first bricks in #3702
2023-06-01 09:56:57 +01:00
Flavian Missi
0207adaa5c registry/storage/driver/gcs: fix code to use updated gcs driver
Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-31 09:28:03 +02:00
Flavian Missi
d0bc83d8e4 registry/storage/driver: receive context on Cancel methods
both oss and gcs driver were missing the context parameter that is
required to satisfy the storagedriver.FileWriter interface.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-31 09:14:51 +02:00
Milos Gajdos
ba46c769b3
Merge pull request #3932 from flavianmissi/fix-azure-test-parameters
registry/storage/driver: test call to Stat(ctx, "/")
2023-05-30 21:49:32 +01:00
Flavian Missi
0d20e7ae9e registry/storage/driver/testsuites: use 4MB for Azure append test
Fixes #3931.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:08 +02:00
Flavian Missi
0c33bb1092 registry/storage/driver/azure: consider CannotVerifyCopySource as 404
Azure will return CannotVerifyCopySource with a 404 status code from a
call to Move when the source blob does not exist.
Details: https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-error-codes.

This fixes the TestMoveNonexistent test case for the Azure driver.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:08 +02:00
Flavian Missi
d2e16fc74a registry/storage/driver/azure: fix driver parameters on tests
fixes "azure: no Host in request URL" tests failure, allowing azure
driver tests to run.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:08 +02:00
Flavian Missi
90ece48d77 registry/storage/driver: add test call to Stat on "/"
Stat(ctx, "/") is called by the registry healthcheck.
Also fixes blob name building in the Azure driver so it no longer
returns empty blob names. This was causing errors in the healthcheck
call to Stat for Azure.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:00 +02:00
Lucas França de Oliveira
035a8ec52a
Fix panic in the s3 backend walk logic
Signed-off-by: Lucas França de Oliveira <lucasfdo@palantir.com>
2023-05-25 14:56:05 -07:00
Sebastiaan van Stijn
f03d966ef7
cloudfront: use consistent names for test-tables, t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:39:58 +02:00
Sebastiaan van Stijn
5301ae14bf
cloudfront: rename vars that collided with type
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:17 +02:00
Milos Gajdos
08f7a47bc9
Merge pull request #2766 from glefloch/remove-testdriver
Remove registry storage testdriver
2023-05-03 21:40:53 +01:00
glefloch
9fb201e23d Remove registry storage testdriver
Signed-off-by: glefloch <glfloch@gmail.com>
2023-05-02 16:37:33 +02:00
Sebastiaan van Stijn
3fa6d5a33b
remove dot-imports for gopkg.in/check.v1
Dot-imports were only used in a couple of places, and replacing them
makes it more explicit what's imported.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-29 17:08:47 +02:00
Milos Gajdos
2fb8dbdeca
Merge pull request #3839 from kirat-singh/feature.azure-sdk-update
Update Azure SDK and support additional authentication schemes
2023-04-25 19:35:34 +01:00
Kirat Singh
ba4a6bbe02 Update Azure SDK and support additional authentication schemes
Microsoft has updated the golang Azure SDK significantly.  Update the
azure storage driver to use the new SDK.  Add support for client
secret and MSI authentication schemes in addition to shared key
authentication.

Implement rootDirectory support for the azure storage driver to mirror
the S3 driver.

Signed-off-by: Kirat Singh <kirat.singh@beacon.io>

Co-authored-by: Cory Snider <corhere@gmail.com>
2023-04-25 17:23:20 +00:00
Milos Gajdos
0c958010ac
Merge pull request #3763 from distribution/multipart-upload-empty-files
Enable pushing empty blobs
2023-03-27 10:18:44 +01:00
Milos Gajdos
5fa926a609
Enable pushing empty blobs
This is an edge case when we are trying to upload an empty chunk of data using
a MultiPart upload. As a result we are trying to complete the MultipartUpload
with an empty slice of `completedUploadedParts` which will always lead to 400
being returned from S3 See: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#CompletedMultipartUpload
Solution: we upload an empty i.e. 0 byte part as a single part and then append it
to the completedUploadedParts slice used to complete the Multipart upload.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-03-27 10:11:07 +01:00
Aaron Lehmann
2074688be9 Fix S3 multipart upload pagination loop condition
The loop that iterates over paginated lists of S3 multipart upload parts
appears to be using the wrong variable in its loop condition. Nothing
inside the loop affects the value of `resp.IsTruncated`, so this loop
will either be wrongly skipped or loop forever.

It looks like this is a regression caused by commit
7736319f2e. The return value of
`ListMultipartUploads` used to be assigned to a variable named `resp`,
but it was renamed to `partsList` without updating the for loop
condition.

I believe this is causing an error we're seeing with large layer uploads
at commit time:

    upload resumed at wrong offset: 5242880000 != 5815706782

Missing parts of the multipart S3 upload would cause an incorrect size
calculation in `newWriter`.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-02-21 20:57:50 -08:00
Kirat Singh
3117e2eb2f
Use default http.Transport for AWS S3 session
Previously we used a custom Transport in order to modify the user agent header.
This prevented the AWS SDK from being able to customize SSL and other client TLS
parameters since it could not understand the Transport type.

Instead we can simply use the SDK function MakeAddToUserAgentFreeFormHandler to
customize the UserAgent if necessary and leave all the TLS configuration to the
AWS SDK.

The only exception being SkipVerify which we have to handle, but we can set it
onto the standard http.Transport which does not interfere with the SDKs ability
to set other options.

Signed-off-by: Kirat Singh <kirat.singh@gmail.com>
2023-02-15 13:37:01 -05:00
Milos Gajdos
3b8fbf9752
Merge pull request #3686 from m5i-work/exp 2022-11-11 17:07:14 +00:00
Wei Meng
35cae1099e Realloc slice exponentially in mfs
`registry/storage/driver/inmemory/driver_test.go` times out after ~10min. The slow test is `testsuites.go:TestWriteReadLargeStreams()` which writes a 5GB file.
Root cause is inefficient slice reallocation algorithm. The slice holding file bytes grows only 32K on each allocation. To fix it, this PR grows slice exponentially.

Signed-off-by: Wei Meng <wemeng@microsoft.com>
2022-11-11 18:18:08 +08:00
Sebastiaan van Stijn
842d4c04f5
cloudfront: use strings.Equalfold()
Minor optimization :)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:16 +01:00
Sebastiaan van Stijn
3b391d3290
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_ = strings.SplitN(s, "=", 2)[0]
		}
	}
}

func BenchmarkCut(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_, _, _ = strings.Cut(s, "=")
		}
	}
}
```

    BenchmarkSplit
    BenchmarkSplit-10    	 8244206	       128.0 ns/op	     128 B/op	       4 allocs/op
    BenchmarkCut
    BenchmarkCut-10      	54411998	        21.80 ns/op	       0 B/op	       0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints;

- for cases where an specific number of items is expected, I used `strings.SplitN()`
  with a suitable limit. This prevents (theoretical) unlimited splits.
- in some cases it we were using `strings.Split()`, but _actually_ were trying to match
  a prefix; for those I replaced the code to just match (and/or strip) the prefix.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:12 +01:00
Sebastiaan van Stijn
f8b3af78fc
replace deprecated io/ioutil
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 23:47:52 +01:00
Hayley Swimelar
e3509fc1de
Merge pull request #3635 from milosgajdos/make-s3-driver-delete-faster
Delete S3 keys incrementally in batches
2022-11-04 16:56:41 +01:00