Commit graph

1165 commits

Author SHA1 Message Date
Sebastiaan van Stijn
999527f978
Ignore SA1019: "schema1 is deprecated" linting errors
We need to use this for backward compatibility.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:17 +02:00
Sebastiaan van Stijn
3c144f2264
registry/auth/token: fix the surrounding loop is unconditionally terminate
registry/auth/token/types_test.go:83:3: SA4004: the surrounding loop is unconditionally terminated (staticcheck)
                   return
                   ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:17 +02:00
Sebastiaan van Stijn
4052d269f5
reference, registry: fix loop variable captured by func literal (govet)
reference/normalize_test.go:274:40: loopclosure: loop variable r captured by func literal (govet)
                                   named, err := ParseNormalizedNamed(r)
                                                                      ^
    reference/normalize_test.go:276:29: loopclosure: loop variable r captured by func literal (govet)
                        t.Fatalf("ref=%s: %v", r, err)
                                               ^
    registry/api/errcode/errors_test.go:45:7: loopclosure: loop variable ec captured by func literal (govet)
                if ec != desc.Code {
                   ^
    registry/api/errcode/errors_test.go:46:66: loopclosure: loop variable ec captured by func literal (govet)
                    t.Fatalf("error code in descriptor isn't correct, %q != %q", ec, desc.Code)
                                                                                 ^
    registry/api/errcode/errors_test.go:49:23: loopclosure: loop variable desc captured by func literal (govet)
                if idToDescriptors[desc.Value].Code != ec {
                                   ^
    registry/api/errcode/errors_test.go:50:80: loopclosure: loop variable desc captured by func literal (govet)
                    t.Fatalf("error code in idToDesc isn't correct, %q != %q", idToDescriptors[desc.Value].Code, ec)
                                                                                               ^
    registry/api/errcode/errors_test.go:53:7: loopclosure: loop variable ec captured by func literal (govet)
                if ec.Message() != desc.Message {
                   ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:08 +02:00
Sebastiaan van Stijn
f03d966ef7
cloudfront: use consistent names for test-tables, t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:39:58 +02:00
Sebastiaan van Stijn
5301ae14bf
cloudfront: rename vars that collided with type
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:17 +02:00
Sebastiaan van Stijn
85028a801b
registry/handlers: use consistent names for test-tables
Also marked assertBlobUploadStateEquals as t.Helper()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:17 +02:00
Sebastiaan van Stijn
f4acd98865
registry/api/v2: checkTestRouter(): use sub-tests, t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:05 +02:00
Sebastiaan van Stijn
2444c3282d
registry/api/v2: use consistent names for test-tables
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:16 +02:00
Sebastiaan van Stijn
9266220c2a
registry/auth/token: TestAudienceList_Unmarshal: t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:16 +02:00
Sebastiaan van Stijn
5e67a40e08
registry/auth/token: use consistent names for test-tables
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:16 +02:00
Sebastiaan van Stijn
f884a079df
registry/api/errorcode: TestErrorCodes: use sub-tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:11 +02:00
Wang Yan
8900e90699
Merge pull request #3895 from pluralsh/bug-fix-5001-server-rebase
Fix the issue that the debug server with port 5001 run twice
2023-05-05 13:21:55 +08:00
Milos Gajdos
08f7a47bc9
Merge pull request #2766 from glefloch/remove-testdriver
Remove registry storage testdriver
2023-05-03 21:40:53 +01:00
David van der Spek
99a8ad00ea fix: rename log to logrus
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-05-03 19:02:03 +02:00
Feng Honglin
f0fdaff0a5 Fix the issue that the debug server with port 5001 run twice
When configuring bugsnag, bugsnag will fork the process, resulting the port 5001 listened twice. The PR fix this error by moving the initialization of prometheus server after the configuration of bugsnag

Signed-off-by: Honglin Feng <tifayuki@gmail.com>
(cherry picked from commit 5a6a2d6ae06453136f5e1cfb5e9efa20c27085d9)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-05-03 18:37:32 +02:00
AdamKorcz
e2a43ec8d3
Fuzzing: Move over two fuzzers from cncf-fuzzing
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-02 23:03:57 +02:00
glefloch
9fb201e23d Remove registry storage testdriver
Signed-off-by: glefloch <glfloch@gmail.com>
2023-05-02 16:37:33 +02:00
Milos Gajdos
788598e23f
Merge pull request #3881 from vvoland/handle-forbidden
registry/errors: Parse http forbidden as denied
2023-05-02 09:38:36 +01:00
Sebastiaan van Stijn
3fa6d5a33b
remove dot-imports for gopkg.in/check.v1
Dot-imports were only used in a couple of places, and replacing them
makes it more explicit what's imported.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-29 17:08:47 +02:00
Paweł Gronowski
5f1df02149
registry/errors: Parse http forbidden as denied
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-04-27 19:48:32 +02:00
Milos Gajdos
29b5e79f82
Merge pull request #3742 from sagikazarmark/fix-aud-claim-list
Accept list of strings in audience claim in token auth
2023-04-26 18:39:26 +01:00
Milos Gajdos
2fb8dbdeca
Merge pull request #3839 from kirat-singh/feature.azure-sdk-update
Update Azure SDK and support additional authentication schemes
2023-04-25 19:35:34 +01:00
Kirat Singh
ba4a6bbe02 Update Azure SDK and support additional authentication schemes
Microsoft has updated the golang Azure SDK significantly.  Update the
azure storage driver to use the new SDK.  Add support for client
secret and MSI authentication schemes in addition to shared key
authentication.

Implement rootDirectory support for the azure storage driver to mirror
the S3 driver.

Signed-off-by: Kirat Singh <kirat.singh@beacon.io>

Co-authored-by: Cory Snider <corhere@gmail.com>
2023-04-25 17:23:20 +00:00
Milos Gajdos
0c958010ac
Merge pull request #3763 from distribution/multipart-upload-empty-files
Enable pushing empty blobs
2023-03-27 10:18:44 +01:00
Milos Gajdos
5fa926a609
Enable pushing empty blobs
This is an edge case when we are trying to upload an empty chunk of data using
a MultiPart upload. As a result we are trying to complete the MultipartUpload
with an empty slice of `completedUploadedParts` which will always lead to 400
being returned from S3 See: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#CompletedMultipartUpload
Solution: we upload an empty i.e. 0 byte part as a single part and then append it
to the completedUploadedParts slice used to complete the Multipart upload.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-03-27 10:11:07 +01:00
Aaron Lehmann
2074688be9 Fix S3 multipart upload pagination loop condition
The loop that iterates over paginated lists of S3 multipart upload parts
appears to be using the wrong variable in its loop condition. Nothing
inside the loop affects the value of `resp.IsTruncated`, so this loop
will either be wrongly skipped or loop forever.

It looks like this is a regression caused by commit
7736319f2e. The return value of
`ListMultipartUploads` used to be assigned to a variable named `resp`,
but it was renamed to `partsList` without updating the for loop
condition.

I believe this is causing an error we're seeing with large layer uploads
at commit time:

    upload resumed at wrong offset: 5242880000 != 5815706782

Missing parts of the multipart S3 upload would cause an incorrect size
calculation in `newWriter`.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-02-21 20:57:50 -08:00
Milos Gajdos
772cef6b4e
Merge pull request #3736 from aaronlehmann/log-username-on-successful-requests
Log username on successful requests
2023-02-19 19:05:35 +00:00
Kirat Singh
3117e2eb2f
Use default http.Transport for AWS S3 session
Previously we used a custom Transport in order to modify the user agent header.
This prevented the AWS SDK from being able to customize SSL and other client TLS
parameters since it could not understand the Transport type.

Instead we can simply use the SDK function MakeAddToUserAgentFreeFormHandler to
customize the UserAgent if necessary and leave all the TLS configuration to the
AWS SDK.

The only exception being SkipVerify which we have to handle, but we can set it
onto the standard http.Transport which does not interfere with the SDKs ability
to set other options.

Signed-off-by: Kirat Singh <kirat.singh@gmail.com>
2023-02-15 13:37:01 -05:00
Aaron Lehmann
a811c1bb57 Log username on successful requests
Currently, "response completed with error" log lines include an
`auth.user.name` key, but successful "response completed" lines do not
include this, because they are logged a few stack frames up where
`auth.user.name` is not present on the `Context`. Move the successful
request logging inside the `dispatcher` closure, where the logger on the
context automatically includes this key.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-01-30 09:32:39 -08:00
Milos Gajdos
ac302d9ce5
Merge pull request #3807 from thaJeztah/replace_types_for_oci_step1
minor fixes and enhancements
2022-11-29 10:49:12 +00:00
Milos Gajdos
8cc58797e8
Merge pull request #3794 from AdamKorcz/fuzz1
Fuzzing: Rewrite existing fuzzers to native go fuzzers
2022-11-29 09:57:09 +00:00
Sebastiaan van Stijn
f2db7faa2f
registry/storage: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Milos Gajdos
21622ca699
Merge pull request #3787 from thaJeztah/simplify_mocks 2022-11-18 08:53:26 +00:00
Milos Gajdos
5357c45703
Merge pull request #3788 from thaJeztah/deprecate_ReadSeekCloser 2022-11-18 08:53:15 +00:00
AdamKorcz
9337b8df66 Fuzzing: Rewrite existing fuzzers to native go fuzzers
Signed-off-by: AdamKorcz <adam@adalogics.com>
2022-11-12 17:30:10 +00:00
Milos Gajdos
3b8fbf9752
Merge pull request #3686 from m5i-work/exp 2022-11-11 17:07:14 +00:00
Wei Meng
35cae1099e Realloc slice exponentially in mfs
`registry/storage/driver/inmemory/driver_test.go` times out after ~10min. The slow test is `testsuites.go:TestWriteReadLargeStreams()` which writes a 5GB file.
Root cause is inefficient slice reallocation algorithm. The slice holding file bytes grows only 32K on each allocation. To fix it, this PR grows slice exponentially.

Signed-off-by: Wei Meng <wemeng@microsoft.com>
2022-11-11 18:18:08 +08:00
Sebastiaan van Stijn
1d8cd5e443
registry/client: use struct literals
Remove some intermediate variables, and use struct literals instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:11:12 +01:00
Sebastiaan van Stijn
d71ad5b3a6
transport.NewHTTPReadSeeker: return concrete type, deprecate ReadSeekCloser
General convention is to define interfaces on the receiver side, and
to return concrete types.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:39 +01:00
Sebastiaan van Stijn
019ead86f5
deprecate ReadSeekCloser in favor of io.ReadSeekCloser
Go's io package in stdlib now defines this interface, so we can switch
to using that instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:32 +01:00
Sebastiaan van Stijn
842d4c04f5
cloudfront: use strings.Equalfold()
Minor optimization :)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:16 +01:00
Sebastiaan van Stijn
3b391d3290
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_ = strings.SplitN(s, "=", 2)[0]
		}
	}
}

func BenchmarkCut(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_, _, _ = strings.Cut(s, "=")
		}
	}
}
```

    BenchmarkSplit
    BenchmarkSplit-10    	 8244206	       128.0 ns/op	     128 B/op	       4 allocs/op
    BenchmarkCut
    BenchmarkCut-10      	54411998	        21.80 ns/op	       0 B/op	       0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints;

- for cases where an specific number of items is expected, I used `strings.SplitN()`
  with a suitable limit. This prevents (theoretical) unlimited splits.
- in some cases it we were using `strings.Split()`, but _actually_ were trying to match
  a prefix; for those I replaced the code to just match (and/or strip) the prefix.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:12 +01:00
Sebastiaan van Stijn
2cd52d5c0c
simplify mocks
Embed the interface that we're mocking; calling any of it's methods
that are not implemented will panic, so should give the same result
as before.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 17:09:24 +01:00
Sebastiaan van Stijn
bf72536440
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 13:51:11 +01:00
Hayley Swimelar
e9a25da7a4
Merge pull request #3772 from thaJeztah/registry_logging_simplify
registry: configureLogging() simplify logic a bit
2022-11-08 09:17:20 +01:00
Sebastiaan van Stijn
b73c038000
registry: configureLogging() simplify logic a bit
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-05 13:26:13 +01:00
Sebastiaan van Stijn
f1dff3e434
registry: use consts for some defaults
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-05 13:26:05 +01:00
Sebastiaan van Stijn
f8b3af78fc
replace deprecated io/ioutil
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 23:47:52 +01:00
Hayley Swimelar
e3509fc1de
Merge pull request #3635 from milosgajdos/make-s3-driver-delete-faster
Delete S3 keys incrementally in batches
2022-11-04 16:56:41 +01:00
Hayley Swimelar
52d948a9f5
Merge pull request #3766 from thaJeztah/gofumpt
format code with gofumpt
2022-11-04 12:19:53 +01:00