Microsoft has updated the golang Azure SDK significantly. Update the
azure storage driver to use the new SDK. Add support for client
secret and MSI authentication schemes in addition to shared key
authentication.
Implement rootDirectory support for the azure storage driver to mirror
the S3 driver.
Signed-off-by: Kirat Singh <kirat.singh@beacon.io>
Co-authored-by: Cory Snider <corhere@gmail.com>
This is an edge case when we are trying to upload an empty chunk of data using
a MultiPart upload. As a result we are trying to complete the MultipartUpload
with an empty slice of `completedUploadedParts` which will always lead to 400
being returned from S3 See: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#CompletedMultipartUpload
Solution: we upload an empty i.e. 0 byte part as a single part and then append it
to the completedUploadedParts slice used to complete the Multipart upload.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
The loop that iterates over paginated lists of S3 multipart upload parts
appears to be using the wrong variable in its loop condition. Nothing
inside the loop affects the value of `resp.IsTruncated`, so this loop
will either be wrongly skipped or loop forever.
It looks like this is a regression caused by commit
7736319f2e. The return value of
`ListMultipartUploads` used to be assigned to a variable named `resp`,
but it was renamed to `partsList` without updating the for loop
condition.
I believe this is causing an error we're seeing with large layer uploads
at commit time:
upload resumed at wrong offset: 5242880000 != 5815706782
Missing parts of the multipart S3 upload would cause an incorrect size
calculation in `newWriter`.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
Previously we used a custom Transport in order to modify the user agent header.
This prevented the AWS SDK from being able to customize SSL and other client TLS
parameters since it could not understand the Transport type.
Instead we can simply use the SDK function MakeAddToUserAgentFreeFormHandler to
customize the UserAgent if necessary and leave all the TLS configuration to the
AWS SDK.
The only exception being SkipVerify which we have to handle, but we can set it
onto the standard http.Transport which does not interfere with the SDKs ability
to set other options.
Signed-off-by: Kirat Singh <kirat.singh@gmail.com>
Currently, "response completed with error" log lines include an
`auth.user.name` key, but successful "response completed" lines do not
include this, because they are logged a few stack frames up where
`auth.user.name` is not present on the `Context`. Move the successful
request logging inside the `dispatcher` closure, where the logger on the
context automatically includes this key.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
golang.org/x/net contains a fix for CVE-2022-41717, which was addressed
in stdlib in go1.19.4 and go1.18.9;
> net/http: limit canonical header cache by bytes, not entries
>
> An attacker can cause excessive memory growth in a Go server accepting
> HTTP/2 requests.
>
> HTTP/2 server connections contain a cache of HTTP header keys sent by
> the client. While the total number of entries in this cache is capped,
> an attacker sending very large keys can cause the server to allocate
> approximately 64 MiB per open connection.
>
> This issue is also fixed in golang.org/x/net/http2 v0.4.0,
> for users manually configuring HTTP/2.
full diff: https://github.com/golang/net/compare/v0.2.0...v0.4.0
other dependency updates (due to (circular) dependencies):
- golang.org/x/sys v0.3.0: https://github.com/golang/sys/compare/3c1f35247d10...v0.3.0
- golang.org/x/text v0.5.0: https://github.com/golang/text/compare/v0.3.7...v0.5.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
commit 9337b8df66 rewrote the fuzzers to
native go fuzzers, so the script was no longer needed. With this, the
script directory is no longer used, so we can remove it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is just cosmetic; alighn the fields with the order in which they appear
in the struct (and JSON output).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I am looking at aligning the types defined in this repository with the
OCI image specification, and potentially exchanging local types with
those from the specification.
This patch is a stepping-stone towards that effort, but as this changes
the format of the serialized JSON, I wanted to put this up first before
proceeding with the other work in case there are concerns.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Docker Image manifest v2, schema version 1 is deprecated since 2015, when
manifest v2, schema version 2 was introduced (2e3f4934a7).
Users should no longer use this specification other than for backward
compatibility.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This makes them easier to find between the non-exported ones, and puts
them as separate sections in the generated docs. While updating, also
extended documentation for some to be more descriptive.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Localhost is treated special when parsing references, and always considered
to be a domain, despite not having a "." nor a ":port". Adding a const for
this, to allow documenting this special case (making it more visible).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This pattern was used in two places, so adding an intermediate variable allows
documenting its purpose. The "remote-name" grammer (which is interchangably
used with "path") also seemed to be missing from the grammar, so adding it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `domain` variable didn't make it clear that this could include port-numbers
as well, so renaming it makes that more visible.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The remaining uses of "expression()" were quite trivial; probably goes without
saying, but just using string-concatenating for these is more performant as well,
and removing the extra abstraction may make it easier to read;
pkg: github.com/distribution/distribution/v3/reference
BenchmarkExpression
BenchmarkExpression-10 27260877 43.10 ns/op 24 B/op 1 allocs/op
BenchmarkConcat
BenchmarkConcat-10 1000000000 0.3154 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/distribution/distribution/v3/reference 1.762s
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
With the exception of ".", none of the literals used required escaping, which made
the function rather redundant (and the extra abstraction made it harder to read).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
