Commit graph

4857 commits

Author SHA1 Message Date
Kirat Singh
ba4a6bbe02 Update Azure SDK and support additional authentication schemes
Microsoft has updated the golang Azure SDK significantly.  Update the
azure storage driver to use the new SDK.  Add support for client
secret and MSI authentication schemes in addition to shared key
authentication.

Implement rootDirectory support for the azure storage driver to mirror
the S3 driver.

Signed-off-by: Kirat Singh <kirat.singh@beacon.io>

Co-authored-by: Cory Snider <corhere@gmail.com>
2023-04-25 17:23:20 +00:00
Jose D. Gomez R
4c1561e9fb
Fix runaway allocation on /v2/_catalog
Introduced a Catalog entry in the configuration struct. With it,
it's possible to control the maximum amount of entries returned
by /v2/catalog (`GetCatalog` in registry/handlers/catalog.go).

It's set to a default value of 1000.

`GetCatalog` returns 100 entries by default if no `n` is
provided. When provided it will be validated to be between `0`
and `MaxEntries` defined in Configuration. When `n` is outside
the aforementioned boundary, an error response is returned.

`GetCatalog` now handles `n=0` gracefully with an empty response
as well.

Signed-off-by: José D. Gómez R. <1josegomezr@gmail.com>
2023-03-31 13:17:43 +02:00
Milos Gajdos
0c958010ac
Merge pull request #3763 from distribution/multipart-upload-empty-files
Enable pushing empty blobs
2023-03-27 10:18:44 +01:00
Milos Gajdos
5fa926a609
Enable pushing empty blobs
This is an edge case when we are trying to upload an empty chunk of data using
a MultiPart upload. As a result we are trying to complete the MultipartUpload
with an empty slice of `completedUploadedParts` which will always lead to 400
being returned from S3 See: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#CompletedMultipartUpload
Solution: we upload an empty i.e. 0 byte part as a single part and then append it
to the completedUploadedParts slice used to complete the Multipart upload.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-03-27 10:11:07 +01:00
Milos Gajdos
e5d5810851
Merge pull request #3847 from aaronlehmann/s3-pagination-loop
Fix S3 multipart upload pagination loop condition
2023-02-23 07:28:52 +00:00
Aaron Lehmann
2074688be9 Fix S3 multipart upload pagination loop condition
The loop that iterates over paginated lists of S3 multipart upload parts
appears to be using the wrong variable in its loop condition. Nothing
inside the loop affects the value of `resp.IsTruncated`, so this loop
will either be wrongly skipped or loop forever.

It looks like this is a regression caused by commit
7736319f2e. The return value of
`ListMultipartUploads` used to be assigned to a variable named `resp`,
but it was renamed to `partsList` without updating the for loop
condition.

I believe this is causing an error we're seeing with large layer uploads
at commit time:

    upload resumed at wrong offset: 5242880000 != 5815706782

Missing parts of the multipart S3 upload would cause an incorrect size
calculation in `newWriter`.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-02-21 20:57:50 -08:00
Milos Gajdos
a38eb86fa6
Merge pull request #3845 from distribution/dependabot/go_modules/golang.org/x/net-0.7.0 2023-02-20 14:00:06 +00:00
Milos Gajdos
772cef6b4e
Merge pull request #3736 from aaronlehmann/log-username-on-successful-requests
Log username on successful requests
2023-02-19 19:05:35 +00:00
dependabot[bot]
9594fbcfeb
build(deps): bump golang.org/x/net from 0.4.0 to 0.7.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.4.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-17 22:42:11 +00:00
Hayley Swimelar
a4d64adabc
Merge pull request #3838 from smorimoto/update-fossa-action
Update fossa-contrib/fossa-action action to v2
2023-02-17 14:28:03 -08:00
Milos Gajdos
e64b08ada6
Merge pull request #3841 from kirat-singh/feature/aws-ca-bundle
Support AWS_CA_BUNDLE when talking to the S3 API
2023-02-17 09:36:20 +00:00
Kirat Singh
3117e2eb2f
Use default http.Transport for AWS S3 session
Previously we used a custom Transport in order to modify the user agent header.
This prevented the AWS SDK from being able to customize SSL and other client TLS
parameters since it could not understand the Transport type.

Instead we can simply use the SDK function MakeAddToUserAgentFreeFormHandler to
customize the UserAgent if necessary and leave all the TLS configuration to the
AWS SDK.

The only exception being SkipVerify which we have to handle, but we can set it
onto the standard http.Transport which does not interfere with the SDKs ability
to set other options.

Signed-off-by: Kirat Singh <kirat.singh@gmail.com>
2023-02-15 13:37:01 -05:00
Milos Gajdos
36d8c594d7
Merge pull request #3823 from drornir/patch-1
add double newlines in ADOPTERS.md
2023-02-14 15:00:26 +00:00
drornir
2179a5f66c add double newlines in ADOPTERS.md
Using double newlines in markdown makes it easier to read in markdown readers, e.g github

Signed-off-by: dror.nir <dror@minutemedia.com>
2023-02-14 14:51:41 +02:00
Sora Morimoto
165fd5f9ac Update fossa-contrib/fossa-action action to v2
Signed-off-by: Sora Morimoto <sora@morimoto.io>
2023-02-14 01:43:23 +09:00
Milos Gajdos
cf87e8d07e
Merge pull request #3818 from ckw017/ckw/fixseparatorregex
Fix separator regex to disallow empty strings
2023-01-31 08:15:13 +00:00
Aaron Lehmann
a811c1bb57 Log username on successful requests
Currently, "response completed with error" log lines include an
`auth.user.name` key, but successful "response completed" lines do not
include this, because they are logged a few stack frames up where
`auth.user.name` is not present on the `Context`. Move the successful
request logging inside the `dispatcher` closure, where the logger on the
context automatically includes this key.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-01-30 09:32:39 -08:00
Milos Gajdos
9b629737cb
Merge pull request #3804 from thaJeztah/deprecate_schema1
manifest/schema1: mark docker manifest v2, schema 1 deprecated
2023-01-30 16:16:38 +00:00
Chris K. Wong
d1c186812e Fix separator regex
Signed-off-by: Chris K. Wong <chriskw.xyz@gmail.com>
2022-12-22 14:50:50 -08:00
Milos Gajdos
362910506b
Merge pull request #3816 from thaJeztah/bump_golang_net
vendor: golang.org/x/net v0.4.0
2022-12-08 16:53:59 +00:00
Sebastiaan van Stijn
345be95498
vendor: golang.org/x/net v0.4.0
golang.org/x/net contains a fix for CVE-2022-41717, which was addressed
in stdlib in go1.19.4 and go1.18.9;

> net/http: limit canonical header cache by bytes, not entries
>
> An attacker can cause excessive memory growth in a Go server accepting
> HTTP/2 requests.
>
> HTTP/2 server connections contain a cache of HTTP header keys sent by
> the client. While the total number of entries in this cache is capped,
> an attacker sending very large keys can cause the server to allocate
> approximately 64 MiB per open connection.
>
> This issue is also fixed in golang.org/x/net/http2 v0.4.0,
> for users manually configuring HTTP/2.

full diff: https://github.com/golang/net/compare/v0.2.0...v0.4.0

other dependency updates (due to (circular) dependencies):

- golang.org/x/sys v0.3.0: https://github.com/golang/sys/compare/3c1f35247d10...v0.3.0
- golang.org/x/text v0.5.0: https://github.com/golang/text/compare/v0.3.7...v0.5.0

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-08 10:39:04 +01:00
Hayley Swimelar
7847817946
Merge pull request #3814 from thaJeztah/remove_fuzz_leftovers
remove script directory and fuzz left-overs
2022-12-05 15:46:40 -08:00
Sebastiaan van Stijn
59f1357775
remove script directory and fuzz left-overs
commit 9337b8df66 rewrote the fuzzers to
native go fuzzers, so the script was no longer needed. With this, the
script directory is no longer used, so we can remove it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-12-03 17:15:16 +01:00
Wang Yan
92d136e113
Merge pull request #3810 from thaJeztah/align_descriptor_with_oci
Descriptor: align field order with OCI image specification
2022-12-01 16:32:18 +08:00
Sebastiaan van Stijn
19233195b7
Align code to match order of fields
This is just cosmetic; alighn the fields with the order in which they appear
in the struct (and JSON output).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-29 23:24:10 +01:00
Sebastiaan van Stijn
86cd830fb3
Descriptor: align field order with OCI image specification
I am looking at aligning the types defined in this repository with the
OCI image specification, and potentially exchanging local types with
those from the specification.

This patch is a stepping-stone towards that effort, but as this changes
the format of the serialized JSON, I wanted to put this up first before
proceeding with the other work in case there are concerns.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-29 23:24:10 +01:00
Milos Gajdos
ac302d9ce5
Merge pull request #3807 from thaJeztah/replace_types_for_oci_step1
minor fixes and enhancements
2022-11-29 10:49:12 +00:00
Milos Gajdos
8cc58797e8
Merge pull request #3794 from AdamKorcz/fuzz1
Fuzzing: Rewrite existing fuzzers to native go fuzzers
2022-11-29 09:57:09 +00:00
Sebastiaan van Stijn
0b4311d5ce
manifest: improve test output and use const
Use consts to make clear these values are fixed, and improve the output
to make it clearer which part is the expected output, and which part
the actual.

Before this:

    === RUN   TestManifest
        manifest_test.go:87: manifest bytes not equal: "{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.oci.image.manifest.v1+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.oci.image.config.v1+json\",\n      \"size\": 985,\n      \"digest\": \"sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b\",\n      \"annotations\": {\n         \"apple\": \"orange\"\n      }\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\",\n         \"size\": 153263,\n         \"digest\": \"sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b\",\n         \"annotations\": {\n            \"lettuce\": \"wrap\"\n         }\n      }\n   ],\n   \"annotations\": {\n      \"hot\": \"potato\"\n   }\n}" != "{\n   \"schemaVersion\": 2,\n   \"mediaType\": \"application/vnd.oci.image.manifest.v1+json\",\n   \"config\": {\n      \"mediaType\": \"application/vnd.oci.image.config.v1+json\",\n      \"size\": 985,\n      \"digest\": \"sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b\",\n      \"annotations\": {\n         \"apple\": \"orange\"\n      }\n   },\n   \"layers\": [\n      {\n         \"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\",\n         \"size\": 153263,\n         \"digest\": \"sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b\",\n         \"annotations\": {\n            \"lettuce\": \"wrap\"\n         }\n      }\n   ],\n   \"annotations\": {\n      \"hot\": \"potato\"\n   }\n}"
    --- FAIL: TestManifest (0.00s)

After this:

    === RUN   TestManifest
        manifest_test.go:72: manifest bytes not equal:
            expected:
            {
               "schemaVersion": 2,
               "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
               "config": {
                  "mediaType": "application/vnd.docker.container.image.v1+json",
                  "size": 985,
                  "digest": "sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b"
               },
               "layers": [
                  {
                     "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                     "size": 153263,
                     "digest": "sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b"
                  }
               ]
            }
            actual:
            {
               "schemaVersion": 2,
               "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
               "config": {
                  "mediaType": "application/vnd.docker.container.image.v1+json",
                  "size": 985,
                  "digest": "sha256:1a9ec845ee94c202b2d5da74a24f0ed2058318bfa9879fa541efaecba272e86b"
               },
               "layers": [
                  {
                     "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                     "size": 153263,
                     "digest": "sha256:62d8908bee94c202b2d35224a221aaa2058318bfa9879fa541efaecba272331b"
                  }
               ]
            }
    --- FAIL: TestManifest (0.00s)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Sebastiaan van Stijn
f2db7faa2f
registry/storage: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Sebastiaan van Stijn
030489ca66
testutil: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Sebastiaan van Stijn
0e3efe749b
manifest: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:02 +01:00
Milos Gajdos
b9f9073d1a
Merge pull request #3805 from crazy-max/update-gha
ci: update github actions
2022-11-27 10:56:15 +00:00
CrazyMax
b91c9a22f4
ci: add concurrency check
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-11-26 16:09:46 +01:00
CrazyMax
2400718d81
ci: update github actions
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-11-26 16:09:46 +01:00
Sebastiaan van Stijn
ff2bce2731
manifest/schema1: mark docker manifest v2, schema 1 deprecated
Docker Image manifest v2, schema version 1 is deprecated since 2015, when
manifest v2, schema version 2 was introduced (2e3f4934a7).

Users should no longer use this specification other than for backward
compatibility.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-26 13:21:06 +01:00
Milos Gajdos
ede90db01c
Merge pull request #3789 from thaJeztah/reference_consts
reference: clean up regular expressions
2022-11-25 12:38:43 +00:00
Sebastiaan van Stijn
02e88c0f15
reference: move exported regexes to separate block
This makes them easier to find between the non-exported ones, and puts
them as separate sections in the generated docs. While updating, also
extended documentation for some to be more descriptive.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-23 00:01:57 +01:00
Sebastiaan van Stijn
a4cec8ca82
reference: introduce const for "localhost"
Localhost is treated special when parsing references, and always considered
to be a domain, despite not having a "." nor a ":port". Adding a const for
this, to allow documenting this special case (making it more visible).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-22 23:45:26 +01:00
Sebastiaan van Stijn
bbd41f40bb
reference: introduce remoteName variable
This pattern was used in two places, so adding an intermediate variable allows
documenting its purpose. The "remote-name" grammer (which is interchangably
used with "path") also seemed to be missing from the grammar, so adding it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-22 23:45:23 +01:00
Sebastiaan van Stijn
71a0666398
reference: optional repeated == any number of times
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-22 23:41:56 +01:00
Sebastiaan van Stijn
919bd8ab09
reference: add const for (optional) port, and rename "domain" variable
The `domain` variable didn't make it clear that this could include port-numbers
as well, so renaming it makes that more visible.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:11:33 +01:00
Sebastiaan van Stijn
f0c7c97e73
reference: remove remaining uses of "expression()"
The remaining uses of "expression()" were quite trivial; probably goes without
saying, but just using string-concatenating for these is more performant as well,
and removing the extra abstraction may make it easier to read;

    pkg: github.com/distribution/distribution/v3/reference
    BenchmarkExpression
    BenchmarkExpression-10    27260877        43.10 ns/op      24 B/op       1 allocs/op
    BenchmarkConcat
    BenchmarkConcat-10      1000000000         0.3154 ns/op     0 B/op       0 allocs/op
    PASS
    ok  	github.com/distribution/distribution/v3/reference	1.762s

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:11:33 +01:00
Sebastiaan van Stijn
04d6592df1
reference: remove "literal()" utility
With the exception of ".", none of the literals used required escaping, which made
the function rather redundant (and the extra abstraction made it harder to read).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:11:27 +01:00
Sebastiaan van Stijn
c786a2bd3e
reference: inline "group()"
It was only used in a couple of places, and more transparent to just
inline it.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn
1d4917d4fb
reference: expression(): use strings.Join()
It's easier to read, and more performant:

    pkg: github.com/distribution/distribution/v3/reference
    BenchmarkExpression
    BenchmarkExpression-10    	10474380	        97.32 ns/op	      64 B/op	       4 allocs/op
    BenchmarkJoin
    BenchmarkJoin-10          	27722588	        42.71 ns/op	      24 B/op	       1 allocs/op
    PASS

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn
a7e7ff933c
reference: align docs and variables with grammar
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn
32a4d8e39f
reference: fix docs for NameRegexp
NameRegexp does not have capturing groups, so updating the documentation
to reflect that.

To verify if this was an unintentional regression, I looked up the commit
that introduced this regex (31a448a628), and
it looks like it never had capturing groups, so this was just a mistake in
the docs.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:25 +01:00
Sebastiaan van Stijn
226b21beb6
reference: make some regexp vars a const, remove intermediate vars
This patch:

- makes regexp strings that are constant a const
- moves some variables closer to where they're used
- removes some intermediate vars
- un-wraps some lines; they're lengthy, but probably more readable than having
  them wrapped over multiple lines.
- touches-up some docs.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00
Sebastiaan van Stijn
10eace9a53
reference: document consts for normalizing and legacy domain
These const could use some documentation, as it won't be clear why
they exist.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00