Commit graph

1377 commits

Author SHA1 Message Date
Geoffrey Hausheer
9721db9504 Add info message regarding socket-activation
Signed-off-by: Geoffrey Hausheer <rc2012@pblue.org>
2023-09-20 09:56:25 -07:00
Geoffrey Hausheer
2435def474 Support systemd socket-activation
Signed-off-by: Geoffrey Hausheer <rc2012@pblue.org>
2023-09-20 09:37:22 -07:00
Milos Gajdos
42ce5d4d51
Merge pull request #3569 from justadogistaken/optimize/avoid-redundant-blob-fetching
optimize: avoid redundant blob fetching
2023-09-18 08:01:14 +01:00
baojiangnan
17952924f3 avoid redundant blob fetching
Signed-off-by: baojiangnan <baojn1998@163.com>
2023-09-18 10:40:25 +08:00
Milos Gajdos
612ad42609
Merge pull request #4040 from thaJeztah/move_api_errors 2023-09-15 09:36:36 +01:00
Sebastiaan van Stijn
c8ba5d7081
registry/client: combine SuccessStatus and HandleErrorResponse
The SuccessStatus acted on the response's status code, and was used to return
early, before checking the same status code with HandleErrorResponse.

This patch combines both functions into a HandleHTTPResponseError, which
returns an error for "non-success" status-codes, which simplifies handling
of responses, and makes some logic slightly more idiomatic.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-08 14:40:41 +02:00
Milos Gajdos
612a30a7e7
Remove libtrust from handler tests
It was used for signing schema v1 manifests in tests which have now been
removed so there is no point in keeping these there anymore.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-05 08:41:29 +01:00
Milos Gajdos
6baa31a273
Remove duplicate code that instruments Redis OTLP
This was somehow overlooked in https://github.com/distribution/distribution/pull/4019

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-04 17:52:38 +01:00
Sebastiaan van Stijn
0104adf4a8
registry/api/errcode: split Register to internal / exported
Use the non-exported function to all errors; there's currently no external
consumers of this function (perhaps it should be deprecated).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-04 18:03:00 +02:00
Sebastiaan van Stijn
292e30bc61
registry/api: move all errors to "errcode" package
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-04 18:02:54 +02:00
Milos Gajdos
9790bc806c
Merge pull request #4037 from milosgajdos/enable-prealloc
Enable prealloc linter
2023-09-04 16:57:29 +01:00
Milos Gajdos
823cd4a370
Add a comment why prealloc linter is disabled when configuring endpoints
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:33:20 +01:00
Milos Gajdos
1089800643
Preallocate created slice in S3 tests
In case drvr.PutContent fails and returns error we'd have
some extra memory allocated, though in this case
(test with known size of the slice being iterated), that's fine.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:26:32 +01:00
Milos Gajdos
a9d31ec7b9
Avoid unnecessary type assertion in mfs driver
We already make sure the node in *dir

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:23:25 +01:00
Milos Gajdos
59fd8656ac
Enable prealloc linter
This will give us nice little performance gains in some code paths.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 22:41:51 +01:00
Milos Gajdos
dcdd8bb740
Propagate storage driver context to S3 API calls
Only some of the S3 storage driver calls were propagating context to the
S3 API calls. This commit updates the S3 storage drivers so the context
is propagated to all the S3 API calls.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 21:54:54 +01:00
Milos Gajdos
db4bd9933e
Merge pull request #4031 from thaJeztah/migrate_reference
deprecate reference package, migrate to github.com/distribution/reference
2023-08-31 15:20:28 +01:00
Sebastiaan van Stijn
152af63ec5
deprecate reference package, migrate to github.com/distribution/reference
This integrates the new module, which was extracted from this repository
at commit b9b19409cf458dcb9e1253ff44ba75bd0620faa6;

    # install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
    brew install git-filter-repo

    # create a temporary clone of docker
    cd ~/Projects
    git clone https://github.com/distribution/distribution.git reference
    cd reference

    # commit taken from
    git rev-parse --verify HEAD
    b9b19409cf

    # remove all code, except for general files, 'reference/', and rename to /
    git filter-repo \
      --path .github/workflows/codeql-analysis.yml \
      --path .github/workflows/fossa.yml \
      --path .golangci.yml \
      --path distribution-logo.svg \
      --path CODE-OF-CONDUCT.md \
      --path CONTRIBUTING.md \
      --path GOVERNANCE.md \
      --path README.md \
      --path LICENSE \
      --path MAINTAINERS \
      --path-glob 'reference/*.*' \
      --path-rename reference/:

    # initialize go.mod
    go mod init github.com/distribution/reference
    go mod tidy -go=1.20

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-31 15:47:06 +02:00
Milos Gajdos
17552d864d
Merge pull request #3685 from Jamstah/aws-paging
Work with the storage driver to minimise work when paging
2023-08-31 08:27:26 +01:00
bin liu
2513dd1f96 fix typos in registry/storage/paths.go
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-30 22:36:14 +08:00
Milos Gajdos
b9b19409cf
Merge pull request #4028 from liubin/delete-duplicated-code 2023-08-30 07:45:51 +01:00
bin liu
eda5fe2d67 remove duplicated code
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-30 07:56:56 +08:00
James Hewitt
e22f7cbc73
Pass the last paging flag to storage drivers
Storage drivers may be able to take advantage of the hint to start
their walk more efficiently.

For S3: The API takes a start-after parameter. Registries with many
repositories can drastically reduce calls to s3 by telling s3 to only
list results lexographically after the last parameter.

For the fallback: We can start deeper in the tree and avoid statting
the files and directories before the hint in a walk. For a filesystem
this improves performance a little, but many of the API based drivers
are currently treated like a filesystem, so this drastically improves
the performance of GCP and Azure blob.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-29 11:27:42 +01:00
Milos Gajdos
3a44c2e10e
Merge pull request #4022 from Jamstah/catalog-optimisation
Don't make a new buffer for catalog listing
2023-08-29 11:24:39 +01:00
James Hewitt
a41613ba3a
Don't make a new buffer for catalog listing
We are given a slice to fill, write catalog entries directly to the
slice until it is full.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-29 11:11:13 +01:00
bin liu
b889cc2eb4 Add username to create redis client
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-29 16:27:43 +08:00
Wang Yan
5f8b59177b
Merge pull request #4019 from milosgajdos/replace-redigo-redis
Replace redigo with redis-go
2023-08-29 09:53:32 +08:00
Milos Gajdos
884cf14d30
Merge pull request #4013 from milosgajdos/nonjson-error-client
Dont parse errors as JSON unless Content-Type is set to JSON
2023-08-28 14:28:48 +01:00
Milos Gajdos
45b7b9cec3
Dont parse errors as JSON unless Content-Type is set to JSON
Client attempts to parse the body of every error it receives as JSON
regardless of the content-type. This commit rectifies by only parsing
he error body as JSON if the Content-Type header is set to
either "application/json" or "application/vnd.api+json".

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-28 12:35:33 +01:00
David van der Spek
c7bdabadcf
add back getKeys + cleanup manifeststore test
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-28 12:44:49 +02:00
David van der Spek
f9bc9220eb
feat(storage)!: remove schema1 except manifeststore_test
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-28 12:44:46 +02:00
James Hewitt
1a3e73cb84
Handle rand deprecations in go 1.20
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-28 09:33:12 +01:00
Milos Gajdos
fcbc25e789
Replace redigo with redis-go
We are replacing the very outdated redigo Go module with the official
redis Go module, go-redis.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-26 07:44:02 +01:00
Milos Gajdos
4f7424c8eb
Merge pull request #4000 from DavidSpek/remove-bugsnag
Remove bugsnag
2023-08-23 15:21:18 +01:00
Milos Gajdos
4a85aab30f
Merge pull request #4012 from liubin/fix-typo
fix typo in comment and log
2023-08-23 15:20:05 +01:00
bin liu
1284c48781 fix typo in comment and log
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-23 19:39:19 +08:00
Milos Gajdos
59dd684cc8
Merge pull request #3713 from Jamstah/s3-tests 2023-08-21 13:48:43 +01:00
Milos Gajdos
1a4638a06c
Merge pull request #4007 from milosgajdos/remove-schema1-client 2023-08-21 13:46:29 +01:00
Sebastiaan van Stijn
5b3be39870
s3: add interface assertion
This was added for the other drivers in 6b388b1ba6,
but it missed the s3 storage driver.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 13:54:13 +02:00
Milos Gajdos
df2787c6cf
Update test asserts
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-21 10:27:46 +01:00
David van der Spek
3e4c4ead4c
Remove bugsnag
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-21 10:24:36 +02:00
Milos Gajdos
11c341a369
Remove schema1 references from registry client
We've replaced all the schema1 references with OCI schema manifest.
Note, there are some TODO items that must be addressed at some point in
the future once the schema1 package is removed completely from the
codebase.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-20 09:07:59 +01:00
Milos Gajdos
3dbfbc7255
Enable bodyclose linter
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-19 09:45:44 +01:00
Milos Gajdos
597e8a6b34
Merge pull request #4001 from DavidSpek/remove-newrelic
Remove NewRelic
2023-08-18 22:35:16 +01:00
James Hewitt
37a213dc4b
Revert "optimize catalog last param"
This reverts commit 65f4ce4d93.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:37 +01:00
James Hewitt
8fd504debe
Revert "Rename catalog funcs and update their godocs."
This reverts commit 230cc72a8b.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:28 +01:00
James Hewitt
ad11105052
Revert "removed redundant check"
This reverts commit 0f846853fe.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:19 +01:00
James Hewitt
8e4a8517c5
Revert "fix: resolve most comments"
This reverts commit 6a5846b32e.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:06 +01:00
David van der Spek
77c33cd243
remove NewRelic
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-18 12:16:18 +02:00
Milos Gajdos
bf3c2df6b2
Merge pull request #3902 from pluralsh/catalog-opti-fix-rebase
Optimise catalog function rebase of #3145
2023-08-18 08:11:29 +01:00
Milos Gajdos
ac32466188
Merge pull request #3993 from DavidSpek/update-golang-lru
Update github.com/hashicorp/golang-lru to v2
2023-08-18 08:09:11 +01:00
Milos Gajdos
40c56bf1b6
Keep returning image for default arch to old clients fetching lists
This puts back the original flow where old clients are fetching manifest
lists schema1 images where we want to try returning some image for the
default architecture. This was incorrectly removed by one of the
previous commits.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-17 20:57:02 +01:00
David van der Spek
0f006548a1
update golang-lru to v2
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-17 13:41:54 +02:00
Milos Gajdos
f517191da1
Add small update to api tests
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-17 09:37:26 +01:00
Milos Gajdos
7e39a7c6dc
Remove references to schema1 pacakge from handlers
schema1 package was deprecated a while ago so we are removing
any references to it from handlers. in preparation to
removing it from the codebase altogether.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-17 08:35:54 +01:00
Milos Gajdos
3f1859af26
Remove oss storage driver and alicdn storage driver middleware
This commit removes `oss` storage driver from distribution as well as
`alicdn` storage middleware which only works with the `oss` driver.

There are several reasons for it:
* no real-life expertise among the maintainers
* oss is compatible with S3 API operations required by S3 storage driver

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-16 08:39:20 +01:00
Milos Gajdos
65b57464f9
Merge pull request #3982 from milosgajdos/remove-swift-storage-driver
Remove SWIFT storage driver
2023-08-16 07:47:42 +01:00
Milos Gajdos
fff194fc46
Merge pull request #3983 from Jamstah/azure-tests
Fix Azure tests
2023-08-15 18:31:05 +01:00
James Hewitt
46ff5f8528
Fix Azure tests
The Azure tests fail if there is no Azure configuration available,
instead they should be skipped.

Also, one of the Azure tests is wrong and doesn't match the code.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:46:36 +01:00
James Hewitt
7622d0a453
Don't return the from of a walk
Other storage drivers will only return children and below, s3 should do
the same. The only reason it was returning was because of the addition
of a / to ensure we treat the from as a directory.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:26:37 +01:00
James Hewitt
f7bdd9127b
Don't test the OUTPOSTS storage class
This test will only work on an s3 bucket on an s3 outpost. Most
developers won't have access to one of these.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:52 +01:00
James Hewitt
6ceb904c3e
Don't check returned storage class if we use NONE
If we haven't set a storage class there's no point in checking the
storage class applied to the object - s3 will choose one.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
2d316a12d3
We don't use gocheck in these tests
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
f78d81e78a
Remove test as S3 does not support empty directories
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:48 +01:00
Milos Gajdos
c6b9944ab1
Remove SWIFT storage driver
This commit removes swift storage driver from distribution.
There are several reasons for it:
* no real life expertise among the maintainers
* swift is compatible with S3 API operations required by S3 storage driver

This will also remove depedencies that are also hard to keep up with.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-15 09:14:11 +01:00
Milos Gajdos
24de708d22
Set Content-Type header in registry client ReadFrom
Client ReadFrom doesn't set Content-Type header leading to server
side implementor to assume it's application/octet-stream. This commit
makes this explicit on the client side.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-15 08:46:48 +01:00
Milos Gajdos
ff46bf1c41
Merge pull request #2897 from SuperQ/cache_metrics
Cleanup storage cache metrics
2023-08-15 07:44:14 +01:00
Hayley Swimelar
f1529a7782
Merge pull request #3977 from milosgajdos/remove-schema1-proxystore
Remove references to schema1 pacakge from proxy package
2023-08-14 16:24:45 -07:00
Milos Gajdos
02a92efba8
Drop docker prefix from storage driver API user agent
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-13 22:27:55 +01:00
Milos Gajdos
ae0001f54d
Remove references to schema1 pacakge from proxy package
schema1 package was deprecated a while ago so we are removing
any references to it from the proxy package in preparation to
removing it from the codebase altogether.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-13 14:57:55 +01:00
David van der Spek
6a5846b32e
fix: resolve most comments
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-10 10:47:39 +02:00
eyjhb
0f846853fe
removed redundant check
Signed-off-by: eyjhb <eyjhbb@gmail.com
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-10 10:47:37 +02:00
Milos Gajdos
230cc72a8b
Rename catalog funcs and update their godocs.
Signed-off-by: Milos Gajdos <milosgajdos83@gmail.com>
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-10 10:47:34 +02:00
eyjhbb@gmail.com
65f4ce4d93
optimize catalog last param
Signed-off-by: eyjhb <eyjhbb@gmail.com>
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-10 10:46:40 +02:00
chlins
32a476b840 feat: added support for redis username configuration
Redis introduced an Access Control List (ACL) mechanism since version 6.0. This commit implements the necessary changes to support configuring the username for Redis. Users can now define a specific username to authenticate with Redis and enhance security through the ACL feature.

Signed-off-by: chlins <chenyuzh@vmware.com>
2023-08-04 15:06:12 +08:00
Ben Kochie
c19adfdf06
Cleanup storage cache metrics
Split request and hit metrics into separate metrics, rather than using
labels. This avoids duplication of data and makes metric math easier.

* Count cache errors separately to avoid weird math.
* Hit ratio: `registry_storage_cache_hits_total / registry_storage_cache_requests_total`
* Miss ratio: `1 - (registry_storage_cache_hits_total / registry_storage_cache_requests_total`
* Misses: `registry_storage_cache_requests_total -
registry_storage_cache_hits_total`

Signed-off-by: Ben Kochie <superq@gmail.com>
2023-08-04 09:03:06 +02:00
Wang Yan
46b3d62016
Merge pull request #3869 from brackendawson/split-oci-index
Split OCI Image Index from Docker Manifest List
2023-07-19 12:02:15 +08:00
Milos Gajdos
003dd5aaa1
Merge pull request #3942 from lavalleeale-forks/main
Added support for specifying ACME-server by using REGISTRY_HTTP_TLS_LETSENCRYPT_DIRECTORYURL
2023-07-14 16:28:34 +01:00
Milos Gajdos
d5c1b39b8b
Merge pull request #3206 from takmatsu/suppurt-path-in-middleware
Make redirect middleware can use path
2023-07-14 10:50:29 +01:00
Milos Gajdos
316e1c6b82
Get rid of unnecessary import alias
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-07-14 10:37:42 +01:00
MATSUMOTO TAKEAKI
a3eb956464
use path.Join() for building path
Signed-off-by: MATSUMOTO TAKEAKI <takeaki.matsumoto@linecorp.com>
2023-07-14 10:37:21 +01:00
MATSUMOTO TAKEAKI
a1cfd267c8
Make redirect middleware can use path
Signed-off-by: MATSUMOTO TAKEAKI <takeaki.matsumoto@linecorp.com>
2023-07-14 10:36:23 +01:00
Milos Gajdos
69023c7f85
Merge pull request #3880 from wzshiming/proxy-cache-configurable 2023-07-14 08:43:19 +01:00
Milos Gajdos
6b388b1ba6
Enable Go build tags
This enables go build tags so the GCS and OSS driver support is
available in the binary distributed via the image build by Dockerfile.

This led to quite a few fixes in the GCS and OSS packages raised as
warning by golang-ci linter.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-06-28 11:41:22 +01:00
Hayley Swimelar
71a6c56fbf
Merge pull request #3948 from flavianmissi/log-healthcheck-error
registry/handlers/app: log healthcheck error before return
2023-06-27 14:06:31 -07:00
Flavian Missi
87081252ba registry/handlers/app: log healthcheck error before return
Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-06-27 09:50:15 +02:00
Milos Gajdos
22725209e3
Merge pull request #3936 from flavianmissi/azure-path-not-found
Fix path not found error in Azure
2023-06-26 13:21:00 +01:00
Flavian Missi
2b72c4d1ca registry/storage/driver/azure: fix Move method
Something seems broken on azure/azure sdk side - it is currently not
possible to copy a blob of type AppendBlob using `CopyFromURL`.
Using the AppendBlob client via NewAppendBlobClient does not work
either.

According to Azure the correct way to do this is by using
StartCopyFromURL. Because this is an async operation, we need to do
polling ourselves. A simple backoff mechanism is used, where during each
iteration, the configured delay is multiplied by the retry number.

Also introduces two new config options for the Azure driver:
copy_status_poll_max_retry, and copy_status_poll_delay.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-06-26 13:47:30 +02:00
Hayley Swimelar
87b280718d
Merge pull request #2752 from davidswu/registry-driver-middleware
Add content cache required changes to distribution
2023-06-21 10:06:13 -07:00
David Wu
db1d0cbf35 Add registry middleware access to storage drivers
Signed-off-by: David Wu <dwu7401@gmail.com>
2023-06-20 17:18:05 -07:00
Mike Truman
4bbe0ba080
Added support for specifying ACME-server by using REGISTRY_HTTP_TLS_LETSENCRYPT_DIRECTORYURL
Signed-off-by: Alex Lavallee <73203142+lavalleeale@users.noreply.github.com>
2023-06-17 21:52:34 -07:00
Wang Yan
6a57630cf4
Merge pull request #3934 from stonezdj/add_caller_info
Add option to enable caller information in logger
2023-06-11 21:53:14 +08:00
Irene Diez
421a97ffab registry/api/v2: fix ContentType in RouteNameBlobUpload
Sets 'application/octet-stream' as the ContentType in the
Body of RouteNameBlobUpload.

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-06-08 11:05:34 +02:00
stonezdj
2338ee4f25 Add option to enable caller information in logger
Signed-off-by: stonezdj <stonezdj@gmail.com>
2023-06-06 13:14:02 +08:00
Laura Brehm
afe5a2a9b7 Support ztsd compression as Content-Encoding
Co-authored-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Co-authored-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
2023-06-01 14:06:25 +01:00
Bracken Dawson
9d1a8fc929
Remove duplicated platform field from oci index
It is desirable to remove Platform from distribution.Descriptor because it doesn't really belong there. However this would be a further breaking change because the References() call would no longer be returning plaform information when it reurns descriptors of manifests, which is started to for OCI Indices after c94f288 and this feature was added to Docker Manifest Lists in 1a059fe. I don't want to take away something people clearly want.

Signed-off-by: Bracken Dawson <abdawson@gmail.com>
2023-06-01 11:38:36 +01:00
Milos Gajdos
5902a24fd0
Merge pull request #3929 from flavianmissi/fix-gcs-oss-file-writer
Fix gcs storage driver

Thanks to @jmontleon who laid the first bricks in #3702
2023-06-01 09:56:57 +01:00
Flavian Missi
0207adaa5c registry/storage/driver/gcs: fix code to use updated gcs driver
Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-31 09:28:03 +02:00
Flavian Missi
d0bc83d8e4 registry/storage/driver: receive context on Cancel methods
both oss and gcs driver were missing the context parameter that is
required to satisfy the storagedriver.FileWriter interface.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-31 09:14:51 +02:00
Milos Gajdos
ba46c769b3
Merge pull request #3932 from flavianmissi/fix-azure-test-parameters
registry/storage/driver: test call to Stat(ctx, "/")
2023-05-30 21:49:32 +01:00
Flavian Missi
0d20e7ae9e registry/storage/driver/testsuites: use 4MB for Azure append test
Fixes #3931.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:08 +02:00
Flavian Missi
0c33bb1092 registry/storage/driver/azure: consider CannotVerifyCopySource as 404
Azure will return CannotVerifyCopySource with a 404 status code from a
call to Move when the source blob does not exist.
Details: https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-error-codes.

This fixes the TestMoveNonexistent test case for the Azure driver.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:08 +02:00
Flavian Missi
d2e16fc74a registry/storage/driver/azure: fix driver parameters on tests
fixes "azure: no Host in request URL" tests failure, allowing azure
driver tests to run.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:08 +02:00
Flavian Missi
90ece48d77 registry/storage/driver: add test call to Stat on "/"
Stat(ctx, "/") is called by the registry healthcheck.
Also fixes blob name building in the Azure driver so it no longer
returns empty blob names. This was causing errors in the healthcheck
call to Stat for Azure.

Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-05-30 15:12:00 +02:00
Lucas França de Oliveira
035a8ec52a
Fix panic in the s3 backend walk logic
Signed-off-by: Lucas França de Oliveira <lucasfdo@palantir.com>
2023-05-25 14:56:05 -07:00
Milos Gajdos
f55a6552b0
Merge pull request from GHSA-hqxw-f8mx-cpmw
Fix runaway allocation on /v2/_catalog
2023-05-09 21:21:54 +01:00
Sebastiaan van Stijn
ebe9d67446
ignore SA1019: ac.(*accessController).rootCerts.Subjects has been deprecated
We need to look into this; can we remove it, or is there a replacement?

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:17 +02:00
Sebastiaan van Stijn
999527f978
Ignore SA1019: "schema1 is deprecated" linting errors
We need to use this for backward compatibility.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:17 +02:00
Sebastiaan van Stijn
3c144f2264
registry/auth/token: fix the surrounding loop is unconditionally terminate
registry/auth/token/types_test.go:83:3: SA4004: the surrounding loop is unconditionally terminated (staticcheck)
                   return
                   ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:17 +02:00
Sebastiaan van Stijn
4052d269f5
reference, registry: fix loop variable captured by func literal (govet)
reference/normalize_test.go:274:40: loopclosure: loop variable r captured by func literal (govet)
                                   named, err := ParseNormalizedNamed(r)
                                                                      ^
    reference/normalize_test.go:276:29: loopclosure: loop variable r captured by func literal (govet)
                        t.Fatalf("ref=%s: %v", r, err)
                                               ^
    registry/api/errcode/errors_test.go:45:7: loopclosure: loop variable ec captured by func literal (govet)
                if ec != desc.Code {
                   ^
    registry/api/errcode/errors_test.go:46:66: loopclosure: loop variable ec captured by func literal (govet)
                    t.Fatalf("error code in descriptor isn't correct, %q != %q", ec, desc.Code)
                                                                                 ^
    registry/api/errcode/errors_test.go:49:23: loopclosure: loop variable desc captured by func literal (govet)
                if idToDescriptors[desc.Value].Code != ec {
                                   ^
    registry/api/errcode/errors_test.go:50:80: loopclosure: loop variable desc captured by func literal (govet)
                    t.Fatalf("error code in idToDesc isn't correct, %q != %q", idToDescriptors[desc.Value].Code, ec)
                                                                                               ^
    registry/api/errcode/errors_test.go:53:7: loopclosure: loop variable ec captured by func literal (govet)
                if ec.Message() != desc.Message {
                   ^

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 16:04:08 +02:00
Sebastiaan van Stijn
f03d966ef7
cloudfront: use consistent names for test-tables, t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:39:58 +02:00
Sebastiaan van Stijn
5301ae14bf
cloudfront: rename vars that collided with type
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:17 +02:00
Sebastiaan van Stijn
85028a801b
registry/handlers: use consistent names for test-tables
Also marked assertBlobUploadStateEquals as t.Helper()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:17 +02:00
Sebastiaan van Stijn
f4acd98865
registry/api/v2: checkTestRouter(): use sub-tests, t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 14:00:05 +02:00
Sebastiaan van Stijn
2444c3282d
registry/api/v2: use consistent names for test-tables
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:16 +02:00
Sebastiaan van Stijn
9266220c2a
registry/auth/token: TestAudienceList_Unmarshal: t.Parallel()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:16 +02:00
Sebastiaan van Stijn
5e67a40e08
registry/auth/token: use consistent names for test-tables
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:16 +02:00
Sebastiaan van Stijn
f884a079df
registry/api/errorcode: TestErrorCodes: use sub-tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-09 13:55:11 +02:00
Wang Yan
8900e90699
Merge pull request #3895 from pluralsh/bug-fix-5001-server-rebase
Fix the issue that the debug server with port 5001 run twice
2023-05-05 13:21:55 +08:00
Milos Gajdos
08f7a47bc9
Merge pull request #2766 from glefloch/remove-testdriver
Remove registry storage testdriver
2023-05-03 21:40:53 +01:00
David van der Spek
99a8ad00ea fix: rename log to logrus
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-05-03 19:02:03 +02:00
Feng Honglin
f0fdaff0a5 Fix the issue that the debug server with port 5001 run twice
When configuring bugsnag, bugsnag will fork the process, resulting the port 5001 listened twice. The PR fix this error by moving the initialization of prometheus server after the configuration of bugsnag

Signed-off-by: Honglin Feng <tifayuki@gmail.com>
(cherry picked from commit 5a6a2d6ae06453136f5e1cfb5e9efa20c27085d9)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-05-03 18:37:32 +02:00
AdamKorcz
e2a43ec8d3
Fuzzing: Move over two fuzzers from cncf-fuzzing
Signed-off-by: AdamKorcz <adam@adalogics.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-02 23:03:57 +02:00
glefloch
9fb201e23d Remove registry storage testdriver
Signed-off-by: glefloch <glfloch@gmail.com>
2023-05-02 16:37:33 +02:00
Milos Gajdos
788598e23f
Merge pull request #3881 from vvoland/handle-forbidden
registry/errors: Parse http forbidden as denied
2023-05-02 09:38:36 +01:00
Sebastiaan van Stijn
3fa6d5a33b
remove dot-imports for gopkg.in/check.v1
Dot-imports were only used in a couple of places, and replacing them
makes it more explicit what's imported.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-04-29 17:08:47 +02:00
Paweł Gronowski
5f1df02149
registry/errors: Parse http forbidden as denied
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
2023-04-27 19:48:32 +02:00
Milos Gajdos
29b5e79f82
Merge pull request #3742 from sagikazarmark/fix-aud-claim-list
Accept list of strings in audience claim in token auth
2023-04-26 18:39:26 +01:00
Milos Gajdos
2fb8dbdeca
Merge pull request #3839 from kirat-singh/feature.azure-sdk-update
Update Azure SDK and support additional authentication schemes
2023-04-25 19:35:34 +01:00
Kirat Singh
ba4a6bbe02 Update Azure SDK and support additional authentication schemes
Microsoft has updated the golang Azure SDK significantly.  Update the
azure storage driver to use the new SDK.  Add support for client
secret and MSI authentication schemes in addition to shared key
authentication.

Implement rootDirectory support for the azure storage driver to mirror
the S3 driver.

Signed-off-by: Kirat Singh <kirat.singh@beacon.io>

Co-authored-by: Cory Snider <corhere@gmail.com>
2023-04-25 17:23:20 +00:00
Bracken Dawson
973bfbb676
Fix Go Idioms
- DRY out SchemaVersion literals
- Better name the predefined Versioned struct for the Image Index
- Var names, declarations, else cases.

Co-authored-by: Milos Gajdos <milosthegajdos@gmail.com>
Signed-off-by: Bracken Dawson <abdawson@gmail.com>
2023-04-21 15:22:37 +01:00
Manish Tomar
8fe4ca4038 Option to configure proxy cache TTL
Currently when registry is run as proxy it tries to cleanup unused blobs
from its cache after 7 days which is hard-coded. This PR makes that
value configurable.

Co-authored-by: Shiming Zhang <wzshiming@foxmail.com>
Co-authored-by: Manish Tomar <manish.tomar@docker.com>
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
2023-04-20 13:03:39 +08:00
Bracken Dawson
88646f54da
Support annotations in the OCI Image Index
Empty platform structs were already supported after splitting OCI Image
Index out from Docker Manifest List.

Signed-off-by: Bracken Dawson <abdawson@gmail.com>
2023-03-31 14:01:30 +01:00
Jose D. Gomez R
4c1561e9fb
Fix runaway allocation on /v2/_catalog
Introduced a Catalog entry in the configuration struct. With it,
it's possible to control the maximum amount of entries returned
by /v2/catalog (`GetCatalog` in registry/handlers/catalog.go).

It's set to a default value of 1000.

`GetCatalog` returns 100 entries by default if no `n` is
provided. When provided it will be validated to be between `0`
and `MaxEntries` defined in Configuration. When `n` is outside
the aforementioned boundary, an error response is returned.

`GetCatalog` now handles `n=0` gracefully with an empty response
as well.

Signed-off-by: José D. Gómez R. <1josegomezr@gmail.com>
2023-03-31 13:17:43 +02:00
Bracken Dawson
e72294d075
Split OCI Image Index from Docker Manifest List
Move implementation of the index from the manifestlist package to the ocischema package so that other modules making empty imports support the manifest types their authors would expect. This is a breaking change to distribution as a library but not the registry.

As OCI 1.0 released the manifest and index together, that is a good package from which to initialise both manifests. The docker manifest and manifest list remain in separate packages because one was released later.

The image index and manifest list still share common code in many functions not intended for import by other modules.

Signed-off-by: Bracken Dawson <abdawson@gmail.com>
2023-03-31 11:43:21 +01:00
Milos Gajdos
0c958010ac
Merge pull request #3763 from distribution/multipart-upload-empty-files
Enable pushing empty blobs
2023-03-27 10:18:44 +01:00
Milos Gajdos
5fa926a609
Enable pushing empty blobs
This is an edge case when we are trying to upload an empty chunk of data using
a MultiPart upload. As a result we are trying to complete the MultipartUpload
with an empty slice of `completedUploadedParts` which will always lead to 400
being returned from S3 See: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#CompletedMultipartUpload
Solution: we upload an empty i.e. 0 byte part as a single part and then append it
to the completedUploadedParts slice used to complete the Multipart upload.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-03-27 10:11:07 +01:00
Aaron Lehmann
2074688be9 Fix S3 multipart upload pagination loop condition
The loop that iterates over paginated lists of S3 multipart upload parts
appears to be using the wrong variable in its loop condition. Nothing
inside the loop affects the value of `resp.IsTruncated`, so this loop
will either be wrongly skipped or loop forever.

It looks like this is a regression caused by commit
7736319f2e. The return value of
`ListMultipartUploads` used to be assigned to a variable named `resp`,
but it was renamed to `partsList` without updating the for loop
condition.

I believe this is causing an error we're seeing with large layer uploads
at commit time:

    upload resumed at wrong offset: 5242880000 != 5815706782

Missing parts of the multipart S3 upload would cause an incorrect size
calculation in `newWriter`.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-02-21 20:57:50 -08:00
Milos Gajdos
772cef6b4e
Merge pull request #3736 from aaronlehmann/log-username-on-successful-requests
Log username on successful requests
2023-02-19 19:05:35 +00:00
Kirat Singh
3117e2eb2f
Use default http.Transport for AWS S3 session
Previously we used a custom Transport in order to modify the user agent header.
This prevented the AWS SDK from being able to customize SSL and other client TLS
parameters since it could not understand the Transport type.

Instead we can simply use the SDK function MakeAddToUserAgentFreeFormHandler to
customize the UserAgent if necessary and leave all the TLS configuration to the
AWS SDK.

The only exception being SkipVerify which we have to handle, but we can set it
onto the standard http.Transport which does not interfere with the SDKs ability
to set other options.

Signed-off-by: Kirat Singh <kirat.singh@gmail.com>
2023-02-15 13:37:01 -05:00
Aaron Lehmann
a811c1bb57 Log username on successful requests
Currently, "response completed with error" log lines include an
`auth.user.name` key, but successful "response completed" lines do not
include this, because they are logged a few stack frames up where
`auth.user.name` is not present on the `Context`. Move the successful
request logging inside the `dispatcher` closure, where the logger on the
context automatically includes this key.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2023-01-30 09:32:39 -08:00
Milos Gajdos
ac302d9ce5
Merge pull request #3807 from thaJeztah/replace_types_for_oci_step1
minor fixes and enhancements
2022-11-29 10:49:12 +00:00
Milos Gajdos
8cc58797e8
Merge pull request #3794 from AdamKorcz/fuzz1
Fuzzing: Rewrite existing fuzzers to native go fuzzers
2022-11-29 09:57:09 +00:00
Sebastiaan van Stijn
f2db7faa2f
registry/storage: rename variables that collided with imports
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-28 10:08:05 +01:00
Milos Gajdos
21622ca699
Merge pull request #3787 from thaJeztah/simplify_mocks 2022-11-18 08:53:26 +00:00
Milos Gajdos
5357c45703
Merge pull request #3788 from thaJeztah/deprecate_ReadSeekCloser 2022-11-18 08:53:15 +00:00
AdamKorcz
9337b8df66 Fuzzing: Rewrite existing fuzzers to native go fuzzers
Signed-off-by: AdamKorcz <adam@adalogics.com>
2022-11-12 17:30:10 +00:00
Milos Gajdos
3b8fbf9752
Merge pull request #3686 from m5i-work/exp 2022-11-11 17:07:14 +00:00
Wei Meng
35cae1099e Realloc slice exponentially in mfs
`registry/storage/driver/inmemory/driver_test.go` times out after ~10min. The slow test is `testsuites.go:TestWriteReadLargeStreams()` which writes a 5GB file.
Root cause is inefficient slice reallocation algorithm. The slice holding file bytes grows only 32K on each allocation. To fix it, this PR grows slice exponentially.

Signed-off-by: Wei Meng <wemeng@microsoft.com>
2022-11-11 18:18:08 +08:00
Sebastiaan van Stijn
1d8cd5e443
registry/client: use struct literals
Remove some intermediate variables, and use struct literals instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:11:12 +01:00