Commit graph

1311 commits

Author SHA1 Message Date
Cory Snider
9157226e7b Extract request utilities into its own package
The RemoteAddr and RemoteIP functions operate on *http.Request values,
not contexts. They have very low cohesion with the rest of the package.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-27 10:58:37 -04:00
Cory Snider
d0f5aa670b Move context package internal
Our context package predates the establishment of current best practices
regarding context usage and it shows. It encourages bad practices such
as using contexts to propagate non-request-scoped values like the
application version and using string-typed keys for context values. Move
the package internal to remove it from the API surface of
distribution/v3@v3.0.0 so we are free to iterate on it without being
constrained by compatibility.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-27 10:58:37 -04:00
Milos Gajdos
daf3d00a32
Add prometheus proxy related metrics (#4047) 2023-10-26 16:11:10 +01:00
James Hewitt
ef8651ec2a
Switch to github.com/google/uuid
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-25 12:15:21 +01:00
Cory Snider
8a86dc61ff Make our UUID package internal
Nowadays there are much, much better UUID implementations to choose
from, such as github.com/google/uuid. Prevent external users from
importing our bespoke implementation so that we can change or migrate
away from it internally without introducing breaking changes.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-24 18:11:41 -04:00
Cory Snider
cc23fdacff Move registry client internal
Our registry client is not currently in a good place to be used as the
reference OCI Distribution client implementation. But the registry proxy
currently depends on it. Make the registry client internal to the
distribution application to remove it from the API surface area (and any
implied compatibility promises) of distribution/v3@v3.0.0 without
breaking the proxy.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-24 12:26:38 -04:00
Dimitar Kostadinov
2ce5c81f47 Address PR review feedback
Signed-off-by: Dimitar Kostadinov <dimitar.kostadinov@sap.com>
2023-10-23 19:59:22 +03:00
Dimitar Kostadinov
9861a46d99 Add prometheus proxy related metrics
Signed-off-by: Dimitar Kostadinov <dimitar.kostadinov@sap.com>
2023-10-23 19:59:22 +03:00
Milos Gajdos
708bc6f3e9
Make S3 tests pass with MinIO (#4107) 2023-10-20 16:20:30 +01:00
Milos Gajdos
dfd191e7d2
Replace docker/libtrust with go-jose/go-jose (#4096) 2023-10-19 15:52:09 +01:00
Milos Gajdos
fe21f43911
feat: replace docker/libtrust with go-jose/go-jose
docker/libtrust repository has been archived for several years now.
This commit replaces all the libtrust JWT machinery with go-jose/go-jose module.
Some of the code has been adopted from libtrust and adjusted for some of
the use cases covered by the token authorization flow especially in the
tests.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-19 15:32:59 +01:00
Milos Gajdos
5aee8e1917
feat: Add context to storagedriver.(Filewriter).Commit() (#4109) 2023-10-19 11:41:55 +01:00
Milos Gajdos
cb0d083d8d
feat: Add context to storagedriver.(Filewriter).Commit()
This commit changes storagedriver.Filewriter interface
by adding context.Context as an argument to its Commit
func.

We pass the context appropriately where need be throughout
the distribution codebase to all the writers and tests.

S3 driver writer unfortunately must maintain the context
passed down to it from upstream so it contnues to
implement io.Writer and io.Closer interfaces which do not
allow accepting the context in any of their funcs.

Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-19 11:27:27 +01:00
Milos Gajdos
ea41722902
refactor: Storage driver errors
Small refactoring of storagedriver errors.
We change the Enclosed field to Detail and make sure
Errors get properly serialized to JSON.
We also add tests.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-18 10:02:21 +01:00
James Hewitt
eac199875e
Remove test for nested file delete on S3
Nested files aren't supported on MinIO, and as our storage layout is
filesystem based, we don't actually use nest files in the code.

Remove the test so that we can support MinIO.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-17 09:13:15 +01:00
James Hewitt
647ec33c33
Bump minio version and test less storage classes
This fixes some of the tests for minio.

The walk tests needs a version of minio that contains https://github.com/minio/minio/pull/18099

The storage classes minio supports are a subset of the s3 classes.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-17 02:10:43 +01:00
Glyn Owen Hanmer
fee6faef70 json encode storage driver enclosed error
Signed-off-by: Glyn Owen Hanmer <1295698+glynternet@users.noreply.github.com>
2023-10-11 17:53:27 -06:00
Milos Gajdos
a70964c2fc
Merge pull request #4076 from flavianmissi/s3-loglevel
registry: add loglevel support for aws s3 storage driver
2023-10-04 14:13:15 +01:00
Milos Gajdos
ed8423176f
Merge pull request #4081 from liubin/fix/refactor-redis
refactor redis cache
2023-10-03 16:01:07 +01:00
Flavian Missi
3df7e28f44 registry: add loglevel support for aws s3 storage driver
based on the work from
https://github.com/distribution/distribution/pull/3057.

Co-authored-by: Simon Compston <compston@gmail.com>
Signed-off-by: Flavian Missi <fmissi@redhat.com>
2023-10-02 15:47:02 +02:00
Milos Gajdos
735c161b53
Merge pull request #4066 from milosgajdos/optimise-s3-push
Optimise push in S3 driver
2023-09-29 13:47:20 +01:00
Milos Gajdos
23083ac9d2
Merge pull request #4077 from liubin/fix/use-manifestTagsPathSpec-all-tag-all
use manifestTagsPathSpec for listing all tags
2023-09-28 17:37:25 +01:00
Milos Gajdos
4fce3c0028
Move completedParts type back to the original position
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-28 15:58:02 +01:00
bin liu
46a9da160e refactor redis cache
This commit removes some `conn` parameters of private functions, which can
be obtain from the struct itself. The `conn` is for the old `redisgo` library,
which is replaced by `go-redis` in #4019.

Signed-off-by: bin liu <liubin0329@gmail.com>
2023-09-28 18:40:50 +08:00
bin liu
dca71db976 fix comment typos
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-09-28 17:48:21 +08:00
bin liu
6c724a1a95 use manifestTagsPathSpec for listing all tags
In terms of results, a`manifestTagsPathSpec{ name: "repo" }` equals
`manifestTagPathSpec{ name: "repo", tag: "" }`, but from the intention,
the `manifestTagsPathSpec` should be used.

Signed-off-by: bin liu <liubin0329@gmail.com>
2023-09-28 10:44:46 +08:00
Milos Gajdos
b888b14b39
Optimise push in S3 driver
This commit cleans up and attempts to optimise the performance of image push in S3 driver.
There are 2 main changes:
* we refactor the S3 driver Writer where instead of using separate bytes
  slices for ready and pending parts which get constantly appended data
  into them causing unnecessary allocations we use optimised bytes
  buffers; we make sure these are used efficiently when written to.
* we introduce a memory pool that is used for allocating the byte
  buffers introduced above

These changes should alleviate high memory pressure on the push path to S3.

Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-27 21:33:22 +01:00
Neil Wilson
71c532e60c
driver testsuite: Add zero byte file checks
Add two new checks to the testsuite that check
the driver can handle zero byte files and appends to zero
byte files correctly

Signed-off-by: Neil Wilson <neil@aldur.co.uk>
2023-09-26 10:48:46 +01:00
Milos Gajdos
f7e792417a
Merge pull request #4070 from liubin/add-repositoriesRootPathSpec
add repositoriesRootPathSpec in pathFor documentation
2023-09-26 08:47:19 +01:00
bin liu
06acf2def5 remove go build directive for older go version
Go 1.4 is too old and should be dropped safely.

Signed-off-by: bin liu <liubin0329@gmail.com>
2023-09-26 15:14:57 +08:00
bin liu
a0d9279e8f add repositoriesRootPathSpec in pathFor documentation
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-09-26 15:07:49 +08:00
Milos Gajdos
f4d5210b25
Do not close HTTP request body in HTTP handler
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-22 16:21:54 +01:00
bin liu
34654f6c4a remove not exist function name in comment
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-09-21 16:53:59 +08:00
Geoffrey Hausheer
a9399e9ea2 Improve socket-activation message
Co-authored-by: James Hewitt <james.hewitt@gmail.com>
Signed-off-by: Geoffrey Hausheer <rc2012@pblue.org>
2023-09-20 10:31:02 -07:00
Geoffrey Hausheer
9721db9504 Add info message regarding socket-activation
Signed-off-by: Geoffrey Hausheer <rc2012@pblue.org>
2023-09-20 09:56:25 -07:00
Geoffrey Hausheer
2435def474 Support systemd socket-activation
Signed-off-by: Geoffrey Hausheer <rc2012@pblue.org>
2023-09-20 09:37:22 -07:00
Milos Gajdos
42ce5d4d51
Merge pull request #3569 from justadogistaken/optimize/avoid-redundant-blob-fetching
optimize: avoid redundant blob fetching
2023-09-18 08:01:14 +01:00
baojiangnan
17952924f3 avoid redundant blob fetching
Signed-off-by: baojiangnan <baojn1998@163.com>
2023-09-18 10:40:25 +08:00
Milos Gajdos
612ad42609
Merge pull request #4040 from thaJeztah/move_api_errors 2023-09-15 09:36:36 +01:00
Sebastiaan van Stijn
c8ba5d7081
registry/client: combine SuccessStatus and HandleErrorResponse
The SuccessStatus acted on the response's status code, and was used to return
early, before checking the same status code with HandleErrorResponse.

This patch combines both functions into a HandleHTTPResponseError, which
returns an error for "non-success" status-codes, which simplifies handling
of responses, and makes some logic slightly more idiomatic.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-08 14:40:41 +02:00
Milos Gajdos
612a30a7e7
Remove libtrust from handler tests
It was used for signing schema v1 manifests in tests which have now been
removed so there is no point in keeping these there anymore.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-05 08:41:29 +01:00
Milos Gajdos
6baa31a273
Remove duplicate code that instruments Redis OTLP
This was somehow overlooked in https://github.com/distribution/distribution/pull/4019

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-04 17:52:38 +01:00
Sebastiaan van Stijn
0104adf4a8
registry/api/errcode: split Register to internal / exported
Use the non-exported function to all errors; there's currently no external
consumers of this function (perhaps it should be deprecated).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-04 18:03:00 +02:00
Sebastiaan van Stijn
292e30bc61
registry/api: move all errors to "errcode" package
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-04 18:02:54 +02:00
Milos Gajdos
9790bc806c
Merge pull request #4037 from milosgajdos/enable-prealloc
Enable prealloc linter
2023-09-04 16:57:29 +01:00
Milos Gajdos
823cd4a370
Add a comment why prealloc linter is disabled when configuring endpoints
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:33:20 +01:00
Milos Gajdos
1089800643
Preallocate created slice in S3 tests
In case drvr.PutContent fails and returns error we'd have
some extra memory allocated, though in this case
(test with known size of the slice being iterated), that's fine.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:26:32 +01:00
Milos Gajdos
a9d31ec7b9
Avoid unnecessary type assertion in mfs driver
We already make sure the node in *dir

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 23:23:25 +01:00
Milos Gajdos
59fd8656ac
Enable prealloc linter
This will give us nice little performance gains in some code paths.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 22:41:51 +01:00
Milos Gajdos
dcdd8bb740
Propagate storage driver context to S3 API calls
Only some of the S3 storage driver calls were propagating context to the
S3 API calls. This commit updates the S3 storage drivers so the context
is propagated to all the S3 API calls.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-09-03 21:54:54 +01:00