Commit graph

5374 commits

Author SHA1 Message Date
Cory Snider
d0f5aa670b Move context package internal
Our context package predates the establishment of current best practices
regarding context usage and it shows. It encourages bad practices such
as using contexts to propagate non-request-scoped values like the
application version and using string-typed keys for context values. Move
the package internal to remove it from the API surface of
distribution/v3@v3.0.0 so we are free to iterate on it without being
constrained by compatibility.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-27 10:58:37 -04:00
Milos Gajdos
852de2c2bb
cleanup: make chunk sizes easier to understand and change writer append
This commit make the S3 driver chunk size constants more straightforward
to understand -- instead of remembering the bit shifts we make this more
explicit.

We are also updating append parameter to the `(writer).Write` to follow
the new convention we are trying to establish.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-27 10:57:54 +01:00
Milos Gajdos
e8e46b2195
cleanup: a small Azure driver cleanup
Just so we can make the code a bit more consistent with the rest of the
storage drivers.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-27 10:44:53 +01:00
Milos Gajdos
6c694cbcf6
ci: add cloud storage driver integration tests to CI (#4121) 2023-10-26 16:39:41 +01:00
Milos Gajdos
daf3d00a32
Add prometheus proxy related metrics (#4047) 2023-10-26 16:11:10 +01:00
Milos Gajdos
d8d14ca363
Switch to github.com/google/uuid (#4132) 2023-10-26 13:36:12 +01:00
Milos Gajdos
5064789f2a
docs: fix rendering issues for raw html and images (#4136) 2023-10-26 12:55:16 +01:00
David Karlsson
e5f16beae7 docs: remove unused images
Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
2023-10-26 13:04:56 +02:00
David Karlsson
57a6fa46b2 docs: fix broken image references
Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
2023-10-26 13:03:53 +02:00
David Karlsson
9d12b47bbb docs: allow unsafe HTML
Enable rendering of unsafe HTML in Markdown files

Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
2023-10-26 12:55:39 +02:00
Milos Gajdos
3612f5e488
Bump google.golang.org/grpc from 1.53.0 to 1.56.3 (#4133) 2023-10-26 08:59:26 +01:00
dependabot[bot]
32316367c8
Bump google.golang.org/grpc from 1.53.0 to 1.56.3
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.53.0 to 1.56.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.53.0...v1.56.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-25 22:26:53 +00:00
Milos Gajdos
da92c34ae9
ci: add cloud storage driver integration tests to CI
* include storage integration tests in the build matrix
* add a new CI job that runs E2E tests backed by S3 storage driver

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-25 17:17:50 +01:00
Milos Gajdos
66bedcf1a3
Delete reference package (#4127) 2023-10-25 13:20:13 +01:00
James Hewitt
ef8651ec2a
Switch to github.com/google/uuid
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-25 12:15:21 +01:00
Milos Gajdos
3831c8ccb4
feat: push distribution images to GHCR
This addition enables pushing distribution images into GHCR.
This is in addition to the Docker Hub push which remains in place.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-25 11:37:01 +01:00
Milos Gajdos
03a778c1f5
Make our UUID package internal (#4129) 2023-10-25 09:19:19 +01:00
Cory Snider
8a86dc61ff Make our UUID package internal
Nowadays there are much, much better UUID implementations to choose
from, such as github.com/google/uuid. Prevent external users from
importing our bespoke implementation so that we can change or migrate
away from it internally without introducing breaking changes.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-24 18:11:41 -04:00
Cory Snider
6999f230d1 Delete reference package
Users should import it from its new location
github.com/distribution/reference.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-24 12:54:43 -04:00
Milos Gajdos
f3ce7c46bd
Move registry client internal (#4126) 2023-10-24 17:44:34 +01:00
Cory Snider
cc23fdacff Move registry client internal
Our registry client is not currently in a good place to be used as the
reference OCI Distribution client implementation. But the registry proxy
currently depends on it. Make the registry client internal to the
distribution application to remove it from the API surface area (and any
implied compatibility promises) of distribution/v3@v3.0.0 without
breaking the proxy.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-24 12:26:38 -04:00
Dimitar Kostadinov
2ce5c81f47 Address PR review feedback
Signed-off-by: Dimitar Kostadinov <dimitar.kostadinov@sap.com>
2023-10-23 19:59:22 +03:00
Dimitar Kostadinov
9861a46d99 Add prometheus proxy related metrics
Signed-off-by: Dimitar Kostadinov <dimitar.kostadinov@sap.com>
2023-10-23 19:59:22 +03:00
Milos Gajdos
708bc6f3e9
Make S3 tests pass with MinIO (#4107) 2023-10-20 16:20:30 +01:00
Milos Gajdos
dfd191e7d2
Replace docker/libtrust with go-jose/go-jose (#4096) 2023-10-19 15:52:09 +01:00
Milos Gajdos
fe21f43911
feat: replace docker/libtrust with go-jose/go-jose
docker/libtrust repository has been archived for several years now.
This commit replaces all the libtrust JWT machinery with go-jose/go-jose module.
Some of the code has been adopted from libtrust and adjusted for some of
the use cases covered by the token authorization flow especially in the
tests.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-19 15:32:59 +01:00
Milos Gajdos
5aee8e1917
feat: Add context to storagedriver.(Filewriter).Commit() (#4109) 2023-10-19 11:41:55 +01:00
Milos Gajdos
cb0d083d8d
feat: Add context to storagedriver.(Filewriter).Commit()
This commit changes storagedriver.Filewriter interface
by adding context.Context as an argument to its Commit
func.

We pass the context appropriately where need be throughout
the distribution codebase to all the writers and tests.

S3 driver writer unfortunately must maintain the context
passed down to it from upstream so it contnues to
implement io.Writer and io.Closer interfaces which do not
allow accepting the context in any of their funcs.

Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-19 11:27:27 +01:00
Milos Gajdos
5ad2c45b8c
update to go1.20.10, test go1.21.3 (#4116) 2023-10-19 10:54:01 +01:00
Sebastiaan van Stijn
46d13ff75b
update to go1.20.10, test go1.21.3
go1.20.10 (released 2023-10-10) includes a security fix to the net/http package.
See the Go 1.20.10 milestone on our issue tracker for details:

https://github.com/golang/go/issues?q=milestone%3AGo1.20.10+label%3ACherryPickApproved

full diff: https://github.com/golang/go/compare/go1.20.9...go1.20.10

From the security mailing:

[security] Go 1.21.3 and Go 1.20.10 are released

Hello gophers,

We have just released Go versions 1.21.3 and 1.20.10, minor point releases.

These minor releases include 1 security fixes following the security policy:

- net/http: rapid stream resets can cause excessive work

  A malicious HTTP/2 client which rapidly creates requests and
  immediately resets them can cause excessive server resource consumption.
  While the total number of requests is bounded to the
  http2.Server.MaxConcurrentStreams setting, resetting an in-progress
  request allows the attacker to create a new request while the existing
  one is still executing.

  HTTP/2 servers now bound the number of simultaneously executing
  handler goroutines to the stream concurrency limit. New requests
  arriving when at the limit (which can only happen after the client
  has reset an existing, in-flight request) will be queued until a
  handler exits. If the request queue grows too large, the server
  will terminate the connection.

  This issue is also fixed in golang.org/x/net/http2 v0.17.0,
  for users manually configuring HTTP/2.

  The default stream concurrency limit is 250 streams (requests)
  per HTTP/2 connection. This value may be adjusted using the
  golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
  setting and the ConfigureServer function.

  This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
  This is also tracked by CVE-2023-44487.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-19 10:45:12 +02:00
Sebastiaan van Stijn
9cc6e5b27f
update to go1.20.9, test go1.21.2
go1.20.9 (released 2023-10-05) includes one security fixes to the cmd/go package,
as well as bug fixes to the go command and the linker. See the Go 1.20.9
milestone on our issue tracker for details:

https://github.com/golang/go/issues?q=milestone%3AGo1.20.9+label%3ACherryPickApproved

full diff: https://github.com/golang/go/compare/go1.20.8...go1.20.9

From the security mailing:

[security] Go 1.21.2 and Go 1.20.9 are released

Hello gophers,

We have just released Go versions 1.21.2 and 1.20.9, minor point releases.

These minor releases include 1 security fixes following the security policy:

- cmd/go: line directives allows arbitrary execution during build

  "//line" directives can be used to bypass the restrictions on "//go:cgo_"
  directives, allowing blocked linker and compiler flags to be passed during
  compliation. This can result in unexpected execution of arbitrary code when
  running "go build". The line directive requires the absolute path of the file in
  which the directive lives, which makes exploting this issue significantly more
  complex.

  This is CVE-2023-39323 and Go issue https://go.dev/issue/63211.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-19 10:44:38 +02:00
Milos Gajdos
5592968cbc
reference: fix broken alias for DomainRegexp (#4113) 2023-10-19 09:25:17 +01:00
Sebastiaan van Stijn
c8c2bc279c
reference: fix broken alias for DomainRegexp
An incorrect alias snuck into 152af63ec5,
and DomainRegexp was aliased to the regex for digests (DigestRegexp).

This didn't affect this repository, as it didn't use the aliases and migrated
to the new module, but does affect user of the old module that depend on the
aliases.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-10-19 10:06:38 +02:00
Wang Yan
d24243730f
refactor: Storage driver errors (#4108) 2023-10-19 01:12:14 +08:00
Milos Gajdos
ea41722902
refactor: Storage driver errors
Small refactoring of storagedriver errors.
We change the Enclosed field to Detail and make sure
Errors get properly serialized to JSON.
We also add tests.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-18 10:02:21 +01:00
Milos Gajdos
915ad2d5a6
json encode storage driver enclosed error (#4099) 2023-10-17 21:44:42 +01:00
James Hewitt
eac199875e
Remove test for nested file delete on S3
Nested files aren't supported on MinIO, and as our storage layout is
filesystem based, we don't actually use nest files in the code.

Remove the test so that we can support MinIO.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-17 09:13:15 +01:00
James Hewitt
647ec33c33
Bump minio version and test less storage classes
This fixes some of the tests for minio.

The walk tests needs a version of minio that contains https://github.com/minio/minio/pull/18099

The storage classes minio supports are a subset of the s3 classes.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-17 02:10:43 +01:00
Milos Gajdos
1d410148ef
Update dockerhub-readme GH Action (#4105) 2023-10-16 19:10:39 +01:00
Milos Gajdos
71217a0c7b
Update docs GHA and renamed a doc file (#4102) 2023-10-16 13:59:03 +01:00
Milos Gajdos
078c0546a4
Add annotation for descriptor (#4106) 2023-10-16 13:36:07 +01:00
Tosone
4dce8b866e Add annotation for descriptor
Signed-off-by: Tosone <i@tosone.cn>
2023-10-16 19:03:12 +08:00
Milos Gajdos
777ad03208
Update docs GHA
Add missing steps to the job, pick up the path automatically, trigger
the job on config file changes.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-15 11:03:23 +01:00
Milos Gajdos
4f506663d3
Update dockerhub-readme GH Action
We were missing GitHub checkout step before running the update.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-14 18:16:01 +01:00
Milos Gajdos
c78d6f99ae
Move dockerhub-readme workflow to the correct path (#4103) 2023-10-14 18:09:52 +01:00
Milos Gajdos
dc07c42810
Move dockerhub-readme workflow to the correct path
We've incorrectly added the dockerhub-readme workflow into .github
path from where it can not be triggered:

https://docs.github.com/en/actions/using-workflows/triggering-a-workflow

This commit addresses it and update the workflow paths.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-10-13 11:05:12 +01:00
Milos Gajdos
ebba01efea
docs: add hugo website (#4101) 2023-10-12 17:08:54 +01:00
David Karlsson
a66f6c37cb ci: add github pages workflow for docs
Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
2023-10-12 13:44:58 +02:00
James Hewitt
c3ae793f85
And other content...
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-12 12:00:15 +01:00
James Hewitt
83dd4ff0a6
Cleanup of naming in docs
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-12 11:39:36 +01:00