The garbage-collect should remove unsed layer link file
P.S. This was originally contributed by @m-masataka, now I would like to take over it.
Thanks @m-masataka efforts with PR https://github.com/distribution/distribution/pull/2288
Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
Huge help from @milosgajdos who figured out how to do the entire
marshalling/unmarshalling for the configs
Signed-off-by: Anders Ingemann <aim@orbit.online>
Enable configuration options that can selectively disable validation
that dependencies exist within the registry before the image index
is uploaded.
This enables sparse indexes, where a registry holds a manifest index that
could be signed (so the digest must not change) but does not hold every
referenced image in the index. The use case for this is when a registry
mirror does not need to mirror all platforms, but does need to maintain
the digests of all manifests either because they are signed or because
they are pulled by digest.
The registry administrator can also select specific image architectures
that must exist in the registry, enabling a registry operator to select
only the platforms they care about and ensure all image indexes uploaded
to the registry are valid for those platforms.
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
Harbor is using the distribution for it's (harbor-registry) registry component.
The harbor GC will call into the registry to delete the manifest, which in turn
then does a lookup for all tags that reference the deleted manifest.
To find the tag references, the registry will iterate every tag in the repository
and read it's link file to check if it matches the deleted manifest (i.e. to see
if uses the same sha256 digest). So, the more tags in repository, the worse the
performance will be (as there will be more s3 API calls occurring for the tag
directory lookups and tag file reads).
Therefore, we can use concurrent lookup and untag to optimize performance as described in https://github.com/goharbor/harbor/issues/12948.
P.S. This optimization was originally contributed by @Antiarchitect, now I would like to take it over.
Thanks @Antiarchitect's efforts with PR https://github.com/distribution/distribution/pull/3890.
Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
it is reasonable to ignore the error that the manifest tag path does not exist when querying
all tags of the specified repository when executing gc.
Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
Currently, the `forcepathstyle` parameter for the s3 storage driver is
considered only if the `regionendpoint` parameter is set. Since setting
a region endpoint explicitly is discouraged with AWS s3, it is not clear
how to enforce path style URLs with AWS s3.
This also means, that the default value (true) only applies if a region
endpoint is configured.
This change makes sure we always forward the `forcepathstyle` parameter
to the aws-sdk if present in the config. This is a breaking change where
a `regionendpoint` is configured but no explicit `forcepathstyle` value
is set.
Signed-off-by: Benjamin Schanzel <benjamin.schanzel@bmw.de>
This reverts https://github.com/distribution/distribution/pull/3556
This feature is currently broken and requires more fundamental changes
in the S3 driver. Until then it's better to remove it.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
Unfortunately one of the changes we merged in broken the support for
http.ProxyFromEnvironment https://pkg.go.dev/net/http#ProxyFromEnvironment
This commit attempts to fix that by cloning the http.DefaultTransport
and updating it accordingly.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
This commit updates (writer).Writer() method in S3 storage driver to
handle the case where an append is attempted to a zer-size content.
S3 does not allow appending to already committed content, so we are
optiing to provide the following case as a narrowed down behaviour:
Writer can only append to zero byte content - in that case, a new S3
MultipartUpload is created that will be used for overriding the already
committed zero size content.
Appending to non-zero size content fails with error.
Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
GCS storage driver used to be conditionally built due to its being
outdated and basically unmaintained. Recently the driver has gone
through a rework and updates. Let's remove the build tag so we have less
headaches dealing with it and try keeping it up to date.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
This linter both prevents parallel test races as well as
suggests parallel tests where appropriate:
See: https://github.com/moricho/tparallel
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
We make sure they're not hiding at the bottom or in the middle
which makes debugging an utter nightmare!
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
If the reference in the API request exceeds the threshold allowed by the
reference package (NOTE: this isn't defined by distribution
specification!) we return 500 back to the client.
This commit makes sure we return 400 and the explanation of the error in
the returned JSON payload.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
This commit refactors the GCS storage driver from the ground up and makes
it more consistent with the rest of the storage drivers.
We are also fixing GCS authentication using default app credentials:
When the default application credentials are used we don't initialize the
GCS storage client which then panics.
Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
For some reason a PR we merged passed the build even though it was
missing various func parameters. This commmit fixes it.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
Several storage drivers and storage middlewares need to introspect the
client HTTP request in order to construct content-redirect URLs. The
request is indirectly passed into the driver interface method URLFor()
through the context argument, which is bad practice. The request should
be passed in as an explicit argument as the method is only called from
request handlers.
Replace the URLFor() method with a RedirectURL() method which takes an
HTTP request as a parameter instead of a context. Drop the options
argument from URLFor() as in practice it only ever encoded the request
method, which can now be fetched directly from the request. No URLFor()
callers ever passed in an "expiry" option, either.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The RemoteAddr and RemoteIP functions operate on *http.Request values,
not contexts. They have very low cohesion with the rest of the package.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Our context package predates the establishment of current best practices
regarding context usage and it shows. It encourages bad practices such
as using contexts to propagate non-request-scoped values like the
application version and using string-typed keys for context values. Move
the package internal to remove it from the API surface of
distribution/v3@v3.0.0 so we are free to iterate on it without being
constrained by compatibility.
Signed-off-by: Cory Snider <csnider@mirantis.com>
This commit make the S3 driver chunk size constants more straightforward
to understand -- instead of remembering the bit shifts we make this more
explicit.
We are also updating append parameter to the `(writer).Write` to follow
the new convention we are trying to establish.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
Nowadays there are much, much better UUID implementations to choose
from, such as github.com/google/uuid. Prevent external users from
importing our bespoke implementation so that we can change or migrate
away from it internally without introducing breaking changes.
Signed-off-by: Cory Snider <csnider@mirantis.com>
