Commit graph

32 commits

Author SHA1 Message Date
Cory Snider
d0f5aa670b Move context package internal
Our context package predates the establishment of current best practices
regarding context usage and it shows. It encourages bad practices such
as using contexts to propagate non-request-scoped values like the
application version and using string-typed keys for context values. Move
the package internal to remove it from the API surface of
distribution/v3@v3.0.0 so we are free to iterate on it without being
constrained by compatibility.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-27 10:58:37 -04:00
James Hewitt
ef8651ec2a
Switch to github.com/google/uuid
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-10-25 12:15:21 +01:00
Cory Snider
8a86dc61ff Make our UUID package internal
Nowadays there are much, much better UUID implementations to choose
from, such as github.com/google/uuid. Prevent external users from
importing our bespoke implementation so that we can change or migrate
away from it internally without introducing breaking changes.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-10-24 18:11:41 -04:00
Sebastiaan van Stijn
152af63ec5
deprecate reference package, migrate to github.com/distribution/reference
This integrates the new module, which was extracted from this repository
at commit b9b19409cf458dcb9e1253ff44ba75bd0620faa6;

    # install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
    brew install git-filter-repo

    # create a temporary clone of docker
    cd ~/Projects
    git clone https://github.com/distribution/distribution.git reference
    cd reference

    # commit taken from
    git rev-parse --verify HEAD
    b9b19409cf

    # remove all code, except for general files, 'reference/', and rename to /
    git filter-repo \
      --path .github/workflows/codeql-analysis.yml \
      --path .github/workflows/fossa.yml \
      --path .golangci.yml \
      --path distribution-logo.svg \
      --path CODE-OF-CONDUCT.md \
      --path CONTRIBUTING.md \
      --path GOVERNANCE.md \
      --path README.md \
      --path LICENSE \
      --path MAINTAINERS \
      --path-glob 'reference/*.*' \
      --path-rename reference/:

    # initialize go.mod
    go mod init github.com/distribution/reference
    go mod tidy -go=1.20

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-31 15:47:06 +02:00
Sebastiaan van Stijn
19233195b7
Align code to match order of fields
This is just cosmetic; alighn the fields with the order in which they appear
in the struct (and JSON output).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-29 23:24:10 +01:00
Sebastiaan van Stijn
1d33874951
go.mod: change imports to github.com/distribution/distribution/v3
Go 1.13 and up enforce import paths to be versioned if a project
contains a go.mod and has released v2 or up.

The current v2.x branches (and releases) do not yet have a go.mod,
and therefore are still allowed to be imported with a non-versioned
import path (go modules add a `+incompatible` annotation in that case).

However, now that this project has a `go.mod` file, incompatible
import paths will not be accepted by go modules, and attempting
to use code from this repository will fail.

This patch uses `v3` for the import-paths (not `v2`), because changing
import paths itself is a breaking change, which means that  the
next release should increment the "major" version to comply with
SemVer (as go modules dictate).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-08 18:30:46 +01:00
Elliot Pahl
800cb95821
Use go-events package
TBD: Queue not converted yet

Signed-off-by: Elliot Pahl <elliot.pahl@gmail.com>
2020-03-04 12:49:32 -08:00
Manish Tomar
da8db4666b Fix gometalint errors
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:01:04 -08:00
Derek McGowan
b12bd4004a
Merge pull request #2639 from andrew-leung/manifesteventlayers
Add configurable layers in manifest events
2018-08-28 16:03:05 -07:00
Andrew Leung
5e4b81a578 Use references terminology instead of layers.
Signed-off-by: Andrew Leung <anwleung@gmail.com>
2018-08-20 10:01:40 -07:00
Manish Tomar
328069bb4d add support for repo deleted event also
by having another interface RepositoryRemover that is implemented by
registry instance and is injected in app context for event tracking

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-02 23:05:05 -07:00
Andrew Leung
54aef6251e Fix typo in manifestLayers variable name
Signed-off-by: Andrew Leung <anwleung@gmail.com>
2018-07-29 08:01:21 -07:00
Manish Tomar
0d8f4ac7b8 add tag deletion event
whenever a tag is deleted an event is sent out via the regular
notification channels

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-07-18 11:34:09 -07:00
Andrew Leung
276fdce3d9 Add configurable layers in manifest events
Signed-off-by: Andrew Leung <anwleung@gmail.com>
2018-06-27 09:27:24 -07:00
Stephen J Day
532ec9f036
digest: migrate to opencontainers/go-digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 15:42:03 -08:00
Richard Scothern
afe2bdd1c5 Propogate tag as a functional argument into the notification system to attach
tags to manifest push and pull event notifications.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 14:57:52 -07:00
Richard Scothern
f37b2ee16e Send manifest and blob delete events to the notifications subsystem.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-08 13:37:35 -08:00
Aaron Lehmann
2b20b0167a Change URLBuilder methods to use references for tags and digests
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:49:29 -08:00
Aaron Lehmann
4441333912 Use reference package internally
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:47:05 -08:00
Richard Scothern
4f822100f7 Ensure the canonical manifest digest and length are sent to notification endpoints.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-20 11:21:04 -08:00
Brian Bland
613cfc861d Fires a new Mount event when blobs are cross-repo mounted
Adds an optional "fromRepository" field to the event target

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:30 -08:00
Richard Scothern
cb6f002350 Implementation of the Manifest Service API refactor.
Add a generic Manifest interface to represent manifests in the registry and
remove references to schema specific manifests.

Add a ManifestBuilder to construct Manifest objects. Concrete manifest builders
will exist for each manifest type and implementations will contain manifest
specific data used to build a manifest.

Remove Signatures() from Repository interface.

Signatures are relevant only to schema1 manifests.  Move access to the signature
store inside the schema1 manifestStore.  Add some API tests to verify
signature roundtripping.

schema1
-------

Change the way data is stored in schema1.Manifest to enable Payload() to be used
to return complete Manifest JSON from the HTTP handler without knowledge of the
schema1 protocol.

tags
----

Move tag functionality to a seperate TagService and update ManifestService
to use the new interfaces.  Implement a driver based tagService to be backward
compatible with the current tag service.

Add a proxyTagService to enable the registry to get a digest for remote manifests
from a tag.

manifest store
--------------

Remove revision store and move all signing functionality into the signed manifeststore.

manifest registration
---------------------

Add a mechanism to register manifest media types and to allow different manifest
types to be Unmarshalled correctly.

client
------

Add ManifestServiceOptions to client functions to allow tags to be passed into Put and
Get for building correct registry URLs.  Change functional arguments to be an interface type
to allow passing data without mutating shared state.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2015-12-17 17:09:14 -08:00
Aaron Lehmann
31047c8113 Simplify digest.FromBytes calling convention
The current implementation of digest.FromBytes returns an error. This
error can never be non-nil, but its presence in the function signature
means each call site needs error handling code for an error that is
always nil.

I verified that none of the hash.Hash implementations in the standard
library can return an error on Write. Nor can any of the hash.Hash
implementations vendored in distribution.

This commit changes digest.FromBytes not to return an error. If Write
returns an error, it will panic, but as discussed above, this should
never happen.

This commit also avoids using a bytes.Reader to feed data into the hash
function in FromBytes. This makes the hypothetical case that would panic
a bit more explicit, and should also be more performant.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-14 14:30:51 -08:00
Stephen J Day
6712e602b0 Move manifest package to schema1
As we begin our march towards multi-arch, we must prepare for the reality of
multiple manifest schemas. This is the beginning of a set of changes to
facilitate this. We are both moving this package into its target position where
it may live peacefully next to other manfiest versions.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-21 16:29:47 -07:00
Stephen J Day
26b7fe4a91 Use "Size" field to describe blobs over "Length"
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-17 17:07:11 -07:00
Stephen J Day
a2d4f51aa4 Notification should send digest URL in event target
Previously, the most accurate reference for a manifest was the tag url. After
adding pull by digest, all event notifications should refer directly to the
digest url. This ensures that event uniquely identifies the target of the
notification. Testing has been added for manifest pull events to check that
this doesn't change.

In addition, the listener interface has been refactored to only use the
repository name, rather than the full repository object.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-27 12:56:12 -07:00
Stephen J Day
36e34a55ad Replace uuid dependency with internal library
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:59:41 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Richard
e21a425f88 Verify IP addresses
Fix lint errors
Add more test
2015-03-25 11:11:46 -07:00
Richard
c6fdfc9cd5 Attempt to identify remote IP addresses for requests which come
through proxies.

Add a function to examine X-Forward-For and X-Real-Ip headers for
originating IP addresses.  Use RemoteAddr for notification request
record and HTTP request context.
2015-03-24 18:45:14 -07:00
Josh Hawn
a877811c0b Update notification event Target fields
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-05 18:01:50 -08:00
Stephen J Day
d2d46fca41 Move notifications package to distribution
Since the notifications package is now decoupled from storage, we are moving it
to the root package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:40:35 -08:00
Renamed from registry/storage/notifications/bridge.go (Browse further)