context.App.repoRemover is single registry instance stored throughout
app run. It was wrapped in another remover when processing each request.
This remover happened to be remover got from previous request. This way
every remover created was stored in infinite linked list causing memory
leak. Fixing it by storing the wrapped remover inside the request context
which will get gced when request context is gced. This was introduced in
PR #2648.
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
at the first iteration, only the following metrics are collected:
- HTTP metrics of each API endpoint
- cache counter for request/hit/miss
- histogram of storage actions, including:
GetContent, PutContent, Stat, List, Move, and Delete
Signed-off-by: tifayuki <tifayuki@gmail.com>
If htpasswd authentication option is configured but the htpasswd file is
missing, populate it with a default user and automatically generated
password.
The password will be printed to stdout.
Signed-off-by: Liron Levin <liron@twistlock.com>
Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.
To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Under certain circumstances, the use of `StorageDriver.GetContent` can
result in unbounded memory allocations. In particualr, this happens when
accessing a layer through the manifests endpoint.
This problem is mitigated by setting a 4MB limit when using to access
content that may have been accepted from a user. In practice, this means
setting the limit with the use of `BlobProvider.Get` by wrapping
`StorageDriver.GetContent` in a helper that uses `StorageDriver.Reader`
with a `limitReader` that returns an error.
When mitigating this security issue, we also noticed that the size of
manifests uploaded to the registry is also unlimited. We apply similar
logic to the request body of payloads that are full buffered.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
If the client doesn't support manifest lists, the registry will
rewrite a manifest list into the old format. The Docker-Content-Digest
header should be updated in this case.
Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
The registry uses partial Named values which the named parsers
no longer support. To allow the registry service to continue
to operate without canonicalization, switch to use WithName.
In the future, the registry should start using fully canonical
values on the backend and WithName should no longer support
creating partial values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
When get manifest, the handler will try to retrieve it from storage driver. When storage driver is cloud storage, it can fail due to various reasons even if the manifest exists
(like 500, 503, etc. from storage server). Currently manifest handler blindly return 404 which can be confusing to user.
This change will return 404 if the manifest blob doesn't exist, and return 500 UnknownError for all other errors (consistent with the behavior of other handlers).
Signed-off-by: Yu Wang (UC) <yuwa@microsoft.com>
Once upon a time, we referred to manifests and images interchangably.
That simple past is no more. As we grow, we update our nomenclature and
so follows our code.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
With token authentication, requiring the "*" action for DELETE requests
makes it impossible to administratively lock a repository against pushes
and pulls but still allow deletion. This change adds a new "delete"
action for DELETE requests to make that possible.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
`app.driver.List` on `"/"` is very expensive if registry contains significant amount of images. And the result isn't used anyways.
In most (if not all) storage drivers, `Stat` has a cheaper implementation, so use it instead to achieve the same goal.
Signed-off-by: yixi zhang <yixi@memsql.com>
Modify manifest builder so it can be used to build
manifests with different configuration media types.
Rename config media type const to image config.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Use whitelist of allowed repository classes to enforce.
By default all repository classes are allowed.
Add authorized resources to context after authorization.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
To allow generic manifest walking, we define an interface method of
`References` that returns the referenced items in the manifest. The
current implementation does not return the config target from schema2,
making this useless for most applications.
The garbage collector has been modified to show the utility of this
correctly formed `References` method. We may be able to make more
generic traversal methods with this, as well.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Context should use type values instead of strings.
Updated direct calls to WithValue, but still other uses of string keys.
Update Acl to ACL in s3 driver.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The Hub registry generates a large volume of notifications, many of
which are uninteresting based on target media type. Discarding them
within the notification endpoint consumes considerable resources that
could be saved by discarding them within the registry. To that end,
this change adds registry configuration options to restrict the
notifications sent to an endpoint based on target media type.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
Until we have some experience hosting foreign layer manifests, the Hub
operators wish to limit foreign layers on Hub. To that end, this change
adds registry configuration options to restrict the URLs that may appear
in pushed manifests.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
In Go's header parsing, the same header multiple times results in multiple entries in the `r.Header[...]` slice, but Go does no further parsing beyond that (and in https://golang.org/cl/4528086 it was determined that until/unless the stdlib itself needs it, Go will not do so).
The consequence here for parsing of `Accept:` headers is that we support the way Go outputs headers, but not all language HTTP libraries have a facility to output multiple headers instead of a single list header.
This change ensures that the following (valid) header blocks all parse to the same result for the purposes of what is being tested here:
```
Accept: a/b
Accept: b/c
Accept: d/e
```
```
Accept: a/b; q=0.5, b/c
Accept: d/e
```
```
Accept: a/b; q=0.1, b/c; q=0.2, d/e; q=0.8
```
Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
This lets us access registry config within middleware for additional
configuration of whatever it is that you're overriding.
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
+ Inmemory
+ Filesystem
+ S3
+ Azure
Signed-off-by: Brian Bland <brian.bland@docker.com>
It is possible for a middlebox to lowercase the URL at somepoint causing a
lookup in the auth challenges table to fail. Lowercase hostname before
using as keys to challenge map.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Middleware code may perform additional checks on blobs written. Allow it
to return access denied errors that will result in 403 Forbidden.
Signed-off-by: Michal Minar <miminar@redhat.com>
Add option for specifying trust key for signing schema1 manifests.
Since schema1 signature key identifiers are not verified anywhere and deprecated, storing signatures is no longer a requirement.
Furthermore in schema2 there is no signature, requiring the registry to already add signatures to generated schema1 manifests.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
When schema2 manifests are rewritten as schema1 currently the etag and docker content digest header keep the value for the schema2 manifest.
Fixes#1444
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Uses docker/goamz instead of AdRoll/goamz
Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well
Signed-off-by: Brian Bland <brian.bland@docker.com>
Prevent using strings throughout the code to reference a string key defined in the auth package.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
