As we begin our march towards multi-arch, we must prepare for the reality of
multiple manifest schemas. This is the beginning of a set of changes to
facilitate this. We are both moving this package into its target position where
it may live peacefully next to other manfiest versions.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To bring ErrorCode into liine with Go conventions, ErrorCode.Error() now
returns the "nice" value of the error code. This ensures error message assembly
works similar to commonly used Go conventions when directly using ErrorCode as
an error.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Also, add timeout and status code parameters to the HTTP checker, and
remove the threshold parameter for the file checker.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Update docs.
Change health_test.go tests to create their own registries and register
the checks there. The tests now call CheckStatus directly instead of
polling the HTTP handler, which returns results from the default
registry.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Add default storagedriver health check to example configuration files
with parameters matching the previous hardcoded configuration.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Add a section to the config file called "health". Within this section,
"filecheckers" and "httpcheckers" list checks to run. Each check
specifies a file or URI, a time interval for the check, and a threshold
specifying how many times the check must fail to reach an unhealthy
state.
Document the new options in docs/configuration.md.
Add unit testing for both types of checkers. Add an UnregisterAll
function in the health package to support the unit tests, and an
Unregister function for consistency with Register.
Fix a string conversion problem in the health package's HTTP checker.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Clean up calling convention for NewRegistryWithDriver to use functional
arguments.
This is a first step towards the refactor described in #215. I plan to
add additional options in the process of moving configurable items from
the App structure to the registry structure.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
A change in #763 to address review comments caused problems. Originally,
instrumentedResponseWriter implemented the CloseNotifier interface, and
would panic if it was wrapping something that did not implement that
interface. This was split into a separate instrumentedResponseWriterCN
type that implements CloseNotifier, so there's a fallback if
instrumentedResponseWriter ever needs to wrap something that does not
implement this interface.
instrumentedResponseWriter's Value method would end up upcasting either
type back to instrumentedResponseWriter, which does not implement the
interface. In effect, instrumentedResponseWriterCN was never visible to
the handler.
This fixes the problem by implementing a wrapper Value method for
instrumentedResponseWriterCN.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Change checkResponse to only expect the configured
X-Content-Type-Options header if it doesn't receive a 405 error, which
means the handler isn't registered for that method.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The use of the pathMapper is no longer needed the way we have organized the
code base. The extra level of indirection has proved unnecessary and confusing
so we've opted to clean it up. In the future, we may require more flexibility,
but now it is simply not required.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Unfortunately, the 2.1 releease has written manfiest links into the wrong
directory. This doesn't affect new 2.1 deployments but fixing this to be 2.0
backwards compatible has broken 2.1.0 compatibility. To ensure we have
compatibility between 2.0, 2.1.0 and future releases, we now check one of
several locations to identify a manifest link.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Unfortunately, the refactor used the incorrect path for manifest links within a
repository. While this didn't stop the registry from working, it did break
compatibility with 2.0 deployments for manifest fetches.
Tests were added to ensure these are locked down to the appropriate paths.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The example configuration files add X-Content-Type-Options: nosniff.
Add coverage in existing registry/handlers unit tests.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
distribution errors. Fill in missing checks for mutations on a registry pull-through
cache. Add unit tests and update documentation.
Also, give v2.ErrorCodeUnsupported an HTTP status code, previously it was
defaulting to 500, now its 405 Method Not Allowed.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Several error codes are generally useful but tied to the v2 specification
definitions. This change moves these error code definitions into the common
package for use by the health package, which is not tied to the v2 API.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To ensure the ensure the web application is properly operating, we've added a
periodic health check for the storage driver. If the health check fails three
times in a row, the registry will serve 503 response status for any request
until the condition is resolved. The condition is reported in the response body
and via the /debug/health endpoint.
To ensure that all drivers will properly operate with this health check, a
function has been added to the driver testsuite.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
When using the RADOS driver, the hierarchy of the files is stored
in OMAPs, but the root OMAP was not created and a call to List("/")
was returning an error instead of returned the first level files
stored. This patches creates an OMAP for "/" and excludes the listed
directory from the list of files returned.
Signed-off-by: Vincent Giersch <vincent@giersch.fr>
Instead, provide a variant of instrumentedResponseWriter that does not
implement CloseNotifier, and use that when necessary. In
copyFullPayload, log instead of panicing when we encounter something
that doesn't implement CloseNotifier.
This is more complicated than I'd like, but it's necessary because
instrumentedResponseWriter must not embed CloseNotifier unless there's
really a CloseNotifier to embed.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
with a new `proxy` section in the configuration file.
Create a new registry type which delegates storage to a proxyBlobStore
and proxyManifestStore. These stores will pull through data if not present
locally. proxyBlobStore takes care not to write duplicate data to disk.
Add a scheduler to cleanup expired content. The scheduler runs as a background
goroutine. When a blob or manifest is pulled through from the remote registry,
an entry is added to the scheduler with a TTL. When the TTL expires the
scheduler calls a pre-specified function to remove the fetched resource.
Add token authentication to the registry middleware. Get a token at startup
and preload the credential store with the username and password supplied in the
config file.
Allow resumable digest functionality to be disabled at runtime and disable
it when the registry is a pull through cache.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
registry/storage/blob_test.go:149: arg d for printf verb %s of wrong type: github.com/docker/distribution.Descriptor
Signed-off-by: Doug Davis <dug@us.ibm.com>
The response code isn't actually sent to the client, because the
connection has already closed by this point. But it causes the status
code to appear as 499 in the logs instead of 0.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
When a client disconnects without completing a HTTP request, we were
attempting to process the partial request, which usually leads to a 400
error. These errors can pollute the logs and make it more difficult to
track down real bugs.
This change uses CloseNotifier to detect disconnects. In combination
with checking Content-Length, we can detect a disconnect before sending
the full payload, and avoid logging a 400 error.
This logic is only applied to PUT, POST, and PATCH endpoints, as these
are the places where disconnects during a request are most likely to
happen.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
includes the http response.
When debugging non-successful registry requests this will place
the error details and http status fields in the same log line
giving easier visibility to what error occured in the request.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Log a warning if the registry generates its own secret.
Update configuration doc, and remove the default secret from the
development config file.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This passed in the #744 before merge, but apparently the test changed
since the PR was created in ways that led to a new failures.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Storage drivers can implement a method called URLFor which can return a direct
url for a given path. The functionality allows the registry to direct clients
to download content directly from the backend storage. This is commonly used
with s3 and cloudfront. Under certain conditions, such as when the registry is
not local to the backend, these redirects can hurt performance and waste
incoming bandwidth on pulls. This feature addition allows one to disable this
feature, if required.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Conflicts:
configuration/configuration.go
registry/handlers/app.go
registry/storage/catalog_test.go
registry/storage/manifeststore_test.go
registry/storage/registry.go
Implement the delete API by implementing soft delete for layers
and blobs by removing link files and updating the blob descriptor
cache. Deletion is configurable - if it is disabled API calls
will return an unsupported error.
We invalidate the blob descriptor cache by changing the linkedBlobStore's
blobStatter to a blobDescriptorService and naming it blobAccessController.
Delete() is added throughout the relevant API to support this functionality.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Because the logger was incorrectly replaced while adding hooks, log output did
not include the version and instance ids. The main issue was the the
logrus.Entry was replaced with the logger, which included no context. Replacing
the logger on the context is not necessary when configuring hooks since we are
configuring the contexts logger directly.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This removes the erroneous http.Handler interface in favor a simple SetHeaders
method that only operattes on the response. Several unnecessary uses of pointer
types were also fixed up.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Some missing descriptions and error code for tags pagination was cleaned up to
ensure clarity. Specifically, we ensure the request variations are named and
the proper error codes are included.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The way Repositories() was initially called was somewhat different than
other parts of the client bindings because there was no way to instantiate a
Namespace. This change implements a NewRegistry() function which changes
it so that Repositories() can be called the way one would expect.
It doesn't implement any of the other functions of Namespaces.
Signed-off-by: Patrick Devine <patrick.devine@docker.com>
This change removes the Catalog Service and replaces it with a more
simplistic Repositories() method for obtaining a catalog of all
repositories. The Repositories method takes a pre-allocated slice
and fills it up to the size of the slice and returns the amount
filled. The catalog is returned lexicographically and will start
being filled from the last entry passed to Repositories(). If there
are no more entries to fill, io.EOF will be returned.
Signed-off-by: Patrick Devine <patrick.devine@docker.com>
Conflicts:
registry/client/repository.go
registry/handlers/api_test.go
This change adds a basic catalog endpoint to the API, which returns a list,
or partial list, of all of the repositories contained in the registry. Calls
to this endpoint are somewhat expensive, as every call requires walking a
large part of the registry.
Instead, to maintain a list of repositories, you would first call the catalog
endpoint to get an initial list, and then use the events API to maintain
any future repositories.
Signed-off-by: Patrick Devine <patrick.devine@docker.com>
Move the specification to use a Link header, rather than a "next" entry in the
json results. This prevents requiring clients from parsing the request body to
issue the next request. It also ensures that the returned response body does
not change in between requests.
The ordering of the specification has been slightly tweaked, as well. Listing
image tags has been moved after the catalog specification. Tag pagination now
heavily references catalog pagination.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This contains a proposal for a catalog API, provided access to the internal
contents of a registry instance. The API endpoint is prefixed with an
underscore, which is illegal in images names, to prevent collisions with
repositories names. To avoid issues with large result sets, a paginated version
of the API is proposed. We make an addition to the tags API to support
pagination to ensure the specification is conistent.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".
Signed-off-by: Stephen J Day <stephen.day@docker.com>
An error level log is already produced within app.authorized() if an
actual unexpected error occurs during authorization, so this warning
level log remains for auditability purposes, but should not be
considered an error condition.
Addresses #704
Signed-off-by: Brian Bland <brian.bland@docker.com>
This changeset provides a common http handler for serving errcodes. This should
unify http responses across webservices in the face of errors.
Several type assertions have been added, as well, to ensure the error interface
is implemented.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Add a functional argument to pass a digest to (ManifestService).GetByTag().
If the digest matches an empty manifest and nil error are returned.
See 1bc740b0d5 for server implementation.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
The main goal of this changeset is to allow repository name components to
consist of a single character. The number of components allowed and the slash
separation requirements have also been clarified.
To go along with this simplification, errant constants and unneeded error types
have been removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Challenger manager interface is used to handle getting authorization challenges from an endpoint as well as extracting challenges from responses.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Replace ping logic with individual functions to extract API version and authorization challenges. The response from a ping operation can be passed into these function. If an error occurs in parsing, the version or challenge will not be used. Sending the ping request is the responsibility of the caller.
APIVersion has been converted from a string to a structure type. A parse function was added to convert from string to the structure type.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Move client auth into a separate package.
Separate ping from the authorizer and export Challenges type.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Timeouts should not be a discrete period of time, because they end
up being arbitrary and may be difficult to gauge correctly against
very large Docker layers. Rather, timeouts should be set at the
transport level using the SetDeadline attribute on a net.Conn
object.
Signed-off-by: Jon Poler <jonathan.poler@apcera.com>
This removes documentation and code related to IPC based storage driver
plugins. The existence of this functionality was an original feature goal but
is now not maintained and actively confusing incoming contributions. We will
likely explore some driver plugin mechanism in the future but we don't need
this laying around in the meantime.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
See: 3ea67df373/registry/handlers/app.go (L498)
Per the comment on line 498, this moves the logic of setting the http
status code into the serveJSON func, leaving the auth.Challenge.ServeHTTP()
func to just set the auth challenge header.
Signed-off-by: Doug Davis <dug@us.ibm.com>
This ensures that rados is not required when building the registry. This was
slightly tricky in that when the flags were applied, the rados package was
completely missing. This led to a problem where rados was basically unlistable
and untestable as a package. This was fixed by simply adding a doc.go file that
is included whether rados is built or not.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.
I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...
This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.
Fixes#620, #175.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
After consideration, the basic authentication implementation has been
simplified to only support bcrypt entries in an htpasswd file. This greatly
increases the security of the implementation by reducing the possibility of
timing attacks and other problems trying to detect the password hash type.
Also, the htpasswd file is only parsed at startup, ensuring that the file can
be edited and not effect ongoing requests. Newly added passwords take effect on
restart. Subsequently, password hash entries are now stored in a map.
Test cases have been modified accordingly.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change refactors the basic authentication implementation to better follow
Go coding standards. Many types are no longer exported. The parser is now a
separate function from the authentication code. The standard functions
(*http.Request).BasicAuth/SetBasicAuth are now used where appropriate.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
In the request parameters lists `tag` was used instead of
`reference` present in the HTTP requests paths
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
This PR is for issue of "email after registry webapp panic" #41, improving my
previous design (closed).
It use self setting up hooks, to catch panic in web application.
And, send email in hooks handle directly, to no use new http server and
handler.
Signed-off-by: xiekeyang <keyangxie@126.com>
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.
We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Ensure that clients can use the blob descriptor cache provider without needing
the redis package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Changes behavior so ping doesn't happen if /v2/ is anywhere in a request path, but instead only at the beginning. This fixes attempts to ping on redirected URLs.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The transport package no longer requires importing distribution for the ReadSeekCloser, instead declares its own.
Added comments on the Authenication handler in session.
Added todo on http seek reader to highlight its lack of belonging to the client transport.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Add check for unauthorized error code and explicitly set the error code if the content could not be parsed.
Updated repository test for unauthorized tests and nit feedback.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Each type no longer requires holding a reference to repository.
Added implementation for signatures get.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Update comments and TODOs
Fix switch style
Updated parse http response to take in reader
Add Cancel implementation
Update blobstore variable name
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.
Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:
* All the objects data are stored with opaque UUID names prefixed
(e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
omap object identifier is the virtual directory name, the keys in
a specific are the relative filenames and the values the blob
object identifier (or empty value for a sub directory).
e.g. For the following hierarchy:
/directory1
/directory1/object1
/directory1/object2
/directory1/directory2/object3
The omap "/directory1" will contains the following key / values:
- "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
- "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
- "directory2" ""
The omap "/directory1/directory2" will contains:
- "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"
* The MOVE is implemented by changing the reference to a specific
blob in its parent virtual directory omap.
This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
Repository creation now just takes in an http.RoundTripper. Authenticated requests or requests which require additional headers should use the NewTransport function along with a request modifier (such an an authentication handler).
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
