Commit graph

265 commits

Author SHA1 Message Date
Derek McGowan
c7f7747368 Update transport package to sever distribution dependency
The transport package no longer requires importing distribution for the ReadSeekCloser, instead declares its own.
Added comments on the Authenication handler in session.
Added todo on http seek reader to highlight its lack of belonging to the client transport.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:09:37 -07:00
Derek McGowan
572ff64d21 Add unauthorized error check
Add check for unauthorized error code and explicitly set the error code if the content could not be parsed.
Updated repository test for unauthorized tests and nit feedback.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:05:44 -07:00
Derek McGowan
13894e8736 Break down type dependencies
Each type no longer requires holding a reference to repository.
Added implementation for signatures get.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:56:27 -07:00
Derek McGowan
a3276fcc5b Feedback update
Update comments and TODOs
Fix switch style
Updated parse http response to take in reader
Add Cancel implementation
Update blobstore variable name

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:18:30 -07:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Jordan Liggitt
32ed2d7df1 Fix typo: respository->repository
Signed-off-by: Jordan Liggitt <jliggitt@redhat.com>
2015-05-19 13:25:39 -04:00
Derek McGowan
006ddd8283 Lint and documentation fixes
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
2874454224 Create client transport package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
94e375c5d1 Remove unused and duplicate error types
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
98836d6267 Update to track refactor updates
Added use of cache blob statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fdf7c8ff15 Open cache interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
70074b2286 Rename layer files to blob
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
9d64e461be Update to use blob interfaces
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fddeb1c8d5 Add missing defer on Tags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
468c5e79ba Simplify configuration and transport
Repository creation now just takes in an http.RoundTripper. Authenticated requests or requests which require additional headers should use the NewTransport function along with a request modifier (such an an authentication handler).

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
a15806ed9c Add base transport to interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
c6b51970cd Removed unused mirror flags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
68d5ecf6bf Update ReadFrom to wrap reader in NopCloser
Wrapping the reader in a NopCloser is necessary to prevent the http library from closing the input reader.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
49f7f54d07 Create authentication handler
Refactory authorizer to take a set of authentication handlers for different authentication schemes returned by an unauthorized HTTP requst.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
d92e5b1096 Add tags implementation
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
da05873b7c Use distribution context instead of google
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
c7ef45130b Cleanup session and config interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
6f9fbf99a9 Split layer and upload from repository
Layer upload moved to its own file with its own unit tests

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
b1ba2183ee Add unit tests for auth challenge and endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
174a732c94 Remove deprecated client interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
ce614b6de8 Add client implementation of distribution interface
Adds functionality to create a Repository client which connects to a remote endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Stephen Day
aeda6f5f8f Merge pull request #518 from RichardScothern/cache-headers
Set cache headers for layers.
2015-05-12 18:32:16 -07:00
Richard
3dc2d849c7 Set cache headers for layers.
- Set an Etag header
     - Check If-None-Match and respond appropriately
     - Set a Cache-Control header with a default of 1 week

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-12 17:49:18 -07:00
Anton Tiurin
ad80cbe1ea [Server] Listen and serve on a unix socket
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-05-11 16:00:14 +03:00
Derek McGowan
2cf40ab790 Modify blob upload API
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.

closes #470

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:43:23 -07:00
Richard
70f853e787 Ensure the instrumentedResponseWriter correctly sets the http
status in the context.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-05 16:46:33 -07:00
Stephen Day
dd58ffe6ed Merge pull request #410 from RichardScothern/driver-context
Add golang/x/net/context.Context to storage driver method calls
2015-05-04 19:18:30 -07:00
Derek McGowan
7d6e6aa980 Update API spec to reference digest instead of tarsum
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-01 17:14:38 -07:00
Stephen J Day
69e477f2d5 Use done channel to avoid goroutine leak
This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-28 14:40:20 -07:00
Richard
ae216e365a Make Storage Driver API calls context aware.
- Change driver interface to take a context as its first argument
     - Make newFileReader take a context as its first argument
     - Make newFileWriter take a context as its first argument
     - Make blobstore exists and delete take a context as a first argument
     - Pass the layerreader's context to the storage layer
     - Pass the app's context to purgeuploads
     - Store the app's context into the blobstore (was previously null)
     - Pass the trace'd context to the storage drivers

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 15:58:58 -07:00
Stephen Day
70427e3d9f Merge pull request #381 from RichardScothern/purge-config
Add configuration for upload purging
2015-04-27 14:33:34 -07:00
Richard
6460ddb2cb Add configuration for upload purging
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 11:06:15 -07:00
xiekeyang
c89e517be1 simplify the embedded method expression of repository
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2015-04-27 16:33:58 +08:00
Josh Hawn
628c7fa77b Updated urlbuilder X-Forwarded-Host logic
According to the Apache mod_proxy docs, X-Forwarded-Host can be a
comma-separated list of hosts, to which each proxy appends the requested
host. We want to grab only the first from this comma-separated list
to get the original requested Host when building URLs.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-24 14:16:49 -07:00
Olivier Gambier
9b6d49fdb0 Merge pull request #430 from stevvooe/address-s3-timeout-error
Attempt to address intermittent s3 RequestTimeout error
2015-04-24 11:20:30 -07:00
Stephen Day
528869ac03 Merge pull request #420 from stevvooe/expose-drivernames
Expose storage driver names for tracing
2015-04-24 10:53:40 -07:00
Stephen J Day
e6715c5cec Attempt to address intermittent s3 RequestTimeout error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 20:07:32 -07:00
Stephen J Day
6dd27cd2cb Correctly check s3 chunksize parameter
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 16:31:41 -07:00
Stephen J Day
f6d75a6139 Return after error in handler
This adds a missing return statement. It is not strictly needed since if the
io.Copy fails, the Finish operation will fail. Currently, the client reports
both errors where this new code will correctly only report the io.Copy error.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 13:13:13 -07:00
Stephen J Day
d7eb17fd72 Include driver name in trace messsages
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:31 -07:00
Stephen J Day
9e146437e4 Require storage drivers to report their name
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:01 -07:00
Stephen Day
73960f4024 Merge pull request #419 from stevvooe/pool-buffers-s3
Pool buffers used in S3.WriteStream
2015-04-22 16:41:28 -07:00
Stephen J Day
1adab029f6 Pool buffers used in S3.WriteStream
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 16:25:53 -07:00
Olivier Gambier
506a5b4a97 Merge pull request #418 from stevvooe/accept-chunksize-parameter
Allow configuration of chunksize parameter
2015-04-22 14:46:40 -07:00
Stephen J Day
72a702d0d9 Allow configuration of chunksize parameter
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.

This code likely never worked. The configuration system should be decoupled
from the object instantiation.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 14:31:34 -07:00
Stephen J Day
4330627c71 Check error returned from io.Copy
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 12:35:08 -07:00
Stephen Day
aa09c6c262 Merge pull request #408 from stevvooe/increase-retry-backoff
Backoff retry on verification to give s3 time to propagate
2015-04-21 13:19:28 -07:00
Stephen J Day
632e46418b Backoff retry on verification to give s3 time to propagate
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 13:04:46 -07:00
Stephen J Day
d8dbcc3efa log canonical digest on verification error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 11:34:18 -07:00
Olivier Gambier
1b975e980b Merge pull request #405 from stevvooe/eventual-consistency
Attempt to deal with eventual consistency by retrying
2015-04-20 18:59:01 -07:00
Stephen J Day
c8345090d9 Attempt to deal with eventual consistency by retrying
Rather than accept the resulting of a layer validation, we retry up to three
times, backing off 100ms after each try. The thought is that we allow s3 files
to make their way into the correct location increasing the liklihood the
verification can proceed, if possible.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-20 18:49:22 -07:00
Richard
3a1029a262 Add logging for generic handler errors.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-20 17:47:52 -07:00
Stephen Day
7c25fcf698 Merge pull request #375 from RichardScothern/path-logging
registry/storage/driver: add path and other info to filesytem trace methods.
2015-04-17 14:09:23 -07:00
Richard
6b1c2e9b1f Add path and other info to filesytem trace methods.
Also fix Delete (was 'Move').
2015-04-17 13:55:18 -07:00
bin liu
e0521d2d01 fix some typos in source comments
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-04-17 12:39:52 +00:00
Stephen Day
cce1641f9b Merge pull request #333 from RichardScothern/purgeuploads
registry/storage: automatically purge old upload files
2015-04-16 11:16:51 -07:00
Richard
e4f07c368b Automatically purge old upload files.
When the registry starts a background timer will periodically
scan the upload directories on the file system every 24 hours
and delete any files older than 1 week. An initial jitter
intends to avoid contention on the filesystem where multiple
registries with the same storage driver are started
simultaneously.
2015-04-16 10:57:34 -07:00
Stephen Day
c5183a446b Merge pull request #369 from stevvooe/http-request-status-context-manager
context, registry/handlers: instantiate http context before dispatch
2015-04-15 23:23:40 -07:00
Stephen Day
75983a4a7f Merge pull request #364 from ncdc/resumable-digest-optional
Use a build flag to disable resumable digests
2015-04-15 23:22:25 -07:00
Stephen J Day
7420a3de10 Instantiate http context before dispatch
Ensure that the status is logged in the context by instantiating before the
request is routed to handlers. While this requires some level of hacking to
acheive, the result is that the context value of "http.request.status" is as
accurate as possible for each request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-15 19:53:36 -07:00
Andy Goldstein
7fdd395653 Use a build flag to disable resumable digests.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-04-16 01:19:57 +00:00
Derek McGowan
21504560e1 Update final upload chunk api doc
Updates description about content length and location

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-15 17:55:15 -07:00
Richard
cb6a6125f6 Add auth.user.name to logging context 2015-04-15 10:34:24 -07:00
Richard Scothern
90af0f9b7a Merge pull request #343 from stevvooe/tracing-driver
context, storagedriver: trace function calls to Base storage driver
2015-04-10 16:55:57 -07:00
Stephen J Day
730ce83cdf Trace function calls to Base storage driver
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-10 16:42:36 -07:00
Stephen Day
41dd70c901 Merge pull request #349 from RichardScothern/348
Prevent Close() from being called after Finish()
2015-04-10 16:21:01 -07:00
Richard
3d529094fa Prevent Close() from being called after Finish() 2015-04-10 16:00:05 -07:00
Derek McGowan
142d62798e Rename top level registry interface to namespace
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-09 19:21:33 -07:00
Stephen J Day
4d91e791c0 Disassociate instance id from application
This moves the instance id out of the app so that it is associated with an
instantiation of the runtime. The instance id is stored on the background
context. This allows allow contexts using the main background context to
include an instance id for log messages. It also simplifies the application
slightly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-09 18:45:39 -07:00
Stephen J Day
84046e03e0 Prevent false sharing in signature fetch
The original implementation wrote to different locations in a shared slice.
While this is theoretically okay, we end up thrashing the cpu cache since
multiple slice members may be on the same cache line. So, even though each
thread has its own memory location, there may be contention over the cache
line. This changes the code to aggregate to a slice in a single goroutine.

In reality, this change likely won't have any performance impact. The theory
proposed above hasn't really even been tested. Either way, we can consider it
and possibly go forward.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-09 14:35:10 -07:00
Olivier Gambier
5fe1039947 Merge pull request #332 from stevvooe/case-sensitive-backend
registry/storage/driver: defer case-sensitive support to storage backend
2015-04-07 16:24:38 -07:00
Stephen J Day
e23ca5ac5f Defer case-sensitive support to storage backend
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:

1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.

The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-07 14:14:45 -07:00
Olivier Gambier
4e2d176d27 Merge pull request #330 from stevvooe/parallelize-signature-fetch
registry/storage: parallelize signature fetch in signature store
2015-04-06 22:30:35 -07:00
Josh Hawn
0eb599329a Handle cloudFront bucket prefix issue
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-06 19:46:05 -07:00
Stephen J Day
1eab4b79bc Parallelize signature fetch in signature store
To avoid compounded round trips leading to slow retrieval of manifests with a
large number of signatures, the fetch of signatures has been parallelized. This
simply spawns a goroutine for each path, coordinated with a sync.WaitGroup.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-06 19:13:15 -07:00
Olivier Gambier
1963b7902e Merge pull request #312 from stevvooe/add-layer-info-cache
registry: integrate layer info cache with registry and storage
2015-04-03 13:53:27 -07:00
Stephen J Day
09d48c7192 Move expvar under the registry section
For consistency with other systems, the redis and caching monitoring data has
been moved under the "registry" section in expvar. This ensures the entire
registry state is kept to a single section.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 21:30:27 -07:00
Stephen J Day
919d972e65 Allow control over which storage cache to use
This allows one to better control the usage of the cache and turn it off
completely. The storage configuration module was modified to allow parameters
to be passed to just the storage implementation, rather than to the driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
44b14ceadc Integrate layer info cache with registry and storage
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.

This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
b1f616cbff Define and implement layer info cache
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.

Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:09 -07:00
Stephen Day
cf5839b019 Merge pull request #295 from jlhawn/use_resumable_digest
digest, registry/storage: use resumable digest
2015-04-02 20:12:57 -07:00
Stephen Day
7c3b73482f Merge pull request #314 from stevvooe/lockdown-uuid-route
registry/api/v2: stronger validation for uuid field in urls
2015-04-02 13:57:17 -07:00
Josh Hawn
18c9a1cdd8 Use resumable digest for efficient upload finish
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-02 11:20:31 -07:00
Stephen J Day
f26a283a48 Avoid crash on invalid Move arguments
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 19:03:02 -07:00
Stephen J Day
1944be9db3 Stronger validation for uuid field in urls
This change adds strong validation for the uuid variable for v2 routes. This is
a minor specification change but is okay since the uuid field is controlled by
the server. The character set is restricted to avoid path traversal, allowing
for alphanumeric values and urlsafe base64 encoding.

This change has no effect on client implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 18:57:59 -07:00
Stephen J Day
3cad3c7b6a Add redis pool to registry webapp
Redis has been integrated with the web application for use with various
services. The configuraiton exposes connection details, timeouts and pool
parameters. Documentation has been updated accordingly.

A few convenience methods have been added to the context package to get loggers
with certain fields, exposing some missing functionality from logrus.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 16:27:24 -07:00
Stephen Day
02b8c7b8cf Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
11db8185bc Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
ce9f500a31 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Josh Hawn
db5689aa86 Refactor Layer interface to return a Handler
... Rather than ServeHTTP directly.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 21:59:07 -07:00
Josh Hawn
e062e66ee1 Insert request method option storage driver URLFor
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 17:06:40 -07:00
Stephen Day
70560cceaf Merge pull request #256 from ncdc/master
Send WWW-Authenticate header for silly auth
2015-03-11 19:01:23 -07:00
Andy Goldstein
4491e755b8 Send WWW-Authenticate header for silly auth
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 15:20:15 -04:00
David Lawrence
3853e66f4b don't panic during a request when configuring repository middleware. Return a 500 with an appropriate error
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:59:02 -07:00
Andy Goldstein
b5a63d75ea Fix Godoc typos
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:29 -07:00
David Lawrence
c858a4ba89 Final polish to cloudfront and larger middleware refactor
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:29 -07:00
Andy Goldstein
a20abfbf3c Middleware!
Convert middleware in the config to be a map of type->[]Middleware

Add support for registry & repository middleware.

Some naming updates as well.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:28 -07:00
David Lawrence
4acda57e05 Refactoring cloudfactory layer handler into a more generic storage
middleware concept.

This also breaks the dependency the storage package had on goamz
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:28 -07:00
Josh Hawn
87959abe8f digest: Minor refactoring
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 16:44:19 -07:00
David Lawrence
6dcec265a0 minor refactor + tests for app.go just to improve test coverage.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-09 16:31:30 -07:00
Andy Goldstein
e1e724860f Remove max repository component length restriction
Fixes #241

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-06 19:55:33 -05:00
Stephen Day
f0ccdd448f Merge pull request #239 from jlhawn/event_target_update
notifications: update notification event Target fields
2015-03-06 16:45:06 -08:00
Stephen Day
e3dd701004 Merge pull request #238 from endophage/canonical_sha256
digest, registry/storage, registry/handlers: switch to SHA256 as canonical digest
2015-03-06 15:27:14 -08:00
David Lawrence
b777e389b9 fixing up tests to work with for non-tarsum future
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-06 14:54:23 -08:00
Stephen Day
81a03041d4 Merge pull request #243 from stevvooe/storagedriver-docs
doc: move storage driver readmes into docs
2015-03-06 11:25:42 -08:00
Josh Hawn
d3bc4c4b38 Switch to SHA256 as canonical digest
Also support client digests linking to canonical digest.
2015-03-06 09:49:05 -08:00
Josh Hawn
a877811c0b Update notification event Target fields
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-05 18:01:50 -08:00
Stephen J Day
41b1e22f97 doc: move storage driver readmes into docs
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-05 17:23:33 -08:00
Stephen J Day
40273b1d36 Implement immutable manifest reference support
This changeset implements immutable manifest references via the HTTP API. Most
of the changes follow from modifications to ManifestService. Once updates were
made across the repo to implement these changes, the http handlers were change
accordingly. The new methods on ManifestService will be broken out into a
tagging service in a later PR.

Unfortunately, due to complexities around managing the manifest tag index in an
eventually consistent manner, direct deletes of manifests have been disabled.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:40:55 -08:00
Stephen J Day
f536633ca8 spec: fetch manifests by tag or digest
Manifests are now fetched by a field called "reference", which may be a tag or
a digest. When using digests to reference a manifest, the data is immutable.
The routes and specification have been updated to allow this.

There are a few caveats to this approach:

1. It may be problematic to rely on data format to differentiate between a tag
   and a digest. Currently, they are disjoint but there may modifications on
   either side that break this guarantee.
2. The caching characteristics of returned content are very different for
   digest versus tag-based references. Digest urls can be cached forever while tag
   urls cannot.

Both of these are minimal caveats that we can live with in the future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:39:36 -08:00
Stephen Day
91403c1b52 Merge pull request #234 from ncdc/signature-service
Expose Signatures() on Repository
2015-03-04 18:24:36 -08:00
Josh Hawn
e249b478d9 Merge pull request #231 from stevvooe/remove-unnecessary-close
Remove unnecessary close in client
2015-03-04 18:19:35 -08:00
Andy Goldstein
6b12e34a4b Expose Signatures() on Repository
Add a SignatureService and expose it via Signatures() on Repository so
external integrations wrapping the registry can access signatures.

Move signature related code from revisionstore.go to signaturestore.go.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-04 20:56:11 -05:00
Stephen J Day
dc0408767a Remove unnecessary close in client
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-03 19:29:12 -08:00
Stephen Day
bfef2046e8 Merge pull request #218 from endophage/DIST-148
registry/storage: buffered wrapper for fileWriter
2015-03-03 18:08:28 -08:00
Stephen Day
72b48812b4 Merge pull request #212 from stevvooe/add-documentation-name-validator
documentation for name validation grammar
2015-03-03 17:32:54 -08:00
David Lawrence
5161d69ebf wrap buffered writer around filewriter
benchmarks added to filewriter_test, demonstrate buffered
version is ~5x faster on my hardware.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-03 16:23:21 -08:00
Stephen J Day
6add5f30d2 documentation for name validation grammar
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-02 10:55:31 -08:00
Stephen Day
16d8b2c34d Merge pull request #213 from stevvooe/docker-upload-uuid
doc/spec, registry/handlers: specify and implement Docker-Upload-UUID
2015-03-02 10:51:15 -08:00
David Lawrence
6f1b92621c Adding test cases to confirm path traversal attempts are mitigated and
bad characters in URI return 404
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-02 09:15:49 -08:00
Stephen J Day
996235dc59 Specify and implement Docker-Upload-UUID
This changeset adds support for a header to identify docker upload uuids. This
id can be used as a key to manage local state for resumable uploads. The goal
is remove the necessity for a client to parse the url to get an upload uuid.
The restrictions for clients to use the location header are still strongly in
place.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 16:43:47 -08:00
David Lawrence
1700f518cb Path prefix support for running registry somewhere other than root of server
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-26 16:04:43 -08:00
Stephen Day
5c4f03d38e Merge pull request #189 from donhcd/rootcertbundle
Rename auth.token.{rootCertBundle -> rootcertbundle}
2015-02-19 19:31:55 -08:00
Stephen Day
c3536ea1c5 Merge pull request #191 from stevvooe/missing-digest-bad-request
A digest missing parameter should result in a bad request
2015-02-19 18:30:58 -08:00
Stephen J Day
c2b20907d9 A digest missing parameter should result in a bad request
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-19 17:14:25 -08:00
Donald Huang
77de18f751 Rename auth.token.rootCertBundle yml field
Renames auth.token.rootCertBundle field in registry config to rootcertbundle so
that the REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE environment variable will override it.

See
()[https://github.com/docker/distribution/blob/master/configuration/parser.go#L155]

Signed-off-by: Donald Huang <don.hcd@gmail.com>
2015-02-20 00:46:24 +00:00
Andrey Kostov
df71f3451a Fix S3 driver's list when the root directory is either "" or "/" 2015-02-19 16:31:34 -08:00
Andrey Kostov
da5510b55e Add an empty root directory s3 driver specific test 2015-02-19 16:28:32 -08:00
Stephen J Day
5d029fb807 Add error return to Repository method on Registry
The method (Registry).Repository may now return an error. This is too allow
certain implementationt to validate the name or opt to not return a repository
under certain conditions.

In conjunction with this change, error declarations have been moved into a
single file in the distribution package. Several error declarations that had
remained in the storage package have been moved into distribution, as well. The
declarations for Layer and LayerUpload have also been moved into the main
registry file, as a result.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-13 16:27:33 -08:00
Stephen J Day
d2d46fca41 Move notifications package to distribution
Since the notifications package is now decoupled from storage, we are moving it
to the root package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:40:35 -08:00
Stephen J Day
286a644948 Remove Name from Layer and LayerUpload interface
A Layer or LayerUpload should not be coupled with the containing repository.
Remove the Name method and correctly reference from the repository where
appropriate.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:27:05 -08:00
Stephen J Day
27b03f2136 Move layer interface definitions to distribution package
After consideration, it has been decided that the interfaces defined in the
storage package provide a good base for interacting with various registry
instances. Whether interacting with a remote API or a local, on-disk registry,
these types have proved flexible. By moving them here, they can become the
central components of interacting with distribution components.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:26:46 -08:00
Ahmet Alp Balkan
76f1abfa07 storagedriver/azure: Add README
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-02-11 14:40:51 -08:00
Stephen J Day
f74b9852fe Run goimports/gofmt on previous changes
After all of the perl refactoring, some import orderings were left asunder.
This commit corrects that.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
65b0d73cb7 Move storagedriver package to registry/storage/driver
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
3468fbd4a8 Move storage package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
2832ec7f5c Move client package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:42:56 -08:00
Stephen J Day
0371f648bf Move auth package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 17:34:04 -08:00
Stephen J Day
e4b811f489 Move registry api definitions under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 17:32:22 -08:00
Stephen J Day
0f08b6961a Move registry package into handler package
The goal is to free up the distribution/registry package to include common
registry types. This moves the webapp definitions out of the way to allow for
this change in the future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 17:25:40 -08:00
Stephen J Day
287de9f60a Correctly return when repo name is not available
The branch that executes after a failed request authorization due to a missing
repo name now correctly returns an error. This is somewhat superficial since
the response would have already been executed. Although, unintended repository
operations may have occurred.

Documentations and comments have also been updated to be in line with
surrounding changes.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-10 15:19:02 -08:00
Stephen J Day
d9d84ae269 Integrate context with storage package
This changeset integrates context with the storage package. Debug messages have
been added to exported methods. Existing log messages will now include
contextual details through logger fields to aid in debugging. This integration
focuses on logging and may be followed up with a metric-oriented change in the
future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-09 14:44:58 -08:00
Stephen J Day
d2bfb5825c Integrate contextual logging with regsitry app
This changeset integrates contextual logging into the registry web application.
Idiomatic context use is attempted within the current webapp layout. The
functionality is centered around making lifecycle objects (application and
request context) into contexts themselves. Relevant data has been moved into
the context where appropriate.  We still have some work to do to factor out the
registry.Context object and the dispatching functionality to remove some
awkward portions.

The api tests were slightly refactored to use a test environment to eliminate
common code.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-06 16:51:53 -08:00
Stephen Day
9b9ea55797 Merge pull request #140 from jlhawn/137_auth_context
Use context for auth access controllers
2015-02-04 19:25:10 -08:00