Commit graph

948 commits

Author SHA1 Message Date
Stephen J Day
87ff947038 Decouple redis dependency from blob descriptor cache
Ensure that clients can use the blob descriptor cache provider without needing
the redis package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 18:58:10 -07:00
Derek McGowan
07ff029506 Merge pull request #543 from tiborvass/fix-typo-in-client
client: fix a typo preventing compilation
2015-05-20 18:13:17 -07:00
Tibor Vass
cb99c076db client: fix a typo preventing compilation
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2015-05-20 21:06:13 -04:00
Stephen Day
f19d1a9d12 Merge pull request #521 from docker/doc-work
Revamp index
2015-05-20 16:17:48 -07:00
Olivier Gambier
8b537e0888 Revamp index
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-05-20 16:15:20 -07:00
Stephen Day
ea54a5560a Merge pull request #387 from dmcgowan/registry-client-update
Add client implementation of distribution interface
2015-05-20 15:55:20 -07:00
Olivier Gambier
b41f93ae95 Merge pull request #541 from stevvooe/fix-debug-endpoint-description
Revise description of debug endpoint
2015-05-20 15:29:34 -07:00
Stephen J Day
dfa8504408 Revise description of debug endpoint
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 15:24:25 -07:00
Derek McGowan
68c1ceac95 Remove error message shortening
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 14:55:59 -07:00
Thomas Sjögren
e39583cc9d sha256 when generating certificates
Small detail, but when generating certificates using sha256 is recommended. See for example http://googleonlinesecurity.blogspot.se/2014/09/gradually-sunsetting-sha-1.html.

Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
2015-05-20 23:44:17 +02:00
Derek McGowan
49369ffe9a Only do auth checks for endpoints starting with v2
Changes behavior so ping doesn't happen if /v2/ is anywhere in a request path, but instead only at the beginning. This fixes attempts to ping on redirected URLs.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 13:35:23 -07:00
Derek McGowan
c7f7747368 Update transport package to sever distribution dependency
The transport package no longer requires importing distribution for the ReadSeekCloser, instead declares its own.
Added comments on the Authenication handler in session.
Added todo on http seek reader to highlight its lack of belonging to the client transport.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:09:37 -07:00
Derek McGowan
572ff64d21 Add unauthorized error check
Add check for unauthorized error code and explicitly set the error code if the content could not be parsed.
Updated repository test for unauthorized tests and nit feedback.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:05:44 -07:00
Derek McGowan
13894e8736 Break down type dependencies
Each type no longer requires holding a reference to repository.
Added implementation for signatures get.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:56:27 -07:00
Derek McGowan
a3276fcc5b Feedback update
Update comments and TODOs
Fix switch style
Updated parse http response to take in reader
Add Cancel implementation
Update blobstore variable name

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:18:30 -07:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Stephen Day
36d4e365aa Merge pull request #534 from liggitt/respository_typo
Fix typo: respository->repository
2015-05-19 12:07:24 -07:00
Jordan Liggitt
32ed2d7df1 Fix typo: respository->repository
Signed-off-by: Jordan Liggitt <jliggitt@redhat.com>
2015-05-19 13:25:39 -04:00
Stephen Day
cde7b93b85 Merge pull request #531 from moxiegirl/484-version-fix
docs: clarify minimum docker version for use with v2 API
2015-05-18 16:49:29 -07:00
Mary Anthony
e1a1e3a8da Fixes #484
Signed-off-by: Mary Anthony <mary@docker.com>
2015-05-18 13:33:29 -07:00
Stephen Day
316f3c5831 Merge pull request #530 from dmcgowan/fix-compose-nginx
Remove set header in compose script
2015-05-18 13:11:28 -07:00
Stephen Day
4d17981206 Merge pull request #529 from moxiegirl/485-issue-fix
docs: add warning about using window's volumes with registry
2015-05-18 13:10:44 -07:00
Mary Anthony
8d407c81ae Closes #485
Signed-off-by: Mary Anthony <mary@docker.com>

Tweak per Stephen

Signed-off-by: Mary Anthony <mary@docker.com>
2015-05-18 12:56:17 -07:00
Derek McGowan
beab2d7fbe Remove set header in compose script
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-18 12:05:28 -07:00
Derek McGowan
006ddd8283 Lint and documentation fixes
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
2874454224 Create client transport package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
94e375c5d1 Remove unused and duplicate error types
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
98836d6267 Update to track refactor updates
Added use of cache blob statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fdf7c8ff15 Open cache interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
70074b2286 Rename layer files to blob
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
9d64e461be Update to use blob interfaces
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fddeb1c8d5 Add missing defer on Tags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
468c5e79ba Simplify configuration and transport
Repository creation now just takes in an http.RoundTripper. Authenticated requests or requests which require additional headers should use the NewTransport function along with a request modifier (such an an authentication handler).

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
a15806ed9c Add base transport to interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
c6b51970cd Removed unused mirror flags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
68d5ecf6bf Update ReadFrom to wrap reader in NopCloser
Wrapping the reader in a NopCloser is necessary to prevent the http library from closing the input reader.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
49f7f54d07 Create authentication handler
Refactory authorizer to take a set of authentication handlers for different authentication schemes returned by an unauthorized HTTP requst.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
d92e5b1096 Add tags implementation
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
da05873b7c Use distribution context instead of google
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
c7ef45130b Cleanup session and config interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
6f9fbf99a9 Split layer and upload from repository
Layer upload moved to its own file with its own unit tests

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
b1ba2183ee Add unit tests for auth challenge and endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
174a732c94 Remove deprecated client interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
ce614b6de8 Add client implementation of distribution interface
Adds functionality to create a Repository client which connects to a remote endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Stephen Day
ace71ea9c9 Merge pull request #519 from stevvooe/blob-service-refactor
Refactor Blob Service API
2015-05-15 17:52:15 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Stephen Day
aeda6f5f8f Merge pull request #518 from RichardScothern/cache-headers
Set cache headers for layers.
2015-05-12 18:32:16 -07:00
Richard
3dc2d849c7 Set cache headers for layers.
- Set an Etag header
     - Check If-None-Match and respond appropriately
     - Set a Cache-Control header with a default of 1 week

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-12 17:49:18 -07:00
Stephen Day
7c2d0ee81d Merge pull request #491 from noxiouz/unix_socket_listener
cmd/registry, registry/listener: listen and serve on a unix socket
2015-05-12 11:34:30 -07:00
Stephen Day
1a829c8f51 Merge pull request #516 from burnettk/refer-to-correct-docs-directory
docs/spec: correct doc/spec to docs/spec everywhere
2015-05-11 14:45:58 -07:00