Commit graph

51 commits

Author SHA1 Message Date
W. Trevor King
e1c553fee9 docs/storagedrivers.md: Update filesystem.go -> driver.go reference
Catch up after 88795e0a (Lots of various golint fixes, 2014-11-17, #1)
and 65b0d73c (Move storagedriver package to registry/storage/driver,
2015-02-10, #167).

Signed-off-by: W. Trevor King <wking@tremily.us>
2015-05-31 15:23:31 -07:00
Stephen Day
318af0b1ce Merge pull request #443 from gierschv/driver-rados
Storage Driver: Ceph Object Storage (RADOS)
2015-05-29 18:10:49 -07:00
Stephen Day
63d2e51529 Merge pull request #540 from konstruktoid/patch-1
sha256 when generating certificates
2015-05-29 17:09:26 -07:00
Oilbeater
b78b395ae2 Fix typo
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
2015-05-26 21:19:49 +08:00
Thomas Sjögren
58981a4535 increase key size to 4096. @diogomonica
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
2015-05-24 22:02:25 +02:00
Olivier Gambier
8b537e0888 Revamp index
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-05-20 16:15:20 -07:00
Stephen J Day
dfa8504408 Revise description of debug endpoint
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 15:24:25 -07:00
Thomas Sjögren
e39583cc9d sha256 when generating certificates
Small detail, but when generating certificates using sha256 is recommended. See for example http://googleonlinesecurity.blogspot.se/2014/09/gradually-sunsetting-sha-1.html.

Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
2015-05-20 23:44:17 +02:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Mary Anthony
e1a1e3a8da Fixes #484
Signed-off-by: Mary Anthony <mary@docker.com>
2015-05-18 13:33:29 -07:00
Mary Anthony
8d407c81ae Closes #485
Signed-off-by: Mary Anthony <mary@docker.com>

Tweak per Stephen

Signed-off-by: Mary Anthony <mary@docker.com>
2015-05-18 12:56:17 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Anton Tiurin
ad80cbe1ea [Server] Listen and serve on a unix socket
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-05-11 16:00:14 +03:00
Adam Enger
0180615a82 Fixing grammatical mistake in docs/deploying.md
Signed-off-by: Adam Enger <adamenger@gmail.com>
2015-05-08 11:27:57 -05:00
Stephen Day
fbd022e452 Merge pull request #475 from dmcgowan/patch-support
Modify blob upload API
2015-05-06 16:12:54 -07:00
Derek McGowan
d9003dfc25 Add documentation for client version header
Add documentation for Docker-Distribution-API-Version header required by clients

closes #99

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:58:48 -07:00
Derek McGowan
e842662ede Move pre-release 2.0 changes into the 2.0 API specification
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:58:48 -07:00
Derek McGowan
2cf40ab790 Modify blob upload API
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.

closes #470

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:43:23 -07:00
Stephen J Day
aac8e0d0bf Remove configuration/README.md
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-06 15:31:12 -07:00
Derek McGowan
7d6e6aa980 Update API spec to reference digest instead of tarsum
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-01 17:14:38 -07:00
Mary Anthony
68c0682e00 Fixes Issue #471 with Publish
- Add sed to Dockerfile; this sed exists on publish script; breaks headings/nav in files without metadata
- Ensure sed runs over storage-driver/ subdir
- Add metadata to all the files (including specs) that don't have it; this ensures they display correctly on publish
- Implement the fix for the showing up in Github
- Update template with GITHUB IGNORES

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-30 15:39:40 -07:00
Mary Anthony
7c93c8c265 Fixing headings
Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-28 15:10:52 -07:00
Stephen Day
1f071786d4 Merge pull request #438 from kelseyhightower/document-osx-setup
docs: document running Registry natively on OS X
2015-04-28 10:36:33 -07:00
Mary Anthony
5043bdbc93 Updating env var documentation
The position was a bit too early. Fleshed out the example.
Also, using the _ underscore for emphasis was confusing because it is also used to indicate
a level change.

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-27 15:49:19 -07:00
Stephen Day
70427e3d9f Merge pull request #381 from RichardScothern/purge-config
Add configuration for upload purging
2015-04-27 14:33:34 -07:00
Stephen Day
7ba2bdb7dc Merge pull request #444 from RichardScothern/docs
Add registry configuration environment variable documentation
2015-04-27 13:41:11 -07:00
Mary Anthony
d5c0300986 Pushing fix for mkdocs.yml introduced
Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-27 11:27:49 -07:00
Richard
937c356585 Add environment variable override instructions
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 11:23:37 -07:00
Richard
6460ddb2cb Add configuration for upload purging
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 11:06:15 -07:00
Kelsey Hightower
cd83b75d4c docs: document running Registry natively on OS X
Signed-off-by: Kelsey Hightower <kelsey.hightower@gmail.com>
2015-04-26 18:41:36 -07:00
Ian Babrou
824e7e8ae2 docs: fixed links for storage drivers
Signed-off-by: Ian Babrou <ibobrik@gmail.com>
2015-04-24 21:57:09 +03:00
Mary Anthony
d588ed13b2 Fixes #391 for registry top page
Fixes #390 to add building link to README
Docker Registry Service to Docker Registry

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-18 17:32:57 -07:00
Stephen Day
a9a9658e41 Merge pull request #388 from mika/master
docs: drop newlines in URLs to avoid 404
2015-04-17 16:54:09 -07:00
Shawn Falkner-Horine
85cc6bc4bb Fix deployment guide typos re: image name consistency.
Signed-off-by: Shawn Falkner-Horine <dreadpirateshawn@gmail.com>
2015-04-17 21:13:24 +00:00
Michael Prokop
647e0a48b3 docs: drop newlines in URLs to avoid 404
The docs as available at
https://github.com/docker/distribution/blob/master/docs/configuration.md
result in 404 errors:

  https://github.com/docker/distribution/blob/master/cmd/registry/%0Aconfig.yml
  http://docs.aws.amazon.com/AWSSecurityCredentials/1.0/%0AAboutAWSCredentials.html#KeyPairs

instead of pointing to the correct ones, being:

  https://github.com/docker/distribution/blob/master/cmd/registry/config.yml
  http://docs.aws.amazon.com/AWSSecurityCredentials/1.0/AboutAWSCredentials.html#KeyPairs

So avoid the newlines in the corresponding source files.

Signed-off-by: Michael Prokop <mika@grml.org>
2015-04-17 23:09:43 +02:00
Spencer Rinehart
570fedb2ab Fix registry link to point to localhost in deploy doc.
Signed-off-by: Spencer Rinehart <anubis@overthemonkey.com>
2015-04-16 17:16:42 -05:00
Ben Firshman
5e4deba5a4 Use registry:2.0 tag in docker run
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
2015-04-16 11:11:03 -07:00
Mary Anthony
b939d6d118 Updating with new RC work
Fixing typos and adding tables
Updating with testing material
Tweak after read through
Adding in Stephen's comments
Adding in Richard's comments. Fixing the broken images
closes issue #363
Another try

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-16 10:44:30 -07:00
Jessie Frazelle
399b11cf29 Merge pull request #367 from bfirsh/update-landing-page
docs: add landing page for Registry to docs
2015-04-16 10:15:54 -07:00
Ben Firshman
ec9b9d560f Add landing page for Registry to docs
- Explain why you would want to use a registry
- Explain difference between Docker Hub and Registry
- List some features of Registry
- Add table of contents for documentation - particularly "deploying a registry" call to action
- Use standard "Getting help" section from orchestration projects

Signed-off-by: Ben Firshman <ben@firshman.co.uk>
2015-04-16 10:01:36 -07:00
Stephen Day
e16c23bb43 Merge pull request #368 from dmcgowan/upload-api-docs-update
Update final upload chunk api doc
2015-04-15 19:09:00 -07:00
Derek McGowan
21504560e1 Update final upload chunk api doc
Updates description about content length and location

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-15 17:55:15 -07:00
Daisuke Fujita
4cacbb4faf Modify markdown format in distribution.md
Signed-off-by: Daisuke Fujita <dtanshi45@gmail.com>
2015-04-16 09:22:24 +09:00
Kenneth Lim
f5a104dbcd server --> serve
Possible typo?
2015-04-13 22:11:48 +08:00
Stephen Day
aa752f68a1 Merge pull request #338 from dmcgowan/nginx_v1_v2_deployment
docs: add nginx configuration for v1 and v2 registry
2015-04-10 16:35:50 -07:00
Stephen Day
ad76574d74 Merge pull request #336 from RichardScothern/docs
doc/spec: documentation for the Image Manifest V2 format
2015-04-10 16:26:27 -07:00
Richard
e564ac59c3 Documentation for the Image Manifest V2 specification 2015-04-10 16:13:32 -07:00
Derek McGowan
6f087829c9 Add nginx configuration for v1 and v2 registry
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-10 15:16:13 -07:00
Derek McGowan
142d62798e Rename top level registry interface to namespace
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-09 19:21:33 -07:00
Mary Anthony
cf9b4ab5e9 Breaking out README
Adding new material
Adding in template chomped in error
Cover install/deploy in README
Adding in Stephen's comments
Fixing you tabs!
Updating with commentary from pr
Updating with last minute comments

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-09 17:50:46 -07:00