Commit graph

83 commits

Author SHA1 Message Date
Adrian Mouat
f670f00058 Fix typo
I think negociate is very occasionally used, but it certainly looks wrong to me.

Signed-off-by: Adrian Mouat <adrian.mouat@gmail.com>
2015-06-11 21:45:26 +01:00
Olivier Gambier
ad23a43bc4 Enhance building doc to reflect the new data default location
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-11 11:08:16 -07:00
Olivier Gambier
8ed0c66745 Enhancing doc to persist data
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-11 01:41:15 -07:00
Olivier Gambier
6e0cfc17dc Saner default data location
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-10 20:54:24 -07:00
Olivier Gambier
b230183b0f Merge pull request #553 from dmp42/docs-rework
Docs rework
2015-06-09 14:48:39 -07:00
Stephen Day
f63313de1f Merge pull request #599 from stevvooe/clarify-deletion-by-digest-constraint
Clarify digest in API specification
2015-06-08 19:04:53 -07:00
Stephen J Day
7e6b4e8c52 Add description of digests to API specification
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-08 19:04:13 -07:00
Olivier Gambier
25bd3fc777 Link to compose
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:57:06 -07:00
Olivier Gambier
b8d785c79e Use bcrypt
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:57:06 -07:00
Olivier Gambier
81e8657d7b Fix here doc conf generation
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:57:05 -07:00
Olivier Gambier
8c1784c838 Keeping in synch
- commenting out both the "JSON" and "token" specs, unless someone thinks they should be here
- added help, glossary, introduction and authentication documents

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:57:05 -07:00
Olivier Gambier
96d79eb30e Adding authentication
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:57:05 -07:00
Olivier Gambier
c1c638d01f Minor keyword add
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:52 -07:00
Olivier Gambier
c405f3717a Minor fixes
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:52 -07:00
Olivier Gambier
56ff32c683 Revising glossary
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:52 -07:00
Olivier Gambier
10ba376a99 Minor cleanup in the index
- adding glossary
- removing empty "advanced"
- commenting out building and architecture for now
- minor text enhancements

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
c3b3802503 Small fixes
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
b7b8e64f1d Moved instructions up
Make it more obvious that environment variables is the way to go.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
26d87ed1a5 No reason to detail here
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
431e829fa5 Simplified index
Simplified index again, to make access to information more obvious and more direct.
Added a TLDR for people in a hurry.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
d9dfe54308 Separated "help"
Help page is separate now.
Removed the bulk of it from various pages.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
4eb8d907b9 Separate introduction
Use cases, generalities, image naming overview.
Removed most of it from index, and some of it from other random pages.

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Olivier Gambier
2fb5c97a97 Deployment rewrote
Strong focus on simplification.
Entirely removed custom build instructions.
Providing sane, one-liner defaults.
Verified, easy-to-use TLS instructions.
Removed hybrid instructions.
Removed authentication with nginx instructions (either wait for native support, or move it to advanced topics).

Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-08 15:55:51 -07:00
Vincent Batts
0165b85861 spec/manifest: clarify the v2-s1 is provisional
Signed-off-by: Vincent Batts <vbatts@redhat.com>
2015-06-08 16:48:48 -04:00
Ahmet Alp Balkan
89f183639a docs: Update azure docs in configuration.md
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-06-05 17:25:17 -07:00
Stephen J Day
c54adb667f Clarify that manifests can only be deleted by digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-04 18:59:08 -07:00
Stephen J Day
93aff60741 Disable building of azure storage driver by default
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-03 18:14:53 -07:00
Olivier Gambier
00388a14aa Merge pull request #591 from stevvooe/building-environment
Build environment requires proper checkout of project
2015-06-03 12:24:44 -07:00
Stephen J Day
4700c27eb5 Build environment requires proper checkout of project
This clarifies the importance of properly setting a Go build environment when
building targets. Typically, users seem to editorialize the checkout location,
either ignoring the first section or have limited experience with the Go
development environment. We clarify the checkout requirements and point to the
documentation on how to setup Go.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-03 11:59:56 -07:00
Vincent Giersch
7b3f571555 doc: coherence between requests and parameters + typo
In the request parameters lists `tag` was used instead of
`reference` present in the HTTP requests paths

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-06-02 14:36:55 +00:00
Stephen Day
0d40913b9a Merge pull request #394 from xiekeyang/feature-panic-hook
Feature: Add Hook for Web Application Panic
2015-06-01 13:23:32 -07:00
W. Trevor King
e1c553fee9 docs/storagedrivers.md: Update filesystem.go -> driver.go reference
Catch up after 88795e0a (Lots of various golint fixes, 2014-11-17, #1)
and 65b0d73c (Move storagedriver package to registry/storage/driver,
2015-02-10, #167).

Signed-off-by: W. Trevor King <wking@tremily.us>
2015-05-31 15:23:31 -07:00
xiekeyang
47aa47e3f6 Feature: Web Panic Reporting via hooks
This PR is for issue of "email after registry webapp panic" #41, improving my
previous design (closed).
It use self setting up hooks, to catch panic in web application.
And, send email in hooks handle directly, to no use new http server and
handler.

Signed-off-by: xiekeyang <keyangxie@126.com>
2015-05-31 14:21:22 +00:00
Stephen Day
318af0b1ce Merge pull request #443 from gierschv/driver-rados
Storage Driver: Ceph Object Storage (RADOS)
2015-05-29 18:10:49 -07:00
Stephen Day
63d2e51529 Merge pull request #540 from konstruktoid/patch-1
sha256 when generating certificates
2015-05-29 17:09:26 -07:00
Oilbeater
b78b395ae2 Fix typo
Signed-off-by: Mengxin Liu <liumengxinfly@gmail.com>
2015-05-26 21:19:49 +08:00
Thomas Sjögren
58981a4535 increase key size to 4096. @diogomonica
Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
2015-05-24 22:02:25 +02:00
Olivier Gambier
8b537e0888 Revamp index
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-05-20 16:15:20 -07:00
Stephen J Day
dfa8504408 Revise description of debug endpoint
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 15:24:25 -07:00
Thomas Sjögren
e39583cc9d sha256 when generating certificates
Small detail, but when generating certificates using sha256 is recommended. See for example http://googleonlinesecurity.blogspot.se/2014/09/gradually-sunsetting-sha-1.html.

Signed-off-by: Thomas Sjögren <konstruktoid@users.noreply.github.com>
2015-05-20 23:44:17 +02:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Mary Anthony
e1a1e3a8da Fixes #484
Signed-off-by: Mary Anthony <mary@docker.com>
2015-05-18 13:33:29 -07:00
Mary Anthony
8d407c81ae Closes #485
Signed-off-by: Mary Anthony <mary@docker.com>

Tweak per Stephen

Signed-off-by: Mary Anthony <mary@docker.com>
2015-05-18 12:56:17 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Anton Tiurin
ad80cbe1ea [Server] Listen and serve on a unix socket
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-05-11 16:00:14 +03:00
Adam Enger
0180615a82 Fixing grammatical mistake in docs/deploying.md
Signed-off-by: Adam Enger <adamenger@gmail.com>
2015-05-08 11:27:57 -05:00
Stephen Day
fbd022e452 Merge pull request #475 from dmcgowan/patch-support
Modify blob upload API
2015-05-06 16:12:54 -07:00
Derek McGowan
d9003dfc25 Add documentation for client version header
Add documentation for Docker-Distribution-API-Version header required by clients

closes #99

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:58:48 -07:00
Derek McGowan
e842662ede Move pre-release 2.0 changes into the 2.0 API specification
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:58:48 -07:00
Derek McGowan
2cf40ab790 Modify blob upload API
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.

closes #470

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:43:23 -07:00