Commit graph

1108 commits

Author SHA1 Message Date
Derek McGowan
6f9fbf99a9 Split layer and upload from repository
Layer upload moved to its own file with its own unit tests

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
b1ba2183ee Add unit tests for auth challenge and endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
174a732c94 Remove deprecated client interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
ce614b6de8 Add client implementation of distribution interface
Adds functionality to create a Repository client which connects to a remote endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Stephen Day
ace71ea9c9 Merge pull request #519 from stevvooe/blob-service-refactor
Refactor Blob Service API
2015-05-15 17:52:15 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Stephen Day
aeda6f5f8f Merge pull request #518 from RichardScothern/cache-headers
Set cache headers for layers.
2015-05-12 18:32:16 -07:00
Richard
3dc2d849c7 Set cache headers for layers.
- Set an Etag header
     - Check If-None-Match and respond appropriately
     - Set a Cache-Control header with a default of 1 week

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-12 17:49:18 -07:00
Stephen Day
7c2d0ee81d Merge pull request #491 from noxiouz/unix_socket_listener
cmd/registry, registry/listener: listen and serve on a unix socket
2015-05-12 11:34:30 -07:00
Stephen Day
1a829c8f51 Merge pull request #516 from burnettk/refer-to-correct-docs-directory
docs/spec: correct doc/spec to docs/spec everywhere
2015-05-11 14:45:58 -07:00
Stephen Day
f2954260a6 Merge pull request #515 from davidxia/fix-link
Fix broken ROADMAP link
2015-05-11 14:33:29 -07:00
burnettk
b0eb769ed4 correct doc/spec to docs/spec everywhere
Signed-off-by: Kevin Burnett <burnettk@gmail.com>
2015-05-11 15:30:56 -04:00
David Xia
c75c57afa6 Fix broken ROADMAP link
Signed-off-by: David Xia <dxia@spotify.com>
2015-05-11 13:54:20 -04:00
Anton Tiurin
ad80cbe1ea [Server] Listen and serve on a unix socket
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-05-11 16:00:14 +03:00
Stephen Day
ced8a0378b Merge pull request #507 from adamenger/patch-1
Fixing grammatical mistake in docs/deploying.md
2015-05-08 15:22:07 -07:00
Adam Enger
0180615a82 Fixing grammatical mistake in docs/deploying.md
Signed-off-by: Adam Enger <adamenger@gmail.com>
2015-05-08 11:27:57 -05:00
Stephen Day
fbd022e452 Merge pull request #475 from dmcgowan/patch-support
Modify blob upload API
2015-05-06 16:12:54 -07:00
Derek McGowan
d9003dfc25 Add documentation for client version header
Add documentation for Docker-Distribution-API-Version header required by clients

closes #99

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:58:48 -07:00
Derek McGowan
e842662ede Move pre-release 2.0 changes into the 2.0 API specification
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:58:48 -07:00
Stephen Day
5c36ab3ab5 Merge pull request #500 from stevvooe/documentation-issues
Remove configuration/README.md
2015-05-06 15:53:48 -07:00
Derek McGowan
2cf40ab790 Modify blob upload API
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.

closes #470

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:43:23 -07:00
Stephen Day
2a87663c19 Merge pull request #490 from pkill/httpd-response-header-2.2-compat
Updating the Request Header directive for compatibility with httpd 2.2
2015-05-06 15:40:33 -07:00
Stephen J Day
aac8e0d0bf Remove configuration/README.md
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-06 15:31:12 -07:00
Stephen Day
f210b090bf Merge pull request #494 from RichardScothern/bad-digest
Ensure the instrumentedResponseWriter correctly sets the http status in the context.
2015-05-05 20:34:56 -07:00
Richard
70f853e787 Ensure the instrumentedResponseWriter correctly sets the http
status in the context.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-05 16:46:33 -07:00
Stephen Day
cf2fa8678f Merge pull request #497 from RichardScothern/disable-coveralls
Disable code coverage in the build system because of coveralls errors.
2015-05-05 16:26:19 -07:00
Richard
c4cac824ed Disabled coveralls reporting: build breaking sending coverage data to coveralls
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-05 15:29:37 -07:00
Stephen Day
dd58ffe6ed Merge pull request #410 from RichardScothern/driver-context
Add golang/x/net/context.Context to storage driver method calls
2015-05-04 19:18:30 -07:00
Alex Elman
7a4e5f3723 Updating the Request Header directive for compatibility with httpd 2.2
Signed-off-by: Alex Elman <aelman@indeed.com>
2015-05-04 15:19:53 -05:00
Stephen Day
5257e2dc9e Merge pull request #476 from dmcgowan/update-spec-sha256
Update API spec to reference digest instead of tarsum
2015-05-01 17:51:45 -07:00
Derek McGowan
7d6e6aa980 Update API spec to reference digest instead of tarsum
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-01 17:14:38 -07:00
Stephen Day
5e9bda2506 Merge pull request #472 from moxiegirl/fix-471-docs-error
Fixes Issue #471 with Publish
2015-04-30 17:26:38 -07:00
Mary Anthony
68c0682e00 Fixes Issue #471 with Publish
- Add sed to Dockerfile; this sed exists on publish script; breaks headings/nav in files without metadata
- Ensure sed runs over storage-driver/ subdir
- Add metadata to all the files (including specs) that don't have it; this ensures they display correctly on publish
- Implement the fix for the showing up in Github
- Update template with GITHUB IGNORES

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-30 15:39:40 -07:00
Stephen Day
3283ef1949 Merge pull request #474 from stevvooe/goroutine-leak-notifications
Address possible goroutine leak in notification library
2015-04-29 18:48:15 -07:00
Stephen J Day
2896c32d39 Address possible goroutine leak in notification library
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-29 18:31:01 -07:00
Stephen Day
5556cd1ba1 Merge pull request #260 from dmcgowan/short-digests
Add short digest implementation
2015-04-29 14:05:41 -07:00
Richard Scothern
b5bae799e3 Merge pull request #463 from stevvooe/free-leaked-goroutine
registry/storage/driver/s3: use done channel to avoid goroutine leak
2015-04-28 15:38:40 -07:00
Stephen Day
4bf55c26f1 Merge pull request #464 from moxiegirl/adjust-storagedriver-heads
Fixing headings
2015-04-28 15:26:29 -07:00
Mary Anthony
7c93c8c265 Fixing headings
Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-28 15:10:52 -07:00
Stephen J Day
69e477f2d5 Use done channel to avoid goroutine leak
This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-28 14:40:20 -07:00
Stephen Day
1f071786d4 Merge pull request #438 from kelseyhightower/document-osx-setup
docs: document running Registry natively on OS X
2015-04-28 10:36:33 -07:00
Stephen Day
de1f95fdba Merge pull request #457 from YannRobert/patch-2
fixed missing word in CONTRIBUTING.md
2015-04-28 10:35:29 -07:00
Yann ROBERT
f5de8ea640 fixed missing word in CONTRIBUTING.md
Signed-off-by: Yann ROBERT <yann.robert@anantaplex.fr>
2015-04-28 18:13:23 +02:00
Stephen Day
34bc659a79 Merge pull request #449 from moxiegirl/update-env-config
Updating env var documentation
2015-04-27 16:01:15 -07:00
Richard
ae216e365a Make Storage Driver API calls context aware.
- Change driver interface to take a context as its first argument
     - Make newFileReader take a context as its first argument
     - Make newFileWriter take a context as its first argument
     - Make blobstore exists and delete take a context as a first argument
     - Pass the layerreader's context to the storage layer
     - Pass the app's context to purgeuploads
     - Store the app's context into the blobstore (was previously null)
     - Pass the trace'd context to the storage drivers

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 15:58:58 -07:00
Mary Anthony
5043bdbc93 Updating env var documentation
The position was a bit too early. Fleshed out the example.
Also, using the _ underscore for emphasis was confusing because it is also used to indicate
a level change.

Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-27 15:49:19 -07:00
Olivier Gambier
fab7e7f2c2 Merge pull request #446 from moxiegirl/update-per-397
Updating configuration with required header
2015-04-27 15:14:11 -07:00
Mary Anthony
8ae13b4725 Updating configuration with required header
Signed-off-by: Mary Anthony <mary@docker.com>
2015-04-27 14:46:33 -07:00
Stephen Day
70427e3d9f Merge pull request #381 from RichardScothern/purge-config
Add configuration for upload purging
2015-04-27 14:33:34 -07:00
Stephen Day
a9f93d6157 Merge pull request #439 from xiekeyang/superfluous-embedded-name
simplify the embedded method expression of repository
2015-04-27 14:31:26 -07:00