Commit graph

1269 commits

Author SHA1 Message Date
Stephen J Day
e809796f59 Initial implementation of Manifest HTTP API
Push, pull and delete of manifest files in the registry have been implemented
on top of the storage services. Basic workflows, including reporting of missing
manifests are tested, including various proposed response codes. Common testing
functionality has been collected into shared methods. A test suite may be
emerging but it might better to capture more edge cases (such as resumable
upload, range requests, etc.) before we commit to a full approach.

To support clearer test cases and simpler handler methods, an application aware
urlBuilder has been added. We may want to export the functionality for use in
the client, which could allow us to abstract away from gorilla/mux.

A few error codes have been added to fill in error conditions missing from the
proposal. Some use cases have identified some problems with the approach to
error reporting that requires more work to reconcile. To resolve this, the
mapping of Go errors into error types needs to pulled out of the handlers and
into the application. We also need to move to type-based errors, with rich
information, rather than value-based errors. ErrorHandlers will probably
replace the http.Handlers to make this work correctly.

Unrelated to the above, the "length" parameter has been migrated to "size" for
completing layer uploads. This change should have gone out before but these
diffs ending up being coupled with the parameter name change due to updates to
the layer unit tests.
2014-11-26 13:35:07 -08:00
Stephen J Day
4a4d403655 Correct ENV declaration in Dockerfile 2014-11-26 13:19:45 -08:00
Stephen J Day
6fead90736 Rich error reporting for manifest push
To provide rich error reporting during manifest pushes, the storage layers
verifyManifest stage has been modified to provide the necessary granularity.
Along with this comes with a partial shift to explicit error types, which
represents a small move in larger refactoring of error handling. Signature
methods from libtrust have been added to the various Manifest types to clean up
the verification code.

A primitive deletion implementation for manifests has been added. It only
deletes the manifest file and doesn't attempt to add some of the richer
features request, such as layer cleanup.
2014-11-26 12:57:14 -08:00
Stephen Day
1f772456b0 Merge pull request #787 from ahmetalpbalkan/next-generation
NG: Fix read offset check for inmemory driver
2014-11-26 10:18:53 -08:00
Ahmet Alp Balkan
c5bb224bf9 Fix read offset check for inmemory driver
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2014-11-25 21:43:12 -08:00
Olivier Gambier
59c399cb25 Merge pull request #782 from stevvooe/cleanup-storage-mapping
Clean up layer storage layout
2014-11-25 10:02:15 -08:00
Stephen J Day
68944ea9cf Clean up layer storage layout
Previously, discussions were still ongoing about different storage layouts that
could support various access models. This changeset removes a layer of
indirection that was in place due to earlier designs. Effectively, this both
associates a layer with a named repository and ensures that content cannot be
accessed across repositories. It also moves to rely on tarsum as a true
content-addressable identifier, removing a layer of indirection during blob
resolution.
2014-11-25 09:57:43 -08:00
Olivier Gambier
756989c011 Merge pull request #783 from stevvooe/next-generation
Correct references after move of Manifest into storage package
2014-11-24 17:32:12 -08:00
Stephen J Day
66d2a13aec Correct references after move of Manifest into storage package 2014-11-24 16:38:33 -08:00
Olivier Gambier
d825559473 Merge pull request #780 from stevvooe/manifest-storage
Initial implementation of image manifest storage
2014-11-24 15:27:48 -08:00
Olivier Gambier
104d203ee0 Merge pull request #750 from BrianBland/ng-push-pull
Allows layers to be partially pulled and resumed
2014-11-24 15:22:47 -08:00
Brian Bland
50d64ac63a Allows layers to be partially pulled and resumed
Adds a sort of contrived test for resumable pulls
2014-11-24 13:55:36 -08:00
Stephen J Day
4decfaa82e Initial implementation of image manifest storage
This change implements the first pass at image manifest storage on top of the
storagedriver. Very similar to LayerService, its much simpler due to less
complexity of pushing and pulling images.

Various components are still missing, such as detailed error reporting on
missing layers during verification, but the base functionality is present.
2014-11-24 13:05:27 -08:00
Stephen Day
73d6e8af84 Merge pull request #774 from docker/test-drone
Fix goverall
2014-11-24 10:24:48 -08:00
Olivier Gambier
a463f48b1a Merge pull request #776 from noxiouz/tiny_cleaninig
[IPC] Tiny cleaning
2014-11-23 19:22:56 -08:00
Anton Tiurin
4635a1e903 [IPC] Tiny cleaning 2014-11-22 23:23:03 +03:00
Olivier Gambier
b60ee6dfa0 Merge pull request #769 from stevvooe/layer-http-api
Initial implementation of Layer API
2014-11-22 10:21:23 -08:00
Olivier Gambier
ebab275b76 Fix goverall
Also simplified things a bit + adding notes about extra tooling.
2014-11-22 10:17:14 -08:00
Stephen J Day
eaadb82e1e Move Manifest type into storage package
This changeset move the Manifest type into the storage package to make the type
accessible to client and registry without import cycles. The structure of the
manifest was also changed to accuratle reflect the stages of the signing
process. A straw man Manifest.Sign method has been added to start testing this
concept out but will probably be accompanied by the more import
SignedManifest.Verify method as the security model develops.

This is probably the start of a concerted effort to consolidate types across
the client and server portions of the code base but we may want to see how such
a handy type, like the Manifest and SignedManifest, would work in docker core.
2014-11-21 19:37:44 -08:00
Stephen J Day
4bbabc6e36 Implement path spec for manifest storage 2014-11-21 19:15:35 -08:00
Stephen J Day
e158e3cd65 Initial implementation of Layer API
The http API has its first set of endpoints to implement the core aspects of
fetching and uploading layers. Uploads can be started and completed in a single
chunk and the content can be fetched via tarsum. Most proposed error conditions
should be represented but edge cases likely remain.

In this version, note that the layers are still called layers, even though the
routes are pointing to blobs. This will change with backend refactoring over
the next few weeks.

The unit tests are a bit of a shamble but these need to be carefully written
along with the core specification process. As the the client-server interaction
solidifies, we can port this into a verification suite for registry providers.
2014-11-21 19:12:20 -08:00
Stephen J Day
195568017a Update error declarations and add missing test
This updates API error codes to coincide with changes to the proposal. Mostly,
redundant error codes were merged and missing ones were added. The set in the
main errors.go file will flow back into the specification.

A test case has been added to ensure ErrorCodeUnknown is included in marshaled
json.
2014-11-21 15:24:14 -08:00
Stephen J Day
3f479b62b4 Refactor layerReader into fileReader
This change separates out the remote file reader functionality from layer
reprsentation data. More importantly, issues with seeking have been fixed and
thoroughly tested.
2014-11-21 15:24:14 -08:00
Stephen Day
74279723c0 Merge pull request #773 from docker/test-drone
Move notifications to dt
2014-11-21 12:34:22 -08:00
Olivier Gambier
96a557c8e7 Move notifications to dt 2014-11-21 12:02:34 -08:00
Stephen Day
5829d8fa6c Merge pull request #770 from noxiouz/fix_leak_in_push
[Client] Fix possible goroutine leak in push.
2014-11-21 11:50:26 -08:00
Anton Tiurin
2071422bea [Client] Fix possible goroutine leak in push.
The same as 5a804ac05b
2014-11-21 11:20:16 +03:00
Stephen Day
1caf81bf5e Merge pull request #756 from docker/test-drone
Move to own dev image
2014-11-20 19:42:31 -08:00
Olivier Gambier
efd350c3e7 Merge pull request #767 from BrianBland/ng-storagedriver-concurrency-test
Adds a test for concurrent storagedriver Write/Read Stream operations
2014-11-20 16:42:06 -08:00
Olivier Gambier
b7190a1e70 Merge pull request #766 from noxiouz/fix_possible_goroutine_leak
[Client] Fix possible goroutine leak in pull
2014-11-20 16:41:48 -08:00
Anton Tiurin
fdd053a618 Typo fix 2014-11-21 03:24:32 +03:00
Brian Bland
18eac89506 Adds a test for concurrent storagedriver Write/Read Stream operations
This test is currently failing and Skipped for IPC drivers
2014-11-20 16:15:55 -08:00
Anton Tiurin
5a804ac05b [Client] Fix possible goroutine leak.
Running goroutines with pullLayer are blocked to send error of a
pull operation. If we abort pulling without notify them about
cancelation they will get stucked forever. To avoid this possible
leak cancelCh was introduced. In case of abort we close that channel
to notify other goroutines about cancelation.
2014-11-21 03:15:34 +03:00
Olivier Gambier
3f6cb52357 Move to own dev image 2014-11-20 15:33:06 -08:00
Olivier Gambier
0111cf2bd4 Merge pull request #758 from BrianBland/ng-client-routes-update
Updates client to newer routes and changes "layer" to "blob"
2014-11-20 15:29:15 -08:00
Olivier Gambier
e98feed770 Merge pull request #764 from BrianBland/ng-storagedriver-misc-fixes
Miscellaneous storagedriver+ipc fixes
2014-11-20 15:13:13 -08:00
Brian Bland
68fd15b688 Miscellaneous storagedriver+ipc fixes
Fixes/tests listing for keys beginning with "/"
No longer extraneously wraps Closers in ioutil.NopClosers
Uses omitempty for all ipc struct type fields
2014-11-20 14:14:55 -08:00
Brian Bland
64c8bd29cc Uses new digest package instead of string digests
Also drops extraneous test package and uses testutil instead
2014-11-19 18:54:11 -08:00
Brian Bland
1336ced030 Updates client to newer routes and changes "layer" to "blob" 2014-11-19 18:37:24 -08:00
Olivier Gambier
b65d8d046e Merge pull request #755 from stevvooe/digest-type
Move Digest type into discrete package
2014-11-19 17:02:54 -08:00
Stephen J Day
56118905b8 Include testutil package needed for tar-based tests 2014-11-19 15:10:23 -08:00
Stephen J Day
c0fe9d72d1 Various adjustments to digest package for govet/golint 2014-11-19 14:59:05 -08:00
Stephen J Day
1a508d67d9 Move storage package to use Digest type
Mostly, we've made superficial changes to the storage package to start using
the Digest type. Many of the exported interface methods have been changed to
reflect this in addition to changes in the way layer uploads will be initiated.

Further work here is necessary but will come with a separate PR.
2014-11-19 14:39:32 -08:00
Stephen J Day
3cfe9aede5 Move Digest type into discrete package
The Digest type will be fairly central for blob and layer management. The type
presented in this package provides a number of core features that should enable
reliable use within the registry. This commit will be followed by others that
convert the storage layer and webapp to use this type as the primary layer/blob
CAS identifier.
2014-11-19 14:25:55 -08:00
Olivier Gambier
ce91eabaab Merge pull request #751 from stevvooe/update-routes
Update routes to reflect specification changes
2014-11-18 23:45:55 -08:00
Stephen J Day
5789ca7629 Update routes to reflect specification changes
To bring the implementation inline with the specification, the names and
structure of the API routes have been updated.

The overloaded term "image" has been replaced with the term "manifest", which
may also be known as "image manifest".  The desire for the layer storage to be
more of a general blob storage is reflected in moving from "layer" api prefixes
to "blob". The "tarsum" path parameter has been replaced by a more general
"digest" parameter and is no longer required to start uploads. Another set of
changes will come along to support this change at the storage service layer.
2014-11-18 20:39:32 -08:00
Stephen Day
ec3d4ccf6d Merge pull request #749 from BrianBland/storagedriver-ipc-errors
Adds ability to unwrap ipc errors into their original type
2014-11-18 19:44:30 -08:00
Olivier Gambier
9dc6fa3765 Merge pull request #741 from stevvooe/layer-service
Initial implementation of registry LayerService
2014-11-18 18:31:15 -08:00
Brian Bland
a3481c5f1c Adds ability to unwrap ipc errors into their original type
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.

Also improves storagedriver tests to enforce proper error types are
returned
2014-11-18 17:41:48 -08:00
Stephen Day
1e8f0ce50a Merge pull request #742 from BrianBland/ng-push-pull
Adds sliding-window parallelization to Push/Pull operations
2014-11-18 15:19:04 -08:00