Commit graph

5294 commits

Author SHA1 Message Date
Brian Bland
18eac89506 Adds a test for concurrent storagedriver Write/Read Stream operations
This test is currently failing and Skipped for IPC drivers
2014-11-20 16:15:55 -08:00
Anton Tiurin
5a804ac05b [Client] Fix possible goroutine leak.
Running goroutines with pullLayer are blocked to send error of a
pull operation. If we abort pulling without notify them about
cancelation they will get stucked forever. To avoid this possible
leak cancelCh was introduced. In case of abort we close that channel
to notify other goroutines about cancelation.
2014-11-21 03:15:34 +03:00
Olivier Gambier
3f6cb52357 Move to own dev image 2014-11-20 15:33:06 -08:00
Olivier Gambier
0111cf2bd4 Merge pull request #758 from BrianBland/ng-client-routes-update
Updates client to newer routes and changes "layer" to "blob"
2014-11-20 15:29:15 -08:00
Olivier Gambier
e98feed770 Merge pull request #764 from BrianBland/ng-storagedriver-misc-fixes
Miscellaneous storagedriver+ipc fixes
2014-11-20 15:13:13 -08:00
Brian Bland
68fd15b688 Miscellaneous storagedriver+ipc fixes
Fixes/tests listing for keys beginning with "/"
No longer extraneously wraps Closers in ioutil.NopClosers
Uses omitempty for all ipc struct type fields
2014-11-20 14:14:55 -08:00
Brian Bland
64c8bd29cc Uses new digest package instead of string digests
Also drops extraneous test package and uses testutil instead
2014-11-19 18:54:11 -08:00
Brian Bland
1336ced030 Updates client to newer routes and changes "layer" to "blob" 2014-11-19 18:37:24 -08:00
Olivier Gambier
b65d8d046e Merge pull request #755 from stevvooe/digest-type
Move Digest type into discrete package
2014-11-19 17:02:54 -08:00
Stephen J Day
56118905b8 Include testutil package needed for tar-based tests 2014-11-19 15:10:23 -08:00
Stephen J Day
c0fe9d72d1 Various adjustments to digest package for govet/golint 2014-11-19 14:59:05 -08:00
Stephen J Day
1a508d67d9 Move storage package to use Digest type
Mostly, we've made superficial changes to the storage package to start using
the Digest type. Many of the exported interface methods have been changed to
reflect this in addition to changes in the way layer uploads will be initiated.

Further work here is necessary but will come with a separate PR.
2014-11-19 14:39:32 -08:00
Stephen J Day
3cfe9aede5 Move Digest type into discrete package
The Digest type will be fairly central for blob and layer management. The type
presented in this package provides a number of core features that should enable
reliable use within the registry. This commit will be followed by others that
convert the storage layer and webapp to use this type as the primary layer/blob
CAS identifier.
2014-11-19 14:25:55 -08:00
Olivier Gambier
ce91eabaab Merge pull request #751 from stevvooe/update-routes
Update routes to reflect specification changes
2014-11-18 23:45:55 -08:00
Stephen J Day
5789ca7629 Update routes to reflect specification changes
To bring the implementation inline with the specification, the names and
structure of the API routes have been updated.

The overloaded term "image" has been replaced with the term "manifest", which
may also be known as "image manifest".  The desire for the layer storage to be
more of a general blob storage is reflected in moving from "layer" api prefixes
to "blob". The "tarsum" path parameter has been replaced by a more general
"digest" parameter and is no longer required to start uploads. Another set of
changes will come along to support this change at the storage service layer.
2014-11-18 20:39:32 -08:00
Stephen Day
ec3d4ccf6d Merge pull request #749 from BrianBland/storagedriver-ipc-errors
Adds ability to unwrap ipc errors into their original type
2014-11-18 19:44:30 -08:00
Olivier Gambier
9dc6fa3765 Merge pull request #741 from stevvooe/layer-service
Initial implementation of registry LayerService
2014-11-18 18:31:15 -08:00
Brian Bland
a3481c5f1c Adds ability to unwrap ipc errors into their original type
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.

Also improves storagedriver tests to enforce proper error types are
returned
2014-11-18 17:41:48 -08:00
Stephen Day
1e8f0ce50a Merge pull request #742 from BrianBland/ng-push-pull
Adds sliding-window parallelization to Push/Pull operations
2014-11-18 15:19:04 -08:00
Stephen J Day
2637e29e18 Initial implementation of registry LayerService
This change contains the initial implementation of the LayerService to power
layer push and pulls on the storagedriver. The interfaces presented in this
package will be used by the http application to drive most features around
efficient pulls and resumable pushes.

The file storage/layer.go defines the interface interactions. LayerService is
the root type and supports methods to access Layer and LayerUpload objects.
Pull operations are supported with LayerService.Fetch and push operations are
supported with LayerService.Upload and LayerService.Resume. Reads and writes of
layers are split between Layer and LayerUpload, respectively.

LayerService is implemented internally with the layerStore object, which takes
a storagedriver.StorageDriver and a pathMapper instance.

LayerUploadState is currently exported and will likely continue to be as the
interaction between it and layerUploadStore are better understood. Likely, the
layerUploadStore lifecycle and implementation will be deferred to the
application.

Image pushes pulls will be implemented in a similar manner without the
discrete, persistent upload.

Much of this change is in place to get something running and working. Caveats
of this change include the following:

1. Layer upload state storage is implemented on the local filesystem, separate
   from the storage driver. This must be replaced with using the proper backend
   and other state storage. This can be removed when we implement resumable
   hashing and tarsum calculations to avoid backend roundtrips.
2. Error handling is rather bespoke at this time. The http API implementation
   should really dictate the error return structure for the future, so we
   intend to refactor this heavily to support these errors. We'd also like to
   collect production data to understand how failures happen in the system as
   a while before moving to a particular edict around error handling.
3. The layerUploadStore, which manages layer upload storage and state is not
   currently exported. This will likely end up being split, with the file
   management portion being pointed at the storagedriver and the state storage
   elsewhere.
4. Access Control provisions are nearly completely missing from this change.
   There are details around how layerindex lookup works that are related with
   access controls. As the auth portions of the new API take shape, these
   provisions will become more clear.

Please see TODOs for details and individual recommendations.
2014-11-17 17:54:07 -08:00
Brian Bland
28b7b82e2d Adds sliding-window parallelization to Push/Pull operations
A layer can only be pushed/pulled if the layer preceding it by the
length of the push/pull window has been successfully pushed.

An error returned from pushing or pulling any layer will cause the full
operation to be aborted.
2014-11-17 17:46:07 -08:00
Olivier Gambier
a2d232aaec Merge pull request #740 from BrianBland/ng-push-pull
WIP: Adds push/pull client functionality
2014-11-17 17:07:33 -08:00
Brian Bland
0e1b1cc04e Adds push/pull client functionality
These methods rely on an ObjectStore interface, which is meant to
approximate the storage behavior of the docker engine. This is very much
subject to change.
2014-11-17 16:50:02 -08:00
Olivier Gambier
de4e976ef2 Merge pull request #739 from BrianBland/drone-lint
Lots of various golint fixes
2014-11-17 16:32:16 -08:00
Brian Bland
88795e0a14 Lots of various golint fixes
Changes some names to match go conventions
Comments all exported methods
Removes dot imports
2014-11-17 15:46:06 -08:00
Stephen Day
b5cf681458 Merge pull request #738 from docker/test-drone
Test
2014-11-17 15:38:31 -08:00
Olivier Gambier
b02ca32ac8 Fix drone 2014-11-17 14:44:36 -08:00
Olivier Gambier
815acfd6e7 Merge pull request #736 from stevvooe/repository-name-clarification
Clarify repository naming constraints for registry API
2014-11-17 14:19:46 -08:00
Stephen Day
e1798d1ffb Merge pull request #737 from BrianBland/drone-vet
Fixes "go vet" for drone CI
2014-11-17 14:19:20 -08:00
Brian Bland
8ad7819b1b Fixes "go vet" for drone CI
Removes "go get" commands for go cmd packages to use the default
versions
Also updates client/client.go to conform to go vet style
2014-11-17 13:35:58 -08:00
Stephen J Day
a650f0f854 Clarify repository naming constraints for registry API
After discussion, it was found that one of the proposed regular expressions
incorrectly limited separator delimited compoonents to two characters. The
desired restriction is to have repository name components limited to two
characters minimum. This changeset accomplishes this by wrapping the regular
expressions in a validation function, returning detailed feedback on the
validation error.

With this change, the repository name regular expressions are no longer enough
to respond with 404s on invalid repo names. Changes to the router will need to
be added to support this.
2014-11-17 11:42:54 -08:00
Olivier Gambier
3127ffcfb1 Merge pull request #733 from BrianBland/drone-fmt
Fixes drone build
2014-11-15 20:01:38 -08:00
Brian Bland
39fee7d40a Fixes formatting in errors_test.go to match gofmt rules 2014-11-14 16:00:41 -08:00
Tibor Vass
6638cd7bc7 Add the possibility of specifying a subnet for --insecure-registry
Signed-off-by: Tibor Vass <teabee89@gmail.com>

Conflicts:
	registry/endpoint.go
2014-11-14 14:20:19 -08:00
Tibor Vass
8065dad50b registry: parse INDEXSERVERADDRESS into a URL for easier check in isSecure
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2014-11-14 14:20:19 -08:00
Tibor Vass
8b0e8b6621 Put mock registry address in insecureRegistries for unit tests
Signed-off-by: Tibor Vass <teabee89@gmail.com>

Conflicts:
	registry/registry_mock_test.go
2014-11-14 14:20:19 -08:00
Tibor Vass
44d97c1fd0 registry: refactor registry.IsSecure calls into registry.NewEndpoint
Signed-off-by: Tibor Vass <teabee89@gmail.com>

Conflicts:
	registry/endpoint.go
	registry/endpoint_test.go
	registry/registry_test.go
2014-11-14 14:05:31 -08:00
Tibor Vass
ae0ebb9d07 Add the possibility of specifying a subnet for --insecure-registry
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2014-11-14 12:31:11 -08:00
Olivier Gambier
8a64db69ec Merge pull request #729 from stevvooe/storage-path-mapper
Initial implementation of storage layer path mapper
2014-11-14 11:06:19 -08:00
Stephen Day
8217760a4d Merge pull request #725 from docker/test-drone
Test drone
2014-11-13 17:42:05 -08:00
Stephen J Day
8e44c1d209 Initial implementation of storage layer path mapper
We've added a path mapper to support simple mapping between path objects used
in the storage layer and the underlying file system. The target of this is to
ensure that paths are only calculated in a single place and their format is
separated from the data that makes up the path components.

This commit only includes spec implementation to support layer reads. Further
specs will come along with their implementations.
2014-11-13 16:02:17 -08:00
Olivier Gambier
0c999bd2da Merge pull request #727 from stevvooe/next-generation
Add route test case with dangerous path
2014-11-13 15:55:48 -08:00
Olivier Gambier
dd47af5feb Test 2014-11-13 15:40:06 -08:00
Stephen J Day
89c6bb2a90 Add route test case with dangerous path 2014-11-13 14:39:13 -08:00
Olivier Gambier
9fd57ab42b Test drone 2014-11-13 13:58:52 -08:00
Olivier Gambier
378256de47 Merge pull request #690 from BrianBland/storagedriver-process-management
Adds logic for tracking ipc storage driver process status
2014-11-13 11:24:46 -08:00
Tibor Vass
f0920e61bf registry: parse INDEXSERVERADDRESS into a URL for easier check in isSecure
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2014-11-13 07:02:24 -08:00
Olivier Gambier
d1bcfd6a08 Merge pull request #721 from stevvooe/disambiguate-url-routes
Disambiguate routing for multi-level repository names
2014-11-12 21:01:49 -08:00
Olivier Gambier
a32680eb53 Merge pull request #722 from BrianBland/vet-fix
Miscellaneous go vet fixes
2014-11-12 21:01:38 -08:00
Tibor Vass
cca910e878 Put mock registry address in insecureRegistries for unit tests
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2014-11-12 20:34:03 -06:00