Commit graph

22 commits

Author SHA1 Message Date
Olivier Gambier
67ca9d10cf Move from docker-registry to distribution 2014-12-23 17:13:02 -08:00
Stephen J Day
a4f42b8eea Relax requirement for size argument during blob upload
During client implementation, it was found that requiring the size argument
made client implementation more complex. The original benefit of the size
argument was to provide an additional check alongside of tarsum to validate
incoming data. For the purposes of the registry, it has been determined that
tarsum should be enough to validate incoming content.

At this time, the size check is optional but we may consider removing it
completely.
2014-12-12 19:08:50 -08:00
Stephen J Day
33b2b80a8c Remove errant log message 2014-12-09 14:19:07 -08:00
Stephen J Day
49d13f9a08 Move manifest store errors to where they happen 2014-12-09 13:40:44 -08:00
Stephen J Day
c71089c653 Implement Tags method on ManifestService 2014-12-09 13:37:21 -08:00
Stephen J Day
1a75fccb43 Address PathNotFoundError in (*manifestStore).Exists
Exists was returning an error when encountering a PathNotFoundError when it
should just return false without an error.
2014-12-05 14:34:54 -08:00
Stephen J Day
70ab06b864 Update storage package to use StorageDriver.Stat
This change updates the backend storage package that consumes StorageDriver to
use the new Stat call, over CurrentSize. It also makes minor updates for using
WriteStream and ReadStream.
2014-12-04 20:55:59 -08:00
Stephen J Day
e6e0219065 Avoid manifest verification errors by using Raw
Because json.Marshal does compaction on returned results, applications must
directly use SignedManifest.Raw when the marshaled value is required.
Otherwise, the returned manifest will fail signature checks.
2014-12-01 17:10:33 -08:00
Stephen J Day
8c7bec72b1 Cleanup image verification error handling
This diff removes a few early outs that caused errors to be unreported and
catches a missed error case for signature verification from libtrust. More work
needs to be done around ensuring consistent error handling but this is enough
to make the API work correctly.
2014-12-01 16:13:01 -08:00
Stephen J Day
b73a6c1998 Use json.MashalIndent for raw manifest json
This provides compatibility with what is in docker core, ensuring that image
manifests generated here have the same formatting. We'll need to automate this
some how.
2014-12-01 16:11:27 -08:00
Stephen J Day
98f5f30e75 Create copy of buffer for SignedManifest.Raw
Without this copy, the buffer may be re-used in the json package, causing
missing or corrupted content for the long-lived SignedManifest object. By
creating a new buffer, owned by the SignedManifest object, the content remains
stable.
2014-12-01 15:57:05 -08:00
Stephen J Day
6fead90736 Rich error reporting for manifest push
To provide rich error reporting during manifest pushes, the storage layers
verifyManifest stage has been modified to provide the necessary granularity.
Along with this comes with a partial shift to explicit error types, which
represents a small move in larger refactoring of error handling. Signature
methods from libtrust have been added to the various Manifest types to clean up
the verification code.

A primitive deletion implementation for manifests has been added. It only
deletes the manifest file and doesn't attempt to add some of the richer
features request, such as layer cleanup.
2014-11-26 12:57:14 -08:00
Stephen J Day
68944ea9cf Clean up layer storage layout
Previously, discussions were still ongoing about different storage layouts that
could support various access models. This changeset removes a layer of
indirection that was in place due to earlier designs. Effectively, this both
associates a layer with a named repository and ensures that content cannot be
accessed across repositories. It also moves to rely on tarsum as a true
content-addressable identifier, removing a layer of indirection during blob
resolution.
2014-11-25 09:57:43 -08:00
Stephen J Day
4decfaa82e Initial implementation of image manifest storage
This change implements the first pass at image manifest storage on top of the
storagedriver. Very similar to LayerService, its much simpler due to less
complexity of pushing and pulling images.

Various components are still missing, such as detailed error reporting on
missing layers during verification, but the base functionality is present.
2014-11-24 13:05:27 -08:00
Stephen J Day
eaadb82e1e Move Manifest type into storage package
This changeset move the Manifest type into the storage package to make the type
accessible to client and registry without import cycles. The structure of the
manifest was also changed to accuratle reflect the stages of the signing
process. A straw man Manifest.Sign method has been added to start testing this
concept out but will probably be accompanied by the more import
SignedManifest.Verify method as the security model develops.

This is probably the start of a concerted effort to consolidate types across
the client and server portions of the code base but we may want to see how such
a handy type, like the Manifest and SignedManifest, would work in docker core.
2014-11-21 19:37:44 -08:00
Stephen J Day
4bbabc6e36 Implement path spec for manifest storage 2014-11-21 19:15:35 -08:00
Stephen J Day
3f479b62b4 Refactor layerReader into fileReader
This change separates out the remote file reader functionality from layer
reprsentation data. More importantly, issues with seeking have been fixed and
thoroughly tested.
2014-11-21 15:24:14 -08:00
Stephen J Day
c0fe9d72d1 Various adjustments to digest package for govet/golint 2014-11-19 14:59:05 -08:00
Stephen J Day
1a508d67d9 Move storage package to use Digest type
Mostly, we've made superficial changes to the storage package to start using
the Digest type. Many of the exported interface methods have been changed to
reflect this in addition to changes in the way layer uploads will be initiated.

Further work here is necessary but will come with a separate PR.
2014-11-19 14:39:32 -08:00
Stephen J Day
2637e29e18 Initial implementation of registry LayerService
This change contains the initial implementation of the LayerService to power
layer push and pulls on the storagedriver. The interfaces presented in this
package will be used by the http application to drive most features around
efficient pulls and resumable pushes.

The file storage/layer.go defines the interface interactions. LayerService is
the root type and supports methods to access Layer and LayerUpload objects.
Pull operations are supported with LayerService.Fetch and push operations are
supported with LayerService.Upload and LayerService.Resume. Reads and writes of
layers are split between Layer and LayerUpload, respectively.

LayerService is implemented internally with the layerStore object, which takes
a storagedriver.StorageDriver and a pathMapper instance.

LayerUploadState is currently exported and will likely continue to be as the
interaction between it and layerUploadStore are better understood. Likely, the
layerUploadStore lifecycle and implementation will be deferred to the
application.

Image pushes pulls will be implemented in a similar manner without the
discrete, persistent upload.

Much of this change is in place to get something running and working. Caveats
of this change include the following:

1. Layer upload state storage is implemented on the local filesystem, separate
   from the storage driver. This must be replaced with using the proper backend
   and other state storage. This can be removed when we implement resumable
   hashing and tarsum calculations to avoid backend roundtrips.
2. Error handling is rather bespoke at this time. The http API implementation
   should really dictate the error return structure for the future, so we
   intend to refactor this heavily to support these errors. We'd also like to
   collect production data to understand how failures happen in the system as
   a while before moving to a particular edict around error handling.
3. The layerUploadStore, which manages layer upload storage and state is not
   currently exported. This will likely end up being split, with the file
   management portion being pointed at the storagedriver and the state storage
   elsewhere.
4. Access Control provisions are nearly completely missing from this change.
   There are details around how layerindex lookup works that are related with
   access controls. As the auth portions of the new API take shape, these
   provisions will become more clear.

Please see TODOs for details and individual recommendations.
2014-11-17 17:54:07 -08:00
Brian Bland
88795e0a14 Lots of various golint fixes
Changes some names to match go conventions
Comments all exported methods
Removes dot imports
2014-11-17 15:46:06 -08:00
Stephen J Day
8e44c1d209 Initial implementation of storage layer path mapper
We've added a path mapper to support simple mapping between path objects used
in the storage layer and the underlying file system. The target of this is to
ensure that paths are only calculated in a single place and their format is
separated from the data that makes up the path components.

This commit only includes spec implementation to support layer reads. Further
specs will come along with their implementations.
2014-11-13 16:02:17 -08:00