Commit graph

143 commits

Author SHA1 Message Date
Stephen J Day
2ebc373d91 Refactor inmemory driver for Stat and WriteStream methods
This change started out as simply updating the existing inmemory driver to
implement the new Stat call. After struggling with the map based
implementation, it has been refactored to be a tree-based implementation.

This process has exposed a few missing error cases in the StorageDriver API
that should be addressed in the coming weeks.
2014-12-04 20:25:14 -08:00
Stephen J Day
ab9570f872 Migrate filesystem driver to new storagedriver calls
The filesystem driver has been migrated to impleemnt the storagedriver
interface changes. Most interetingly, this provides a filesystem-based
implementation of the Stat driver call. With this comes some refactoring of
Reads and Write to be much simpler and more robust.

The IPC tests have been disabled to stability problems that we'll have to
troubleshoot at a later date.
2014-12-03 16:47:31 -08:00
Stephen J Day
2037b1d6bf Update testsuite with storagedriver interface changes
This change updates the testsuite to migrate to the new driver interface. This
includes the new Stat call, changes to int64 over uint64 and the changes to the
WriteStream signature. Several test cases have been added to vet
implementations against various assumptions.
2014-12-03 16:37:46 -08:00
Stephen J Day
2e3ecdca37 Remove size argument and using io.Reader for StorageDriver.WriteStream
We are change the the rpc call for WriteStream to not require the size
argument, opting to drive the process with io.Reader. The main issue was that
io.Reader may return io.EOF before reaching size, making the error handling
around this condition for callers more complex. To complement this, WriteStream
now returns the number of successfully written bytes.

The method no longer requires an io.ReadCloser, opting to require just an
io.Reader. This keeps the reader under the control of the caller, which
provides more flexibility.

This also begins to address some of the problems described in #791.
2014-12-02 21:47:28 -08:00
Stephen J Day
ac660e72bf Replace StorageLayer.CurrentSize interface call with Stat
To support single-flight Size and ModTime queries against backend storage file,
we are replacing the CurrentSize call with a Stat call. A FileInfo interface is
provided for backends to provide a type, with a default implementation called
FileInfoInternal, for use by driver implementations.

More work needs to follow this change to update all the driver implementations.
2014-12-02 21:00:42 -08:00
Stephen J Day
b047c92e1c Use sync.WaitGroup to control concurrent tests 2014-12-02 20:43:31 -08:00
Stephen J Day
66107df1af Use int64 for ReadStream and WriteStream offsets
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
2014-12-02 19:01:00 -08:00
Olivier Gambier
511827efe2 Merge pull request #801 from ahmetalpbalkan/next-generation-azure-storagedriver
NG: Microsoft Azure storage driver
2014-12-02 13:45:52 -08:00
Olivier Gambier
e0414278d3 Merge pull request #806 from stevvooe/ng-registry-main
NG: Add registry main cmd
2014-12-02 12:34:59 -08:00
Olivier Gambier
efa4e03519 Merge pull request #807 from stevvooe/ng-nonzero-exit-error
StorageDrivers should exit non-zero on error (closes #803)
2014-12-02 12:33:08 -08:00
Olivier Gambier
fc1db3ad10 Merge pull request #805 from stevvooe/ng-manifest-bugs
NG: Various Manifest Oriented bugfixes
2014-12-01 18:45:12 -08:00
Ahmet Alp Balkan
4054cd3e73 Azure storage driver implementation
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2014-12-01 18:38:23 -08:00
Stephen J Day
54c0290cda StorageDrivers should exit non-zero on error (closes #803) 2014-12-01 18:25:10 -08:00
Stephen J Day
17b32e0aa0 Add TODO about manifest tampering test 2014-12-01 17:40:14 -08:00
Stephen J Day
3301626830 Add registry main cmd 2014-12-01 17:36:20 -08:00
Stephen J Day
dd8eb6a8df configuration.Parse should take io.Reader 2014-12-01 17:32:31 -08:00
Stephen J Day
e6e0219065 Avoid manifest verification errors by using Raw
Because json.Marshal does compaction on returned results, applications must
directly use SignedManifest.Raw when the marshaled value is required.
Otherwise, the returned manifest will fail signature checks.
2014-12-01 17:10:33 -08:00
Stephen J Day
8c7bec72b1 Cleanup image verification error handling
This diff removes a few early outs that caused errors to be unreported and
catches a missed error case for signature verification from libtrust. More work
needs to be done around ensuring consistent error handling but this is enough
to make the API work correctly.
2014-12-01 16:13:01 -08:00
Stephen J Day
b73a6c1998 Use json.MashalIndent for raw manifest json
This provides compatibility with what is in docker core, ensuring that image
manifests generated here have the same formatting. We'll need to automate this
some how.
2014-12-01 16:11:27 -08:00
Stephen J Day
98f5f30e75 Create copy of buffer for SignedManifest.Raw
Without this copy, the buffer may be re-used in the json package, causing
missing or corrupted content for the long-lived SignedManifest object. By
creating a new buffer, owned by the SignedManifest object, the content remains
stable.
2014-12-01 15:57:05 -08:00
Olivier Gambier
f497bceffa Merge pull request #794 from stevvooe/ng-client-race-condition
Address race condition in client tests (closes #784)
2014-12-01 10:10:43 -08:00
Olivier Gambier
af72f6cc94 Merge pull request #792 from stevvooe/next-generation
Initial implementation of Manifest HTTP API
2014-12-01 10:10:35 -08:00
Stephen J Day
c706657502 Address race condition in client tests (closes #784) 2014-11-26 15:37:27 -08:00
Olivier Gambier
9da63d0bd2 Merge pull request #793 from stevvooe/update-go-version
Correct ENV declaration in Dockerfile
2014-11-26 13:53:59 -08:00
Stephen J Day
e809796f59 Initial implementation of Manifest HTTP API
Push, pull and delete of manifest files in the registry have been implemented
on top of the storage services. Basic workflows, including reporting of missing
manifests are tested, including various proposed response codes. Common testing
functionality has been collected into shared methods. A test suite may be
emerging but it might better to capture more edge cases (such as resumable
upload, range requests, etc.) before we commit to a full approach.

To support clearer test cases and simpler handler methods, an application aware
urlBuilder has been added. We may want to export the functionality for use in
the client, which could allow us to abstract away from gorilla/mux.

A few error codes have been added to fill in error conditions missing from the
proposal. Some use cases have identified some problems with the approach to
error reporting that requires more work to reconcile. To resolve this, the
mapping of Go errors into error types needs to pulled out of the handlers and
into the application. We also need to move to type-based errors, with rich
information, rather than value-based errors. ErrorHandlers will probably
replace the http.Handlers to make this work correctly.

Unrelated to the above, the "length" parameter has been migrated to "size" for
completing layer uploads. This change should have gone out before but these
diffs ending up being coupled with the parameter name change due to updates to
the layer unit tests.
2014-11-26 13:35:07 -08:00
Stephen J Day
4a4d403655 Correct ENV declaration in Dockerfile 2014-11-26 13:19:45 -08:00
Stephen J Day
6fead90736 Rich error reporting for manifest push
To provide rich error reporting during manifest pushes, the storage layers
verifyManifest stage has been modified to provide the necessary granularity.
Along with this comes with a partial shift to explicit error types, which
represents a small move in larger refactoring of error handling. Signature
methods from libtrust have been added to the various Manifest types to clean up
the verification code.

A primitive deletion implementation for manifests has been added. It only
deletes the manifest file and doesn't attempt to add some of the richer
features request, such as layer cleanup.
2014-11-26 12:57:14 -08:00
Stephen Day
1f772456b0 Merge pull request #787 from ahmetalpbalkan/next-generation
NG: Fix read offset check for inmemory driver
2014-11-26 10:18:53 -08:00
Ahmet Alp Balkan
c5bb224bf9 Fix read offset check for inmemory driver
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2014-11-25 21:43:12 -08:00
Olivier Gambier
59c399cb25 Merge pull request #782 from stevvooe/cleanup-storage-mapping
Clean up layer storage layout
2014-11-25 10:02:15 -08:00
Stephen J Day
68944ea9cf Clean up layer storage layout
Previously, discussions were still ongoing about different storage layouts that
could support various access models. This changeset removes a layer of
indirection that was in place due to earlier designs. Effectively, this both
associates a layer with a named repository and ensures that content cannot be
accessed across repositories. It also moves to rely on tarsum as a true
content-addressable identifier, removing a layer of indirection during blob
resolution.
2014-11-25 09:57:43 -08:00
Olivier Gambier
756989c011 Merge pull request #783 from stevvooe/next-generation
Correct references after move of Manifest into storage package
2014-11-24 17:32:12 -08:00
Stephen J Day
66d2a13aec Correct references after move of Manifest into storage package 2014-11-24 16:38:33 -08:00
Olivier Gambier
d825559473 Merge pull request #780 from stevvooe/manifest-storage
Initial implementation of image manifest storage
2014-11-24 15:27:48 -08:00
Olivier Gambier
104d203ee0 Merge pull request #750 from BrianBland/ng-push-pull
Allows layers to be partially pulled and resumed
2014-11-24 15:22:47 -08:00
Brian Bland
50d64ac63a Allows layers to be partially pulled and resumed
Adds a sort of contrived test for resumable pulls
2014-11-24 13:55:36 -08:00
Stephen J Day
4decfaa82e Initial implementation of image manifest storage
This change implements the first pass at image manifest storage on top of the
storagedriver. Very similar to LayerService, its much simpler due to less
complexity of pushing and pulling images.

Various components are still missing, such as detailed error reporting on
missing layers during verification, but the base functionality is present.
2014-11-24 13:05:27 -08:00
Stephen Day
73d6e8af84 Merge pull request #774 from docker/test-drone
Fix goverall
2014-11-24 10:24:48 -08:00
Olivier Gambier
a463f48b1a Merge pull request #776 from noxiouz/tiny_cleaninig
[IPC] Tiny cleaning
2014-11-23 19:22:56 -08:00
Anton Tiurin
4635a1e903 [IPC] Tiny cleaning 2014-11-22 23:23:03 +03:00
Olivier Gambier
b60ee6dfa0 Merge pull request #769 from stevvooe/layer-http-api
Initial implementation of Layer API
2014-11-22 10:21:23 -08:00
Olivier Gambier
ebab275b76 Fix goverall
Also simplified things a bit + adding notes about extra tooling.
2014-11-22 10:17:14 -08:00
Stephen J Day
eaadb82e1e Move Manifest type into storage package
This changeset move the Manifest type into the storage package to make the type
accessible to client and registry without import cycles. The structure of the
manifest was also changed to accuratle reflect the stages of the signing
process. A straw man Manifest.Sign method has been added to start testing this
concept out but will probably be accompanied by the more import
SignedManifest.Verify method as the security model develops.

This is probably the start of a concerted effort to consolidate types across
the client and server portions of the code base but we may want to see how such
a handy type, like the Manifest and SignedManifest, would work in docker core.
2014-11-21 19:37:44 -08:00
Stephen J Day
4bbabc6e36 Implement path spec for manifest storage 2014-11-21 19:15:35 -08:00
Stephen J Day
e158e3cd65 Initial implementation of Layer API
The http API has its first set of endpoints to implement the core aspects of
fetching and uploading layers. Uploads can be started and completed in a single
chunk and the content can be fetched via tarsum. Most proposed error conditions
should be represented but edge cases likely remain.

In this version, note that the layers are still called layers, even though the
routes are pointing to blobs. This will change with backend refactoring over
the next few weeks.

The unit tests are a bit of a shamble but these need to be carefully written
along with the core specification process. As the the client-server interaction
solidifies, we can port this into a verification suite for registry providers.
2014-11-21 19:12:20 -08:00
Stephen J Day
195568017a Update error declarations and add missing test
This updates API error codes to coincide with changes to the proposal. Mostly,
redundant error codes were merged and missing ones were added. The set in the
main errors.go file will flow back into the specification.

A test case has been added to ensure ErrorCodeUnknown is included in marshaled
json.
2014-11-21 15:24:14 -08:00
Stephen J Day
3f479b62b4 Refactor layerReader into fileReader
This change separates out the remote file reader functionality from layer
reprsentation data. More importantly, issues with seeking have been fixed and
thoroughly tested.
2014-11-21 15:24:14 -08:00
Stephen Day
74279723c0 Merge pull request #773 from docker/test-drone
Move notifications to dt
2014-11-21 12:34:22 -08:00
Olivier Gambier
96a557c8e7 Move notifications to dt 2014-11-21 12:02:34 -08:00
Stephen Day
5829d8fa6c Merge pull request #770 from noxiouz/fix_leak_in_push
[Client] Fix possible goroutine leak in push.
2014-11-21 11:50:26 -08:00