This change updates the backend storage package that consumes StorageDriver to
use the new Stat call, over CurrentSize. It also makes minor updates for using
WriteStream and ReadStream.
This change separates out the remote file reader functionality from layer
reprsentation data. More importantly, issues with seeking have been fixed and
thoroughly tested.
Mostly, we've made superficial changes to the storage package to start using
the Digest type. Many of the exported interface methods have been changed to
reflect this in addition to changes in the way layer uploads will be initiated.
Further work here is necessary but will come with a separate PR.
This change contains the initial implementation of the LayerService to power
layer push and pulls on the storagedriver. The interfaces presented in this
package will be used by the http application to drive most features around
efficient pulls and resumable pushes.
The file storage/layer.go defines the interface interactions. LayerService is
the root type and supports methods to access Layer and LayerUpload objects.
Pull operations are supported with LayerService.Fetch and push operations are
supported with LayerService.Upload and LayerService.Resume. Reads and writes of
layers are split between Layer and LayerUpload, respectively.
LayerService is implemented internally with the layerStore object, which takes
a storagedriver.StorageDriver and a pathMapper instance.
LayerUploadState is currently exported and will likely continue to be as the
interaction between it and layerUploadStore are better understood. Likely, the
layerUploadStore lifecycle and implementation will be deferred to the
application.
Image pushes pulls will be implemented in a similar manner without the
discrete, persistent upload.
Much of this change is in place to get something running and working. Caveats
of this change include the following:
1. Layer upload state storage is implemented on the local filesystem, separate
from the storage driver. This must be replaced with using the proper backend
and other state storage. This can be removed when we implement resumable
hashing and tarsum calculations to avoid backend roundtrips.
2. Error handling is rather bespoke at this time. The http API implementation
should really dictate the error return structure for the future, so we
intend to refactor this heavily to support these errors. We'd also like to
collect production data to understand how failures happen in the system as
a while before moving to a particular edict around error handling.
3. The layerUploadStore, which manages layer upload storage and state is not
currently exported. This will likely end up being split, with the file
management portion being pointed at the storagedriver and the state storage
elsewhere.
4. Access Control provisions are nearly completely missing from this change.
There are details around how layerindex lookup works that are related with
access controls. As the auth portions of the new API take shape, these
provisions will become more clear.
Please see TODOs for details and individual recommendations.
Olivier Gambier
Stephen J Day