This changeset integrates context with the storage package. Debug messages have
been added to exported methods. Existing log messages will now include
contextual details through logger fields to aid in debugging. This integration
focuses on logging and may be followed up with a metric-oriented change in the
future.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.
In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
In support of making the storage API ready for supporting notifications and
mirroring, we've begun the process of paring down the storage model. The
process started by creating a central Registry interface. From there, the
common name argument on the LayerService and ManifestService was factored into
a Repository interface. The rest of the changes directly follow from this.
An interface wishlist was added, suggesting a direction to take the registry
package that should support the distribution project's future goals. As these
objects move out of the storage package and we implement a Registry backed by
the http client, these design choices will start getting validation.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This change refactors the storage backend to use the new path layout. To
facilitate this, manifest storage has been separated into a revision store and
tag store, supported by a more general blob store. The blob store is a hybrid
object, effectively providing both small object access, keyed by content
address, as well as methods that can be used to manage and traverse links to
underlying blobs. This covers common operations used in the revision store and
tag store, such as linking and traversal. The blob store can also be updated to
better support layer reading but this refactoring has been left for another
day.
The revision store and tag store support the manifest store's compound view of
data. These underlying stores provide facilities for richer access models, such
as content-addressable access and a richer tagging model. The highlight of this
change is the ability to sign a manifest from different hosts and have the
registry merge and serve those signatures as part of the manifest package.
Various other items, such as the delegate layer handler, were updated to more
directly use the blob store or other mechanism to fit with the changes.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To smooth initial implementation, uploads were spooled to local file storage,
validated, then pushed to remote storage. That approach was flawed in that it
present easy clustering of registry services that share a remote storage
backend. The original plan was to implement resumable hashes then implement
remote upload storage. After some thought, it was found to be better to get
remote spooling working, then optimize with resumable hashes.
Moving to this approach has tradeoffs: after storing the complete upload
remotely, the node must fetch the content and validate it before moving it to
the final location. This can double bandwidth usage to the remote backend.
Modifying the verification and upload code to store intermediate hashes should
be trivial once the layer digest format has settled.
The largest changes for users of the storage package (mostly the registry app)
are the LayerService interface and the LayerUpload interface. The LayerService
now takes qualified repository names to start and resume uploads. In corallry,
the concept of LayerUploadState has been complete removed, exposing all aspects
of that state as part of the LayerUpload object. The LayerUpload object has
been modified to work as an io.WriteSeeker and includes a StartedAt time, to
allow for upload timeout policies. Finish now only requires a digest, eliding
the requirement for a size parameter.
Resource cleanup has taken a turn for the better. Resources are cleaned up
after successful uploads and during a cancel call. Admittedly, this is probably
not completely where we want to be. It's recommend that we bolster this with a
periodic driver utility script that scans for partial uploads and deletes the
underlying data. As a small benefit, we can leave these around to better
understand how and why these uploads are failing, at the cost of some extra
disk space.
Many other changes follow from the changes above. The webapp needs to be
updated to meet the new interface requirements.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the common package no longer exists, the testutil package is being moved
up to the root. Ideally, we don't have large omnibus packages, like testutil,
but we can fix that in another refactoring round.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To support clustered registry, upload UUIDs must be recognizable by
registries that did not issue the UUID. By creating an HMAC verifiable
upload state token, registries can validate upload requests that other
instances authorized. The tokenProvider interface could also use a redis
store or other system for token handling in the future.
To provide rich error reporting during manifest pushes, the storage layers
verifyManifest stage has been modified to provide the necessary granularity.
Along with this comes with a partial shift to explicit error types, which
represents a small move in larger refactoring of error handling. Signature
methods from libtrust have been added to the various Manifest types to clean up
the verification code.
A primitive deletion implementation for manifests has been added. It only
deletes the manifest file and doesn't attempt to add some of the richer
features request, such as layer cleanup.
Previously, discussions were still ongoing about different storage layouts that
could support various access models. This changeset removes a layer of
indirection that was in place due to earlier designs. Effectively, this both
associates a layer with a named repository and ensures that content cannot be
accessed across repositories. It also moves to rely on tarsum as a true
content-addressable identifier, removing a layer of indirection during blob
resolution.
This change separates out the remote file reader functionality from layer
reprsentation data. More importantly, issues with seeking have been fixed and
thoroughly tested.
Mostly, we've made superficial changes to the storage package to start using
the Digest type. Many of the exported interface methods have been changed to
reflect this in addition to changes in the way layer uploads will be initiated.
Further work here is necessary but will come with a separate PR.
This change contains the initial implementation of the LayerService to power
layer push and pulls on the storagedriver. The interfaces presented in this
package will be used by the http application to drive most features around
efficient pulls and resumable pushes.
The file storage/layer.go defines the interface interactions. LayerService is
the root type and supports methods to access Layer and LayerUpload objects.
Pull operations are supported with LayerService.Fetch and push operations are
supported with LayerService.Upload and LayerService.Resume. Reads and writes of
layers are split between Layer and LayerUpload, respectively.
LayerService is implemented internally with the layerStore object, which takes
a storagedriver.StorageDriver and a pathMapper instance.
LayerUploadState is currently exported and will likely continue to be as the
interaction between it and layerUploadStore are better understood. Likely, the
layerUploadStore lifecycle and implementation will be deferred to the
application.
Image pushes pulls will be implemented in a similar manner without the
discrete, persistent upload.
Much of this change is in place to get something running and working. Caveats
of this change include the following:
1. Layer upload state storage is implemented on the local filesystem, separate
from the storage driver. This must be replaced with using the proper backend
and other state storage. This can be removed when we implement resumable
hashing and tarsum calculations to avoid backend roundtrips.
2. Error handling is rather bespoke at this time. The http API implementation
should really dictate the error return structure for the future, so we
intend to refactor this heavily to support these errors. We'd also like to
collect production data to understand how failures happen in the system as
a while before moving to a particular edict around error handling.
3. The layerUploadStore, which manages layer upload storage and state is not
currently exported. This will likely end up being split, with the file
management portion being pointed at the storagedriver and the state storage
elsewhere.
4. Access Control provisions are nearly completely missing from this change.
There are details around how layerindex lookup works that are related with
access controls. As the auth portions of the new API take shape, these
provisions will become more clear.
Please see TODOs for details and individual recommendations.
