Commit graph

2898 commits

Author SHA1 Message Date
Stephen J Day
83d62628fc Refactor storage to use new backend layout
This change refactors the storage backend to use the new path layout. To
facilitate this, manifest storage has been separated into a revision store and
tag store, supported by a more general blob store. The blob store is a hybrid
object, effectively providing both small object access, keyed by content
address, as well as methods that can be used to manage and traverse links to
underlying blobs. This covers common operations used in the revision store and
tag store, such as linking and traversal. The blob store can also be updated to
better support layer reading but this refactoring has been left for another
day.

The revision store and tag store support the manifest store's compound view of
data. These underlying stores provide facilities for richer access models, such
as content-addressable access and a richer tagging model. The highlight of this
change is the ability to sign a manifest from different hosts and have the
registry merge and serve those signatures as part of the manifest package.

Various other items, such as the delegate layer handler, were updated to more
directly use the blob store or other mechanism to fit with the changes.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-15 10:32:18 -08:00
Olivier Gambier
82a7101f21 Merge pull request #66 from stevvooe/autogenerate-authors-file
Autogenerate AUTHORS file for distribution repo
2015-01-15 10:07:21 -08:00
Stephen J Day
deddda5d29 Autogenerate AUTHORS file for distribution repo
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-14 16:25:27 -08:00
Andrey Kostov
bdd5d35622 Add functionality to make a url signed for a HEAD request to S4 driver 2015-01-14 13:37:39 -08:00
Stephen Day
d64e70b0e1 Merge pull request #59 from stevvooe/use-godep-for-dependencies
Use Godep to vendor distribution dependencies
2015-01-14 12:13:25 -08:00
Stephen J Day
3277d9fc74 Redesign path layout for backend storage
Several requirements for storing registry data have been compiled and the
backend layout has been refactored to comply. Specifically, we now store most
data as blobs that are linked from repositories. All data access is traversed
through repositories. Manifest updates are no longer destructive and support
references by digest or tag. Signatures for manifests are now stored externally
to the manifest payload to allow merging of signatures posted at different
time.

The design is detailed in the documentation for pathMapper.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-14 11:34:47 -08:00
Stephen J Day
10a4605ec2 Explicitly select digest method for content
Detecting tar files then falling back for calculating digests turned out to be
fairly unreliable. Likely, the implementation was broken for content that was
not a tarfile. Also, for the use case of the registry, it is really not needed.
This functionality has been removed in FromReader and FromBytes. FromTarArchive
has been added for convenience.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-14 10:35:42 -08:00
Stephen Day
6b18639eac Merge pull request #60 from BrianBland/s3-docs
Clarifies s3 driver "secure" flag documentation
2015-01-13 19:39:47 -08:00
Brian Bland
14c8e2b5f0 Clarifies s3 driver "secure" flag documentation
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-01-13 11:38:19 -08:00
Stephen J Day
01bc08351d Ignore Godeps for gofmt and golint
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-12 14:01:13 -08:00
Stephen J Day
fc2a840e8f Use Godep to vendor distribution dependencies
As we get closer to release, we need to ensure that builds are repeatable.
Godep provides a workable solution to managing dependencies in Go to support
this requirement. This commit should be bolstered by updates to documentation
and build configuration.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-12 14:01:04 -08:00
Stephen Day
972a95f077 Merge pull request #58 from BrianBland/s3-enable
Enables s3 driver
2015-01-12 13:41:35 -08:00
Brian Bland
16a258ce69 Enables s3 driver, limits permission check to scope of rootdirectory
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-01-12 13:34:49 -08:00
Stephen Day
df4441d03c Merge pull request #18 from AndreyKostov/ng-add-s3driver-features
Add features to the s3 driver
2015-01-12 13:10:33 -08:00
Stephen Day
e9e26bd362 Merge pull request #56 from stevvooe/blob-upload-unknown
Correctly handle missing layer upload
2015-01-12 12:14:32 -08:00
Olivier Gambier
bbc94aef39 Clarify maturity of this
Fixes https://github.com/docker/docker-registry/issues/894
2015-01-12 11:10:13 -08:00
Stephen Day
16e1c441f1 Merge pull request #57 from icecrime/groups_in_readme
Add Google Groups to README.md
2015-01-12 11:09:18 -08:00
Arnaud Porterie
cbb493e881 Add Google Groups to README.md
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
2015-01-12 08:31:04 -08:00
Stephen J Day
c78335a023 Correctly handle missing layer upload
Because we guarded the error check, nil Upload on the handler was getting
through to unexpected branches. This directly handles the missing upload
ensuring its set as expected.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 16:09:45 -08:00
Stephen Day
21a69f53b5 Merge pull request #53 from stevvooe/spool-uploads-remotely
Spool uploads remotely
2015-01-09 15:40:22 -08:00
Stephen J Day
4aa7837f83 Directly manage layerUploadState in webapp
Most of this change follows from the modifications to the storage api. The
driving factor is the separation of layerUploadState from the storage backend,
leaving it to the web application to store and update it. As part of the
updates to meet changes in the storage api, support for the size parameter has
been completely removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:51:13 -08:00
Stephen J Day
2f2445a335 Refactor handling of hmac state packing
This refactors the hmac state token to take control of the layerUploadState
json message, which has been removed from the storage backend. It also moves
away from the concept of a LayerUploadStateStore callback object, which was
short-lived. This allows for upload offset to be managed by the web application
logic in the face of an inconsistent backend. By controlling the upload offset
externally, we reduce the possibility of misreporting upload state to a client.

We may still want to modify the way this works after getting production
experience.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:50:39 -08:00
Stephen J Day
ba6b774aea Spool layer uploads to remote storage
To smooth initial implementation, uploads were spooled to local file storage,
validated, then pushed to remote storage. That approach was flawed in that it
present easy clustering of registry services that share a remote storage
backend. The original plan was to implement resumable hashes then implement
remote upload storage. After some thought, it was found to be better to get
remote spooling working, then optimize with resumable hashes.

Moving to this approach has tradeoffs: after storing the complete upload
remotely, the node must fetch the content and validate it before moving it to
the final location. This can double bandwidth usage to the remote backend.
Modifying the verification and upload code to store intermediate hashes should
be trivial once the layer digest format has settled.

The largest changes for users of the storage package (mostly the registry app)
are the LayerService interface and the LayerUpload interface. The LayerService
now takes qualified repository names to start and resume uploads. In corallry,
the concept of LayerUploadState has been complete removed, exposing all aspects
of that state as part of the LayerUpload object. The LayerUpload object has
been modified to work as an io.WriteSeeker and includes a StartedAt time, to
allow for upload timeout policies. Finish now only requires a digest, eliding
the requirement for a size parameter.

Resource cleanup has taken a turn for the better. Resources are cleaned up
after successful uploads and during a cancel call. Admittedly, this is probably
not completely where we want to be. It's recommend that we bolster this with a
periodic driver utility script that scans for partial uploads and deletes the
underlying data. As a small benefit, we can leave these around to better
understand how and why these uploads are failing, at the cost of some extra
disk space.

Many other changes follow from the changes above. The webapp needs to be
updated to meet the new interface requirements.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:50:39 -08:00
Stephen J Day
219bd48c24 Add path mapper definitions for upload locations
This change updates the path mapper to be able to specify upload management
locations. This includes a startedat file, which contains the RFC3339 formatted
start time of the upload and the actual data file.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:49:07 -08:00
Stephen J Day
75c5916dde Release lock during WriteStream for inmemory driver
While reading from the input in WriteStream, the inmemory driver can deadlock
if the reader is from the same instance. To fix this, the write lock is
released before reading into a local buffer. The lock is re-acquired to
finish the actual write.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:49:06 -08:00
Stephen J Day
09522d8535 Implement a remote file writer for use with StorageDriver
This changeset implements a fileWriter type that can be used to managed writes
to remote files in a StorageDriver. Basically, it manages a local seek position
for a remote path. An efficient use of this implementation will write data in
large blocks.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:49:06 -08:00
Stephen Day
e8714b9977 Merge pull request #55 from BrianBland/layerhandler
Adds support for content redirects for layer downloads
2015-01-09 14:44:54 -08:00
Brian Bland
f22ad79d36 Factors out resolveBlobPath, renames expires -> expiry 2015-01-08 17:56:45 -08:00
Brian Bland
cc3c648f44 Fixes tests, moves layerhandler in config file 2015-01-08 17:29:22 -08:00
Brian Bland
abb901e4ab Adds options map for storagedriver URLFor() method 2015-01-08 17:10:32 -08:00
Brian Bland
17915e1b01 Adds support for content redirects for layer downloads
Includes a delegate implementation which redirects to the URL generated
by the storagedriver, and a cloudfront implementation.
Satisfies proposal #49
2015-01-08 17:01:28 -08:00
Stephen Day
65863802d7 Merge pull request #51 from AndreyKostov/ng-add-storagedriver-urlfor
Add the URLFor optional method to the storagedriver api
2015-01-08 15:33:15 -08:00
Andrey Kostov
bc9509d85f Consolidate the s3 driver parameters with a struct 2015-01-08 00:46:48 +02:00
Stephen Day
69da73f69d Merge pull request #48 from stevvooe/fix-client-hmac-state-support
Correctly assemble URL during blob upload
2015-01-07 14:24:08 -08:00
Andrey Kostov
a2b294f444 Add the URLFor optional method to the storagedriver api
We now also have a storagedriver error variable for identifying
api calls that are not implemented by drivers (the URLFor method
is not implemented by either the filesystem or inmemory drivers)
2015-01-07 18:44:01 +02:00
Andrey Kostov
7c9112fc3c Make the encrpyt parameter default to false 2015-01-07 11:51:29 +02:00
Andrey Kostov
a0ef0d6aad Add the v4auth parameter
v4auth will default to true and if the frankfurt (eu-central-1) region
is selected with v4auth set to false explicitly, the driver will error
out upon initialization.
2015-01-07 11:50:32 +02:00
Andrey Kostov
031c388543 Add the secure flag for the s3 driver
The secure flag will be true by default and will change the
s3 endpoint of the region to http instead of https when selected as false.
The main benefits of running with secure being false is that it apparently
has a roughly 33% performance boost (even on pure data transfer, not only
connection setup which is what I would have expected).
2015-01-07 11:24:09 +02:00
Stephen Day
fadd5dfcfb Merge pull request #50 from BrianBland/storagedriver-path-fix
Relaxes storagedriver path constraints
2015-01-06 19:52:52 -08:00
Brian Bland
f6aadc2028 Relaxes storagedriver path constraints (#47)
No longer requires that file paths match the repository naming scheme,
but instead allows path components as short as a single character, as to
accommodate for single-character tag names.
2015-01-06 17:18:42 -08:00
Stephen J Day
83977857f8 Correctly assemble URL during blob upload
When adding parameters to a location header, the client must not destroy
parameters already present. This change ensures that parameters are added,
rather than replaced when assembling the url.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-06 16:43:17 -08:00
Olivier Gambier
5f863238c0 Merge pull request #26 from stevvooe/move-registry-package
Move registry package out of repo root
2015-01-06 14:19:39 -08:00
Olivier Gambier
891f6a5ba5 Merge pull request #27 from BrianBland/config-readme
Updates configuration README with explanations of more fields
2015-01-06 14:19:28 -08:00
Stephen Day
70b352d1c6 Merge pull request #17 from AndreyKostov/ng-add-s3driver-readme
Add a README file and make encrypt and rootdirectory optional parameters
2015-01-06 11:43:50 -08:00
Brian Bland
dbc3ecdff5 Updates configuration README with explanations of more fields
Documents auth, reporting, and http
2015-01-06 10:59:52 -08:00
Stephen J Day
ab4a9f0480 Move registry package out of repo root
Since the repo is no longer just the registry, we are moving the registry web
application package out of the repo root into a sub-package. We may break down
the registry package further to separate webapp components and bring the client
package under it. This change accomplishes the task of freeing up the repo root
for a distribution-oriented package. A stub doc.go file is left in place to
declare intent.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-06 10:40:22 -08:00
Stephen Day
fdea60af05 Merge pull request #24 from stevvooe/breakup-common
Breakup common package
2015-01-06 10:08:10 -08:00
Stephen Day
42577c6bde Merge pull request #20 from stevvooe/api-specification-detail
Fill in detail for V2 registry api specification
2015-01-06 10:04:06 -08:00
Stephen J Day
aea52c7fb5 Remove exported StringSet type and collections package
The exported StringSet type is not necessary for the current use case of
validating issues and audiences. The exported fields on VerifyOptions have been
changed to require string slices. The collections package has been removed and
the StringSet has been moved to the token package, where it is used.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-05 18:21:03 -08:00
Stephen Day
a20b493372 Merge pull request #23 from stevvooe/master
Outline constraints for interface from distribution to core
2015-01-05 17:02:22 -08:00