Commit graph

23 commits

Author SHA1 Message Date
Stephen J Day
44da954565 Refactor specification of supported digests
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:08:23 -07:00
Stephen J Day
eee6cad2cf Remove digest package's dependency on external sha implementation
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.

We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 23:44:08 -07:00
Stephen Day
5556cd1ba1 Merge pull request #260 from dmcgowan/short-digests
Add short digest implementation
2015-04-29 14:05:41 -07:00
bin liu
e0521d2d01 fix some typos in source comments
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-04-17 12:39:52 +00:00
Andy Goldstein
7fdd395653 Use a build flag to disable resumable digests.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-04-16 01:19:57 +00:00
Josh Hawn
60b6748c95 Use resumable hash in digest package
Vendored resumable sha256/sha512 library. Digest package new exports a
resumable variant of the Digester.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-01 17:11:47 -07:00
Derek McGowan
7258fda98a Add digest set implementation
Set represents a unique set of digests which allow for efficient lookup.
Dumping short codes is a function which takes in a digest set.
Any operation involving short codes may be considered secure if the list of digests added to the set is the complete list of referenceable digests.
Contains benchmarks for Add, Lookup, and Dump.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-03-24 08:17:43 -07:00
Stephen Day
2ced84f31d Remove unnecessary error pointer formats in test
The linter was complaining about these and they are quite unnecessary.
2015-03-18 19:02:32 -07:00
Stephen J Day
8c254edb9a Correctly close pipe after error in tarsum verification
This addresses a subtle deadlock where an error during a copy prevented pipe
closure to propagate correctly. By closing down the read end of the pipe rather
than the write end, the waiting writer is properly signaled. A nice side-effect
of this change is that errors encountered by io.Copy are no propagated to the
verifier's Write method.

A test to ensure validation errors for unsupported digest types has been added,
as well.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-18 18:26:09 -07:00
xiekeyang
04f8bfbede supplement of digest refactoring 2015-03-11 14:13:29 +08:00
Josh Hawn
87959abe8f digest: Minor refactoring
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 16:44:19 -07:00
David Lawrence
b777e389b9 fixing up tests to work with for non-tarsum future
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-06 14:54:23 -08:00
xiekeyang
8081a13cdf :func Validate in digest doesn't filter no-hex data 2015-03-05 10:36:48 +08:00
Derek McGowan
2289b1f2e8 Replace unsupported hashes with supported
Remote md5 and sha1 hashes which are not supported by distribution.
Add more secure hashes sha384 and sha512.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-02-19 17:55:05 -08:00
Stephen J Day
0270bec916 Handle empty blob files more appropriately
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.

In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 13:01:49 -08:00
Stephen J Day
10a4605ec2 Explicitly select digest method for content
Detecting tar files then falling back for calculating digests turned out to be
fairly unreliable. Likely, the implementation was broken for content that was
not a tarfile. Also, for the use case of the registry, it is really not needed.
This functionality has been removed in FromReader and FromBytes. FromTarArchive
has been added for convenience.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-14 10:35:42 -08:00
Stephen J Day
adaa2246e7 Move testutil package to top-level
Since the common package no longer exists, the testutil package is being moved
up to the root. Ideally, we don't have large omnibus packages, like testutil,
but we can fix that in another refactoring round.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-05 16:53:13 -08:00
Stephen J Day
8be20212f1 Move tarsum utilities out of common package
In preparation for removing the common package, the tarsum utilities are being
moved to the more relevant digest package. This functionality will probably go
away in the future, but it's maintained here for the time being.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-05 16:04:30 -08:00
Olivier Gambier
67ca9d10cf Move from docker-registry to distribution 2014-12-23 17:13:02 -08:00
Stephen J Day
06ebc514a7 Automatically generate V2 API specification
This changeset provides data structures and definitions describing the routes
available in the V2 registry API. These route descriptors are structured to
provide automated registration, for creating routers, in addition to complete
documentation duty. It's also a possibility that this could be used to
enumerate test coverage for server implementation.

Using this functionality, we've also developed a template to automatically
generate and API specification for submission into docker core.
2014-12-18 21:29:56 -08:00
Stephen J Day
68944ea9cf Clean up layer storage layout
Previously, discussions were still ongoing about different storage layouts that
could support various access models. This changeset removes a layer of
indirection that was in place due to earlier designs. Effectively, this both
associates a layer with a named repository and ensures that content cannot be
accessed across repositories. It also moves to rely on tarsum as a true
content-addressable identifier, removing a layer of indirection during blob
resolution.
2014-11-25 09:57:43 -08:00
Stephen J Day
c0fe9d72d1 Various adjustments to digest package for govet/golint 2014-11-19 14:59:05 -08:00
Stephen J Day
3cfe9aede5 Move Digest type into discrete package
The Digest type will be fairly central for blob and layer management. The type
presented in this package provides a number of core features that should enable
reliable use within the registry. This commit will be followed by others that
convert the storage layer and webapp to use this type as the primary layer/blob
CAS identifier.
2014-11-19 14:25:55 -08:00