Commit graph

40 commits

Author SHA1 Message Date
Stephen J Day
ecb2ee3824
digest: better test coverage
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 17:33:33 -08:00
Derek McGowan
729b8c5b91 Merge pull request #2105 from stevvooe/algorithm-own-file
digest: cleanup digester and verifier creation
2016-12-15 16:55:37 -08:00
Stephen J Day
9159833265
digest: remove error return from Digest.Verifier
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 16:43:59 -08:00
Stephen J Day
e37baed88e
digest: cleanup digester and verifier creation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:17:21 -08:00
Stephen J Day
d5cc235c48
digest: remove stuttering ParseDigest function
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 14:58:54 -08:00
Stephen J Day
85994fcfc0
digest: add FromString helper to Digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-07 13:46:47 -08:00
Aaron Lehmann
aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Stephen J Day
9f5ebd0a94 digest: add NewDigestFromBytes for completeness
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-01-06 18:00:20 -08:00
Stephen Day
e63ad1e3d6 Merge pull request #1306 from stevvooe/from-bytes-algorithm
digest: make FromBytes available on digest.Algorithm
2015-12-29 18:59:47 -08:00
Stephen J Day
95bee1895a digest: panic on unavailable hash algorithm
After running into a few nil pointer errors during development, it is clear
that having this function return nil when a hash is not available is the wrong
approach. Nearly every time, this lack of availability was due to a missing
import statement for the hash. This is always a programming error.

To avoid future confusion, we now appropriately panic when the hash function is
not imported by the application. More dynamic uses of the package should call
Algorithm.Available() before calling Algorithm.Hash() to avoid this panic.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-29 15:24:32 -08:00
Stephen J Day
4646282448 digest: make FromBytes available on digest.Algorithm
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-29 15:16:56 -08:00
Aaron Lehmann
4c850e7165 Remove tarsum support for digest package
tarsum is not actually used by the registry. Remove support for it.

Convert numerous uses in unit tests to SHA256.

Update docs to remove mentions of tarsums (which were often inaccurate).

Remove tarsum dependency.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-15 17:22:18 -08:00
Aaron Lehmann
31047c8113 Simplify digest.FromBytes calling convention
The current implementation of digest.FromBytes returns an error. This
error can never be non-nil, but its presence in the function signature
means each call site needs error handling code for an error that is
always nil.

I verified that none of the hash.Hash implementations in the standard
library can return an error on Write. Nor can any of the hash.Hash
implementations vendored in distribution.

This commit changes digest.FromBytes not to return an error. If Write
returns an error, it will panic, but as discussed above, this should
never happen.

This commit also avoids using a bytes.Reader to feed data into the hash
function in FromBytes. This makes the hypothetical case that would panic
a bit more explicit, and should also be more performant.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-14 14:30:51 -08:00
Tonis Tiigi
f015982f0f Validate digest length on parsing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2015-12-02 16:04:06 -08:00
Aaron Lehmann
5608db616c Correct two golint comment issues
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-19 16:42:12 -07:00
Derek McGowan
dc6c54489c Add remove and list functions to digest set
Add mutex protection around set access

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-09-29 10:04:15 -07:00
Stephen J Day
7835d261d8 Add generic content digest tool
Previously a useful gist, this changeset polishes the original tarsum tool into
a utility that can be used to calculate content digests. Any algorithm from the
digest package is supported with additional support from tarsum.

This tool is very useful for quickly checking backend digests and verifying
correctness.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-20 14:55:34 -07:00
Stephen J Day
44da954565 Refactor specification of supported digests
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:08:23 -07:00
Stephen J Day
eee6cad2cf Remove digest package's dependency on external sha implementation
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.

We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 23:44:08 -07:00
Stephen Day
5556cd1ba1 Merge pull request #260 from dmcgowan/short-digests
Add short digest implementation
2015-04-29 14:05:41 -07:00
bin liu
e0521d2d01 fix some typos in source comments
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-04-17 12:39:52 +00:00
Andy Goldstein
7fdd395653 Use a build flag to disable resumable digests.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-04-16 01:19:57 +00:00
Josh Hawn
60b6748c95 Use resumable hash in digest package
Vendored resumable sha256/sha512 library. Digest package new exports a
resumable variant of the Digester.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-01 17:11:47 -07:00
Derek McGowan
7258fda98a Add digest set implementation
Set represents a unique set of digests which allow for efficient lookup.
Dumping short codes is a function which takes in a digest set.
Any operation involving short codes may be considered secure if the list of digests added to the set is the complete list of referenceable digests.
Contains benchmarks for Add, Lookup, and Dump.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-03-24 08:17:43 -07:00
Stephen Day
2ced84f31d Remove unnecessary error pointer formats in test
The linter was complaining about these and they are quite unnecessary.
2015-03-18 19:02:32 -07:00
Stephen J Day
8c254edb9a Correctly close pipe after error in tarsum verification
This addresses a subtle deadlock where an error during a copy prevented pipe
closure to propagate correctly. By closing down the read end of the pipe rather
than the write end, the waiting writer is properly signaled. A nice side-effect
of this change is that errors encountered by io.Copy are no propagated to the
verifier's Write method.

A test to ensure validation errors for unsupported digest types has been added,
as well.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-18 18:26:09 -07:00
xiekeyang
04f8bfbede supplement of digest refactoring 2015-03-11 14:13:29 +08:00
Josh Hawn
87959abe8f digest: Minor refactoring
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 16:44:19 -07:00
David Lawrence
b777e389b9 fixing up tests to work with for non-tarsum future
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-06 14:54:23 -08:00
xiekeyang
8081a13cdf :func Validate in digest doesn't filter no-hex data 2015-03-05 10:36:48 +08:00
Derek McGowan
2289b1f2e8 Replace unsupported hashes with supported
Remote md5 and sha1 hashes which are not supported by distribution.
Add more secure hashes sha384 and sha512.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-02-19 17:55:05 -08:00
Stephen J Day
0270bec916 Handle empty blob files more appropriately
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.

In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 13:01:49 -08:00
Stephen J Day
10a4605ec2 Explicitly select digest method for content
Detecting tar files then falling back for calculating digests turned out to be
fairly unreliable. Likely, the implementation was broken for content that was
not a tarfile. Also, for the use case of the registry, it is really not needed.
This functionality has been removed in FromReader and FromBytes. FromTarArchive
has been added for convenience.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-14 10:35:42 -08:00
Stephen J Day
adaa2246e7 Move testutil package to top-level
Since the common package no longer exists, the testutil package is being moved
up to the root. Ideally, we don't have large omnibus packages, like testutil,
but we can fix that in another refactoring round.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-05 16:53:13 -08:00
Stephen J Day
8be20212f1 Move tarsum utilities out of common package
In preparation for removing the common package, the tarsum utilities are being
moved to the more relevant digest package. This functionality will probably go
away in the future, but it's maintained here for the time being.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-05 16:04:30 -08:00
Olivier Gambier
67ca9d10cf Move from docker-registry to distribution 2014-12-23 17:13:02 -08:00
Stephen J Day
06ebc514a7 Automatically generate V2 API specification
This changeset provides data structures and definitions describing the routes
available in the V2 registry API. These route descriptors are structured to
provide automated registration, for creating routers, in addition to complete
documentation duty. It's also a possibility that this could be used to
enumerate test coverage for server implementation.

Using this functionality, we've also developed a template to automatically
generate and API specification for submission into docker core.
2014-12-18 21:29:56 -08:00
Stephen J Day
68944ea9cf Clean up layer storage layout
Previously, discussions were still ongoing about different storage layouts that
could support various access models. This changeset removes a layer of
indirection that was in place due to earlier designs. Effectively, this both
associates a layer with a named repository and ensures that content cannot be
accessed across repositories. It also moves to rely on tarsum as a true
content-addressable identifier, removing a layer of indirection during blob
resolution.
2014-11-25 09:57:43 -08:00
Stephen J Day
c0fe9d72d1 Various adjustments to digest package for govet/golint 2014-11-19 14:59:05 -08:00
Stephen J Day
3cfe9aede5 Move Digest type into discrete package
The Digest type will be fairly central for blob and layer management. The type
presented in this package provides a number of core features that should enable
reliable use within the registry. This commit will be followed by others that
convert the storage layer and webapp to use this type as the primary layer/blob
CAS identifier.
2014-11-19 14:25:55 -08:00