Commit graph

668 commits

Author SHA1 Message Date
Anton Tiurin
d4435b79d9 Fix comment for PathRegexp
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-12-04 22:14:21 +03:00
Sylvain Baubeau
ef4ab7a885 Use bulk delete to remove segments in Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-12-04 12:16:49 +01:00
Stephen Day
4cf93714f3 Merge pull request #1231 from tonistiigi/digest-length-check
Validate digest length on parsing
2015-12-03 12:17:16 -08:00
Stephen Day
d46b6a8796 Merge pull request #1211 from tt/remove-name-verification
Remove name verification
2015-12-03 11:59:21 -08:00
Stephen Day
b66bb1287c Merge pull request #1227 from stevvooe/walk-sorted
storage: enforce sorted traversal during Walk
2015-12-02 18:37:44 -08:00
Tonis Tiigi
f015982f0f Validate digest length on parsing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2015-12-02 16:04:06 -08:00
Aaron Lehmann
6beeb935cd Avoid stat round-trips when fetching a blob
Without this commit, three round-trips are required to fetch a blob with
a progress bar. The first is a call to Stat (HEAD request), to determine
the size. Then Open is called, which also calls Stat, and finally
performs a GET request.

Only the GET request is actually needed. The size of the blob can be
sniffed from Content-Length in the GET response.

This commit changes HTTPReadSeeker to automatically detect the size from
Content-Length instead of requiring it to be passed in. The Stat call is
removed from Open because it is no longer necessary.

HTTPReadSeeker now takes an additional errorHandler callback argument which
translates an unsuccessful HTTP response into an appropriate API-level
error. Using a callback for this makes it possible to avoid leaking the
repsonse body to Read's caller, which would make lifecycle management
problematic.

Fixes #1223

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-02 14:21:13 -08:00
Stephen J Day
8aa3ee6923 storage: add further tests for Walk implementation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:55:10 -08:00
Stephen J Day
5d576bc0cd storage: enforce sorted traversal during Walk
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:25:30 -08:00
Stephen J Day
6ad10796ef storage: correctly handle error during Walk
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-01 16:25:29 -08:00
Troels Thomsen
300ce35c12 Map error type to error code
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-12-01 22:26:37 +01:00
Troels Thomsen
1ece510198 Use well-known error type
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-12-01 22:23:47 +01:00
Troels Thomsen
d309bce2d1 Verify manifest name format
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-26 10:28:35 +01:00
Troels Thomsen
34c8194c95 Verify manifest name length
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-26 10:28:28 +01:00
Troels Thomsen
e8f8f4034e Remove name verification
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-25 21:16:28 +01:00
Troels Thomsen
ca9f0451a6 Use case of type name
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-11-25 21:12:49 +01:00
Stephen Day
0a3acb2625 Merge pull request #1166 from treed/master
De-obfuscate error message
2015-11-11 17:18:25 -08:00
Richard Scothern
bb745757de Merge pull request #1125 from rustyconover/fix-url-path-escaping
Fix failing test case for URL escaping problem.
2015-11-10 16:50:55 -08:00
Richard Scothern
a33095af80 Merge pull request #1141 from lebauce/swift-no-missing-segment
Ensure read after write for segments
2015-11-10 10:39:43 -08:00
Aaron Lehmann
6664a1384e Fix empty delete requests with Swift fs driver
The Delete method lists objects under the given path and tries to delete
all of them with a bulk delete request. If the path has no objects
underneath it, the body of this request will be empty, which causes
HTTP-level issues. Specifically, Go's HTTP client senses the empty
request buffer and doesn't include a Content-Length, which causes the
Swift server to fail the request.

This commit fixes the problem by avoiding sending empty bulk delete
requests. This is the correct thing to do anyway, since there's no
reason to request deletion of zero objects.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-11-09 15:33:05 -08:00
Ted Reed
fddbf73a57 De-obfuscate error message
Previously, this error message would stringify as a pointer address,
which isn't particularly helpful.

This change breaks out the elements of the challenge object such that
the error is appropriately represented.

Signed-off-by: Ted Reed <ted.reed@gmail.com>
2015-11-06 17:10:28 -08:00
Richard Scothern
362ae9cc41 Merge pull request #1156 from RichardScothern/manifest-verification
Manifest Verification
2015-11-04 14:09:12 -08:00
Richard Scothern
dd32fbe615 Before allowing a schema1 manifest to be stored in the registry, ensure that it
contains equal length History and FSLayer arrays.

This is required to prevent malformed manifests being put to the registry and
failing external verification checks.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-03 15:19:46 -08:00
Richard Scothern
bc6e4cdceb Add a generic error type to capture non-typed errors
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-11-03 11:19:44 -08:00
amitshukla
9637cb40cd Fix for issue 664: https://github.com/docker/distribution/issues/664
Errors thrown by storage drivers don't have the name of the driver, causing user
confusion about whether the error is coming from Docker or from a storage driver.
This change adds the storage driver name to each error message.

This required changing ErrUnsupportedDriver to a type, leading to code changes
whenever ErrUnsupportedDriver is used.  The tests check whether the driver name
appears in the error message.

Signed-off-by: Amit Shukla <amit.shukla@docker.com>
2015-11-03 11:19:17 -08:00
Richard Scothern
a9da0e5100 Merge pull request #1114 from lebauce/swift-temp-url
Redirect support in Swift driver
2015-11-03 09:34:01 -08:00
Sylvain Baubeau
3ff8af326b Ensure read after write for segments
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 17:09:52 +01:00
Sylvain Baubeau
7c3281861f Add support for temporary URL for Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 09:59:50 +01:00
Stephen J Day
46dd057f5c registry/storage: close filereader after allocation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-11-02 11:55:11 -08:00
Richard Scothern
33428c37e1 Buffer writing the scheduler entry state to disk by periodically checking for
changes to the entries index and saving it to the filesystem.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-29 11:35:52 -07:00
Richard Scothern
9e8aaf7b40 Merge pull request #756 from aibaars/storage-gcs
Storage driver for Google Cloud Storage
2015-10-29 10:41:17 -07:00
Richard Scothern
987a69dd05 Merge pull request #1096 from aaronlehmann/simplify-proxy-scheduler
Simplify proxy scheduler
2015-10-28 13:23:00 -07:00
Arthur Baars
59784ecdd0 Storage driver for: Google Cloud Storage (gcs)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2015-10-28 20:21:56 +00:00
Matt Moore
0c7cd3fcf8 Add expires_in and issued_at to the auth spec.
This extends the specification for the Bearer token response to include
information pertaining to when an issued Bearer token will expire.

This also allows the client to accept `access_token` as an alias for `token`.

Signed-off-by: Matt Moore <mattmoor@google.com>
2015-10-27 11:05:33 -07:00
Rusty Conover
5f2205981b Fix failing test case
When building a URL don't include path components in the host parameter.

Closes #1124

Signed-off-by: Rusty Conover <rusty@luckydinosaur.com>
2015-10-25 13:03:22 -04:00
Stephen Day
e5d9837970 Merge pull request #1098 from denverdino/master
Redundant digest verification in validateBlob when pushing a new layer
2015-10-21 14:50:18 -07:00
Tianon Gravi
39d47e8eea Update "type auth.Challenge" comment example code
This interface was changed in a0fdfb9d4d, but the comment wasn't ever updated to match.

Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
2015-10-21 11:54:13 -07:00
Aaron Lehmann
5608db616c Correct two golint comment issues
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-19 16:42:12 -07:00
Richard Scothern
d4910831a3 Merge pull request #1006 from RichardScothern/proxy-serialize
Fix a race condition in pull through cache population
2015-10-16 17:13:26 -07:00
Stephen Day
dfe60f4cb1 Merge pull request #827 from aaronlehmann/read-only-mode-2
Add a read-only mode as a configuration option
2015-10-15 11:50:31 -07:00
Aaron Lehmann
c56e8c2533 Simplify proxy scheduler
The proxy scheduler implemented its own timer state machine. It's
simpler and more efficient to leverage the Go runtime's timer heap by
using time.AfterFunc.

This commit adds a time.Timer to each scheduler entry, and starts and
stops those timers as necessary. Then the mainloop goroutine and its
associated logic are not needed.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-15 09:18:21 -07:00
Li Yi
ad14545745 Redundant digest verification in validateBlob
Change-Id: I03764edadae529db2cc3acf7ecca329570f18659
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-10-15 22:21:14 +08:00
Stephen J Day
76624704c3 Correct unmarshal order for SignedManifest
To ensure that we only unmarshal the verified payload into the contained
manifest, we first copy the entire incoming buffer into Raw and then unmarshal
only the Payload portion of the incoming bytes. If the contents is later
verified, the caller can then be sure that the contents of the Manifest fields
can be trusted.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-10-14 14:37:34 -07:00
Richard Scothern
4c4112bdcf Merge pull request #1035 from aaronlehmann/host-parameter-alternate
Add http.host parameter
2015-10-13 14:03:48 -07:00
Stephen Day
bfd1f423c5 Merge pull request #1033 from jlhawn/api_spec_update_auth_errors
[api spec] Update authN and authZ errors
2015-10-13 13:52:18 -07:00
Derek McGowan
31a448a628 Update to provide small and clear interfaces
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-10-09 16:05:49 -07:00
Tibor Vass
f8c09b6a7d Add a new reference package abstracting repositories, tags and digests
There seems to be a need for a type that represents a way of pointing
to an image, irrespective of the implementation.

This patch defines a Reference interface and provides 3 implementations:
- TagReference: when only a tag is provided
- DigestReference: when a digest (according to the digest package) is
  provided, can include optional tag as well

Validation of references are purely syntactic.

There is also a strong type for tags, analogous to digests, as well
as a strong type for Repository from which clients can access the
hostname alone, or the repository name without the hostname, or both
together via the String() method.

For Repository, the files names.go and names_test.go were moved from
the v2 package.

Signed-off-by: Tibor Vass <tibor@docker.com>
2015-10-09 16:05:34 -07:00
Richard Scothern
515b9596e7 Fix a race condition in pull through cache population by removing the functionality
of readers joining current downloads.  Concurrent requests for the same blob
will not block, but only the first instance will be comitted locally.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-10-08 16:12:29 -07:00
Matt Moore
74c32a0a51 Allow hostname components in component names.
Fixes https://github.com/docker/distribution/issues/1062

This relaxes the naming restrictions places on Docker images to permit
valid hostnames according to [RFC-2396](https://www.ietf.org/rfc/rfc2396.txt).

It deviates from the RFC in the following ways:
 1) Allow underscores where we allow hyphens (hostnames don't allow
 underscores, which we must for backwards compatibility).
 2) Leave "top-level" name segments unrestricted (domains require an
 alpha character to begin a top-level domain, e.g. "com").
 3) DO NOT allow a trailing dot, as permitted by FQDNs.

Signed-off-by: Matt Moore <mattmoor@google.com>
2015-10-08 11:29:54 -07:00
Richard Scothern
61a4a1362c Merge pull request #950 from jackpgriffin/master
Skip creating swift container if already exists
2015-10-01 10:15:54 -07:00
Aaron Lehmann
0a6988195e Add http.host parameter
This allows the administrator to specify an externally-reachable URL for
the registry. It takes precedence over the X-Forwarded-Proto and
X-Forwarded-Host headers, and the hostname in the request.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-30 15:12:01 -07:00
Josh Hawn
0f670bdc91 [api spec] Update authN and authZ errors
Associate HTTP 401s with Authentication errors rather than Authorization
errors. Changes the meaning of the UNAUTHORIZED error to be authentication
specific.

Defines DENIED error code to be associated with authorization
errors which result in HTTP 403 responses.

Add 'No Such Repository' errors to more endpoints.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-09-30 09:12:31 -07:00
Stephen Day
ca2156d558 Merge pull request #1024 from aaronlehmann/avoid-testing-import
Avoid importing "testing" in externally-facing code
2015-09-30 13:14:42 +02:00
Aaron Lehmann
6acb3a0d7e Avoid importing "testing" in externally-facing code
The "testing" package adds some flags in its init function, so utilities
that import distribution code may print a page of extra testing flags in
their help output.

This commit solves the issue by moving an import of "testing" in the
registry/storage/cache package to a new
registry/storage/cache/cachecheck package, which is only imported by
tests.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-29 10:06:39 -07:00
Aaron Lehmann
14a4b4d9d6 Merge pull request #1001 from RichardScothern/fix-atrocity
Don't return nil, nil from functions
2015-09-28 13:34:11 -07:00
Aaron Lehmann
a601f92336 Add an "enabled" parameter under "readonly", and make it as if the mutable handlers don't exist when read-only mode is enabled
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-22 15:49:26 -07:00
Aaron Lehmann
c9bb330b71 Add a read-only mode as a configuration option
Add "readonly" under the storage/maintenance section. When this is set
to true, uploads and deletions will return 503 Service Unavailable
errors.

Document the parameter and add some unit testing.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-22 15:47:48 -07:00
Nuutti Kotivuori
26d8344872 Remove initial access check from S3 driver
In the S3 storage driver there is currently an initial access permission check by listing the bucket. If this check fails, registry will panic and exit.

However, this check is broken in two ways. First of all it strips the final slash from the root directory path, meaning that any access permissions which limit access to a single directory will fail, because S3 treats the path as strict prefix match. Secondly it fails to strip any leading slash that might be present, unlike the other access places, which means that the path used is different as a leading slash is allowed and significant in a filename in S3.

Since there is also a periodic health check which correctly checks access permissions and shows the error more cleanly, the best solution seems to be to just remove this initial access check.

Signed-off-by: Nuutti Kotivuori <nuutti.kotivuori@poplatek.fi>
2015-09-19 12:47:12 +03:00
Richard Scothern
f36ab5a834 Don't return a nil array and a nil error if the Tags endpoint cannot be found
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-09-18 16:19:41 -07:00
Richard Scothern
924913b4c3 Avoid returning nil, nil when fetching a manifest by tag by introducing a new
error ErrManifestNotModified which can be checked by clients.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-09-18 16:19:37 -07:00
Stephen J Day
530afa5234 Add WithVersion to context and other cleanup
By adding WithVersion to the context package, we can simplify context setup in
the application. This avoids some odd bugs where instantiation order can lead
to missing instance.id or version from log messages.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-09-14 17:14:31 -07:00
Stephen J Day
360c24d975 Allow interface{} keys when using logger
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-09-11 11:33:55 -07:00
Aaron Lehmann
c4cf32c797 Only use the distribution/context package in registry.go
This solves a issue from #909 where instance.id was not printed in logs,
because this file was using the background context from
golang.org/x/net/context instead of
github.com/docker/distribution/context.

It's cleaner to standardize on one package, so this commit removes the
import of golang.org/x/net/context entirely. The Context interfaces
defined in both packages are the same, so other code using
golang.org/x/net/context can still pass its context to NewRegistry.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-11 09:54:15 -07:00
Aaron Lehmann
cbc9957e29 Add a cobra command that implements the entire main function for registry
Use this command in cmd/registry/main.go.

Move debug server to the main command, and change Serve to be a
ListenAndServe function.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-09 14:39:31 -07:00
Aaron Lehmann
9b69e40c93 Move initialization code from main.go to the registry package
This makes it easier to embed a registry instance inside another
application.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-09 14:39:31 -07:00
Jack Griffin
2749bcdc28 Skip creating swift container if already exists
Signed-off-by: Jack Griffin <jackpg14@gmail.com>
2015-09-04 18:53:17 +00:00
Hua Wang
ddb689ef8b Add TrustId parameter to swift driver
github/ncw/swift has added support for trust, so let's add it.

Signed-off-by: Hua Wang <wanghua.humble@gmail.com>
2015-09-02 15:05:29 +08:00
Stephen Day
1927c6c0f2 Merge pull request #934 from stevvooe/sanitize-url-correctly
Correctly sanitize location url preserving parameters
2015-08-27 22:30:21 -07:00
Stephen J Day
7232daf692 Correctly sanitize location url preserving parameters
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-27 17:06:03 -07:00
Stephen J Day
6712e602b0 Move manifest package to schema1
As we begin our march towards multi-arch, we must prepare for the reality of
multiple manifest schemas. This is the beginning of a set of changes to
facilitate this. We are both moving this package into its target position where
it may live peacefully next to other manfiest versions.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-21 16:29:47 -07:00
Stephen J Day
3f7f956f80 More consistent return from ErrorCode.Error()
To bring ErrorCode into liine with Go conventions, ErrorCode.Error() now
returns the "nice" value of the error code. This ensures error message assembly
works similar to commonly used Go conventions when directly using ErrorCode as
an error.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-20 17:36:24 -07:00
Aaron Lehmann
b67aab2f60 Add headers parameter for HTTP checker
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 15:11:16 -07:00
Aaron Lehmann
e8f088fea6 Add a TCP health checker
Also, add timeout and status code parameters to the HTTP checker, and
remove the threshold parameter for the file checker.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 15:11:15 -07:00
Aaron Lehmann
b9b9cafa8f Expose a Registry type in health package, so unit tests can stay isolated from each other
Update docs.

Change health_test.go tests to create their own registries and register
the checks there. The tests now call CheckStatus directly instead of
polling the HTTP handler, which returns results from the default
registry.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 15:07:54 -07:00
Aaron Lehmann
79959f578a Switch tests to import "github.com/docker/distribution/context"
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 14:50:12 -07:00
Aaron Lehmann
216df32510 Add storagedriver section to health check configuration
Add default storagedriver health check to example configuration files
with parameters matching the previous hardcoded configuration.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 14:50:12 -07:00
Aaron Lehmann
b09b0ffcf9 Add configurable file-existence and HTTP health checks
Add a section to the config file called "health". Within this section,
"filecheckers" and "httpcheckers" list checks to run. Each check
specifies a file or URI, a time interval for the check, and a threshold
specifying how many times the check must fail to reach an unhealthy
state.

Document the new options in docs/configuration.md.

Add unit testing for both types of checkers. Add an UnregisterAll
function in the health package to support the unit tests, and an
Unregister function for consistency with Register.

Fix a string conversion problem in the health package's HTTP checker.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-20 14:50:12 -07:00
Stephen Day
e4b93d1e6d Merge pull request #893 from aaronlehmann/functional-opts
Functional options for NewRegistryWithDriver
2015-08-19 14:41:08 -07:00
Aaron Lehmann
5bef618ace Add a unit test which verifies the ResponseWriter endpoints see implements CloseNotifier
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-19 11:37:53 -07:00
Aaron Lehmann
dbbcf5fe42 Functional options for NewRegistryWithDriver
Clean up calling convention for NewRegistryWithDriver to use functional
arguments.

This is a first step towards the refactor described in #215. I plan to
add additional options in the process of moving configurable items from
the App structure to the registry structure.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-19 11:06:30 -07:00
Aaron Lehmann
0e7462f1dd Fix CloseNotifier handling and avoid "the ResponseWriter does not implement CloseNotifier" warnings in logs
A change in #763 to address review comments caused problems. Originally,
instrumentedResponseWriter implemented the CloseNotifier interface, and
would panic if it was wrapping something that did not implement that
interface. This was split into a separate instrumentedResponseWriterCN
type that implements CloseNotifier, so there's a fallback if
instrumentedResponseWriter ever needs to wrap something that does not
implement this interface.

instrumentedResponseWriter's Value method would end up upcasting either
type back to instrumentedResponseWriter, which does not implement the
interface. In effect, instrumentedResponseWriterCN was never visible to
the handler.

This fixes the problem by implementing a wrapper Value method for
instrumentedResponseWriterCN.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-18 17:28:51 -07:00
Aaron Lehmann
98d9b5f589 Fix tests after #846
Change checkResponse to only expect the configured
X-Content-Type-Options header if it doesn't receive a 405 error, which
means the handler isn't registered for that method.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-18 13:33:26 -07:00
Richard Scothern
f169359798 Merge pull request #846 from aaronlehmann/http-header-configuration
Add a section to the config file for HTTP headers to add to responses
2015-08-18 12:53:05 -07:00
Stephen J Day
641cdf3ba6 Remove pathMapper object
The use of the pathMapper is no longer needed the way we have organized the
code base. The extra level of indirection has proved unnecessary and confusing
so we've opted to clean it up. In the future, we may require more flexibility,
but now it is simply not required.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-17 18:51:05 -07:00
Richard Scothern
44c7fb9e6e Merge pull request #864 from stevvooe/use-correct-manifest-link
registry/storage: use correct manifest link
2015-08-12 15:13:45 -07:00
Stephen J Day
06a098c632 Maintain manifest link compatibility
Unfortunately, the 2.1 releease has written manfiest links into the wrong
directory. This doesn't affect new 2.1 deployments but fixing this to be 2.0
backwards compatible has broken 2.1.0 compatibility. To ensure we have
compatibility between 2.0, 2.1.0 and future releases, we now check one of
several locations to identify a manifest link.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-12 13:11:13 -07:00
Stephen J Day
d00586de9f Use correct path for manifest revision path
Unfortunately, the refactor used the incorrect path for manifest links within a
repository. While this didn't stop the registry from working, it did break
compatibility with 2.0 deployments for manifest fetches.

Tests were added to ensure these are locked down to the appropriate paths.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-12 13:10:15 -07:00
Aaron Lehmann
9c3bed6b88 Add a section to the config file for HTTP headers to add to responses
The example configuration files add X-Content-Type-Options: nosniff.

Add coverage in existing registry/handlers unit tests.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-11 16:41:38 -07:00
Richard Scothern
776a4ffbe8 Change some incorrect error types in proxy stores from API errors to
distribution errors.  Fill in missing checks for mutations on a registry pull-through
cache.  Add unit tests and update documentation.

Also, give v2.ErrorCodeUnsupported an HTTP status code, previously it was
defaulting to 500, now its 405 Method Not Allowed.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-11 14:16:24 -07:00
Stephen J Day
f141480d98 Move common error codes to errcode package
Several error codes are generally useful but tied to the v2 specification
definitions. This change moves these error code definitions into the common
package for use by the health package, which is not tied to the v2 API.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-11 11:50:58 -07:00
Stephen J Day
6ba799b69e Provide simple storage driver health check
To ensure the ensure the web application is properly operating, we've added a
periodic health check for the storage driver. If the health check fails three
times in a row, the registry will serve 503 response status for any request
until the condition is resolved. The condition is reported in the response body
and via the /debug/health endpoint.

To ensure that all drivers will properly operate with this health check, a
function has been added to the driver testsuite.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-11 11:50:52 -07:00
Vincent Giersch
6c39af6708 fix(rados): Create OMAP for root directory
When using the RADOS driver, the hierarchy of the files is stored
in OMAPs, but the root OMAP was not created and a call to List("/")
was returning an error instead of returned the first level files
stored. This patches creates an OMAP for "/" and excludes the listed
directory from the list of files returned.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-08-10 23:46:33 +02:00
Aaron Lehmann
10f602b158 Don't panic when a http.ResponseWriter does not implement CloseNotifier
Instead, provide a variant of instrumentedResponseWriter that does not
implement CloseNotifier, and use that when necessary. In
copyFullPayload, log instead of panicing when we encounter something
that doesn't implement CloseNotifier.

This is more complicated than I'd like, but it's necessary because
instrumentedResponseWriter must not embed CloseNotifier unless there's
really a CloseNotifier to embed.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-08-06 15:50:54 -07:00
Stephen Day
a0c63372fa Merge pull request #779 from RichardScothern/pull-through-cache
Add pull through cache ability to the Registry.
2015-08-04 17:04:56 -07:00
Richard Scothern
94935f39bc Add pull through cache functionality to the Registry which can be configured
with a new `proxy` section in the configuration file.

Create a new registry type which delegates storage to a proxyBlobStore
and proxyManifestStore.  These stores will pull through data if not present
locally.  proxyBlobStore takes care not to write duplicate data to disk.

Add a scheduler to cleanup expired content. The scheduler runs as a background
goroutine.  When a blob or manifest is pulled through from the remote registry,
an entry is added to the scheduler with a TTL.  When the TTL expires the
scheduler calls a pre-specified function to remove the fetched resource.

Add token authentication to the registry middleware.  Get a token at startup
and preload the credential store with the username and password supplied in the
config file.

Allow resumable digest functionality to be disabled at runtime and disable
it when the registry is a pull through cache.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-04 16:09:55 -07:00
Stephen Day
ef1c72b978 Merge pull request #805 from RichardScothern/811
Spelling corrections
2015-08-04 14:50:32 -07:00
Aaron Lehmann
3a414deddb Merge pull request #514 from denverdino/master
Storage Driver for Aliyun OSS
2015-08-04 12:01:02 -07:00
Richard Scothern
db12c889e1 Merge pull request #763 from aaronlehmann/close-notifier
Use CloseNotifier to supress spurious HTTP 400 errors on early disconnect
2015-08-03 13:57:32 -07:00
Doug Davis
633eec0f91 Fix vet issue
registry/storage/blob_test.go:149: arg d for printf verb %s of wrong type: github.com/docker/distribution.Descriptor

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-08-03 12:24:34 -07:00
Richard Scothern
fb9662ad5a Add blob delete entry to api description and regenerate api.md
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-08-03 11:59:19 -07:00
Aaron Lehmann
609efa79e4 Set the response code to 499 when a client disconnects during an upload
The response code isn't actually sent to the client, because the
connection has already closed by this point. But it causes the status
code to appear as 499 in the logs instead of 0.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:39:30 -07:00
Aaron Lehmann
9c58954a6e Factor CloseNotifier use into a new function
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:30:26 -07:00
Aaron Lehmann
6cb5670ba5 Use CloseNotifier to supress spurious HTTP 400 errors on early disconnect
When a client disconnects without completing a HTTP request, we were
attempting to process the partial request, which usually leads to a 400
error. These errors can pollute the logs and make it more difficult to
track down real bugs.

This change uses CloseNotifier to detect disconnects. In combination
with checking Content-Length, we can detect a disconnect before sending
the full payload, and avoid logging a 400 error.

This logic is only applied to PUT, POST, and PATCH endpoints, as these
are the places where disconnects during a request are most likely to
happen.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:30:26 -07:00
Richard Scothern
bffa20d6bb Spelling corrections
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-31 16:43:11 -07:00
tgic
72c794a810 remove unused code and fix todo format
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
e53d99a837 fix goimports
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
tgic
0058b08eeb add include_oss build tag
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:56 +08:00
Li Yi
708ad28114 Update the comment for the consistency model
Change-Id: Iee49afeda1c11d6af8c0f26c96d8ccc328c22757
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
Li Yi
539c7f5311 Update the comments for consistence model
Change-Id: I161522ee51f247fb17e42844b3699bd9031e34e8
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
8e5c901a26 fix testcase TestReadStreamWithOffset incompatible with oss
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
2a8535626f fix oss: got 403 in TestContinueStreamAppendLarge
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
Li Yi
813543d6e3 Update the OSS test case for latest code change
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
3a240de22e check access key and secret before run 2015-07-31 13:22:55 +08:00
tgic
3d30cb38f6 add endpoint support 2015-07-31 13:22:54 +08:00
Li Yi
c3b42db014 Add the secure access with HTTPS
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
b8a276f2db Fix the warning of golint
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Li Yi
9e4975d8ff Support OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Stephen Day
2b74c46795 Merge pull request #791 from stevvooe/add-double-domain-image-name
Add image name tests around hostnames
2015-07-30 14:35:27 -07:00
Stephen Day
ad1b7d8f35 Merge pull request #740 from stevvooe/disable-storage-redirects
Allow disabling of storage driver redirects
2015-07-30 14:28:31 -07:00
Stephen J Day
bb5466245c Add image name tests around hostnames
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-30 14:26:21 -07:00
Stephen Day
9038e48c3b Merge pull request #609 from RichardScothern/single-log-line
Log a single line when a response completes.
2015-07-29 17:52:13 -07:00
Richard Scothern
54f7e5d54b When a request completes ensure only one log line is printed which
includes the http response.

When debugging non-successful registry requests this will place
the error details and http status fields in the same log line
giving easier visibility to what error occured in the request.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-29 16:52:47 -07:00
Aaron Lehmann
70d9a9d543 Automatically generate a HTTP secret if none is provided
Log a warning if the registry generates its own secret.

Update configuration doc, and remove the default secret from the
development config file.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-29 14:28:56 -07:00
Aaron Lehmann
73702a8707 Fix for api_test.go
This passed in the #744 before merge, but apparently the test changed
since the PR was created in ways that led to a new failures.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-28 10:59:11 -07:00
Derek McGowan
0355c3026c Merge pull request #744 from aaronlehmann/manifest-put-response-code
Manifest PUT should return 201 Created
2015-07-28 10:42:54 -07:00
Stephen J Day
9f9a7f230b Allow disabling of starage driver redirects
Storage drivers can implement a method called URLFor which can return a direct
url for a given path. The functionality allows the registry to direct clients
to download content directly from the backend storage. This is commonly used
with s3 and cloudfront. Under certain conditions, such as when the registry is
not local to the backend, these redirects can hurt performance and waste
incoming bandwidth on pulls. This feature addition allows one to disable this
feature, if required.

Signed-off-by: Stephen J Day <stephen.day@docker.com>

Conflicts:
	configuration/configuration.go
	registry/handlers/app.go
	registry/storage/catalog_test.go
	registry/storage/manifeststore_test.go
	registry/storage/registry.go
2015-07-24 16:59:35 -07:00
Aaron Lehmann
be404d7557 Make the registry client more tolerant about HTTP status codes
Generally, all 2xx and 3xx codes should be treated as success.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-24 16:14:04 -07:00
Stephen Day
060465882b Merge pull request #736 from stevvooe/authorization-interface-cleanup
Authorization interface cleanup
2015-07-24 15:39:49 -07:00
Stephen Day
b49d77a42f Merge pull request #739 from stevvooe/etags-must-be-quoted
Etags must be quoted according to http spec
2015-07-24 15:08:27 -07:00
Stephen J Day
338e645f20 Etags must be quoted according to http spec
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-24 13:07:38 -07:00
Stephen Day
9e9e9a2859 Merge pull request #677 from RichardScothern/soft-delete-remove-links
Manifest and layer soft deletion
2015-07-24 13:02:36 -07:00
Aaron Lehmann
cf32056218 Manifest PUT should return 201 Created
Change handler, update descriptors table, regenerate API spec, and
update test.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-24 12:58:16 -07:00
Richard
9c1dd69439 Manifest and layer soft deletion.
Implement the delete API by implementing soft delete for layers
and blobs by removing link files and updating the blob descriptor
cache.  Deletion is configurable - if it is disabled API calls
will return an unsupported error.

We invalidate the blob descriptor cache by changing the linkedBlobStore's
blobStatter to a blobDescriptorService and naming it blobAccessController.

Delete() is added throughout the relevant API to support this functionality.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-24 09:57:20 -07:00
Stephen J Day
0b89cdfcd4 Do not replace logger when adding hooks
Because the logger was incorrectly replaced while adding hooks, log output did
not include the version and instance ids. The main issue was the the
logrus.Entry was replaced with the logger, which included no context. Replacing
the logger on the context is not necessary when configuring hooks since we are
configuring the contexts logger directly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-23 20:51:11 -07:00
Stephen J Day
d31f9fd5b1 auth.AccessController interface now uses distribution/context
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-23 19:48:47 -07:00
Stephen J Day
a0fdfb9d4d Simplify auth.Challenge interface to SetHeaders
This removes the erroneous http.Handler interface in favor a simple SetHeaders
method that only operattes on the response. Several unnecessary uses of pointer
types were also fixed up.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-23 19:47:57 -07:00
Stephen Day
984037f7fc Merge pull request #729 from stevvooe/pagination-spec-cleanup
Clean up pagination specification
2015-07-23 15:46:32 -07:00
Doug Davis
307f34eeef Remove dead code
thanks to @tiborvass for noticing

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-07-23 07:09:48 -07:00
Stephen Day
34e5b1849c Merge pull request #493 from nevermosby/storage-support-openstack-swift
Storage support openstack swift
2015-07-22 23:53:36 -07:00
Stephen J Day
88831b0523 Clean up pagination specification
Some missing descriptions and error code for tags pagination was cleaned up to
ensure clarity. Specifically, we ensure the request variations are named and
the proper error codes are included.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-22 20:00:28 -07:00
Stephen Day
76f29c2630 Merge pull request #653 from pdevine/catalog-api
Catalog for V2 API Implementation
2015-07-22 18:54:48 -07:00
Aaron Lehmann
4703e9980c Merge pull request #724 from aaronlehmann/repository-name-regexp-tests
Add additional test coverage for the regexp contained in RepositoryNameRegexp
2015-07-22 18:41:44 -07:00
Aaron Lehmann
0d27f70d0c Unify the testcases for the two tests in names_test.go
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-22 18:33:15 -07:00
Patrick Devine
14749fdce4 Add Registry to client bindings for Repositories
The way Repositories() was initially called was somewhat different than
other parts of the client bindings because there was no way to instantiate a
Namespace.  This change implements a NewRegistry() function which changes
it so that Repositories() can be called the way one would expect.

It doesn't implement any of the other functions of Namespaces.

Signed-off-by: Patrick Devine <patrick.devine@docker.com>
2015-07-22 17:48:15 -07:00
Doug Davis
f5283cc5a2 Make Error.Error() return the post-arg-substitution Message
Missed this during the removal of the args property

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-07-22 12:44:59 -07:00
Stephen Day
e04ea232b5 Merge pull request #709 from hopkings2008/master
Close reader after the test is finished.
2015-07-22 12:22:22 -07:00
Patrick Devine
b7e26bac74 Create Repositories method
This change removes the Catalog Service and replaces it with a more
simplistic Repositories() method for obtaining a catalog of all
repositories.  The Repositories method takes a pre-allocated slice
and fills it up to the size of the slice and returns the amount
filled.  The catalog is returned lexicographically and will start
being filled from the last entry passed to Repositories().  If there
are no more entries to fill, io.EOF will be returned.

Signed-off-by: Patrick Devine <patrick.devine@docker.com>

Conflicts:
	registry/client/repository.go
	registry/handlers/api_test.go
2015-07-21 21:45:14 -07:00
Patrick Devine
74563efe98 Catalog for V2 API Implementation
This change adds a basic catalog endpoint to the API, which returns a list,
or partial list, of all of the repositories contained in the registry.  Calls
to this endpoint are somewhat expensive, as every call requires walking a
large part of the registry.

Instead, to maintain a list of repositories, you would first call the catalog
endpoint to get an initial list, and then use the events API to maintain
any future repositories.

Signed-off-by: Patrick Devine <patrick.devine@docker.com>
2015-07-21 21:45:14 -07:00
Stephen J Day
006214d902 Paginate catalog and tag results with Link header
Move the specification to use a Link header, rather than a "next" entry in the
json results. This prevents requiring clients from parsing the request body to
issue the next request. It also ensures that the returned response body does
not change in between requests.

The ordering of the specification has been slightly tweaked, as well. Listing
image tags has been moved after the catalog specification. Tag pagination now
heavily references catalog pagination.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-21 21:45:14 -07:00
Stephen J Day
aebe850f73 Catalog V2 API specification proposal
This contains a proposal for a catalog API, provided access to the internal
contents of a registry instance. The API endpoint is prefixed with an
underscore, which is illegal in images names, to prevent collisions with
repositories names. To avoid issues with large result sets, a paginated version
of the API is proposed. We make an addition to the tags API to support
pagination to ensure the specification is conistent.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-21 21:45:14 -07:00
Aaron Lehmann
9423b38d5f Add additional test coverage for the regexp contained in RepositoryNameRegexp
This was inspired by problems found with new regexps proposed in PR #690

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-21 17:10:36 -07:00
davidli
d1b4bae078 Remove IPC support from test file
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
cca15a76f1 Catch either missing containers or objects
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
b4cf6c053b Do not use Swift server side copy for manifests to handle >5G files
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
326c3a9c49 Inline Swift errors handling
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:10 +02:00
Sylvain Baubeau
15d567671b Retrieve all the objects using pagination
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
49582a6188 Do not create objects for directories
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
ed08d8d4e0 Refactor segment path concatenation code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
91e17e10cd Use http.StatusRequestedRangeNotSatisfiable instead of error code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f4ca709d70 Use file instead of filepath as it may cause troubles on Windows
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
78d722e708 Protect against deletion of objects with the same prefix
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
61e3bce794 Show distribution version in User-Agent
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f256797cac Do not use suite style testing for Swift specific tests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
53e80944f1 Rename environment variables to run Swift testsuite
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
d0dd36125c Check file has been opened before closing it
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1a80ec340b Rename DriverParameters structure to Parameters
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
2846913552 Change folder mime type to application/vnc.swift.directory
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
33028c11a7 Add code documentation
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f21cbcebc6 Do not read segment if no padding is necessary
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3242378e1b Improve 404 errors handling
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
57cef57e1b Use only one Swift container for both files and manifests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
a40502ec02 Increase default chunk size to 20M
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
davidli
af99dbd6bf Add support for Openstack Identity v3 API
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3fb42a1502 Use 'prefix' parameter instead of 'path' when listing files
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1c01ca4b39 Create full folder hierarchy instead of just the top level folder
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
c4d845cb6c Remove one level of indentation in swift path handling code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
4497d6c973 Use mitchellh/mapstructure library to parse Swift parameters
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
2632fd9dc8 Move Dynamic Large Object handling to dedicated methods
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
d8d50fb0e0 Simplify code that handles non existing manifests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
5e650a820a Handle error during copy of original content
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
nevermosby
7372c13f29 Update the import path for swift driver test
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
davidli
6dc1596be4 Use gofmt to format the code of swift driver.
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
ea7c53df08 Add Openstack Swift storage driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:08 +02:00
Stephen Day
5e9d0702a1 Merge pull request #713 from stevvooe/descriptor-size-field
Use "Size" field to describe blobs over "Length"
2015-07-17 22:06:59 -07:00
Stephen J Day
26b7fe4a91 Use "Size" field to describe blobs over "Length"
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-17 17:07:11 -07:00
yuzou
0fc7babf76 Close reader after the test is finished.
Signed-off-by: yuzou <zouyu7@huawei.com>
2015-07-17 14:55:31 +08:00
Stephen Day
fed58bd2d3 Merge pull request #705 from stevvooe/export-servejson-errors
Export ServeJSON for serving error codes
2015-07-16 14:35:27 -07:00
Brian Bland
c25f4af00f Reduces log level of auth error lines from error->warn
An error level log is already produced within app.authorized() if an
actual unexpected error occurs during authorization, so this warning
level log remains for auditability purposes, but should not be
considered an error condition.

Addresses #704

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-07-16 12:43:33 -07:00
Stephen J Day
86ffe515df Export ServeJSON for serving error codes
This changeset provides a common http handler for serving errcodes. This should
unify http responses across webservices in the face of errors.

Several type assertions have been added, as well, to ensure the error interface
is implemented.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-16 12:14:14 -07:00
Richard
f331da2daa Allow Manifest Service to be configured with function arguments
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-15 12:25:16 -07:00
Stephen Day
2b88cb9413 Merge pull request #651 from duglin/ErrArgs
Add ability to pass in substitution args into an Error
2015-07-15 11:22:59 -07:00
Doug Davis
94e2e9f4a0 Add ability to pass in substitution args into an Error
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-07-15 11:02:10 -07:00
Derek McGowan
0cda2f61e8 Merge pull request #699 from RichardScothern/client-manifest-etags-clean
Allow conditional fetching of manifests with the registry client.
2015-07-14 17:29:59 -07:00
Richard Scothern
48a2010ac3 Allow conditional fetching of manifests with the registry client.
Add a functional argument to pass a digest to (ManifestService).GetByTag().
If the digest matches an empty manifest and nil error are returned.

See 1bc740b0d5 for server implementation.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-14 16:52:52 -07:00
Richard Scothern
7e6fda610d Fix build when using build tag 'noresumabledigest'
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-14 11:14:09 -07:00
Stephen J Day
b54cc5ed47 Allow single character repository names
The main goal of this changeset is to allow repository name components to
consist of a single character. The number of components allowed and the slash
separation requirements have also been clarified.

To go along with this simplification, errant constants and unneeded error types
have been removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-10 12:10:42 -06:00
Derek McGowan
8fc782ae09 Fix typo in Version doc
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-08 11:02:47 -07:00
Derek McGowan
3531b22b46 Add challenge manager interface
Challenger manager interface is used to handle getting authorization challenges from an endpoint as well as extracting challenges from responses.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-01 15:00:25 -07:00
Derek McGowan
c8fac94617 Separate version and challenge parsing from ping
Replace ping logic with individual functions to extract API version and authorization challenges. The response from a ping operation can be passed into these function. If an error occurs in parsing, the version or challenge will not be used. Sending the ping request is the responsibility of the caller.
APIVersion has been converted from a string to a structure type. A parse function was added to convert from string to the structure type.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-01 15:00:25 -07:00
Derek McGowan
5a7dab4670 Refactor client auth
Move client auth into a separate package.
Separate ping from the authorizer and export Challenges type.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-01 15:00:25 -07:00
Stephen Day
c56e28826e Merge pull request #670 from stevvooe/remove-ipc
Remove half-baked Storage Driver IPC support
2015-06-30 19:21:54 -07:00
Jon Poler
f09051fe54 Remove timeout for http.Client in registry/client/repository.go.
Timeouts should not be a discrete period of time, because they end
up being arbitrary and may be difficult to gauge correctly against
very large Docker layers. Rather, timeouts should be set at the
transport level using the SetDeadline attribute on a net.Conn
object.

Signed-off-by: Jon Poler <jonathan.poler@apcera.com>
2015-06-30 17:35:24 -07:00
Stephen Day
c1423aa2aa Merge pull request #675 from RichardScothern/fix-ctx
Pass correct context into tracer
2015-06-30 11:18:51 -07:00
Richard Scothern
3629aac84f Pass correct context into tracer
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-30 10:28:14 -07:00
Stephen J Day
d3d4423ff7 Remove half-baked Storage Driver IPC support
This removes documentation and code related to IPC based storage driver
plugins. The existence of this functionality was an original feature goal but
is now not maintained and actively confusing incoming contributions. We will
likely explore some driver plugin mechanism in the future but we don't need
this laying around in the meantime.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-29 16:52:12 -07:00
Richard Scothern
0edb91185b Prevent the ErrUnsupportedMethod error from being returned up the stack.
It eventually causes the go http library to do a double WriteHeader()
which is an error

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-29 16:44:09 -07:00
Stephen Day
b6faf3fa9f Merge pull request #645 from RichardScothern/manifest-cache-headers
Cache headers for manifests.
2015-06-25 13:20:39 -07:00
Richard Scothern
1bc740b0d5 Add Etag header for manifests.
Return 304 (Not Modified) if retrieved with If-None-Match header

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-19 10:44:21 -07:00
Doug Davis
7bb98181ab Add 'message' back to BlobTest sample json
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-18 18:24:54 -07:00
Doug Davis
cb45ec56ff Add back in the "errors" wrapper in the Errors serialization
See: https://github.com/docker/distribution/pull/548/files#r32794064

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-18 18:00:26 -07:00
Stephen Day
d2ca423500 Merge pull request #638 from duglin/MoveHTTPRC
Move challenge http status code logic
2015-06-18 12:03:29 -07:00
Doug Davis
c4eb195cc1 Move challenge http status code logic
See: 3ea67df373/registry/handlers/app.go (L498)

Per the comment on line 498, this moves the logic of setting the http
status code into the serveJSON func, leaving the auth.Challenge.ServeHTTP()
func to just set the auth challenge header.

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-17 18:23:55 -07:00
Stephen J Day
b65a4a8713 Ensure that rados is disabled without build tag
This ensures that rados is not required when building the registry. This was
slightly tricky in that when the flags were applied, the rados package was
completely missing. This led to a problem where rados was basically unlistable
and untestable as a package. This was fixed by simply adding a doc.go file that
is included whether rados is built or not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-16 20:00:02 -07:00
Stephen Day
3ea67df373 Merge pull request #623 from ahmetalpbalkan/azure-vendor
storage/driver/azure: Update vendored Azure SDK
2015-06-16 17:41:38 -07:00
Ahmet Alp Balkan
daa22cacba storage/driver/azure: Update vendored Azure SDK
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.

I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...

This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.

Fixes #620, #175.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-06-16 17:13:44 -07:00
Olivier Gambier
d678e3dc57 Merge pull request #608 from dmcgowan/http-basic-auth
Implementation of a basic authentication scheme using standard .htpasswd
2015-06-15 17:40:58 -07:00
Stephen Day
e57e731821 Merge pull request #548 from duglin/MoveErrors
Move ErrorCode logic to new errcode package
2015-06-15 14:33:28 -07:00
Doug Davis
441f7cac87 Round 4
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-11 21:33:35 -07:00
Stephen J Day
d9d55bcbab Minor formatting fixes related to htpasswd auth
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-11 17:06:35 -07:00
Olivier Gambier
6e0cfc17dc Saner default data location
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-10 20:54:24 -07:00
Stephen J Day
0f654c25ac Rename the basic access controller to htpasswd
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-10 19:41:35 -07:00
Stephen J Day
ffd3662982 Harden basic auth implementation
After consideration, the basic authentication implementation has been
simplified to only support bcrypt entries in an htpasswd file. This greatly
increases the security of the implementation by reducing the possibility of
timing attacks and other problems trying to detect the password hash type.

Also, the htpasswd file is only parsed at startup, ensuring that the file can
be edited and not effect ongoing requests. Newly added passwords take effect on
restart. Subsequently, password hash entries are now stored in a map.

Test cases have been modified accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-10 19:38:56 -07:00
Stephen J Day
ffe56ebe41 Refactor Basic Authentication package
This change refactors the basic authentication implementation to better follow
Go coding standards. Many types are no longer exported. The parser is now a
separate function from the authentication code. The standard functions
(*http.Request).BasicAuth/SetBasicAuth are now used where appropriate.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
abd142855a Unexported function to comply with golint
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
e4c3ab4377 Removed dashes from comments, unexported htpasswd struct
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
04f6a4811d Fixed golint, gofmt warning advice.
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
c50dfb7dae Added support for bcrypt, plaintext; extension points for other htpasswd hash methods.
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
c4849bb99a Aligned formatting with gofmt
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
0ecaa7f40a Fixed WWW-Authenticate: header, added example config and import into main, fixed golint warnings
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
BadZen
8a204f59e7 Implementation of a basic authentication scheme using standard .htpasswd files
Signed-off-by: BadZen <dave.trombley@gmail.com>
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:37 -07:00
Stephen Day
f63313de1f Merge pull request #599 from stevvooe/clarify-deletion-by-digest-constraint
Clarify digest in API specification
2015-06-08 19:04:53 -07:00
Stephen J Day
c54adb667f Clarify that manifests can only be deleted by digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-04 18:59:08 -07:00
Stephen J Day
4bff4910b5 Add more repository name validation test cases
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-04 16:12:35 -07:00
Doug Davis
38393b63b7 Round 3 - Add Register function
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-02 10:01:21 -07:00
Vincent Giersch
7b3f571555 doc: coherence between requests and parameters + typo
In the request parameters lists `tag` was used instead of
`reference` present in the HTTP requests paths

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-06-02 14:36:55 +00:00
Stephen Day
0d40913b9a Merge pull request #394 from xiekeyang/feature-panic-hook
Feature: Add Hook for Web Application Panic
2015-06-01 13:23:32 -07:00
Vincent Giersch
3c9a2a9bac Fix rados build, remove uuid dependency
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-06-01 09:57:40 +00:00
xiekeyang
47aa47e3f6 Feature: Web Panic Reporting via hooks
This PR is for issue of "email after registry webapp panic" #41, improving my
previous design (closed).
It use self setting up hooks, to catch panic in web application.
And, send email in hooks handle directly, to no use new http server and
handler.

Signed-off-by: xiekeyang <keyangxie@126.com>
2015-05-31 14:21:22 +00:00
Philip Misiowiec
1f31f595d9 fixed typos
Signed-off-by: Phil Misiowiec <phil.misiowiec@atlashealth.com>
2015-05-30 18:22:41 -07:00
Stephen Day
318af0b1ce Merge pull request #443 from gierschv/driver-rados
Storage Driver: Ceph Object Storage (RADOS)
2015-05-29 18:10:49 -07:00
Doug Davis
8a0827f799 Round 2
Make Errors a []Error

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-05-26 17:18:32 -07:00
Doug Davis
0a6a6f5b81 Move ErrorCode logic to new errcode package
Make HTTP status codes match the ErrorCode by looking it up in the Descriptors

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-05-26 13:18:54 -07:00
Stephen Day
a3006e8e45 Merge pull request #556 from stevvooe/remove-uuid-dependency
Replace uuid dependency with internal library
2015-05-26 10:39:35 -07:00
Stephen J Day
36e34a55ad Replace uuid dependency with internal library
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:59:41 -07:00
Stephen Day
8ce4dcaef8 Merge pull request #546 from stevvooe/resumable-digest-refactor
Remove digest package's dependency on external sha implementation
2015-05-22 18:15:37 -07:00
Stephen J Day
44da954565 Refactor specification of supported digests
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:08:23 -07:00
Stephen J Day
ff52d7f53a Better error message when failing to get AWS auth
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 16:45:45 -07:00
Stephen J Day
eee6cad2cf Remove digest package's dependency on external sha implementation
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.

We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 23:44:08 -07:00
Stephen J Day
87ff947038 Decouple redis dependency from blob descriptor cache
Ensure that clients can use the blob descriptor cache provider without needing
the redis package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 18:58:10 -07:00
Tibor Vass
cb99c076db client: fix a typo preventing compilation
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2015-05-20 21:06:13 -04:00
Stephen Day
ea54a5560a Merge pull request #387 from dmcgowan/registry-client-update
Add client implementation of distribution interface
2015-05-20 15:55:20 -07:00
Derek McGowan
68c1ceac95 Remove error message shortening
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 14:55:59 -07:00
Derek McGowan
49369ffe9a Only do auth checks for endpoints starting with v2
Changes behavior so ping doesn't happen if /v2/ is anywhere in a request path, but instead only at the beginning. This fixes attempts to ping on redirected URLs.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 13:35:23 -07:00
Derek McGowan
c7f7747368 Update transport package to sever distribution dependency
The transport package no longer requires importing distribution for the ReadSeekCloser, instead declares its own.
Added comments on the Authenication handler in session.
Added todo on http seek reader to highlight its lack of belonging to the client transport.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:09:37 -07:00
Derek McGowan
572ff64d21 Add unauthorized error check
Add check for unauthorized error code and explicitly set the error code if the content could not be parsed.
Updated repository test for unauthorized tests and nit feedback.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:05:44 -07:00
Derek McGowan
13894e8736 Break down type dependencies
Each type no longer requires holding a reference to repository.
Added implementation for signatures get.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:56:27 -07:00
Derek McGowan
a3276fcc5b Feedback update
Update comments and TODOs
Fix switch style
Updated parse http response to take in reader
Add Cancel implementation
Update blobstore variable name

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:18:30 -07:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Jordan Liggitt
32ed2d7df1 Fix typo: respository->repository
Signed-off-by: Jordan Liggitt <jliggitt@redhat.com>
2015-05-19 13:25:39 -04:00
Derek McGowan
006ddd8283 Lint and documentation fixes
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
2874454224 Create client transport package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
94e375c5d1 Remove unused and duplicate error types
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
98836d6267 Update to track refactor updates
Added use of cache blob statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fdf7c8ff15 Open cache interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
70074b2286 Rename layer files to blob
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
9d64e461be Update to use blob interfaces
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fddeb1c8d5 Add missing defer on Tags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
468c5e79ba Simplify configuration and transport
Repository creation now just takes in an http.RoundTripper. Authenticated requests or requests which require additional headers should use the NewTransport function along with a request modifier (such an an authentication handler).

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
a15806ed9c Add base transport to interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
c6b51970cd Removed unused mirror flags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
68d5ecf6bf Update ReadFrom to wrap reader in NopCloser
Wrapping the reader in a NopCloser is necessary to prevent the http library from closing the input reader.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
49f7f54d07 Create authentication handler
Refactory authorizer to take a set of authentication handlers for different authentication schemes returned by an unauthorized HTTP requst.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
d92e5b1096 Add tags implementation
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
da05873b7c Use distribution context instead of google
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
c7ef45130b Cleanup session and config interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
6f9fbf99a9 Split layer and upload from repository
Layer upload moved to its own file with its own unit tests

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
b1ba2183ee Add unit tests for auth challenge and endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
174a732c94 Remove deprecated client interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
ce614b6de8 Add client implementation of distribution interface
Adds functionality to create a Repository client which connects to a remote endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Stephen Day
aeda6f5f8f Merge pull request #518 from RichardScothern/cache-headers
Set cache headers for layers.
2015-05-12 18:32:16 -07:00
Richard
3dc2d849c7 Set cache headers for layers.
- Set an Etag header
     - Check If-None-Match and respond appropriately
     - Set a Cache-Control header with a default of 1 week

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-12 17:49:18 -07:00
Anton Tiurin
ad80cbe1ea [Server] Listen and serve on a unix socket
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-05-11 16:00:14 +03:00
Derek McGowan
2cf40ab790 Modify blob upload API
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.

closes #470

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:43:23 -07:00
Richard
70f853e787 Ensure the instrumentedResponseWriter correctly sets the http
status in the context.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-05 16:46:33 -07:00
Stephen Day
dd58ffe6ed Merge pull request #410 from RichardScothern/driver-context
Add golang/x/net/context.Context to storage driver method calls
2015-05-04 19:18:30 -07:00
Derek McGowan
7d6e6aa980 Update API spec to reference digest instead of tarsum
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-01 17:14:38 -07:00
Stephen J Day
69e477f2d5 Use done channel to avoid goroutine leak
This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-28 14:40:20 -07:00
Richard
ae216e365a Make Storage Driver API calls context aware.
- Change driver interface to take a context as its first argument
     - Make newFileReader take a context as its first argument
     - Make newFileWriter take a context as its first argument
     - Make blobstore exists and delete take a context as a first argument
     - Pass the layerreader's context to the storage layer
     - Pass the app's context to purgeuploads
     - Store the app's context into the blobstore (was previously null)
     - Pass the trace'd context to the storage drivers

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 15:58:58 -07:00
Stephen Day
70427e3d9f Merge pull request #381 from RichardScothern/purge-config
Add configuration for upload purging
2015-04-27 14:33:34 -07:00
Richard
6460ddb2cb Add configuration for upload purging
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 11:06:15 -07:00
xiekeyang
c89e517be1 simplify the embedded method expression of repository
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2015-04-27 16:33:58 +08:00
Josh Hawn
628c7fa77b Updated urlbuilder X-Forwarded-Host logic
According to the Apache mod_proxy docs, X-Forwarded-Host can be a
comma-separated list of hosts, to which each proxy appends the requested
host. We want to grab only the first from this comma-separated list
to get the original requested Host when building URLs.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-24 14:16:49 -07:00
Olivier Gambier
9b6d49fdb0 Merge pull request #430 from stevvooe/address-s3-timeout-error
Attempt to address intermittent s3 RequestTimeout error
2015-04-24 11:20:30 -07:00
Stephen Day
528869ac03 Merge pull request #420 from stevvooe/expose-drivernames
Expose storage driver names for tracing
2015-04-24 10:53:40 -07:00
Stephen J Day
e6715c5cec Attempt to address intermittent s3 RequestTimeout error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 20:07:32 -07:00
Stephen J Day
6dd27cd2cb Correctly check s3 chunksize parameter
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 16:31:41 -07:00
Stephen J Day
f6d75a6139 Return after error in handler
This adds a missing return statement. It is not strictly needed since if the
io.Copy fails, the Finish operation will fail. Currently, the client reports
both errors where this new code will correctly only report the io.Copy error.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 13:13:13 -07:00
Stephen J Day
d7eb17fd72 Include driver name in trace messsages
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:31 -07:00
Stephen J Day
9e146437e4 Require storage drivers to report their name
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:01 -07:00