Commit graph

133 commits

Author SHA1 Message Date
Stephen J Day
e4794ff73d Allow configuration of chunksize parameter
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.

This code likely never worked. The configuration system should be decoupled
from the object instantiation.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 14:31:34 -07:00
Stephen Day
06fcc9213a Merge pull request #408 from stevvooe/increase-retry-backoff
Backoff retry on verification to give s3 time to propagate
2015-04-21 13:19:28 -07:00
Stephen J Day
36ffe0c134 Backoff retry on verification to give s3 time to propagate
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 13:04:46 -07:00
Stephen Day
5f7693da80 Merge pull request #407 from stevvooe/log-canonical-digest
log canonical digest on verification error
2015-04-21 13:03:32 -07:00
Stephen J Day
77b30cfb25 log canonical digest on verification error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 11:34:18 -07:00
Olivier Gambier
4f41f0be06 Merge pull request #405 from stevvooe/eventual-consistency
Attempt to deal with eventual consistency by retrying
2015-04-20 18:59:01 -07:00
Stephen Day
85338448a3 Merge pull request #404 from RichardScothern/digest-log
Add logging for generic handler errors.
2015-04-20 18:51:59 -07:00
Stephen J Day
d4ce8f5ef8 Attempt to deal with eventual consistency by retrying
Rather than accept the resulting of a layer validation, we retry up to three
times, backing off 100ms after each try. The thought is that we allow s3 files
to make their way into the correct location increasing the liklihood the
verification can proceed, if possible.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-20 18:49:22 -07:00
Richard
431811056b Add logging for generic handler errors.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-20 17:47:52 -07:00
Stephen Day
3802701cdc Merge pull request #375 from RichardScothern/path-logging
registry/storage/driver: add path and other info to filesytem trace methods.
2015-04-17 14:09:23 -07:00
Richard
bc2b6efaa6 Add path and other info to filesytem trace methods.
Also fix Delete (was 'Move').
2015-04-17 13:55:18 -07:00
Stephen Day
57341dbb0d Merge pull request #384 from liubin/fixtypos
fix some typos in source comments
2015-04-17 11:55:19 -07:00
bin liu
f3f46307f2 fix some typos in source comments
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-04-17 12:39:52 +00:00
Stephen Day
3bf989585c Merge pull request #333 from RichardScothern/purgeuploads
registry/storage: automatically purge old upload files
2015-04-16 11:16:51 -07:00
Richard
0b2feaf611 Automatically purge old upload files.
When the registry starts a background timer will periodically
scan the upload directories on the file system every 24 hours
and delete any files older than 1 week. An initial jitter
intends to avoid contention on the filesystem where multiple
registries with the same storage driver are started
simultaneously.
2015-04-16 10:57:34 -07:00
Stephen Day
10d875d50d Merge pull request #369 from stevvooe/http-request-status-context-manager
context, registry/handlers: instantiate http context before dispatch
2015-04-15 23:23:40 -07:00
Stephen Day
78baf3f2e1 Merge pull request #364 from ncdc/resumable-digest-optional
Use a build flag to disable resumable digests
2015-04-15 23:22:25 -07:00
Stephen J Day
136f0ed8bb Instantiate http context before dispatch
Ensure that the status is logged in the context by instantiating before the
request is routed to handlers. While this requires some level of hacking to
acheive, the result is that the context value of "http.request.status" is as
accurate as possible for each request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-15 19:53:36 -07:00
Andy Goldstein
480d864fc4 Use a build flag to disable resumable digests.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-04-16 01:19:57 +00:00
Derek McGowan
16174241d1 Update final upload chunk api doc
Updates description about content length and location

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-15 17:55:15 -07:00
Richard
9898552656 Add auth.user.name to logging context 2015-04-15 10:34:24 -07:00
Richard Scothern
e5eddbc762 Merge pull request #343 from stevvooe/tracing-driver
context, storagedriver: trace function calls to Base storage driver
2015-04-10 16:55:57 -07:00
Stephen J Day
12bf470b2f Trace function calls to Base storage driver
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-10 16:42:36 -07:00
Stephen Day
65f19a7e02 Merge pull request #349 from RichardScothern/348
Prevent Close() from being called after Finish()
2015-04-10 16:21:01 -07:00
Richard
4ac515fde4 Prevent Close() from being called after Finish() 2015-04-10 16:00:05 -07:00
Stephen Day
664a827bfb Merge pull request #307 from dmcgowan/namespace-interface
Rename top level registry interface to namespace
2015-04-10 12:58:01 -07:00
Derek McGowan
e83e37618f Rename top level registry interface to namespace
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-09 19:21:33 -07:00
Stephen J Day
36a076995b Disassociate instance id from application
This moves the instance id out of the app so that it is associated with an
instantiation of the runtime. The instance id is stored on the background
context. This allows allow contexts using the main background context to
include an instance id for log messages. It also simplifies the application
slightly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-09 18:45:39 -07:00
Stephen J Day
250e61e2a1 Prevent false sharing in signature fetch
The original implementation wrote to different locations in a shared slice.
While this is theoretically okay, we end up thrashing the cpu cache since
multiple slice members may be on the same cache line. So, even though each
thread has its own memory location, there may be contention over the cache
line. This changes the code to aggregate to a slice in a single goroutine.

In reality, this change likely won't have any performance impact. The theory
proposed above hasn't really even been tested. Either way, we can consider it
and possibly go forward.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-09 14:35:10 -07:00
Olivier Gambier
44fa39e4ad Merge pull request #332 from stevvooe/case-sensitive-backend
registry/storage/driver: defer case-sensitive support to storage backend
2015-04-07 16:24:38 -07:00
Stephen J Day
2b4ad94cee Defer case-sensitive support to storage backend
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:

1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.

The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-07 14:14:45 -07:00
Olivier Gambier
7721fe5a65 Merge pull request #330 from stevvooe/parallelize-signature-fetch
registry/storage: parallelize signature fetch in signature store
2015-04-06 22:30:35 -07:00
Josh Hawn
8c0859e39c Handle cloudFront bucket prefix issue
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-06 19:46:05 -07:00
Stephen J Day
def60f3426 Parallelize signature fetch in signature store
To avoid compounded round trips leading to slow retrieval of manifests with a
large number of signatures, the fetch of signatures has been parallelized. This
simply spawns a goroutine for each path, coordinated with a sync.WaitGroup.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-06 19:13:15 -07:00
Olivier Gambier
e56124d343 Merge pull request #312 from stevvooe/add-layer-info-cache
registry: integrate layer info cache with registry and storage
2015-04-03 13:53:27 -07:00
Stephen J Day
6b748a74ef Move expvar under the registry section
For consistency with other systems, the redis and caching monitoring data has
been moved under the "registry" section in expvar. This ensures the entire
registry state is kept to a single section.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 21:30:27 -07:00
Stephen J Day
4e1ecad6cc Allow control over which storage cache to use
This allows one to better control the usage of the cache and turn it off
completely. The storage configuration module was modified to allow parameters
to be passed to just the storage implementation, rather than to the driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
6ab228f798 Integrate layer info cache with registry and storage
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.

This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
a7c2dceea5 Define and implement layer info cache
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.

Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:09 -07:00
Stephen Day
dff57726f9 Merge pull request #295 from jlhawn/use_resumable_digest
digest, registry/storage: use resumable digest
2015-04-02 20:12:57 -07:00
Stephen Day
721b39d0cb Merge pull request #314 from stevvooe/lockdown-uuid-route
registry/api/v2: stronger validation for uuid field in urls
2015-04-02 13:57:17 -07:00
Josh Hawn
b96de45be8 Use resumable digest for efficient upload finish
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-02 11:20:31 -07:00
Stephen J Day
06acde06cb Avoid crash on invalid Move arguments
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 19:03:02 -07:00
Stephen J Day
6eb804a1ec Stronger validation for uuid field in urls
This change adds strong validation for the uuid variable for v2 routes. This is
a minor specification change but is okay since the uuid field is controlled by
the server. The character set is restricted to avoid path traversal, allowing
for alphanumeric values and urlsafe base64 encoding.

This change has no effect on client implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 18:57:59 -07:00
Stephen J Day
38ae1cb461 Add redis pool to registry webapp
Redis has been integrated with the web application for use with various
services. The configuraiton exposes connection details, timeouts and pool
parameters. Documentation has been updated accordingly.

A few convenience methods have been added to the context package to get loggers
with certain fields, exposing some missing functionality from logrus.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 16:27:24 -07:00
Stephen Day
9ee35877e3 Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
dffd1babd2 Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
594f733e03 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Stephen Day
af4f5aa7d2 Merge pull request #261 from jlhawn/url_for_method
Refactor Layer interface to return a Handler
2015-03-13 17:27:42 -07:00
Josh Hawn
6d14019368 Refactor Layer interface to return a Handler
... Rather than ServeHTTP directly.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 21:59:07 -07:00