Commit graph

163 commits

Author SHA1 Message Date
Stephen J Day
d7eb17fd72 Include driver name in trace messsages
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:31 -07:00
Stephen J Day
9e146437e4 Require storage drivers to report their name
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:01 -07:00
Stephen Day
73960f4024 Merge pull request #419 from stevvooe/pool-buffers-s3
Pool buffers used in S3.WriteStream
2015-04-22 16:41:28 -07:00
Stephen J Day
1adab029f6 Pool buffers used in S3.WriteStream
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 16:25:53 -07:00
Stephen J Day
72a702d0d9 Allow configuration of chunksize parameter
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.

This code likely never worked. The configuration system should be decoupled
from the object instantiation.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 14:31:34 -07:00
Stephen Day
aa09c6c262 Merge pull request #408 from stevvooe/increase-retry-backoff
Backoff retry on verification to give s3 time to propagate
2015-04-21 13:19:28 -07:00
Stephen J Day
632e46418b Backoff retry on verification to give s3 time to propagate
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 13:04:46 -07:00
Stephen J Day
d8dbcc3efa log canonical digest on verification error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 11:34:18 -07:00
Stephen J Day
c8345090d9 Attempt to deal with eventual consistency by retrying
Rather than accept the resulting of a layer validation, we retry up to three
times, backing off 100ms after each try. The thought is that we allow s3 files
to make their way into the correct location increasing the liklihood the
verification can proceed, if possible.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-20 18:49:22 -07:00
Stephen Day
7c25fcf698 Merge pull request #375 from RichardScothern/path-logging
registry/storage/driver: add path and other info to filesytem trace methods.
2015-04-17 14:09:23 -07:00
Richard
6b1c2e9b1f Add path and other info to filesytem trace methods.
Also fix Delete (was 'Move').
2015-04-17 13:55:18 -07:00
bin liu
e0521d2d01 fix some typos in source comments
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-04-17 12:39:52 +00:00
Stephen Day
cce1641f9b Merge pull request #333 from RichardScothern/purgeuploads
registry/storage: automatically purge old upload files
2015-04-16 11:16:51 -07:00
Richard
e4f07c368b Automatically purge old upload files.
When the registry starts a background timer will periodically
scan the upload directories on the file system every 24 hours
and delete any files older than 1 week. An initial jitter
intends to avoid contention on the filesystem where multiple
registries with the same storage driver are started
simultaneously.
2015-04-16 10:57:34 -07:00
Andy Goldstein
7fdd395653 Use a build flag to disable resumable digests.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-04-16 01:19:57 +00:00
Richard Scothern
90af0f9b7a Merge pull request #343 from stevvooe/tracing-driver
context, storagedriver: trace function calls to Base storage driver
2015-04-10 16:55:57 -07:00
Stephen J Day
730ce83cdf Trace function calls to Base storage driver
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-10 16:42:36 -07:00
Stephen Day
41dd70c901 Merge pull request #349 from RichardScothern/348
Prevent Close() from being called after Finish()
2015-04-10 16:21:01 -07:00
Richard
3d529094fa Prevent Close() from being called after Finish() 2015-04-10 16:00:05 -07:00
Derek McGowan
142d62798e Rename top level registry interface to namespace
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-04-09 19:21:33 -07:00
Stephen J Day
84046e03e0 Prevent false sharing in signature fetch
The original implementation wrote to different locations in a shared slice.
While this is theoretically okay, we end up thrashing the cpu cache since
multiple slice members may be on the same cache line. So, even though each
thread has its own memory location, there may be contention over the cache
line. This changes the code to aggregate to a slice in a single goroutine.

In reality, this change likely won't have any performance impact. The theory
proposed above hasn't really even been tested. Either way, we can consider it
and possibly go forward.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-09 14:35:10 -07:00
Olivier Gambier
5fe1039947 Merge pull request #332 from stevvooe/case-sensitive-backend
registry/storage/driver: defer case-sensitive support to storage backend
2015-04-07 16:24:38 -07:00
Stephen J Day
e23ca5ac5f Defer case-sensitive support to storage backend
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:

1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.

The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-07 14:14:45 -07:00
Olivier Gambier
4e2d176d27 Merge pull request #330 from stevvooe/parallelize-signature-fetch
registry/storage: parallelize signature fetch in signature store
2015-04-06 22:30:35 -07:00
Josh Hawn
0eb599329a Handle cloudFront bucket prefix issue
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-06 19:46:05 -07:00
Stephen J Day
1eab4b79bc Parallelize signature fetch in signature store
To avoid compounded round trips leading to slow retrieval of manifests with a
large number of signatures, the fetch of signatures has been parallelized. This
simply spawns a goroutine for each path, coordinated with a sync.WaitGroup.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-06 19:13:15 -07:00
Olivier Gambier
1963b7902e Merge pull request #312 from stevvooe/add-layer-info-cache
registry: integrate layer info cache with registry and storage
2015-04-03 13:53:27 -07:00
Stephen J Day
09d48c7192 Move expvar under the registry section
For consistency with other systems, the redis and caching monitoring data has
been moved under the "registry" section in expvar. This ensures the entire
registry state is kept to a single section.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 21:30:27 -07:00
Stephen J Day
44b14ceadc Integrate layer info cache with registry and storage
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.

This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
b1f616cbff Define and implement layer info cache
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.

Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:09 -07:00
Stephen Day
cf5839b019 Merge pull request #295 from jlhawn/use_resumable_digest
digest, registry/storage: use resumable digest
2015-04-02 20:12:57 -07:00
Josh Hawn
18c9a1cdd8 Use resumable digest for efficient upload finish
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-02 11:20:31 -07:00
Stephen J Day
f26a283a48 Avoid crash on invalid Move arguments
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 19:03:02 -07:00
Stephen Day
02b8c7b8cf Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
11db8185bc Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
ce9f500a31 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Josh Hawn
db5689aa86 Refactor Layer interface to return a Handler
... Rather than ServeHTTP directly.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 21:59:07 -07:00
Josh Hawn
e062e66ee1 Insert request method option storage driver URLFor
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 17:06:40 -07:00
David Lawrence
3853e66f4b don't panic during a request when configuring repository middleware. Return a 500 with an appropriate error
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:59:02 -07:00
Andy Goldstein
b5a63d75ea Fix Godoc typos
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:29 -07:00
David Lawrence
c858a4ba89 Final polish to cloudfront and larger middleware refactor
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:29 -07:00
Andy Goldstein
a20abfbf3c Middleware!
Convert middleware in the config to be a map of type->[]Middleware

Add support for registry & repository middleware.

Some naming updates as well.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:28 -07:00
David Lawrence
4acda57e05 Refactoring cloudfactory layer handler into a more generic storage
middleware concept.

This also breaks the dependency the storage package had on goamz
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:28 -07:00
Josh Hawn
87959abe8f digest: Minor refactoring
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 16:44:19 -07:00
Stephen Day
f0ccdd448f Merge pull request #239 from jlhawn/event_target_update
notifications: update notification event Target fields
2015-03-06 16:45:06 -08:00
Stephen Day
e3dd701004 Merge pull request #238 from endophage/canonical_sha256
digest, registry/storage, registry/handlers: switch to SHA256 as canonical digest
2015-03-06 15:27:14 -08:00
David Lawrence
b777e389b9 fixing up tests to work with for non-tarsum future
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-06 14:54:23 -08:00
Stephen Day
81a03041d4 Merge pull request #243 from stevvooe/storagedriver-docs
doc: move storage driver readmes into docs
2015-03-06 11:25:42 -08:00
Josh Hawn
d3bc4c4b38 Switch to SHA256 as canonical digest
Also support client digests linking to canonical digest.
2015-03-06 09:49:05 -08:00
Josh Hawn
a877811c0b Update notification event Target fields
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-05 18:01:50 -08:00
Stephen J Day
41b1e22f97 doc: move storage driver readmes into docs
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-05 17:23:33 -08:00
Stephen J Day
40273b1d36 Implement immutable manifest reference support
This changeset implements immutable manifest references via the HTTP API. Most
of the changes follow from modifications to ManifestService. Once updates were
made across the repo to implement these changes, the http handlers were change
accordingly. The new methods on ManifestService will be broken out into a
tagging service in a later PR.

Unfortunately, due to complexities around managing the manifest tag index in an
eventually consistent manner, direct deletes of manifests have been disabled.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:40:55 -08:00
Andy Goldstein
6b12e34a4b Expose Signatures() on Repository
Add a SignatureService and expose it via Signatures() on Repository so
external integrations wrapping the registry can access signatures.

Move signature related code from revisionstore.go to signaturestore.go.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-04 20:56:11 -05:00
David Lawrence
5161d69ebf wrap buffered writer around filewriter
benchmarks added to filewriter_test, demonstrate buffered
version is ~5x faster on my hardware.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-03 16:23:21 -08:00
Andrey Kostov
df71f3451a Fix S3 driver's list when the root directory is either "" or "/" 2015-02-19 16:31:34 -08:00
Andrey Kostov
da5510b55e Add an empty root directory s3 driver specific test 2015-02-19 16:28:32 -08:00
Stephen J Day
5d029fb807 Add error return to Repository method on Registry
The method (Registry).Repository may now return an error. This is too allow
certain implementationt to validate the name or opt to not return a repository
under certain conditions.

In conjunction with this change, error declarations have been moved into a
single file in the distribution package. Several error declarations that had
remained in the storage package have been moved into distribution, as well. The
declarations for Layer and LayerUpload have also been moved into the main
registry file, as a result.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-13 16:27:33 -08:00
Stephen J Day
d2d46fca41 Move notifications package to distribution
Since the notifications package is now decoupled from storage, we are moving it
to the root package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:40:35 -08:00
Stephen J Day
286a644948 Remove Name from Layer and LayerUpload interface
A Layer or LayerUpload should not be coupled with the containing repository.
Remove the Name method and correctly reference from the repository where
appropriate.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:27:05 -08:00
Stephen J Day
27b03f2136 Move layer interface definitions to distribution package
After consideration, it has been decided that the interfaces defined in the
storage package provide a good base for interacting with various registry
instances. Whether interacting with a remote API or a local, on-disk registry,
these types have proved flexible. By moving them here, they can become the
central components of interacting with distribution components.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-12 14:26:46 -08:00
Ahmet Alp Balkan
76f1abfa07 storagedriver/azure: Add README
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-02-11 14:40:51 -08:00
Stephen J Day
65b0d73cb7 Move storagedriver package to registry/storage/driver
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Stephen J Day
3468fbd4a8 Move storage package under registry package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00