Commit graph

91 commits

Author SHA1 Message Date
Stephen J Day
730ce83cdf Trace function calls to Base storage driver
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-10 16:42:36 -07:00
Stephen J Day
4d91e791c0 Disassociate instance id from application
This moves the instance id out of the app so that it is associated with an
instantiation of the runtime. The instance id is stored on the background
context. This allows allow contexts using the main background context to
include an instance id for log messages. It also simplifies the application
slightly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-09 18:45:39 -07:00
Olivier Gambier
5fe1039947 Merge pull request #332 from stevvooe/case-sensitive-backend
registry/storage/driver: defer case-sensitive support to storage backend
2015-04-07 16:24:38 -07:00
Stephen J Day
e23ca5ac5f Defer case-sensitive support to storage backend
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:

1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.

The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-07 14:14:45 -07:00
Olivier Gambier
4e2d176d27 Merge pull request #330 from stevvooe/parallelize-signature-fetch
registry/storage: parallelize signature fetch in signature store
2015-04-06 22:30:35 -07:00
Josh Hawn
0eb599329a Handle cloudFront bucket prefix issue
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-06 19:46:05 -07:00
Stephen J Day
1eab4b79bc Parallelize signature fetch in signature store
To avoid compounded round trips leading to slow retrieval of manifests with a
large number of signatures, the fetch of signatures has been parallelized. This
simply spawns a goroutine for each path, coordinated with a sync.WaitGroup.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-06 19:13:15 -07:00
Olivier Gambier
1963b7902e Merge pull request #312 from stevvooe/add-layer-info-cache
registry: integrate layer info cache with registry and storage
2015-04-03 13:53:27 -07:00
Stephen J Day
09d48c7192 Move expvar under the registry section
For consistency with other systems, the redis and caching monitoring data has
been moved under the "registry" section in expvar. This ensures the entire
registry state is kept to a single section.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 21:30:27 -07:00
Stephen J Day
919d972e65 Allow control over which storage cache to use
This allows one to better control the usage of the cache and turn it off
completely. The storage configuration module was modified to allow parameters
to be passed to just the storage implementation, rather than to the driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
44b14ceadc Integrate layer info cache with registry and storage
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.

This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:16 -07:00
Stephen J Day
b1f616cbff Define and implement layer info cache
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.

Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-02 20:15:09 -07:00
Stephen Day
cf5839b019 Merge pull request #295 from jlhawn/use_resumable_digest
digest, registry/storage: use resumable digest
2015-04-02 20:12:57 -07:00
Stephen Day
7c3b73482f Merge pull request #314 from stevvooe/lockdown-uuid-route
registry/api/v2: stronger validation for uuid field in urls
2015-04-02 13:57:17 -07:00
Josh Hawn
18c9a1cdd8 Use resumable digest for efficient upload finish
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-02 11:20:31 -07:00
Stephen J Day
f26a283a48 Avoid crash on invalid Move arguments
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 19:03:02 -07:00
Stephen J Day
1944be9db3 Stronger validation for uuid field in urls
This change adds strong validation for the uuid variable for v2 routes. This is
a minor specification change but is okay since the uuid field is controlled by
the server. The character set is restricted to avoid path traversal, allowing
for alphanumeric values and urlsafe base64 encoding.

This change has no effect on client implementations.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 18:57:59 -07:00
Stephen J Day
3cad3c7b6a Add redis pool to registry webapp
Redis has been integrated with the web application for use with various
services. The configuraiton exposes connection details, timeouts and pool
parameters. Documentation has been updated accordingly.

A few convenience methods have been added to the context package to get loggers
with certain fields, exposing some missing functionality from logrus.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 16:27:24 -07:00
Stephen Day
02b8c7b8cf Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
11db8185bc Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
ce9f500a31 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Josh Hawn
db5689aa86 Refactor Layer interface to return a Handler
... Rather than ServeHTTP directly.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 21:59:07 -07:00
Josh Hawn
e062e66ee1 Insert request method option storage driver URLFor
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 17:06:40 -07:00
Stephen Day
70560cceaf Merge pull request #256 from ncdc/master
Send WWW-Authenticate header for silly auth
2015-03-11 19:01:23 -07:00
Andy Goldstein
4491e755b8 Send WWW-Authenticate header for silly auth
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 15:20:15 -04:00
David Lawrence
3853e66f4b don't panic during a request when configuring repository middleware. Return a 500 with an appropriate error
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:59:02 -07:00
Andy Goldstein
b5a63d75ea Fix Godoc typos
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:29 -07:00
David Lawrence
c858a4ba89 Final polish to cloudfront and larger middleware refactor
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:29 -07:00
Andy Goldstein
a20abfbf3c Middleware!
Convert middleware in the config to be a map of type->[]Middleware

Add support for registry & repository middleware.

Some naming updates as well.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:28 -07:00
David Lawrence
4acda57e05 Refactoring cloudfactory layer handler into a more generic storage
middleware concept.

This also breaks the dependency the storage package had on goamz
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:28 -07:00
Josh Hawn
87959abe8f digest: Minor refactoring
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 16:44:19 -07:00
David Lawrence
6dcec265a0 minor refactor + tests for app.go just to improve test coverage.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-09 16:31:30 -07:00
Andy Goldstein
e1e724860f Remove max repository component length restriction
Fixes #241

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-06 19:55:33 -05:00
Stephen Day
f0ccdd448f Merge pull request #239 from jlhawn/event_target_update
notifications: update notification event Target fields
2015-03-06 16:45:06 -08:00
Stephen Day
e3dd701004 Merge pull request #238 from endophage/canonical_sha256
digest, registry/storage, registry/handlers: switch to SHA256 as canonical digest
2015-03-06 15:27:14 -08:00
David Lawrence
b777e389b9 fixing up tests to work with for non-tarsum future
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-06 14:54:23 -08:00
Stephen Day
81a03041d4 Merge pull request #243 from stevvooe/storagedriver-docs
doc: move storage driver readmes into docs
2015-03-06 11:25:42 -08:00
Josh Hawn
d3bc4c4b38 Switch to SHA256 as canonical digest
Also support client digests linking to canonical digest.
2015-03-06 09:49:05 -08:00
Josh Hawn
a877811c0b Update notification event Target fields
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-05 18:01:50 -08:00
Stephen J Day
41b1e22f97 doc: move storage driver readmes into docs
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-05 17:23:33 -08:00
Stephen J Day
40273b1d36 Implement immutable manifest reference support
This changeset implements immutable manifest references via the HTTP API. Most
of the changes follow from modifications to ManifestService. Once updates were
made across the repo to implement these changes, the http handlers were change
accordingly. The new methods on ManifestService will be broken out into a
tagging service in a later PR.

Unfortunately, due to complexities around managing the manifest tag index in an
eventually consistent manner, direct deletes of manifests have been disabled.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:40:55 -08:00
Stephen J Day
f536633ca8 spec: fetch manifests by tag or digest
Manifests are now fetched by a field called "reference", which may be a tag or
a digest. When using digests to reference a manifest, the data is immutable.
The routes and specification have been updated to allow this.

There are a few caveats to this approach:

1. It may be problematic to rely on data format to differentiate between a tag
   and a digest. Currently, they are disjoint but there may modifications on
   either side that break this guarantee.
2. The caching characteristics of returned content are very different for
   digest versus tag-based references. Digest urls can be cached forever while tag
   urls cannot.

Both of these are minimal caveats that we can live with in the future.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-04 21:39:36 -08:00
Stephen Day
91403c1b52 Merge pull request #234 from ncdc/signature-service
Expose Signatures() on Repository
2015-03-04 18:24:36 -08:00
Josh Hawn
e249b478d9 Merge pull request #231 from stevvooe/remove-unnecessary-close
Remove unnecessary close in client
2015-03-04 18:19:35 -08:00
Andy Goldstein
6b12e34a4b Expose Signatures() on Repository
Add a SignatureService and expose it via Signatures() on Repository so
external integrations wrapping the registry can access signatures.

Move signature related code from revisionstore.go to signaturestore.go.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-04 20:56:11 -05:00
Stephen J Day
dc0408767a Remove unnecessary close in client
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-03 19:29:12 -08:00
Stephen Day
bfef2046e8 Merge pull request #218 from endophage/DIST-148
registry/storage: buffered wrapper for fileWriter
2015-03-03 18:08:28 -08:00
Stephen Day
72b48812b4 Merge pull request #212 from stevvooe/add-documentation-name-validator
documentation for name validation grammar
2015-03-03 17:32:54 -08:00
David Lawrence
5161d69ebf wrap buffered writer around filewriter
benchmarks added to filewriter_test, demonstrate buffered
version is ~5x faster on my hardware.
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-03 16:23:21 -08:00
Stephen J Day
6add5f30d2 documentation for name validation grammar
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-02 10:55:31 -08:00