Commit graph

356 commits

Author SHA1 Message Date
Sylvain Baubeau
f256797cac Do not use suite style testing for Swift specific tests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
53e80944f1 Rename environment variables to run Swift testsuite
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
d0dd36125c Check file has been opened before closing it
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1a80ec340b Rename DriverParameters structure to Parameters
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
2846913552 Change folder mime type to application/vnc.swift.directory
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
33028c11a7 Add code documentation
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f21cbcebc6 Do not read segment if no padding is necessary
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3242378e1b Improve 404 errors handling
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
57cef57e1b Use only one Swift container for both files and manifests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
a40502ec02 Increase default chunk size to 20M
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
davidli
af99dbd6bf Add support for Openstack Identity v3 API
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3fb42a1502 Use 'prefix' parameter instead of 'path' when listing files
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1c01ca4b39 Create full folder hierarchy instead of just the top level folder
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
c4d845cb6c Remove one level of indentation in swift path handling code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
4497d6c973 Use mitchellh/mapstructure library to parse Swift parameters
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
2632fd9dc8 Move Dynamic Large Object handling to dedicated methods
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
d8d50fb0e0 Simplify code that handles non existing manifests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
5e650a820a Handle error during copy of original content
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
nevermosby
7372c13f29 Update the import path for swift driver test
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
davidli
6dc1596be4 Use gofmt to format the code of swift driver.
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
ea7c53df08 Add Openstack Swift storage driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:08 +02:00
Stephen Day
5e9d0702a1 Merge pull request #713 from stevvooe/descriptor-size-field
Use "Size" field to describe blobs over "Length"
2015-07-17 22:06:59 -07:00
Stephen J Day
26b7fe4a91 Use "Size" field to describe blobs over "Length"
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-17 17:07:11 -07:00
yuzou
0fc7babf76 Close reader after the test is finished.
Signed-off-by: yuzou <zouyu7@huawei.com>
2015-07-17 14:55:31 +08:00
Stephen Day
fed58bd2d3 Merge pull request #705 from stevvooe/export-servejson-errors
Export ServeJSON for serving error codes
2015-07-16 14:35:27 -07:00
Brian Bland
c25f4af00f Reduces log level of auth error lines from error->warn
An error level log is already produced within app.authorized() if an
actual unexpected error occurs during authorization, so this warning
level log remains for auditability purposes, but should not be
considered an error condition.

Addresses #704

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-07-16 12:43:33 -07:00
Stephen J Day
86ffe515df Export ServeJSON for serving error codes
This changeset provides a common http handler for serving errcodes. This should
unify http responses across webservices in the face of errors.

Several type assertions have been added, as well, to ensure the error interface
is implemented.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-16 12:14:14 -07:00
Richard
f331da2daa Allow Manifest Service to be configured with function arguments
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-15 12:25:16 -07:00
Stephen Day
2b88cb9413 Merge pull request #651 from duglin/ErrArgs
Add ability to pass in substitution args into an Error
2015-07-15 11:22:59 -07:00
Doug Davis
94e2e9f4a0 Add ability to pass in substitution args into an Error
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-07-15 11:02:10 -07:00
Derek McGowan
0cda2f61e8 Merge pull request #699 from RichardScothern/client-manifest-etags-clean
Allow conditional fetching of manifests with the registry client.
2015-07-14 17:29:59 -07:00
Richard Scothern
48a2010ac3 Allow conditional fetching of manifests with the registry client.
Add a functional argument to pass a digest to (ManifestService).GetByTag().
If the digest matches an empty manifest and nil error are returned.

See 1bc740b0d5 for server implementation.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-14 16:52:52 -07:00
Richard Scothern
7e6fda610d Fix build when using build tag 'noresumabledigest'
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-07-14 11:14:09 -07:00
Stephen J Day
b54cc5ed47 Allow single character repository names
The main goal of this changeset is to allow repository name components to
consist of a single character. The number of components allowed and the slash
separation requirements have also been clarified.

To go along with this simplification, errant constants and unneeded error types
have been removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-07-10 12:10:42 -06:00
Derek McGowan
8fc782ae09 Fix typo in Version doc
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-08 11:02:47 -07:00
Derek McGowan
3531b22b46 Add challenge manager interface
Challenger manager interface is used to handle getting authorization challenges from an endpoint as well as extracting challenges from responses.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-01 15:00:25 -07:00
Derek McGowan
c8fac94617 Separate version and challenge parsing from ping
Replace ping logic with individual functions to extract API version and authorization challenges. The response from a ping operation can be passed into these function. If an error occurs in parsing, the version or challenge will not be used. Sending the ping request is the responsibility of the caller.
APIVersion has been converted from a string to a structure type. A parse function was added to convert from string to the structure type.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-01 15:00:25 -07:00
Derek McGowan
5a7dab4670 Refactor client auth
Move client auth into a separate package.
Separate ping from the authorizer and export Challenges type.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-07-01 15:00:25 -07:00
Stephen Day
c56e28826e Merge pull request #670 from stevvooe/remove-ipc
Remove half-baked Storage Driver IPC support
2015-06-30 19:21:54 -07:00
Jon Poler
f09051fe54 Remove timeout for http.Client in registry/client/repository.go.
Timeouts should not be a discrete period of time, because they end
up being arbitrary and may be difficult to gauge correctly against
very large Docker layers. Rather, timeouts should be set at the
transport level using the SetDeadline attribute on a net.Conn
object.

Signed-off-by: Jon Poler <jonathan.poler@apcera.com>
2015-06-30 17:35:24 -07:00
Stephen Day
c1423aa2aa Merge pull request #675 from RichardScothern/fix-ctx
Pass correct context into tracer
2015-06-30 11:18:51 -07:00
Richard Scothern
3629aac84f Pass correct context into tracer
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-30 10:28:14 -07:00
Stephen J Day
d3d4423ff7 Remove half-baked Storage Driver IPC support
This removes documentation and code related to IPC based storage driver
plugins. The existence of this functionality was an original feature goal but
is now not maintained and actively confusing incoming contributions. We will
likely explore some driver plugin mechanism in the future but we don't need
this laying around in the meantime.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-29 16:52:12 -07:00
Richard Scothern
0edb91185b Prevent the ErrUnsupportedMethod error from being returned up the stack.
It eventually causes the go http library to do a double WriteHeader()
which is an error

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-29 16:44:09 -07:00
Stephen Day
b6faf3fa9f Merge pull request #645 from RichardScothern/manifest-cache-headers
Cache headers for manifests.
2015-06-25 13:20:39 -07:00
Richard Scothern
1bc740b0d5 Add Etag header for manifests.
Return 304 (Not Modified) if retrieved with If-None-Match header

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-19 10:44:21 -07:00
Doug Davis
7bb98181ab Add 'message' back to BlobTest sample json
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-18 18:24:54 -07:00
Doug Davis
cb45ec56ff Add back in the "errors" wrapper in the Errors serialization
See: https://github.com/docker/distribution/pull/548/files#r32794064

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-18 18:00:26 -07:00
Stephen Day
d2ca423500 Merge pull request #638 from duglin/MoveHTTPRC
Move challenge http status code logic
2015-06-18 12:03:29 -07:00
Doug Davis
c4eb195cc1 Move challenge http status code logic
See: 3ea67df373/registry/handlers/app.go (L498)

Per the comment on line 498, this moves the logic of setting the http
status code into the serveJSON func, leaving the auth.Challenge.ServeHTTP()
func to just set the auth challenge header.

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-17 18:23:55 -07:00
Stephen J Day
b65a4a8713 Ensure that rados is disabled without build tag
This ensures that rados is not required when building the registry. This was
slightly tricky in that when the flags were applied, the rados package was
completely missing. This led to a problem where rados was basically unlistable
and untestable as a package. This was fixed by simply adding a doc.go file that
is included whether rados is built or not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-16 20:00:02 -07:00
Stephen Day
3ea67df373 Merge pull request #623 from ahmetalpbalkan/azure-vendor
storage/driver/azure: Update vendored Azure SDK
2015-06-16 17:41:38 -07:00
Ahmet Alp Balkan
daa22cacba storage/driver/azure: Update vendored Azure SDK
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.

I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...

This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.

Fixes #620, #175.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-06-16 17:13:44 -07:00
Olivier Gambier
d678e3dc57 Merge pull request #608 from dmcgowan/http-basic-auth
Implementation of a basic authentication scheme using standard .htpasswd
2015-06-15 17:40:58 -07:00
Stephen Day
e57e731821 Merge pull request #548 from duglin/MoveErrors
Move ErrorCode logic to new errcode package
2015-06-15 14:33:28 -07:00
Doug Davis
441f7cac87 Round 4
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-11 21:33:35 -07:00
Stephen J Day
d9d55bcbab Minor formatting fixes related to htpasswd auth
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-11 17:06:35 -07:00
Olivier Gambier
6e0cfc17dc Saner default data location
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-10 20:54:24 -07:00
Stephen J Day
0f654c25ac Rename the basic access controller to htpasswd
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-10 19:41:35 -07:00
Stephen J Day
ffd3662982 Harden basic auth implementation
After consideration, the basic authentication implementation has been
simplified to only support bcrypt entries in an htpasswd file. This greatly
increases the security of the implementation by reducing the possibility of
timing attacks and other problems trying to detect the password hash type.

Also, the htpasswd file is only parsed at startup, ensuring that the file can
be edited and not effect ongoing requests. Newly added passwords take effect on
restart. Subsequently, password hash entries are now stored in a map.

Test cases have been modified accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-10 19:38:56 -07:00
Stephen J Day
ffe56ebe41 Refactor Basic Authentication package
This change refactors the basic authentication implementation to better follow
Go coding standards. Many types are no longer exported. The parser is now a
separate function from the authentication code. The standard functions
(*http.Request).BasicAuth/SetBasicAuth are now used where appropriate.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
abd142855a Unexported function to comply with golint
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
e4c3ab4377 Removed dashes from comments, unexported htpasswd struct
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
04f6a4811d Fixed golint, gofmt warning advice.
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
c50dfb7dae Added support for bcrypt, plaintext; extension points for other htpasswd hash methods.
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
c4849bb99a Aligned formatting with gofmt
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Dave Trombley
0ecaa7f40a Fixed WWW-Authenticate: header, added example config and import into main, fixed golint warnings
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
BadZen
8a204f59e7 Implementation of a basic authentication scheme using standard .htpasswd files
Signed-off-by: BadZen <dave.trombley@gmail.com>
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:37 -07:00
Stephen Day
f63313de1f Merge pull request #599 from stevvooe/clarify-deletion-by-digest-constraint
Clarify digest in API specification
2015-06-08 19:04:53 -07:00
Stephen J Day
c54adb667f Clarify that manifests can only be deleted by digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-04 18:59:08 -07:00
Stephen J Day
4bff4910b5 Add more repository name validation test cases
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-04 16:12:35 -07:00
Doug Davis
38393b63b7 Round 3 - Add Register function
Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-06-02 10:01:21 -07:00
Vincent Giersch
7b3f571555 doc: coherence between requests and parameters + typo
In the request parameters lists `tag` was used instead of
`reference` present in the HTTP requests paths

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-06-02 14:36:55 +00:00
Stephen Day
0d40913b9a Merge pull request #394 from xiekeyang/feature-panic-hook
Feature: Add Hook for Web Application Panic
2015-06-01 13:23:32 -07:00
Vincent Giersch
3c9a2a9bac Fix rados build, remove uuid dependency
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-06-01 09:57:40 +00:00
xiekeyang
47aa47e3f6 Feature: Web Panic Reporting via hooks
This PR is for issue of "email after registry webapp panic" #41, improving my
previous design (closed).
It use self setting up hooks, to catch panic in web application.
And, send email in hooks handle directly, to no use new http server and
handler.

Signed-off-by: xiekeyang <keyangxie@126.com>
2015-05-31 14:21:22 +00:00
Philip Misiowiec
1f31f595d9 fixed typos
Signed-off-by: Phil Misiowiec <phil.misiowiec@atlashealth.com>
2015-05-30 18:22:41 -07:00
Stephen Day
318af0b1ce Merge pull request #443 from gierschv/driver-rados
Storage Driver: Ceph Object Storage (RADOS)
2015-05-29 18:10:49 -07:00
Doug Davis
8a0827f799 Round 2
Make Errors a []Error

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-05-26 17:18:32 -07:00
Doug Davis
0a6a6f5b81 Move ErrorCode logic to new errcode package
Make HTTP status codes match the ErrorCode by looking it up in the Descriptors

Signed-off-by: Doug Davis <dug@us.ibm.com>
2015-05-26 13:18:54 -07:00
Stephen Day
a3006e8e45 Merge pull request #556 from stevvooe/remove-uuid-dependency
Replace uuid dependency with internal library
2015-05-26 10:39:35 -07:00
Stephen J Day
36e34a55ad Replace uuid dependency with internal library
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:59:41 -07:00
Stephen Day
8ce4dcaef8 Merge pull request #546 from stevvooe/resumable-digest-refactor
Remove digest package's dependency on external sha implementation
2015-05-22 18:15:37 -07:00
Stephen J Day
44da954565 Refactor specification of supported digests
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:08:23 -07:00
Stephen J Day
ff52d7f53a Better error message when failing to get AWS auth
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 16:45:45 -07:00
Stephen J Day
eee6cad2cf Remove digest package's dependency on external sha implementation
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.

We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 23:44:08 -07:00
Stephen J Day
87ff947038 Decouple redis dependency from blob descriptor cache
Ensure that clients can use the blob descriptor cache provider without needing
the redis package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 18:58:10 -07:00
Tibor Vass
cb99c076db client: fix a typo preventing compilation
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2015-05-20 21:06:13 -04:00
Stephen Day
ea54a5560a Merge pull request #387 from dmcgowan/registry-client-update
Add client implementation of distribution interface
2015-05-20 15:55:20 -07:00
Derek McGowan
68c1ceac95 Remove error message shortening
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 14:55:59 -07:00
Derek McGowan
49369ffe9a Only do auth checks for endpoints starting with v2
Changes behavior so ping doesn't happen if /v2/ is anywhere in a request path, but instead only at the beginning. This fixes attempts to ping on redirected URLs.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 13:35:23 -07:00
Derek McGowan
c7f7747368 Update transport package to sever distribution dependency
The transport package no longer requires importing distribution for the ReadSeekCloser, instead declares its own.
Added comments on the Authenication handler in session.
Added todo on http seek reader to highlight its lack of belonging to the client transport.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:09:37 -07:00
Derek McGowan
572ff64d21 Add unauthorized error check
Add check for unauthorized error code and explicitly set the error code if the content could not be parsed.
Updated repository test for unauthorized tests and nit feedback.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-20 10:05:44 -07:00
Derek McGowan
13894e8736 Break down type dependencies
Each type no longer requires holding a reference to repository.
Added implementation for signatures get.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:56:27 -07:00
Derek McGowan
a3276fcc5b Feedback update
Update comments and TODOs
Fix switch style
Updated parse http response to take in reader
Add Cancel implementation
Update blobstore variable name

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-19 19:18:30 -07:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Jordan Liggitt
32ed2d7df1 Fix typo: respository->repository
Signed-off-by: Jordan Liggitt <jliggitt@redhat.com>
2015-05-19 13:25:39 -04:00
Derek McGowan
006ddd8283 Lint and documentation fixes
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
2874454224 Create client transport package
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
94e375c5d1 Remove unused and duplicate error types
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
98836d6267 Update to track refactor updates
Added use of cache blob statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fdf7c8ff15 Open cache interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
70074b2286 Rename layer files to blob
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
9d64e461be Update to use blob interfaces
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
fddeb1c8d5 Add missing defer on Tags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
468c5e79ba Simplify configuration and transport
Repository creation now just takes in an http.RoundTripper. Authenticated requests or requests which require additional headers should use the NewTransport function along with a request modifier (such an an authentication handler).

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
a15806ed9c Add base transport to interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
c6b51970cd Removed unused mirror flags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:28 -07:00
Derek McGowan
68d5ecf6bf Update ReadFrom to wrap reader in NopCloser
Wrapping the reader in a NopCloser is necessary to prevent the http library from closing the input reader.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
49f7f54d07 Create authentication handler
Refactory authorizer to take a set of authentication handlers for different authentication schemes returned by an unauthorized HTTP requst.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
d92e5b1096 Add tags implementation
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
da05873b7c Use distribution context instead of google
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
c7ef45130b Cleanup session and config interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
6f9fbf99a9 Split layer and upload from repository
Layer upload moved to its own file with its own unit tests

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
b1ba2183ee Add unit tests for auth challenge and endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
174a732c94 Remove deprecated client interface
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Derek McGowan
ce614b6de8 Add client implementation of distribution interface
Adds functionality to create a Repository client which connects to a remote endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-15 17:56:27 -07:00
Stephen J Day
593bbccdb5 Refactor Blob Service API
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.

The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.

"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.

Recommend Review Approach
-------------------------

This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.

To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.

Storage
-------

The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links.  Eventually, we
will fully consolidate this storage.

The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.

The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.

Cache
-----

Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.

One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.

Handlers
--------

The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.

Configuration
-------------

One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.

Notifications
-------------

Changes the `notification` package are simply to support the interface
changes.

Context
-------

A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-15 17:05:18 -07:00
Stephen Day
aeda6f5f8f Merge pull request #518 from RichardScothern/cache-headers
Set cache headers for layers.
2015-05-12 18:32:16 -07:00
Richard
3dc2d849c7 Set cache headers for layers.
- Set an Etag header
     - Check If-None-Match and respond appropriately
     - Set a Cache-Control header with a default of 1 week

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-12 17:49:18 -07:00
Anton Tiurin
ad80cbe1ea [Server] Listen and serve on a unix socket
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-05-11 16:00:14 +03:00
Derek McGowan
2cf40ab790 Modify blob upload API
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.

closes #470

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-06 15:43:23 -07:00
Richard
70f853e787 Ensure the instrumentedResponseWriter correctly sets the http
status in the context.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-05 16:46:33 -07:00
Stephen Day
dd58ffe6ed Merge pull request #410 from RichardScothern/driver-context
Add golang/x/net/context.Context to storage driver method calls
2015-05-04 19:18:30 -07:00
Derek McGowan
7d6e6aa980 Update API spec to reference digest instead of tarsum
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-01 17:14:38 -07:00
Stephen J Day
69e477f2d5 Use done channel to avoid goroutine leak
This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-28 14:40:20 -07:00
Richard
ae216e365a Make Storage Driver API calls context aware.
- Change driver interface to take a context as its first argument
     - Make newFileReader take a context as its first argument
     - Make newFileWriter take a context as its first argument
     - Make blobstore exists and delete take a context as a first argument
     - Pass the layerreader's context to the storage layer
     - Pass the app's context to purgeuploads
     - Store the app's context into the blobstore (was previously null)
     - Pass the trace'd context to the storage drivers

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 15:58:58 -07:00
Stephen Day
70427e3d9f Merge pull request #381 from RichardScothern/purge-config
Add configuration for upload purging
2015-04-27 14:33:34 -07:00
Richard
6460ddb2cb Add configuration for upload purging
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 11:06:15 -07:00
xiekeyang
c89e517be1 simplify the embedded method expression of repository
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2015-04-27 16:33:58 +08:00
Josh Hawn
628c7fa77b Updated urlbuilder X-Forwarded-Host logic
According to the Apache mod_proxy docs, X-Forwarded-Host can be a
comma-separated list of hosts, to which each proxy appends the requested
host. We want to grab only the first from this comma-separated list
to get the original requested Host when building URLs.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-24 14:16:49 -07:00
Olivier Gambier
9b6d49fdb0 Merge pull request #430 from stevvooe/address-s3-timeout-error
Attempt to address intermittent s3 RequestTimeout error
2015-04-24 11:20:30 -07:00
Stephen Day
528869ac03 Merge pull request #420 from stevvooe/expose-drivernames
Expose storage driver names for tracing
2015-04-24 10:53:40 -07:00
Stephen J Day
e6715c5cec Attempt to address intermittent s3 RequestTimeout error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 20:07:32 -07:00
Stephen J Day
6dd27cd2cb Correctly check s3 chunksize parameter
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 16:31:41 -07:00
Stephen J Day
f6d75a6139 Return after error in handler
This adds a missing return statement. It is not strictly needed since if the
io.Copy fails, the Finish operation will fail. Currently, the client reports
both errors where this new code will correctly only report the io.Copy error.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 13:13:13 -07:00
Stephen J Day
d7eb17fd72 Include driver name in trace messsages
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:31 -07:00
Stephen J Day
9e146437e4 Require storage drivers to report their name
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:01 -07:00
Stephen Day
73960f4024 Merge pull request #419 from stevvooe/pool-buffers-s3
Pool buffers used in S3.WriteStream
2015-04-22 16:41:28 -07:00
Stephen J Day
1adab029f6 Pool buffers used in S3.WriteStream
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 16:25:53 -07:00
Olivier Gambier
506a5b4a97 Merge pull request #418 from stevvooe/accept-chunksize-parameter
Allow configuration of chunksize parameter
2015-04-22 14:46:40 -07:00
Stephen J Day
72a702d0d9 Allow configuration of chunksize parameter
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.

This code likely never worked. The configuration system should be decoupled
from the object instantiation.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 14:31:34 -07:00
Stephen J Day
4330627c71 Check error returned from io.Copy
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 12:35:08 -07:00
Stephen Day
aa09c6c262 Merge pull request #408 from stevvooe/increase-retry-backoff
Backoff retry on verification to give s3 time to propagate
2015-04-21 13:19:28 -07:00
Stephen J Day
632e46418b Backoff retry on verification to give s3 time to propagate
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 13:04:46 -07:00
Stephen J Day
d8dbcc3efa log canonical digest on verification error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-21 11:34:18 -07:00
Olivier Gambier
1b975e980b Merge pull request #405 from stevvooe/eventual-consistency
Attempt to deal with eventual consistency by retrying
2015-04-20 18:59:01 -07:00
Stephen J Day
c8345090d9 Attempt to deal with eventual consistency by retrying
Rather than accept the resulting of a layer validation, we retry up to three
times, backing off 100ms after each try. The thought is that we allow s3 files
to make their way into the correct location increasing the liklihood the
verification can proceed, if possible.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-20 18:49:22 -07:00
Richard
3a1029a262 Add logging for generic handler errors.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-20 17:47:52 -07:00
Stephen Day
7c25fcf698 Merge pull request #375 from RichardScothern/path-logging
registry/storage/driver: add path and other info to filesytem trace methods.
2015-04-17 14:09:23 -07:00