Commit graph

627 commits

Author SHA1 Message Date
Richard Scothern
2c635d1f7e Merge pull request #1532 from RichardScothern/azure-error-types
Update missing blob error checking with latest Azure API
2016-03-15 10:08:36 -07:00
Aaron Lehmann
3a2231fe39 Include status code in UnexpectedHTTPResponseError
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-15 09:03:56 -07:00
Richard Scothern
a5d372535e Merge pull request #1502 from RichardScothern/auth-url-endpoints
URL parse auth endpoints to normalize hostname to lowercase.
2016-03-14 16:28:24 -07:00
Richard Scothern
772e6f4057 Update missing blob error checking with latest Azure API
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Aaron Lehmann
cd6482ecb8 Don't return empty errcode.Errors slices
If this slice ends up empty after parsing the HTTP response body, it
means the body is not well-formed. We've probably encountered an error
message produced by something that uses a different JSON schema, or
an error that just happens to validate as JSON.

An empty errcode.Errors slice is not a very useful thing to return,
since its Error() output is just `<nil>`. Detect this case, and instend
return an UnexpectedHTTPResponseError.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-14 10:27:01 -07:00
Matt Duch
fcb247dfce registry/storage/driver/s3-aws kms support
Signed-off-by: Matt Duch <matt@learnmetrics.com>
2016-03-11 17:19:01 -06:00
Brian Bland
c03b5fc5ee Merge pull request #1438 from BrianBland/newStorageDriverWriter
Adds new StorageDriver.FileWriter interface
2016-03-11 15:06:07 -08:00
Brian Bland
5967d33342 Removes ceph rados driver in favor of Swift API gateway support
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Brian Bland
ab49f85db8 Merge pull request #1512 from kmala/master
Added support to specifiy custom endpoint
2016-03-10 13:48:59 -08:00
Keerthan Mala
2be1b4ef4f Added support to specifiy custom endpoint
Signed-off-by: Keerthan Reddy Mala <keerthan.mala@gmail.com>
2016-03-09 16:12:20 -07:00
Derek McGowan
a59448784b Add client ID to token fetch to GET endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-09 12:44:55 -08:00
Brian Bland
7fd1db9312 Updates Swift driver to support new storagedriver.FileWriter interface
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:38:39 -08:00
Li Yi
a9bf7a2aae Support FileWriter interface for OSS storage driver
Change-Id: Ie5533ad85f944800499ca1040fd67bf1378815e0
Signed-off-by: Li Yi <denverdino@gmail.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
307504713f Storagedriver: GCS: add chunksize parameter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
7162cb19c6 Storagedriver: GCS: implement resumable uploads
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
666273d9f6 StorageDriver: Testsuite: call Close before getting Size
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:38 -08:00
Brian Bland
ff03381d49 Adds new storagedriver.FileWriter interface
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Aaron Lehmann
87e34bd307 Fix two misspellings in source code comments
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-08 15:13:24 -08:00
Richard Scothern
8bd613aa11 URL parse auth endpoints to normalize hostname to lowercase.
It is possible for a middlebox to lowercase the URL at somepoint causing a
lookup in the auth challenges table to fail.  Lowercase hostname before
using as keys to challenge map.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-07 14:08:21 -08:00
Brian Bland
302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Derek McGowan
a8aa2f78de Fix oauth cross repository push
Cross repository push tokens were not being cached and could not be used,
now any returned token will be used and the caching is hidden in the getToken function.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 15:13:27 -08:00
Derek McGowan
efd7ea4624 Add offline token option
Login needs to add an offline token flag to ensure a refresh token is returned by the token endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 14:32:51 -08:00
Derek McGowan
2ef7a872de Add options struct to initialize handler
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 11:32:48 -08:00
Derek McGowan
c9880e6b05 Fetch token by credentials and refresh token
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 00:34:17 -08:00
Richard Scothern
62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars
1593c2413f StorageDriver: GCS: remove support for directory Moves
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Richard Scothern
c8dff1bb57 Merge pull request #1364 from aaronlehmann/content-type-in-api-docs
Add information about manifest content types to API spec
2016-03-01 13:53:49 -08:00
Richard Scothern
5cbac00dd4 Merge pull request #1386 from andrewnguyen/garbage_collect_pick_from_1050
garbage collection
2016-02-29 15:01:48 -08:00
Andrew T Nguyen
feab4aafbc Implements garbage collection subcommand
- Includes a change in the command to run the registry. The registry
  server itself is now started up as a subcommand.
- Includes changes to the high level interfaces to support enumeration
  of various registry objects.

Signed-off-by: Andrew T Nguyen <andrew.nguyen@docker.com>
2016-02-29 14:15:21 -08:00
Brian Bland
3ae504384c Merge pull request #1473 from miminar/pass-blob-size
Commit uploaded blob with size
2016-02-29 10:26:45 -08:00
Aaron Lehmann
a5bd3e6144 Merge pull request #1474 from miminar/respect-errors
Respect errors returned from middleware code
2016-02-26 23:28:37 -08:00
Brian Bland
e430d77342 Merge pull request #1482 from stweil/master
Fix some typos in comments and strings
2016-02-23 14:08:21 -08:00
Stefan Weil
615c6dfced Fix some typos in comments and strings
All of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-23 22:33:38 +01:00
Aaron Lehmann
7571572305 Merge pull request #1471 from RichardScothern/tag-headers
Enable proxying registries to downgrade fetched manifests to Schema 1.
2016-02-23 11:08:46 -08:00
Michal Minar
21a0878665 Commit blob with known size
Signed-off-by: Michal Minar <miminar@redhat.com>
2016-02-23 08:44:06 +01:00
Richard Scothern
7144936612 Enable proxying registries to downgrade fetched manifests to Schema 1.
Ensure Accept headers are sent with TagService.Get (which hits manifest
endpoints).  Add support for remote Get and Put for the proxied blobstore.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-22 17:56:48 -08:00
Michal Minar
4d15bf071c Defined ErrAccessDenied error
Middleware code may perform additional checks on blobs written. Allow it
to return access denied errors that will result in 403 Forbidden.

Signed-off-by: Michal Minar <miminar@redhat.com>
2016-02-22 21:12:59 +01:00
xiekeyang
d58188c4d7 compare error output in tagstore unit test
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-02-22 17:34:22 +08:00
Richard Scothern
ad1b181291 Merge pull request #1466 from RichardScothern/proxy-lazy-auth
Lazily evaluate auth challenges
2016-02-19 15:06:35 -08:00
Richard Scothern
4ce15476bd Extend authChallenger interface to remove type cast.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-17 13:07:14 -08:00
Brian Bland
5806f275bf Merge pull request #1469 from noxiouz/fix_storage_interface_desc
Fix description of StorageDriver.WriteStream
2016-02-17 10:46:11 -08:00
Anton Tiurin
a048a4c8d5 Fix description of StorageDriver.WriteStream
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2016-02-17 13:57:20 +03:00
HuKeping
5408d4e0a5 Cleanup: remove unused log
Signed-off-by: Hu Keping <hukeping@huawei.com>
2016-02-17 14:11:20 +08:00
Brian Bland
c770f28f04 [driver/s3aws] Fix TestStorageClass
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-16 17:50:55 -08:00
Richard Scothern
56af60ad24 To avoid any network use unless necessary, delay establishing authorization
challenges with the upstream until any proxied data is found not to be local.

Implement auth challenges behind an interface and add to unit tests.  Also,
remove a non-sensical unit test.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-16 11:53:42 -08:00
Brian Bland
b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
2016-02-16 11:19:15 -08:00
liuchang0812
0758653b42 fix gofmt
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:42:09 +08:00
liuchang0812
dbba9cf299 closes #1461, enhance log message of oss driver
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:17:09 +08:00
Brian Bland
1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Olivier Gambier
7b66c50bb7 Merge pull request #1452 from aaronlehmann/export-no-credentials-error
Export "no basic auth credentials" as an error value
2016-02-11 10:09:22 -08:00
Olivier Gambier
3c80f6fb5e Merge pull request #1451 from aaronlehmann/typo-fixes
Typo fixes in comments
2016-02-10 17:12:00 -08:00
Aaron Lehmann
b046861e40 Export "no basic auth credentials" as an error value
Making this an exported error value will allow users of the
registry/client/auth module to have consistent behavior between
authentication failures and cases where no credentials are provided.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:46:57 -08:00
Aaron Lehmann
d6a9b20971 Merge pull request #1445 from dmcgowan/fix-manifest-digest-header
Fix schema1 manifest etag and docker content digest header
2016-02-10 16:27:18 -08:00
Aaron Lehmann
aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Aaron Lehmann
a3213ff331 Merge pull request #1420 from dmcgowan/configurable-trust-key
Add option to disable signatures
2016-02-10 16:15:59 -08:00
Derek McGowan
b34e571bff Add option to disable signatures
Add option for specifying trust key for signing schema1 manifests.
Since schema1 signature key identifiers are not verified anywhere and deprecated, storing signatures is no longer a requirement.
Furthermore in schema2 there is no signature, requiring the registry to already add signatures to generated schema1 manifests.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-10 15:20:39 -08:00
Derek McGowan
350234898a Fix schema1 manifest etag and docker content digest header
When schema2 manifests are rewritten as schema1 currently the etag and docker content digest header keep the value for the schema2 manifest.

Fixes #1444

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-09 18:28:43 -08:00
Brian Bland
52b4a328dd Improves flexibility of configuration handling for S3 driver
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 17:27:16 -08:00
Richard Scothern
dd58349b35 Merge pull request #1419 from aaronlehmann/safer-header-copying
On redirect, only copy headers when they don't already exist in the redirected request
2016-02-05 12:14:01 -08:00
Aaron Lehmann
7761a70389 Add information about manifest content types to API spec
Bring the spec up to date for schema2 changes.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-05 11:27:20 -08:00
Richard Scothern
4a1d36c22d Correct type for repo reference
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 17:32:55 -08:00
Derek McGowan
ab9b433fca Merge pull request #1417 from RichardScothern/token-print
Print the correct token expiration time
2016-02-04 16:57:08 -08:00
Richard Scothern
579981b979 Merge pull request #1350 from aibaars/storage-filewriter-pointer
Storage: remove bufferedFileWriter (dead code)
2016-02-04 16:47:52 -08:00
Richard Scothern
2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass
Adds "storageclass" configuration parameter for S3 driver.
2016-02-04 16:44:13 -08:00
Richard Scothern
b737de4856 Merge pull request #1408 from aaronlehmann/repository-interface
Rename Name method of Repository to Named
2016-02-04 16:43:35 -08:00
Aaron Lehmann
b0989446eb Rename Name method of Repository to Named
This makes code that gets the name as a string read like
repo.Named().Name() instead of repo.Name().Name().

Requested in
https://github.com/docker/docker/pull/19887#discussion_r51479753

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-04 09:55:36 -08:00
yuzou
44c0b9ae87 read the actual number of bytes according to the initial size.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-02-04 16:14:35 +08:00
Aaron Lehmann
4354f0a107 On redirect, only copy headers when they don't already exist in the redirected request
A changeset under consideration for Go 1.7 would automatically copy
headers on redirect. This change future-proofs our code so we won't make
duplicate copies of the headers if net/http does it automatically in the
future.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-03 13:19:44 -08:00
Richard Scothern
f5823ff0be Print the correct token expiration time
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-03 10:42:32 -08:00
Olivier Gambier
830bef975d Merge pull request #1410 from aaronlehmann/failured
Correct ErrAuthenticationFailure message
2016-02-01 19:20:35 -08:00
Aaron Lehmann
e387b63001 Correct ErrAuthenticationFailure message
This was "authentication failured". Change it to "authentication
failure".

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-01 17:03:41 -08:00
Aaron Lehmann
e10efb2ecc Merge pull request #1403 from dmcgowan/auth-const-keys
Update auth context keys to use constant
2016-02-01 16:29:07 -08:00
Aaron Lehmann
d3fb5c7064 Merge pull request #1390 from dmcgowan/token-server
Simple integration test token server
2016-02-01 16:27:49 -08:00
Brian Bland
695bac7b0f Adds test for S3 storage class configuration option
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:55 -08:00
Brian Bland
4688d1adc2 Adds "storageclass" configuration parameter for S3 driver.
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:19 -08:00
Aaron Lehmann
2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString
Adds custom registry User-Agent header to s3 HTTP requests
2016-02-01 15:40:36 -08:00
Brian Bland
2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Derek McGowan
648a1343db Update auth context keys to use constant
Prevent using strings throughout the code to reference a string key defined in the auth package.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-28 17:02:09 -08:00
Aaron Lehmann
74c4a0e3e3 Merge pull request #1393 from RichardScothern/putbydgst
Add manifest put by digest to the registry client
2016-01-28 14:41:07 -08:00
Derek McGowan
70e85dde26 Merge pull request #1392 from aaronlehmann/client-range-requests
Support range requests in the client's httpReadSeeker
2016-01-28 13:44:43 -08:00
Aaron Lehmann
8fd84c4a57 Add a CheckRedirect function to the HTTP client
Use it to preserve Accept and Range headers that were added to the
original request.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-27 16:11:36 -08:00
Aaron Lehmann
ccf8154a44 Support range requests in the client's httpReadSeeker
Remove buffering on the reader, because it's not useful. Also remove
artificial io.EOF return.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-27 16:11:36 -08:00
Richard Scothern
a8861549cf Invalidate the blob store descriptor caches when content is removed from from
the proxy.  Also, switch to reference in the scheduler API.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:59 -08:00
Richard Scothern
584c9b517c Correct test digest lengths and enable all unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:46 -08:00
Richard Scothern
9f72e8442d Add manifest put by digest to the registry client
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 14:20:23 -08:00
Derek McGowan
fd17443988 Update token header struct to use json.RawMessage pointer
Since RawMessage json receivers take a pointer type, the Header structure should use points in order to call the json.RawMessage marshal and unmarshal functions

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-25 20:11:41 -08:00
Arthur Baars
3d67dae525 Storage: remove bufferedFileWriter (dead code)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:08 +00:00
Arthur Baars
26e7268a86 Storage: blobwriter.Write/Seek test case
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:07 +00:00
Aaron Lehmann
2b20b0167a Change URLBuilder methods to use references for tags and digests
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:49:29 -08:00
Aaron Lehmann
4441333912 Use reference package internally
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:47:05 -08:00
Aaron Lehmann
c01fe47231 If the media type for a manifest is unrecognized, default to schema1
This is needed for compatibility with some third-party registries that
send an inappropriate Content-Type header such as text/html.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-21 09:34:06 -08:00
Aaron Lehmann
ec636bbfd2 Handle nonstandard token endpoint errors
https://github.com/docker/distribution/pull/1249 changed token fetching
to parse HTTP error response bodies as serialized errcodes. However,
Docker Hub's authentication endpoint does not return error bodies in
this format. To work around this, convert its format into
ErrCodeUnauthorized or ErrCodeUnknown.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-20 14:45:08 -08:00
Arthur Baars
2a4345ca4b StorageDriver: GCS: retry all api calls
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Arthur Baars
6ee339464c StorageDriver: Test suite: improve cleanup
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Aaron Lehmann
8c1a000799 Fix content type for schema1 signed manifests
The Payload function for schema1 currently returns a signed manifest,
but indicates the content type is that of a manifest that isn't signed.

Note that this breaks compatibility with Registry 2.3 alpha 1 and
Docker 1.10-rc1, because they use the incorrect content type.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-18 16:08:02 -08:00
Richard Scothern
3cb403ae5b Merge pull request #1363 from aaronlehmann/media-type-charset
Do not require "charset=utf-8" for a schema1 with content type application/json
2016-01-18 15:20:01 -08:00
Richard Scothern
fc7ee6d789 Merge pull request #1355 from hopkings2008/master
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
2016-01-18 12:38:26 -08:00
Aaron Lehmann
5a2664e0b1 Do not require "charset=utf-8" for a schema1 with content type application/json
For compatibility with other registries that don't use this exact
variant of the Content-Type header, we need to be more flexible about
what we accept. Any form of "application/json" should be allowed. The
charset should not be included in the comparison.

See docker/docker#19400.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-18 09:59:50 -08:00
Arthur Baars
19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-15 11:47:47 +00:00
yuzou
cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-01-15 17:22:43 +08:00