Commit Graph

609 Commits (v2.4.1-rc.2)

Author SHA1 Message Date
Richard Scothern 8bd613aa11 URL parse auth endpoints to normalize hostname to lowercase.
It is possible for a middlebox to lowercase the URL at somepoint causing a
lookup in the auth challenges table to fail.  Lowercase hostname before
using as keys to challenge map.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-07 14:08:21 -08:00
Brian Bland 302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Derek McGowan a8aa2f78de Fix oauth cross repository push
Cross repository push tokens were not being cached and could not be used,
now any returned token will be used and the caching is hidden in the getToken function.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 15:13:27 -08:00
Derek McGowan efd7ea4624 Add offline token option
Login needs to add an offline token flag to ensure a refresh token is returned by the token endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 14:32:51 -08:00
Derek McGowan 2ef7a872de Add options struct to initialize handler
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 11:32:48 -08:00
Derek McGowan c9880e6b05 Fetch token by credentials and refresh token
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 00:34:17 -08:00
Richard Scothern 62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars 1593c2413f StorageDriver: GCS: remove support for directory Moves
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Richard Scothern c8dff1bb57 Merge pull request #1364 from aaronlehmann/content-type-in-api-docs
Add information about manifest content types to API spec
2016-03-01 13:53:49 -08:00
Richard Scothern 5cbac00dd4 Merge pull request #1386 from andrewnguyen/garbage_collect_pick_from_1050
garbage collection
2016-02-29 15:01:48 -08:00
Andrew T Nguyen feab4aafbc Implements garbage collection subcommand
- Includes a change in the command to run the registry. The registry
  server itself is now started up as a subcommand.
- Includes changes to the high level interfaces to support enumeration
  of various registry objects.

Signed-off-by: Andrew T Nguyen <andrew.nguyen@docker.com>
2016-02-29 14:15:21 -08:00
Brian Bland 3ae504384c Merge pull request #1473 from miminar/pass-blob-size
Commit uploaded blob with size
2016-02-29 10:26:45 -08:00
Aaron Lehmann a5bd3e6144 Merge pull request #1474 from miminar/respect-errors
Respect errors returned from middleware code
2016-02-26 23:28:37 -08:00
Brian Bland e430d77342 Merge pull request #1482 from stweil/master
Fix some typos in comments and strings
2016-02-23 14:08:21 -08:00
Stefan Weil 615c6dfced Fix some typos in comments and strings
All of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-23 22:33:38 +01:00
Aaron Lehmann 7571572305 Merge pull request #1471 from RichardScothern/tag-headers
Enable proxying registries to downgrade fetched manifests to Schema 1.
2016-02-23 11:08:46 -08:00
Michal Minar 21a0878665 Commit blob with known size
Signed-off-by: Michal Minar <miminar@redhat.com>
2016-02-23 08:44:06 +01:00
Richard Scothern 7144936612 Enable proxying registries to downgrade fetched manifests to Schema 1.
Ensure Accept headers are sent with TagService.Get (which hits manifest
endpoints).  Add support for remote Get and Put for the proxied blobstore.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-22 17:56:48 -08:00
Michal Minar 4d15bf071c Defined ErrAccessDenied error
Middleware code may perform additional checks on blobs written. Allow it
to return access denied errors that will result in 403 Forbidden.

Signed-off-by: Michal Minar <miminar@redhat.com>
2016-02-22 21:12:59 +01:00
xiekeyang d58188c4d7 compare error output in tagstore unit test
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-02-22 17:34:22 +08:00
Richard Scothern ad1b181291 Merge pull request #1466 from RichardScothern/proxy-lazy-auth
Lazily evaluate auth challenges
2016-02-19 15:06:35 -08:00
Richard Scothern 4ce15476bd Extend authChallenger interface to remove type cast.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-17 13:07:14 -08:00
Brian Bland 5806f275bf Merge pull request #1469 from noxiouz/fix_storage_interface_desc
Fix description of StorageDriver.WriteStream
2016-02-17 10:46:11 -08:00
Anton Tiurin a048a4c8d5 Fix description of StorageDriver.WriteStream
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2016-02-17 13:57:20 +03:00
HuKeping 5408d4e0a5 Cleanup: remove unused log
Signed-off-by: Hu Keping <hukeping@huawei.com>
2016-02-17 14:11:20 +08:00
Brian Bland c770f28f04 [driver/s3aws] Fix TestStorageClass
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-16 17:50:55 -08:00
Richard Scothern 56af60ad24 To avoid any network use unless necessary, delay establishing authorization
challenges with the upstream until any proxied data is found not to be local.

Implement auth challenges behind an interface and add to unit tests.  Also,
remove a non-sensical unit test.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-02-16 11:53:42 -08:00
Brian Bland b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
2016-02-16 11:19:15 -08:00
liuchang0812 0758653b42 fix gofmt
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:42:09 +08:00
liuchang0812 dbba9cf299 closes #1461, enhance log message of oss driver
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:17:09 +08:00
Brian Bland 1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Olivier Gambier 7b66c50bb7 Merge pull request #1452 from aaronlehmann/export-no-credentials-error
Export "no basic auth credentials" as an error value
2016-02-11 10:09:22 -08:00
Olivier Gambier 3c80f6fb5e Merge pull request #1451 from aaronlehmann/typo-fixes
Typo fixes in comments
2016-02-10 17:12:00 -08:00
Aaron Lehmann b046861e40 Export "no basic auth credentials" as an error value
Making this an exported error value will allow users of the
registry/client/auth module to have consistent behavior between
authentication failures and cases where no credentials are provided.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:46:57 -08:00
Aaron Lehmann d6a9b20971 Merge pull request #1445 from dmcgowan/fix-manifest-digest-header
Fix schema1 manifest etag and docker content digest header
2016-02-10 16:27:18 -08:00
Aaron Lehmann aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Aaron Lehmann a3213ff331 Merge pull request #1420 from dmcgowan/configurable-trust-key
Add option to disable signatures
2016-02-10 16:15:59 -08:00
Derek McGowan b34e571bff Add option to disable signatures
Add option for specifying trust key for signing schema1 manifests.
Since schema1 signature key identifiers are not verified anywhere and deprecated, storing signatures is no longer a requirement.
Furthermore in schema2 there is no signature, requiring the registry to already add signatures to generated schema1 manifests.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-10 15:20:39 -08:00
Derek McGowan 350234898a Fix schema1 manifest etag and docker content digest header
When schema2 manifests are rewritten as schema1 currently the etag and docker content digest header keep the value for the schema2 manifest.

Fixes #1444

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-09 18:28:43 -08:00
Brian Bland 52b4a328dd Improves flexibility of configuration handling for S3 driver
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 17:27:16 -08:00
Richard Scothern dd58349b35 Merge pull request #1419 from aaronlehmann/safer-header-copying
On redirect, only copy headers when they don't already exist in the redirected request
2016-02-05 12:14:01 -08:00
Aaron Lehmann 7761a70389 Add information about manifest content types to API spec
Bring the spec up to date for schema2 changes.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-05 11:27:20 -08:00
Richard Scothern 4a1d36c22d Correct type for repo reference
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 17:32:55 -08:00
Derek McGowan ab9b433fca Merge pull request #1417 from RichardScothern/token-print
Print the correct token expiration time
2016-02-04 16:57:08 -08:00
Richard Scothern 579981b979 Merge pull request #1350 from aibaars/storage-filewriter-pointer
Storage: remove bufferedFileWriter (dead code)
2016-02-04 16:47:52 -08:00
Richard Scothern 2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass
Adds "storageclass" configuration parameter for S3 driver.
2016-02-04 16:44:13 -08:00
Richard Scothern b737de4856 Merge pull request #1408 from aaronlehmann/repository-interface
Rename Name method of Repository to Named
2016-02-04 16:43:35 -08:00
Aaron Lehmann b0989446eb Rename Name method of Repository to Named
This makes code that gets the name as a string read like
repo.Named().Name() instead of repo.Name().Name().

Requested in
https://github.com/docker/docker/pull/19887#discussion_r51479753

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-04 09:55:36 -08:00
yuzou 44c0b9ae87 read the actual number of bytes according to the initial size.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-02-04 16:14:35 +08:00
Aaron Lehmann 4354f0a107 On redirect, only copy headers when they don't already exist in the redirected request
A changeset under consideration for Go 1.7 would automatically copy
headers on redirect. This change future-proofs our code so we won't make
duplicate copies of the headers if net/http does it automatically in the
future.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-03 13:19:44 -08:00
Richard Scothern f5823ff0be Print the correct token expiration time
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-03 10:42:32 -08:00
Olivier Gambier 830bef975d Merge pull request #1410 from aaronlehmann/failured
Correct ErrAuthenticationFailure message
2016-02-01 19:20:35 -08:00
Aaron Lehmann e387b63001 Correct ErrAuthenticationFailure message
This was "authentication failured". Change it to "authentication
failure".

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-01 17:03:41 -08:00
Aaron Lehmann e10efb2ecc Merge pull request #1403 from dmcgowan/auth-const-keys
Update auth context keys to use constant
2016-02-01 16:29:07 -08:00
Aaron Lehmann d3fb5c7064 Merge pull request #1390 from dmcgowan/token-server
Simple integration test token server
2016-02-01 16:27:49 -08:00
Brian Bland 695bac7b0f Adds test for S3 storage class configuration option
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:55 -08:00
Brian Bland 4688d1adc2 Adds "storageclass" configuration parameter for S3 driver.
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:19 -08:00
Aaron Lehmann 2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString
Adds custom registry User-Agent header to s3 HTTP requests
2016-02-01 15:40:36 -08:00
Brian Bland 2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Derek McGowan 648a1343db Update auth context keys to use constant
Prevent using strings throughout the code to reference a string key defined in the auth package.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-28 17:02:09 -08:00
Aaron Lehmann 74c4a0e3e3 Merge pull request #1393 from RichardScothern/putbydgst
Add manifest put by digest to the registry client
2016-01-28 14:41:07 -08:00
Derek McGowan 70e85dde26 Merge pull request #1392 from aaronlehmann/client-range-requests
Support range requests in the client's httpReadSeeker
2016-01-28 13:44:43 -08:00
Aaron Lehmann 8fd84c4a57 Add a CheckRedirect function to the HTTP client
Use it to preserve Accept and Range headers that were added to the
original request.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-27 16:11:36 -08:00
Aaron Lehmann ccf8154a44 Support range requests in the client's httpReadSeeker
Remove buffering on the reader, because it's not useful. Also remove
artificial io.EOF return.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-27 16:11:36 -08:00
Richard Scothern a8861549cf Invalidate the blob store descriptor caches when content is removed from from
the proxy.  Also, switch to reference in the scheduler API.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:59 -08:00
Richard Scothern 584c9b517c Correct test digest lengths and enable all unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:46 -08:00
Richard Scothern 9f72e8442d Add manifest put by digest to the registry client
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 14:20:23 -08:00
Derek McGowan fd17443988 Update token header struct to use json.RawMessage pointer
Since RawMessage json receivers take a pointer type, the Header structure should use points in order to call the json.RawMessage marshal and unmarshal functions

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-01-25 20:11:41 -08:00
Arthur Baars 3d67dae525 Storage: remove bufferedFileWriter (dead code)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:08 +00:00
Arthur Baars 26e7268a86 Storage: blobwriter.Write/Seek test case
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:07 +00:00
Aaron Lehmann 2b20b0167a Change URLBuilder methods to use references for tags and digests
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:49:29 -08:00
Aaron Lehmann 4441333912 Use reference package internally
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:47:05 -08:00
Aaron Lehmann c01fe47231 If the media type for a manifest is unrecognized, default to schema1
This is needed for compatibility with some third-party registries that
send an inappropriate Content-Type header such as text/html.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-21 09:34:06 -08:00
Aaron Lehmann ec636bbfd2 Handle nonstandard token endpoint errors
https://github.com/docker/distribution/pull/1249 changed token fetching
to parse HTTP error response bodies as serialized errcodes. However,
Docker Hub's authentication endpoint does not return error bodies in
this format. To work around this, convert its format into
ErrCodeUnauthorized or ErrCodeUnknown.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-20 14:45:08 -08:00
Arthur Baars 2a4345ca4b StorageDriver: GCS: retry all api calls
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Arthur Baars 6ee339464c StorageDriver: Test suite: improve cleanup
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Aaron Lehmann 8c1a000799 Fix content type for schema1 signed manifests
The Payload function for schema1 currently returns a signed manifest,
but indicates the content type is that of a manifest that isn't signed.

Note that this breaks compatibility with Registry 2.3 alpha 1 and
Docker 1.10-rc1, because they use the incorrect content type.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-18 16:08:02 -08:00
Richard Scothern 3cb403ae5b Merge pull request #1363 from aaronlehmann/media-type-charset
Do not require "charset=utf-8" for a schema1 with content type application/json
2016-01-18 15:20:01 -08:00
Richard Scothern fc7ee6d789 Merge pull request #1355 from hopkings2008/master
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
2016-01-18 12:38:26 -08:00
Aaron Lehmann 5a2664e0b1 Do not require "charset=utf-8" for a schema1 with content type application/json
For compatibility with other registries that don't use this exact
variant of the Content-Type header, we need to be more flexible about
what we accept. Any form of "application/json" should be allowed. The
charset should not be included in the comparison.

See docker/docker#19400.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-18 09:59:50 -08:00
Arthur Baars 19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-15 11:47:47 +00:00
yuzou cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-01-15 17:22:43 +08:00
Richard Scothern 7378e21678 Merge pull request #1332 from RichardScothern/gcs-params
Change the parameters to the GCS drivers to allow CircleCI testing.
2016-01-14 15:50:09 -08:00
Brian Bland 8c5a6c13c0 Splits up blob create options definitions to be package-specific
Redefines privately in both storage and client packages

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-14 10:47:33 -08:00
Brian Bland 3ef917c269 Fixes cross-repo blob mounting in the BlobUploadHandler
Accidentally checked for err != nil instead of err == nil :/
Also now ensures that only a non-nil option is appended to the create
options slice

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 19:21:45 -08:00
Richard Scothern 81dbebc7d3 Change the parameters to the GCS drivers to allow CircleCI testing.
Remove the requirement of file system access to run GCS unit tests.  Deconstruct
the input parameters to take the private key and email which can be specified on
the build system via environment variables.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-13 18:00:59 -08:00
Brian Bland ce88d8a6f4 Adds functional options arguments to the Blobs Create method
Removes the Mount operation and instead implements this behavior as part
of Create a From option is provided, which in turn returns a rich
ErrBlobMounted indicating that a blob upload session was not initiated,
but instead the blob was mounted from another repository

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 16:42:59 -08:00
Richard Scothern 3ea0d70896 Fix manifest API unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-11 12:52:21 -08:00
Richard Scothern 3a1220de01 Merge pull request #1319 from RichardScothern/update-tags
Remove tags referencing deleted manifests.
2016-01-11 11:33:22 -08:00
Richard Scothern 93d9070c8b Merge pull request #1269 from BrianBland/crossRepositoryPush
Adds cross-repository blob mounting behavior
2016-01-08 14:37:00 -08:00
Brian Bland 3a35a2d953 Allows token authentication handler to request additional scopes
When an auth request provides the "from" query parameter, the token
handler will add a "pull" scope for the provided repository, refreshing
the token if the overall scope has increased

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:23 -08:00
Brian Bland 5df21570a7 Adds cross-repository blob mounting behavior
Extends blob upload POST endpoint to support mount and from query
parameters as described in #634

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:18 -08:00
Richard Scothern cc0b6b86ea Merge pull request #1276 from denverdino/oss-test
Support large layer for OSS driver
2016-01-08 13:26:09 -08:00
Stephen Day a7ae88da45 Merge pull request #1281 from aaronlehmann/new-manifest
Implement schema2 manifest formats
2016-01-07 17:19:56 -08:00
Stephen Day 5ebf6391a7 Merge pull request #911 from stevvooe/consistent-error-string
More consistent return from ErrorCode.Error()
2016-01-07 17:06:12 -08:00
Aaron Lehmann 6d17423a6d Move MediaType into manifest.Versioned
This makes content type sniffing cleaner. The document just needs to be
decoded into a manifest.Versioned structure. It's no longer a two-step
process.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann 697af09566 Recognize clients that don't support manifest lists
Convert a default platform's manifest to schema1 on the fly.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann 9c416f0e94 Add support for manifest list ("fat manifest")
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann 9284810356 Add API unit testing for schema2 manifest
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Aaron Lehmann 3f746a8207 Recognize clients that don't support schema2, and convert manifests to schema1 on the fly
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00