Commit graph

869 commits

Author SHA1 Message Date
Ahmet Alp Balkan
2ab25288a2
Update vendored azure-sdk-for-go
Updating to a recent version of Azure Storage SDK to be
able to patch some memory leaks through configurable HTTP client
changes which were made possible by recent patches to it.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-03 13:24:57 -07:00
Derek McGowan
fbe6e8d212 Merge pull request #2008 from miminar/honor-x-forwarded-port
Honor X-Forwarded-Port header
2016-11-02 12:56:17 -07:00
Pierre-Yves Ritschard
f1cf7de788 fixup! v2 signer: correctly sort headers
Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:07:02 +01:00
Pierre-Yves Ritschard
775cc6d632 v2 signer: correctly sort headers
The current code determines the header order for the
"string-to-sign" payload by sorting on the concatenation
of headers and values, whereas it should only happen on the
key.

During multipart uploads, since `x-amz-copy-source-range` and
`x-amz-copy-source` headers are present, V2 signatures fail to
validate since header order is swapped.

This patch reverts to the expected behavior.

Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:01:34 +01:00
Michal Minář
1b43e1e30d
Honor X-Forwarded-Port and Forwarded headers
Prefer non-standard headers like X-Forwarded-Proto, X-Forwarded-Host and
X-Forwarded-Port over the standard Forwarded header to maintain
backwards compatibility.

If a port is not specified neither in Host nor in forwarded headers but
it is specified just with X-Forwarded-Port, use its value in base urls
for redirects.

Forwarded header is defined in rfc7239.

X-Forwarded-Port is a non-standard header. Here's a description copied
from "HTTP Headers and Elastic Load Balancing" of AWS ELB docs:

> The X-Forwarded-Port request header helps you identify the port that
> an HTTP or HTTPS load balancer uses to connect to the client.

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-11-02 16:49:42 +01:00
Ahmet Alp Balkan
a994f35657
driver/swift: Fix go vet warning
Driver was passing connections by copying. Storing
`swift.Connection` as pointer to fix the warnings.

Ref: #2030.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-31 11:41:53 -07:00
Ahmet Alp Balkan
6d2a0bafcd
storagedriver/azure: close leaking response body
In GetContent() we read the bytes from a blob but do not close
the underlying response body.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-28 15:13:22 -07:00
Antonio Murdaca
16f086a0ec
registry/client: set Accept: identity header when getting layers
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-10-25 10:33:11 +02:00
Stephen J Day
c9aaff00f8
manifest: references should cover all children
To allow generic manifest walking, we define an interface method of
`References` that returns the referenced items in the manifest. The
current implementation does not return the config target from schema2,
making this useless for most applications.

The garbage collector has been modified to show the utility of this
correctly formed `References` method. We may be able to make more
generic traversal methods with this, as well.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-10-18 11:43:33 -07:00
Richard Scothern
96f63cbe7f Merge pull request #2010 from mbentley/add-new-s3-region
Added new us-east-2 region for S3
2016-10-18 10:25:30 -07:00
Richard Scothern
5bd2cfaddd Merge pull request #2000 from jessfraz/1950-comment-on-non-dryrun-as-well
storage/garbagecollect: add emit comments for when dryRun is false
2016-10-18 10:00:04 -07:00
Matt Bentley
3857f50825
Added new us-east-2 region for S3
Signed-off-by: Matt Bentley <mbentley@mbentley.net>
2016-10-18 12:30:34 -04:00
Richard Scothern
d0cdc4802b Merge pull request #2002 from lnr0626/1996-instance-roles-with-regionendpoint
Allow using ec2 roles when specifying region endpoint
2016-10-17 13:50:02 -07:00
Richard Scothern
a621a86cb4 Fix aliyun OSS Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-17 09:43:15 -07:00
Noah Treuhaft
12e73f01d2 Fix s3-goamz Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-17 09:43:15 -07:00
Richard Scothern
bd91493195 Satisfy the latest go lint rules
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-14 17:03:08 -07:00
Lloyd Ramey
c8ea7840d3 Allow using ec2 roles when specifying region endpoint
Signed-off-by: Lloyd Ramey <lnr0626@gmail.com>
2016-10-13 18:07:37 -04:00
Jess Frazelle
809774c33f
storage/garbagecollect: add emit commits for when dryRun is false
Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-13 08:04:19 -07:00
Richard Scothern
252cc27ab1 Merge pull request #1988 from nwt/fix-s3-delete
Fix S3 Delete method's notion of subpaths
2016-10-11 11:14:29 -07:00
Richard Scothern
1921dde3f1 Merge pull request #1942 from sergeyfd/auth-proxy
Support for custom authentication URL in proxying registry
2016-10-10 10:40:51 -07:00
HuKeping
b984f91b32 Fix unit test
Signed-off-by: Hu Keping <hukeping@huawei.com>
2016-10-10 11:37:30 +08:00
Noah Treuhaft
76226c61a9 Fix S3 Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-06 11:21:55 -07:00
Derek McGowan
d35d94dcec
Update to fix lint errors
Context should use type values instead of strings.
Updated direct calls to WithValue, but still other uses of string keys.
Update Acl to ACL in s3 driver.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-10-05 17:47:12 -07:00
Richard Scothern
431cfa3179 Merge pull request #1966 from nwt/fix-redis-tests
Fix connection pool exhaustion in Redis tests
2016-09-19 16:27:56 -07:00
Richard Scothern
8cb75f692f Merge pull request #1868 from stanhu/fix-missing-credentials-with-default-port
Fix missing auth headers with PATCH HTTP request when pushing to default port
2016-09-19 15:55:12 -07:00
Noah Treuhaft
cafeae4ecd Fix connection pool exhaustion in Redis tests
The Redis tests were failing with a "connection pool exhausted" error
from Redigo. Closing the connection used for FLUSHDB fixes the problem.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-19 15:30:32 -07:00
Stan Hu
6d952c92cf Rename CanonicalAddr -> canonicalAddr
Update comment for hasPort

Signed-off-by: Stan Hu <stanhu@gmail.com>
2016-09-19 14:16:02 -07:00
Richard Scothern
cfad4321c1 Merge pull request #1957 from nwt/notification-filtering
Add notification filtering by target media type
2016-09-19 10:41:03 -07:00
Noah Treuhaft
ad6bb66faf Add notification filtering by target media type
The Hub registry generates a large volume of notifications, many of
which are uninteresting based on target media type.  Discarding them
within the notification endpoint consumes considerable resources that
could be saved by discarding them within the registry.  To that end,
this change adds registry configuration options to restrict the
notifications sent to an endpoint based on target media type.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-16 12:01:03 -07:00
Richard Scothern
b6e0cfbdaa Merge pull request #1959 from nwt/disable-access-logging
Add configuration option to disable access logging
2016-09-15 09:45:56 -07:00
Noah Treuhaft
4034ff65f0 Add configuration option to disable access logging
Access logging is great.  Access logging you can turn off is even
better.  This change adds a configuration option for that.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-14 14:33:30 -07:00
Richard Scothern
835b04c004 Merge pull request #1958 from spacexnice/master
remoteURL parameter error cause  challenge cache not working which eventually lead to client manifest request timeout
2016-09-14 12:33:18 -07:00
Serge Dubrouski
81c5870c86 Support for custom authentication URL in proxying registry
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-09-13 17:07:20 -06:00
Derek McGowan
17fb0bb6b3 Merge pull request #1934 from jheiss/token_ssl_pem_type
Check PEM block type when reading token cert file
2016-09-13 09:45:06 -07:00
spacexnice
3c4b551a7c remoteURL parameter error cause challenge cache not working,critical area consume to much time, which eventually lead to client manifest request timeout(90s) in massive(which i test 80 concurrent) concurrent manifest request situation.
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-09-13 09:58:12 +08:00
Stephen Day
5e8d18f615 Merge pull request #1945 from stevvooe/better-report-on-invalid-secret
handlers: provide better log message on mismatched secret
2016-09-07 12:55:48 -07:00
Noah Treuhaft
91f268e5a5 Downgrade token auth JWT logging from error to info
The token auth package logs JWT validation and verification failures at
the `error` level.  But from the server's perspective, these aren't
errors.  They're the expected response to bad input.  Logging them at
the `info` level better reflects that distinction.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-07 10:45:06 -07:00
Stephen J Day
668b0a5f40
handlers: provide better log message on mismatched secret
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-09-02 15:24:35 -07:00
Stan Hu
462bb55c3f Fix missing auth headers with PATCH HTTP request when pushing to default port
If a user specifies `mydomain.com:443` in the `Host` configuration, the
PATCH request for the layer upload will fail because the challenge does not
appear to be in the map. To fix this, we normalize the map keys to always
use the Host:Port combination.

Closes https://github.com/docker/docker/issues/18469

Signed-off-by: Stan Hu <stanhu@gmail.com>
2016-09-01 14:46:31 -07:00
Fabio Berchtold
7dcac52f18 Add v2 signature signing to S3 storage driver (#1800)
* Add v2 signature signing to S3 storage driver

Closes #1796
Closes #1606

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* use Logrus for debug logging

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* use 'date' instead of 'x-amz-date' in request header

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* only allow v4 signature signing against AWS S3

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>
2016-09-01 13:52:40 -07:00
Jason Heiss
d04481e388 Check PEM block type when reading token cert file
closes #1909

Signed-off-by: Jason Heiss <jheiss@twosigma.com>
2016-09-01 16:48:55 -04:00
Richard Scothern
49da29ee46 Merge pull request #1925 from dmcgowan/reenable-race-detector
Re-enable race detector in circleci
2016-09-01 13:36:12 -07:00
Derek McGowan
41f383fb9a Merge pull request #1928 from ghostplant/master
Dynamically Parsing the Latest HTPassword File
2016-08-30 10:29:35 -07:00
Derek McGowan
205e606a4c Update proxy scheduler test to account for race
Running with the race detector may cause some parts
of the code to run slower causing a race in the scheduler
ordering.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-29 14:27:32 -07:00
Derek McGowan
b3e276ff93 Add hash map locking to proxy tests
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-29 13:39:24 -07:00
Derek McGowan
a50ce1ab93 Add locking to repository access in memory cache
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 18:10:42 -07:00
Derek McGowan
dabdc5e52b Fix access race in proxy scheduler
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 17:40:21 -07:00
Derek McGowan
b1b100cf01 Merge pull request #1923 from spacexnice/master
fix simpleAuthChallge concurrent problem
2016-08-26 17:31:09 -07:00
Derek McGowan
798b7331ca Add shutdown to test environment
Ensures test http server is fully closed before moving on with test.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 16:23:07 -07:00
Derek McGowan
38ee36eee4 Disable upload purging during test
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 16:23:07 -07:00
a-palchikov
82609180a1 tag service: properly handle error responses on HEAD requests by (#1918)
* tag service: properly handle error responses on HEAD requests by
re-issuing requests as GET for proper error details.

Fixes #1911.

Signed-off-by: dmitri <deemok@gmail.com>

* Simplify handling of failing HEAD requests in TagService and
make a GET request for cases:
  - if the server does not handle HEAD
  - if the response was an error to get error details

Signed-off-by: dmitri <deemok@gmail.com>

* Add a missing http.Response.Body.Close call for the GET request.

Signed-off-by: dmitri <deemok@gmail.com>
2016-08-26 14:35:04 -07:00
cuiwei13
169ff1d098 Dynamically Parsing the Latest HTPassword File
To parse the latest account list dynamically instead of restarting the distribution service frequently.

Signed-off-by: CUI Wei <ghostplant@qq.com>
2016-08-27 04:28:33 +08:00
spacexnice
68243306fb correct onerror behavior
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-26 09:41:54 +08:00
spacexnice
22cf89c02b gofmt -s do code format
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-25 07:34:36 +08:00
Matthew Green
dea554fc7c Swift driver now bulk deletes in chunks specified by the server (#1915)
Swift driver now bulk deletes in chunks specified by the server

Signed-off-by: Matthew Green <matthew.green@uk.ibm.com>
2016-08-24 10:09:25 -07:00
spacexnice
ca2b78dc0a adjust for loop
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-24 09:31:27 +08:00
spacexnice
44b8157572 fix simpleAuthChallge concurrent problem
Signed-off-by: spacexnice <yaoyao.xyy@alibaba-inc.com>
2016-08-23 22:09:18 +08:00
Richard Scothern
010e063270 Merge pull request #1906 from nwt/s3-multipart-copy
Use multipart upload API in S3 Move method
2016-08-17 15:22:32 -07:00
Richard Scothern
c4297ef9da Merge pull request #1839 from adamvduke/adamvduke/allow-http2-registry-clients
Allow registry clients to connect via http2
2016-08-16 11:48:06 -07:00
Noah Treuhaft
63468ef4a8 Use multipart upload API in S3 Move method
This change to the S3 Move method uses S3's multipart upload API to copy
objects whose size exceeds a threshold.  Parts are copied concurrently.
The level of concurrency, part size, and threshold are all configurable
with reasonable defaults.

Using the multipart upload API has two benefits.

* The S3 Move method can now handle objects over 5 GB, fixing #886.

* Moving most objects, and espectially large ones, is faster.  For
  example, moving a 1 GB object averaged 30 seconds but now averages 10.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-08-16 10:53:24 -07:00
Stefan Majewsky
a7c6bfd59f [swift] support different user-domain and tenant-domain
This is already supported by ncw/swift, so we just need to pass the
parameters from the storage driver.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-08-15 11:21:42 +02:00
Adam Duke
ac009c86f1 Allow registry clients to connect via http2
Http2 will be enabled by default and can be disabled with a configuration option.

Signed-off-by: Adam Duke <adam.v.duke@gmail.com>
2016-08-13 22:07:42 -04:00
Richard Scothern
c9fd26e9ef Merge pull request #1895 from hinshun/improve-enumerate
Improve catalog enumerate runtime by an order of magnitude
2016-08-11 10:53:36 -07:00
Richard Scothern
baca174469 Merge pull request #1892 from stevvooe/use-math-rand-reader
testutil, storage: use math/rand.Read where possible
2016-08-11 10:04:36 -07:00
Stephen J Day
040db51795
testutil, storage: use math/rand.Read where possible
Use the much faster math/rand.Read function where cryptographic
guarantees are not required. The unit test suite should speed up a
little bit but we've already optimized around this, so it may not
matter.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-10 14:26:12 -07:00
Edgar Lee
2f81b3b058 Improve catalog enumerate runtime by an order of magnitude
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-09 17:42:26 -07:00
Richard Scothern
9000745401 Merge pull request #1888 from stevvooe/more-efficient-path-comparison
registry/storage: more efficient path compare in catalog
2016-08-09 13:45:07 -07:00
Stephen J Day
308faf00f1
catalog: add benchmarks for overridden path comparison
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 17:09:49 -07:00
Stephen J Day
bba5a0d05c
registry/storage: more efficient path compare in catalog
Previous component-wise path comparison is recursive and generates a
large amount of garbage. This more efficient version simply replaces the
path comparison with the zero-value to sort before everything. We do
this by replacing the byte-wise comparison that swaps a single character
inline for the separator comparison, such that separators sort first.

The resulting implementation provides component-wise path comparison
with no cost incurred for allocation or stack frame.

Direction of the comparison is also reversed to match Go style.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 15:55:26 -07:00
Edgar Lee
379312c148 Stop ErrFinishedWalk from escaping from Repositories walk
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-05 17:21:48 -07:00
Richard Scothern
bfa0a9c097 Merge pull request #1877 from spacexnice/master
function copyContent() cause socket leak.
2016-08-02 09:15:23 -07:00
yaoyao.xyy
64c8dd3252 when deploy registry as a pull through cache ,function copeContent() may cause a socket leak when docker user canceled its pull operation.
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-08-02 11:10:10 +08:00
Michal Minář
7365003236 Provide stat descriptor for Create method during cross-repo mount (#1857)
* Allow precomputed stats on cross-mounted blobs

Signed-off-by: Michal Minář <miminar@redhat.com>

* Extended cross-repo mount tests

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-08-01 19:07:11 -07:00
Frank Chen
87917f3052 Add 'objectAcl' Option to the S3 Storage Backend (#1867)
* Add Object ACL Support to the S3 Storage Backend

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Made changes per @RichardScothern's comments

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Fix Typos

Signed-off-by: Frank Chen <frankchn@gmail.com>
2016-07-27 12:26:57 -07:00
Richard Scothern
c2a201dabf Merge pull request #1860 from Seb-Solon/master
Fix #1854
2016-07-22 10:04:01 -07:00
Richard Scothern
2287f1c42f Fix the build.
Pass the manifestURL directly into the schema2 manifest handler instead of
accessing through the repository as it has since the reference is now an
interface.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-21 17:16:47 -07:00
Richard Scothern
2b72dd3927 Merge pull request #1829 from nwt/foreign-layer-host-whitelist
Add a foreign layer URL host whitelist
2016-07-21 16:02:20 -07:00
Aaron Lehmann
f56b143dc8 Merge pull request #1858 from RichardScothern/decouple
Decouple storage components by redefining dependencies as interfaces
2016-07-21 11:12:05 -07:00
Sebastien Coavoux
a9c9daf0f2 Fix: Compare path properly when list repository in catalog. #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:46:00 -04:00
Sebastien Coavoux
3290e2bef5 Enh: Test - Edit catalog_test to reproduce #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:23:19 -04:00
Richard Scothern
2a5fcacdf0 Decouple storage components by redefining dependencies as interfaces instead of concrete types
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-20 15:09:11 -07:00
Richard Scothern
857d0f15c0 Merge pull request #1757 from miminar/cross-mount-middleware-repository
Allow to use repo middleware wrapper during cross-repo mounts
2016-07-19 17:43:10 -07:00
Richard Scothern
dd355e95af Merge pull request #1686 from marcusmartins/nbf-leeway
Add leeway to JWT nbf checking
2016-07-19 17:41:57 -07:00
Noah Treuhaft
61e5803b56 Add control over validation of URLs in pushed manifests
Until we have some experience hosting foreign layer manifests, the Hub
operators wish to limit foreign layers on Hub. To that end, this change
adds registry configuration options to restrict the URLs that may appear
in pushed manifests.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-19 14:38:42 -07:00
Richard Scothern
f27ceb7ab5 Merge pull request #1710 from majewsky/swift/wait-for-dlo-segments-during-read
[Swift] add simple heuristic to detect incomplete DLOs during read ops
2016-07-19 09:07:44 -07:00
Marcus Martins
db1bf93098
Add leeway to JWT nbf and exp checking
Adds a constant leeway (60 seconds) to the nbf and exp claim check to
account for clock skew between the registry servers and the
authentication server that generated the JWT.

The leeway of 60 seconds is a bit arbitrary but based on the RFC
recommendation and hub.docker.com logs/metrics where we don't see
drifts of more than a second on our servers running ntpd.

I didn't attempt to make the leeway configurable as it would add extra
complexity to the PR and I am not sure how Distribution prefer to
handle runtime flags like that.

Also, I am simplifying the exp and nbf check for readability as the
previous `NOT (A AND B)` with cmp operators was not very friendly.

Ref:
https://tools.ietf.org/html/rfc7519#section-4.1.5

Signed-off-by: Marcus Martins <marcus@docker.com>
2016-07-18 17:47:30 -07:00
Derek McGowan
f8083b7ff3 Merge pull request #1836 from hinshun/catalog-walk-return-error
Fix storage drivers dropping non EOF errors when listing repositories
2016-07-15 14:29:50 -07:00
Derek McGowan
42a7e6e171 Merge pull request #1837 from RichardScothern/catalog-test
Use registry APIs to generate catalog test environment
2016-07-15 14:23:22 -07:00
Edgar Lee
a82f661ef0 Use typecast over reflect for error type checking
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 15:03:18 -07:00
Edgar Lee
3bfd03cbe6 Refactor errVal named parameter for catalog repositories to err
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:28:08 -07:00
Edgar Lee
aeb9a29499 Handle new errors returned from catalog repository listing
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:24:16 -07:00
Richard Scothern
b126d6643e Use distribution APIs to generate catalog test environment
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-14 09:58:19 -07:00
Edgar Lee
5a0b35ca10 Fix storage drivers dropping non EOF errors when listing repositories
This fixes errors other than io.EOF from being dropped when a storage driver
lists repositories. For example, filesystem driver may point to a missing
directory and errors, which then gets subsequently dropped.

Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-13 16:41:51 -07:00
Derek McGowan
37b5e3e81b Merge pull request #1833 from aaronlehmann/document-toomanyrequests
Document TOOMANYREQUESTS error code
2016-07-13 10:41:01 -07:00
Richard Scothern
07f32ac183 Merge pull request #1830 from dmcgowan/allow-v2-auth-on-v1-search
Add support for using v2 ping challenges for v1
2016-07-12 17:28:54 -07:00
Derek McGowan
145abeea7b Add support for using v2 ping challenges for v1
Allows using v2 for v1 endpoints.
The primary use case being for search which does not have a v2 specification.
Added a user scope for allowing v2 search

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-12 17:15:56 -07:00
Aaron Lehmann
b0099004e2 Document TOOMANYREQUESTS error code
Add entries with this error code in registry/api/v2/descriptors.go.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-07-12 12:18:54 -06:00
Richard Scothern
2052f29be6 Merge pull request #1827 from docker/ec2-creds
Allow EC2 IAM roles to be used when authorizing region endpoints
2016-07-11 13:05:41 -07:00
Noah Treuhaft
042bc06175 Properly validate multi-URL foreign layers
The existing code effectively ignored errors from all but the last of a
foreign layer's URLs.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-11 12:13:42 -07:00
Richard Scothern
3da5f9088d Allow EC2 IAM roles to be used when authorizing region endpoints
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-11 10:54:57 -07:00
Stefan Majewsky
1f03d4e77d [Swift] add simple heuristic to detect incomplete DLOs during read ops
This is similar to waitForSegmentsToShowUp which is called during
Close/Commit. Intuitively, you wouldn't expect missing segments to be a
problem during read operations, since the previous Close/Commit
confirmed that all segments are there.

But due to the distributed nature of Swift, the read request could be
hitting a different storage node of the Swift cluster, where the
segments are still missing.

Load tests on my team's staging Swift cluster have shown this to occur
about once every 100-200 layer uploads when the Swift proxies are under
high load. The retry logic, borrowed from waitForSegmentsToShowUp, fixes
this temporary inconsistency.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-07-08 13:47:41 +02:00
Michal Minář
3f1434525b Export storage.CreateOptions in top-level package
Let the options for `BlobStore.Create()` be modified in middleware
wrappers.

Signed-off-by: Michal Minar <miminar@redhat.com>
2016-07-06 16:40:46 +02:00
Cezar Sa Espinola
e015cbadd6
registry: avoid formatting errors with %#v
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-07-01 11:30:58 -03:00
Derek McGowan
c24b072e7d Merge pull request #1808 from BrianBland/clientAllTagsFollowsLinks
Changes the client Tags All() method to follow links
2016-06-29 12:02:48 -07:00
Brian Bland
a1f9f71e67 Changes the client Tags All() method to follow links
This returns all tags even when the registry forces pagination.

Signed-off-by: Brian Bland <brian.t.bland@gmail.com>
2016-06-28 15:49:14 -07:00
Josh Chorlton
2d0a5ecc0e fixed s3 Delete bug due to read-after-delete inconsistency
Signed-off-by: Josh Chorlton <josh.chorlton@docker.com>
2016-06-28 14:22:15 -07:00
bin liu
913e12c8ff fix typos
Signed-off-by: bin liu <liubin0329@gmail.com>
2016-06-22 12:40:21 +08:00
Richard Scothern
edd7cb5249 Merge pull request #1739 from cezarsa/master
[Swift] Expose EndpointType parameter in driver
2016-06-15 10:33:48 -07:00
Richard Scothern
ccfa25cf00 If resumable digest support is disabled, detct this when closing the blobwriter
and allow the close to continue.  Also update the name of the function.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 17:35:06 -07:00
Richard Scothern
1fc752c718 Merge pull request #1706 from aibaars/registry-size-close
Blobwriter: call BlobWriter.Size after BlobWriter.Close
2016-06-13 16:29:35 -07:00
Cezar Sa Espinola
7f72092940
Expose EndpointType parameter in swift storage driver
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:28:45 -03:00
Richard Scothern
fb106e167a Merge pull request #1465 from dmcgowan/token-server-oauth
Integration token server supporting oauth
2016-06-13 15:01:06 -07:00
Richard Scothern
4e08e7d1d6 Merge pull request #1775 from dmcgowan/get-content-digest
Add option to get content digest from manifest get
2016-06-13 14:59:05 -07:00
Richard Scothern
1bf52359f1 Merge pull request #1782 from tianon/accept-lists
Update "Accept" header parsing for list values
2016-06-13 14:33:46 -07:00
Aaron Lehmann
9198d642ba Merge pull request #1779 from dmcgowan/letsencrypt-support
Let's Encrypt support
2016-06-13 10:48:55 -10:00
Derek McGowan
be2ed961aa Add support for Let's Encrypt
Add configuration and certificate manager to use letsencrypt

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-13 11:30:42 -07:00
Tianon Gravi
8907f7d189 Update "Accept" header parsing for list values
In Go's header parsing, the same header multiple times results in multiple entries in the `r.Header[...]` slice, but Go does no further parsing beyond that (and in https://golang.org/cl/4528086 it was determined that until/unless the stdlib itself needs it, Go will not do so).

The consequence here for parsing of `Accept:` headers is that we support the way Go outputs headers, but not all language HTTP libraries have a facility to output multiple headers instead of a single list header.

This change ensures that the following (valid) header blocks all parse to the same result for the purposes of what is being tested here:

```
Accept: a/b
Accept: b/c
Accept: d/e
```

```
Accept: a/b; q=0.5, b/c
Accept: d/e
```

```
Accept: a/b; q=0.1, b/c; q=0.2, d/e; q=0.8
```

Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
2016-06-10 16:52:27 -07:00
Derek McGowan
125f4ff7d7 Add option to get content digest from manifest get
The client may need the content digest to delete a manifest using the digest used by the registry.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-08 17:02:29 -07:00
Richard Scothern
75882f079c Merge pull request #1774 from RichardScothern/catalog-clarifcation
Clarify API documentation around catalog fetch behavior
2016-06-08 12:59:09 -07:00
Richard Scothern
352924cd85 Clarify API documentation around catalog fetch behavior
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-08 10:39:17 -07:00
Richard Scothern
79102ecf85 Merge pull request #1772 from runcom/go1.6const
registry: use const for status code 429
2016-06-08 10:22:57 -07:00
Antonio Murdaca
4901f1b278 registry: use const for status code 429
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-06-08 10:19:15 +02:00
allencloud
db90724ab0 fix typos
Signed-off-by: allencloud <allen.sun@daocloud.io>
2016-06-02 23:03:27 +08:00
Richard Scothern
df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url
Swift auth version param
2016-06-01 11:23:23 -07:00
Richard Scothern
641f102967 Merge pull request #1687 from RichardScothern/signature-store
Remove signature store from registry.
2016-05-31 09:09:39 -07:00
Richard Scothern
0c15ab6952 Remove signature store from registry. Return a generated signature for manifest
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 13:19:26 -07:00
Richard Scothern
a5f72a6032 Merge pull request #1688 from tt/pass-through-known-errors
Pass through known errors
2016-05-27 11:35:34 -07:00
Richard Scothern
596ca8b86a Merge pull request #1744 from tonyhb/add-config-to-middleware
Pass in `app` as context to apply{N}Middleware
2016-05-25 14:08:09 -07:00
Tony Holdstock-Brown
166c4a957f Pass in app as context to apply{N}Middleware
This lets us access registry config within middleware for additional
configuration of whatever it is that you're overriding.

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-24 11:08:57 -07:00
Richard Scothern
32e9779faf Merge pull request #1734 from legionus/blob-access-controller
Add support for  blobAccessController middleware
2016-05-24 10:34:36 -07:00
John Starks
f0052b8434 Add support for layers from foreign sources
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6 Add support for blobAccessController middleware
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
2016-05-19 14:02:15 +02:00
Richard Scothern
4f2ee029a2 Add 'us-gov-west-1' to the valid region list.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-09 16:38:16 +01:00
Arthur Baars
eca581cf36 StorageDriver: GCS: allow Cancel on a closed FileWriter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
0490ff450b Blobwriter: call BlobWriter.Size after BlobWriter.Close
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
1d782c38f2 StorageDriver: Test case for #1698
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Richard Scothern
c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem
Add regulator to filesystem
2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown
c9c62380ff Don't wrap thead limits when using a negative int
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 16:03:44 -07:00
Antonio Murdaca
f60f275c29 registry: do not use http.StatusTooManyRequests
go1.5 doesn't export http.StatusTooManyRequests while
go1.6 does. Fix this by hardcoding the status code for now.

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 21:24:45 +02:00
Antonio Murdaca
a264f9ae29 registry: type too many requests error
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 18:41:54 +02:00
Tony Holdstock-Brown
33c448f147 Implement regulator in filesystem driver
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:22 -07:00
Richard Scothern
5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments
[Swift] wait for DLO segments to show up when Close()ing the writer
2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit
Clean uploads
2016-04-29 16:18:29 -07:00
Troels Thomsen
3730470b64 Pass through known errors
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2016-04-29 23:34:24 +02:00
Richard Scothern
ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0 When a blob upload is committed prevent writing out hashstate in the
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03 Add a test with a missing _manifests directory
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a Move garbage collect code into storage package
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown
a5aaae1f06 Ensure GC continues marking if _manifests is nonexistent
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-27 13:33:36 -07:00