Commit graph

2262 commits

Author SHA1 Message Date
Noah Treuhaft
63468ef4a8 Use multipart upload API in S3 Move method
This change to the S3 Move method uses S3's multipart upload API to copy
objects whose size exceeds a threshold.  Parts are copied concurrently.
The level of concurrency, part size, and threshold are all configurable
with reasonable defaults.

Using the multipart upload API has two benefits.

* The S3 Move method can now handle objects over 5 GB, fixing #886.

* Moving most objects, and espectially large ones, is faster.  For
  example, moving a 1 GB object averaged 30 seconds but now averages 10.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-08-16 10:53:24 -07:00
Richard Scothern
c810308d1b Merge pull request #1902 from majewsky/swift/support-tenant-domain-parameter
[swift] support different user-domain and tenant-domain
2016-08-15 10:45:12 -07:00
Stefan Majewsky
a7c6bfd59f [swift] support different user-domain and tenant-domain
This is already supported by ncw/swift, so we just need to pass the
parameters from the storage driver.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-08-15 11:21:42 +02:00
Adam Duke
ac009c86f1 Allow registry clients to connect via http2
Http2 will be enabled by default and can be disabled with a configuration option.

Signed-off-by: Adam Duke <adam.v.duke@gmail.com>
2016-08-13 22:07:42 -04:00
Anusha Ragunathan
c202010f8a Add plugin mediatype to distribution manifest.
This is required for github.com/docker/docker/pull/25582

Signed-off-by: Anusha Ragunathan <anusha@docker.com>
2016-08-11 19:11:11 -07:00
Richard Scothern
c9fd26e9ef Merge pull request #1895 from hinshun/improve-enumerate
Improve catalog enumerate runtime by an order of magnitude
2016-08-11 10:53:36 -07:00
Richard Scothern
baca174469 Merge pull request #1892 from stevvooe/use-math-rand-reader
testutil, storage: use math/rand.Read where possible
2016-08-11 10:04:36 -07:00
Stephen J Day
040db51795
testutil, storage: use math/rand.Read where possible
Use the much faster math/rand.Read function where cryptographic
guarantees are not required. The unit test suite should speed up a
little bit but we've already optimized around this, so it may not
matter.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-10 14:26:12 -07:00
Edgar Lee
2f81b3b058 Improve catalog enumerate runtime by an order of magnitude
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-09 17:42:26 -07:00
Richard Scothern
9000745401 Merge pull request #1888 from stevvooe/more-efficient-path-comparison
registry/storage: more efficient path compare in catalog
2016-08-09 13:45:07 -07:00
Stephen Day
c270829c6e Merge pull request #1887 from hinshun/fix-catalog-enumerate
Stop ErrFinishedWalk from escaping from Repositories walk
2016-08-09 13:44:20 -07:00
Derek McGowan
2f356d69c8 Merge pull request #1894 from RichardScothern/circle
Remove godep check from build and increase test timout
2016-08-09 12:06:30 -07:00
Richard Scothern
a8ef842a9b Remove godep check from build and increase test timout
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-08-09 11:50:44 -07:00
Stephen J Day
308faf00f1
catalog: add benchmarks for overridden path comparison
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 17:09:49 -07:00
Stephen J Day
bba5a0d05c
registry/storage: more efficient path compare in catalog
Previous component-wise path comparison is recursive and generates a
large amount of garbage. This more efficient version simply replaces the
path comparison with the zero-value to sort before everything. We do
this by replacing the byte-wise comparison that swaps a single character
inline for the separator comparison, such that separators sort first.

The resulting implementation provides component-wise path comparison
with no cost incurred for allocation or stack frame.

Direction of the comparison is also reversed to match Go style.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 15:55:26 -07:00
Edgar Lee
379312c148 Stop ErrFinishedWalk from escaping from Repositories walk
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-05 17:21:48 -07:00
Richard Scothern
bfa0a9c097 Merge pull request #1877 from spacexnice/master
function copyContent() cause socket leak.
2016-08-02 09:15:23 -07:00
yaoyao.xyy
64c8dd3252 when deploy registry as a pull through cache ,function copeContent() may cause a socket leak when docker user canceled its pull operation.
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-08-02 11:10:10 +08:00
Michal Minář
7365003236 Provide stat descriptor for Create method during cross-repo mount (#1857)
* Allow precomputed stats on cross-mounted blobs

Signed-off-by: Michal Minář <miminar@redhat.com>

* Extended cross-repo mount tests

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-08-01 19:07:11 -07:00
Frank Chen
87917f3052 Add 'objectAcl' Option to the S3 Storage Backend (#1867)
* Add Object ACL Support to the S3 Storage Backend

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Made changes per @RichardScothern's comments

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Fix Typos

Signed-off-by: Frank Chen <frankchn@gmail.com>
2016-07-27 12:26:57 -07:00
Richard Scothern
4abae2ae56 Merge pull request #1861 from adamvduke/adamvduke/reorder-dockerfile-for-layer-caching-optimization
reorder Dockerfile steps for better layer caching
2016-07-22 11:07:24 -07:00
Adam Duke
41c87559c6 reorder Dockerfile steps for better layer caching
Running `apk add` before copying source into the image takes better
adavantage of layer caching when developing and regularly building the
image. This avoids source code changes invalidating the `apk add` layer
and causing that step to run on every image build.

Signed-off-by: Adam Duke <adam.v.duke@gmail.com>
2016-07-22 13:51:31 -04:00
Richard Scothern
c2a201dabf Merge pull request #1860 from Seb-Solon/master
Fix #1854
2016-07-22 10:04:01 -07:00
Richard Scothern
08d0d3756c Merge pull request #1862 from RichardScothern/fix-build
Fix the build.
2016-07-22 09:38:27 -07:00
Richard Scothern
2287f1c42f Fix the build.
Pass the manifestURL directly into the schema2 manifest handler instead of
accessing through the repository as it has since the reference is now an
interface.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-21 17:16:47 -07:00
Richard Scothern
2b72dd3927 Merge pull request #1829 from nwt/foreign-layer-host-whitelist
Add a foreign layer URL host whitelist
2016-07-21 16:02:20 -07:00
Aaron Lehmann
f56b143dc8 Merge pull request #1858 from RichardScothern/decouple
Decouple storage components by redefining dependencies as interfaces
2016-07-21 11:12:05 -07:00
Sebastien Coavoux
a9c9daf0f2 Fix: Compare path properly when list repository in catalog. #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:46:00 -04:00
Sebastien Coavoux
3290e2bef5 Enh: Test - Edit catalog_test to reproduce #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:23:19 -04:00
Richard Scothern
2a5fcacdf0 Decouple storage components by redefining dependencies as interfaces instead of concrete types
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-20 15:09:11 -07:00
Richard Scothern
857d0f15c0 Merge pull request #1757 from miminar/cross-mount-middleware-repository
Allow to use repo middleware wrapper during cross-repo mounts
2016-07-19 17:43:10 -07:00
Richard Scothern
dd355e95af Merge pull request #1686 from marcusmartins/nbf-leeway
Add leeway to JWT nbf checking
2016-07-19 17:41:57 -07:00
Noah Treuhaft
61e5803b56 Add control over validation of URLs in pushed manifests
Until we have some experience hosting foreign layer manifests, the Hub
operators wish to limit foreign layers on Hub. To that end, this change
adds registry configuration options to restrict the URLs that may appear
in pushed manifests.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-19 14:38:42 -07:00
Richard Scothern
7b5958798e Merge pull request #1852 from jfindley/load-middleware-redirect
Load the middleware/redirect driver
2016-07-19 09:10:25 -07:00
Richard Scothern
96a1aec65d Merge pull request #1850 from runcom/typo
manifest: versioned: fix typo
2016-07-19 09:08:42 -07:00
Richard Scothern
f27ceb7ab5 Merge pull request #1710 from majewsky/swift/wait-for-dlo-segments-during-read
[Swift] add simple heuristic to detect incomplete DLOs during read ops
2016-07-19 09:07:44 -07:00
James Findley
6b98c68d12 Load the middleware/redirect driver
Signed-off-by: James Findley <jfindley@fastmail.com>
2016-07-19 15:37:10 +01:00
Antonio Murdaca
2ea0a41872
manifest: versioned: fix typo
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-07-19 13:54:41 +02:00
Marcus Martins
db1bf93098
Add leeway to JWT nbf and exp checking
Adds a constant leeway (60 seconds) to the nbf and exp claim check to
account for clock skew between the registry servers and the
authentication server that generated the JWT.

The leeway of 60 seconds is a bit arbitrary but based on the RFC
recommendation and hub.docker.com logs/metrics where we don't see
drifts of more than a second on our servers running ntpd.

I didn't attempt to make the leeway configurable as it would add extra
complexity to the PR and I am not sure how Distribution prefer to
handle runtime flags like that.

Also, I am simplifying the exp and nbf check for readability as the
previous `NOT (A AND B)` with cmp operators was not very friendly.

Ref:
https://tools.ietf.org/html/rfc7519#section-4.1.5

Signed-off-by: Marcus Martins <marcus@docker.com>
2016-07-18 17:47:30 -07:00
Richard Scothern
aa3de8fd4f Merge pull request #1847 from RichardScothern/go-1.7
Fix vet errors with go 1.7
2016-07-18 14:01:47 -07:00
Richard Scothern
fcea6145f5 Fix vet errors with go 1.7
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-18 13:41:19 -07:00
Derek McGowan
f8083b7ff3 Merge pull request #1836 from hinshun/catalog-walk-return-error
Fix storage drivers dropping non EOF errors when listing repositories
2016-07-15 14:29:50 -07:00
Derek McGowan
b1ab3bfde5 Merge pull request #1838 from dmcgowan/search-v2-auth-test
Search v2 auth test
2016-07-15 14:27:56 -07:00
Derek McGowan
42a7e6e171 Merge pull request #1837 from RichardScothern/catalog-test
Use registry APIs to generate catalog test environment
2016-07-15 14:23:22 -07:00
Edgar Lee
a82f661ef0 Use typecast over reflect for error type checking
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 15:03:18 -07:00
Edgar Lee
3bfd03cbe6 Refactor errVal named parameter for catalog repositories to err
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:28:08 -07:00
Edgar Lee
aeb9a29499 Handle new errors returned from catalog repository listing
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:24:16 -07:00
Richard Scothern
b126d6643e Use distribution APIs to generate catalog test environment
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-14 09:58:19 -07:00
Derek McGowan
fc07e0380e Add v1 search with v2 token auth test
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-13 17:03:14 -07:00
Edgar Lee
5a0b35ca10 Fix storage drivers dropping non EOF errors when listing repositories
This fixes errors other than io.EOF from being dropped when a storage driver
lists repositories. For example, filesystem driver may point to a missing
directory and errors, which then gets subsequently dropped.

Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-13 16:41:51 -07:00