Commit graph

566 commits

Author SHA1 Message Date
Michal Fojtik
9e510d67f5 Add more regions to registry S3 storage driver
Namely adding ca-central-1, ap-south-1 and eu-west-1.

Signed-off-by: Michal Fojtik <mfojtik@redhat.com>
2017-01-11 22:38:24 +01:00
Ahmet Alp Balkan
0a1ce58e2c
azure: revendor + remove hacky solution in is404
Removing the temporary workaround in is404() method by re-vendoring
the azure-sdk-for-go.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2017-01-09 17:22:28 -08:00
Stephen Day
7dba427612 Merge pull request #2111 from stevvooe/use-digest-package
digest: use new go-digest package
2017-01-06 16:25:57 -08:00
Stephen J Day
532ec9f036
digest: migrate to opencontainers/go-digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 15:42:03 -08:00
yixi zhang
8e915d69f4 Use app.driver.Stat for registry health check
`app.driver.List` on `"/"` is very expensive if registry contains significant amount of images. And the result isn't used anyways.
In most (if not all) storage drivers, `Stat` has a cheaper implementation, so use it instead to achieve the same goal.

Signed-off-by: yixi zhang <yixi@memsql.com>
2016-12-21 17:12:43 -08:00
Derek McGowan
729b8c5b91 Merge pull request #2105 from stevvooe/algorithm-own-file
digest: cleanup digester and verifier creation
2016-12-15 16:55:37 -08:00
Stephen J Day
9159833265
digest: remove error return from Digest.Verifier
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 16:43:59 -08:00
Stephen J Day
e37baed88e
digest: cleanup digester and verifier creation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:17:21 -08:00
Stephen J Day
01dfa0fcb9
digest: use digest.Parse over ParseDigest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:07:42 -08:00
Derek McGowan
8867e8fac3
Update schema2 builder to take media type
Modify manifest builder so it can be used to build
manifests with different configuration media types.
Rename config media type const to image config.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2016-12-14 16:17:20 -08:00
Derek McGowan
15dc1296af Merge pull request #2088 from ahmetalpbalkan/pr-upstream-azure-race-fix
azure: fix race condition in PutContent()
2016-12-06 14:07:53 -08:00
Ahmet Alp Balkan
78d0660319
azure: fix race condition in PutContent()
See #2077 for background.

The PR #1438 which was not reviewed by azure folks basically introduced
a race condition around uploads to the same blob by multiple clients
concurrently as it used the "writer" type for PutContent(), introduced in #1438.
This does chunked upload of blobs using "AppendBlob" type, which was not atomic.

Usage of "writer" type and thus AppendBlobs on metadata files is currently not
concurrency-safe and generally, they are not the right type of blob for the job.

This patch fixes PutContent() to use the atomic upload operation that works
for uploads smaller than 64 MB and creates blobs with "BlockBlob" type. To be
backwards compatible, we query the type of the blob first and if it is not
a "BlockBlob" we delete the blob first before doing an atomic PUT. This
creates a small inconsistency/race window "only once". Once the blob is made
"BlockBlob", it is overwritten with a single PUT atomicallly next time.

Therefore, going forward, PutContent() will be producing BlockBlobs and it
will silently migrate the AppendBlobs introduced in #1438 to BlockBlobs with
this patch.

Tested with existing code side by side, both registries with and without this
patch work fine without breaking each other. So this should be good from a
backwards/forward compatiblity perspective, with a cost of doing an extra
HEAD checking the blob type.

Fixes #2077.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-30 12:40:43 -08:00
Kira
4accc8f2be filter listResponse.Contents in driver.List()
Signed-off-by: Kira <me@imkira.com>
2016-11-17 10:38:56 +08:00
Derek McGowan
8b71d39c34
Remove newlines from end of error strings
Golint now checks for new lines at the end of go error strings,
remove these unneeded new lines.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-14 14:03:01 -08:00
Richard Scothern
6e62b39842 Merge pull request #2036 from pyr/fix/sort-v2-headers
v2 signer: correctly sort headers
2016-11-10 15:31:24 -08:00
Richard Scothern
4d65dd513e Merge pull request #2038 from spacexnice/master
fix: oss driver would get connection reset by peer when upload large image layer.
2016-11-10 14:44:32 -08:00
yaoyao.xyy
a4a227e351 oss native large file copy consume too much time which will eventually lead to client timeout because of no data transmit throughout native copy. change maxCopySize to 128MB, ensure only sm all medium size file use oss native copy to avoid connection reset by peer. And fix Move function with CopyLargeFileInParallel to optimize oss upload copy
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-11-08 12:14:13 +08:00
Derek McGowan
a2611c7520 Merge pull request #2027 from ahmetalpbalkan/pr-azure-memleak2
Update vendored azure-sdk-for-go
2016-11-04 10:08:40 -07:00
Ahmet Alp Balkan
2ab25288a2
Update vendored azure-sdk-for-go
Updating to a recent version of Azure Storage SDK to be
able to patch some memory leaks through configurable HTTP client
changes which were made possible by recent patches to it.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-03 13:24:57 -07:00
Pierre-Yves Ritschard
f1cf7de788 fixup! v2 signer: correctly sort headers
Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:07:02 +01:00
Pierre-Yves Ritschard
775cc6d632 v2 signer: correctly sort headers
The current code determines the header order for the
"string-to-sign" payload by sorting on the concatenation
of headers and values, whereas it should only happen on the
key.

During multipart uploads, since `x-amz-copy-source-range` and
`x-amz-copy-source` headers are present, V2 signatures fail to
validate since header order is swapped.

This patch reverts to the expected behavior.

Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:01:34 +01:00
Ahmet Alp Balkan
a994f35657
driver/swift: Fix go vet warning
Driver was passing connections by copying. Storing
`swift.Connection` as pointer to fix the warnings.

Ref: #2030.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-31 11:41:53 -07:00
Ahmet Alp Balkan
6d2a0bafcd
storagedriver/azure: close leaking response body
In GetContent() we read the bytes from a blob but do not close
the underlying response body.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-28 15:13:22 -07:00
Stephen J Day
c9aaff00f8
manifest: references should cover all children
To allow generic manifest walking, we define an interface method of
`References` that returns the referenced items in the manifest. The
current implementation does not return the config target from schema2,
making this useless for most applications.

The garbage collector has been modified to show the utility of this
correctly formed `References` method. We may be able to make more
generic traversal methods with this, as well.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-10-18 11:43:33 -07:00
Richard Scothern
96f63cbe7f Merge pull request #2010 from mbentley/add-new-s3-region
Added new us-east-2 region for S3
2016-10-18 10:25:30 -07:00
Richard Scothern
5bd2cfaddd Merge pull request #2000 from jessfraz/1950-comment-on-non-dryrun-as-well
storage/garbagecollect: add emit comments for when dryRun is false
2016-10-18 10:00:04 -07:00
Matt Bentley
3857f50825
Added new us-east-2 region for S3
Signed-off-by: Matt Bentley <mbentley@mbentley.net>
2016-10-18 12:30:34 -04:00
Richard Scothern
d0cdc4802b Merge pull request #2002 from lnr0626/1996-instance-roles-with-regionendpoint
Allow using ec2 roles when specifying region endpoint
2016-10-17 13:50:02 -07:00
Richard Scothern
a621a86cb4 Fix aliyun OSS Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-17 09:43:15 -07:00
Noah Treuhaft
12e73f01d2 Fix s3-goamz Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-17 09:43:15 -07:00
Richard Scothern
bd91493195 Satisfy the latest go lint rules
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-14 17:03:08 -07:00
Lloyd Ramey
c8ea7840d3 Allow using ec2 roles when specifying region endpoint
Signed-off-by: Lloyd Ramey <lnr0626@gmail.com>
2016-10-13 18:07:37 -04:00
Jess Frazelle
809774c33f
storage/garbagecollect: add emit commits for when dryRun is false
Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-13 08:04:19 -07:00
Richard Scothern
252cc27ab1 Merge pull request #1988 from nwt/fix-s3-delete
Fix S3 Delete method's notion of subpaths
2016-10-11 11:14:29 -07:00
HuKeping
b984f91b32 Fix unit test
Signed-off-by: Hu Keping <hukeping@huawei.com>
2016-10-10 11:37:30 +08:00
Noah Treuhaft
76226c61a9 Fix S3 Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-06 11:21:55 -07:00
Derek McGowan
d35d94dcec
Update to fix lint errors
Context should use type values instead of strings.
Updated direct calls to WithValue, but still other uses of string keys.
Update Acl to ACL in s3 driver.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-10-05 17:47:12 -07:00
Noah Treuhaft
cafeae4ecd Fix connection pool exhaustion in Redis tests
The Redis tests were failing with a "connection pool exhausted" error
from Redigo. Closing the connection used for FLUSHDB fixes the problem.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-09-19 15:30:32 -07:00
Fabio Berchtold
7dcac52f18 Add v2 signature signing to S3 storage driver (#1800)
* Add v2 signature signing to S3 storage driver

Closes #1796
Closes #1606

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* use Logrus for debug logging

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* use 'date' instead of 'x-amz-date' in request header

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>

* only allow v4 signature signing against AWS S3

Signed-off-by: Fabio Berchtold <fabio.berchtold@swisscom.com>
2016-09-01 13:52:40 -07:00
Richard Scothern
49da29ee46 Merge pull request #1925 from dmcgowan/reenable-race-detector
Re-enable race detector in circleci
2016-09-01 13:36:12 -07:00
Derek McGowan
a50ce1ab93 Add locking to repository access in memory cache
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-08-26 18:10:42 -07:00
Matthew Green
dea554fc7c Swift driver now bulk deletes in chunks specified by the server (#1915)
Swift driver now bulk deletes in chunks specified by the server

Signed-off-by: Matthew Green <matthew.green@uk.ibm.com>
2016-08-24 10:09:25 -07:00
Noah Treuhaft
63468ef4a8 Use multipart upload API in S3 Move method
This change to the S3 Move method uses S3's multipart upload API to copy
objects whose size exceeds a threshold.  Parts are copied concurrently.
The level of concurrency, part size, and threshold are all configurable
with reasonable defaults.

Using the multipart upload API has two benefits.

* The S3 Move method can now handle objects over 5 GB, fixing #886.

* Moving most objects, and espectially large ones, is faster.  For
  example, moving a 1 GB object averaged 30 seconds but now averages 10.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-08-16 10:53:24 -07:00
Stefan Majewsky
a7c6bfd59f [swift] support different user-domain and tenant-domain
This is already supported by ncw/swift, so we just need to pass the
parameters from the storage driver.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-08-15 11:21:42 +02:00
Richard Scothern
c9fd26e9ef Merge pull request #1895 from hinshun/improve-enumerate
Improve catalog enumerate runtime by an order of magnitude
2016-08-11 10:53:36 -07:00
Richard Scothern
baca174469 Merge pull request #1892 from stevvooe/use-math-rand-reader
testutil, storage: use math/rand.Read where possible
2016-08-11 10:04:36 -07:00
Stephen J Day
040db51795
testutil, storage: use math/rand.Read where possible
Use the much faster math/rand.Read function where cryptographic
guarantees are not required. The unit test suite should speed up a
little bit but we've already optimized around this, so it may not
matter.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-10 14:26:12 -07:00
Edgar Lee
2f81b3b058 Improve catalog enumerate runtime by an order of magnitude
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-09 17:42:26 -07:00
Richard Scothern
9000745401 Merge pull request #1888 from stevvooe/more-efficient-path-comparison
registry/storage: more efficient path compare in catalog
2016-08-09 13:45:07 -07:00
Stephen J Day
308faf00f1
catalog: add benchmarks for overridden path comparison
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 17:09:49 -07:00
Stephen J Day
bba5a0d05c
registry/storage: more efficient path compare in catalog
Previous component-wise path comparison is recursive and generates a
large amount of garbage. This more efficient version simply replaces the
path comparison with the zero-value to sort before everything. We do
this by replacing the byte-wise comparison that swaps a single character
inline for the separator comparison, such that separators sort first.

The resulting implementation provides component-wise path comparison
with no cost incurred for allocation or stack frame.

Direction of the comparison is also reversed to match Go style.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-08 15:55:26 -07:00
Edgar Lee
379312c148 Stop ErrFinishedWalk from escaping from Repositories walk
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-05 17:21:48 -07:00
Michal Minář
7365003236 Provide stat descriptor for Create method during cross-repo mount (#1857)
* Allow precomputed stats on cross-mounted blobs

Signed-off-by: Michal Minář <miminar@redhat.com>

* Extended cross-repo mount tests

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-08-01 19:07:11 -07:00
Frank Chen
87917f3052 Add 'objectAcl' Option to the S3 Storage Backend (#1867)
* Add Object ACL Support to the S3 Storage Backend

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Made changes per @RichardScothern's comments

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Fix Typos

Signed-off-by: Frank Chen <frankchn@gmail.com>
2016-07-27 12:26:57 -07:00
Richard Scothern
c2a201dabf Merge pull request #1860 from Seb-Solon/master
Fix #1854
2016-07-22 10:04:01 -07:00
Richard Scothern
2287f1c42f Fix the build.
Pass the manifestURL directly into the schema2 manifest handler instead of
accessing through the repository as it has since the reference is now an
interface.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-21 17:16:47 -07:00
Richard Scothern
2b72dd3927 Merge pull request #1829 from nwt/foreign-layer-host-whitelist
Add a foreign layer URL host whitelist
2016-07-21 16:02:20 -07:00
Aaron Lehmann
f56b143dc8 Merge pull request #1858 from RichardScothern/decouple
Decouple storage components by redefining dependencies as interfaces
2016-07-21 11:12:05 -07:00
Sebastien Coavoux
a9c9daf0f2 Fix: Compare path properly when list repository in catalog. #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:46:00 -04:00
Sebastien Coavoux
3290e2bef5 Enh: Test - Edit catalog_test to reproduce #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-07-21 11:23:19 -04:00
Richard Scothern
2a5fcacdf0 Decouple storage components by redefining dependencies as interfaces instead of concrete types
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-20 15:09:11 -07:00
Richard Scothern
857d0f15c0 Merge pull request #1757 from miminar/cross-mount-middleware-repository
Allow to use repo middleware wrapper during cross-repo mounts
2016-07-19 17:43:10 -07:00
Noah Treuhaft
61e5803b56 Add control over validation of URLs in pushed manifests
Until we have some experience hosting foreign layer manifests, the Hub
operators wish to limit foreign layers on Hub. To that end, this change
adds registry configuration options to restrict the URLs that may appear
in pushed manifests.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-19 14:38:42 -07:00
Richard Scothern
f27ceb7ab5 Merge pull request #1710 from majewsky/swift/wait-for-dlo-segments-during-read
[Swift] add simple heuristic to detect incomplete DLOs during read ops
2016-07-19 09:07:44 -07:00
Derek McGowan
f8083b7ff3 Merge pull request #1836 from hinshun/catalog-walk-return-error
Fix storage drivers dropping non EOF errors when listing repositories
2016-07-15 14:29:50 -07:00
Derek McGowan
42a7e6e171 Merge pull request #1837 from RichardScothern/catalog-test
Use registry APIs to generate catalog test environment
2016-07-15 14:23:22 -07:00
Edgar Lee
3bfd03cbe6 Refactor errVal named parameter for catalog repositories to err
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-14 13:28:08 -07:00
Richard Scothern
b126d6643e Use distribution APIs to generate catalog test environment
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-14 09:58:19 -07:00
Edgar Lee
5a0b35ca10 Fix storage drivers dropping non EOF errors when listing repositories
This fixes errors other than io.EOF from being dropped when a storage driver
lists repositories. For example, filesystem driver may point to a missing
directory and errors, which then gets subsequently dropped.

Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-07-13 16:41:51 -07:00
Richard Scothern
2052f29be6 Merge pull request #1827 from docker/ec2-creds
Allow EC2 IAM roles to be used when authorizing region endpoints
2016-07-11 13:05:41 -07:00
Noah Treuhaft
042bc06175 Properly validate multi-URL foreign layers
The existing code effectively ignored errors from all but the last of a
foreign layer's URLs.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-07-11 12:13:42 -07:00
Richard Scothern
3da5f9088d Allow EC2 IAM roles to be used when authorizing region endpoints
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-07-11 10:54:57 -07:00
Stefan Majewsky
1f03d4e77d [Swift] add simple heuristic to detect incomplete DLOs during read ops
This is similar to waitForSegmentsToShowUp which is called during
Close/Commit. Intuitively, you wouldn't expect missing segments to be a
problem during read operations, since the previous Close/Commit
confirmed that all segments are there.

But due to the distributed nature of Swift, the read request could be
hitting a different storage node of the Swift cluster, where the
segments are still missing.

Load tests on my team's staging Swift cluster have shown this to occur
about once every 100-200 layer uploads when the Swift proxies are under
high load. The retry logic, borrowed from waitForSegmentsToShowUp, fixes
this temporary inconsistency.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-07-08 13:47:41 +02:00
Michal Minář
3f1434525b Export storage.CreateOptions in top-level package
Let the options for `BlobStore.Create()` be modified in middleware
wrappers.

Signed-off-by: Michal Minar <miminar@redhat.com>
2016-07-06 16:40:46 +02:00
Cezar Sa Espinola
e015cbadd6
registry: avoid formatting errors with %#v
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-07-01 11:30:58 -03:00
Josh Chorlton
2d0a5ecc0e fixed s3 Delete bug due to read-after-delete inconsistency
Signed-off-by: Josh Chorlton <josh.chorlton@docker.com>
2016-06-28 14:22:15 -07:00
bin liu
913e12c8ff fix typos
Signed-off-by: bin liu <liubin0329@gmail.com>
2016-06-22 12:40:21 +08:00
Richard Scothern
edd7cb5249 Merge pull request #1739 from cezarsa/master
[Swift] Expose EndpointType parameter in driver
2016-06-15 10:33:48 -07:00
Richard Scothern
ccfa25cf00 If resumable digest support is disabled, detct this when closing the blobwriter
and allow the close to continue.  Also update the name of the function.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 17:35:06 -07:00
Richard Scothern
1fc752c718 Merge pull request #1706 from aibaars/registry-size-close
Blobwriter: call BlobWriter.Size after BlobWriter.Close
2016-06-13 16:29:35 -07:00
Cezar Sa Espinola
7f72092940
Expose EndpointType parameter in swift storage driver
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:28:45 -03:00
allencloud
db90724ab0 fix typos
Signed-off-by: allencloud <allen.sun@daocloud.io>
2016-06-02 23:03:27 +08:00
Richard Scothern
df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url
Swift auth version param
2016-06-01 11:23:23 -07:00
Richard Scothern
0c15ab6952 Remove signature store from registry. Return a generated signature for manifest
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 13:19:26 -07:00
Richard Scothern
32e9779faf Merge pull request #1734 from legionus/blob-access-controller
Add support for  blobAccessController middleware
2016-05-24 10:34:36 -07:00
John Starks
f0052b8434 Add support for layers from foreign sources
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6 Add support for blobAccessController middleware
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
2016-05-19 14:02:15 +02:00
Richard Scothern
4f2ee029a2 Add 'us-gov-west-1' to the valid region list.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-09 16:38:16 +01:00
Arthur Baars
eca581cf36 StorageDriver: GCS: allow Cancel on a closed FileWriter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
0490ff450b Blobwriter: call BlobWriter.Size after BlobWriter.Close
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
1d782c38f2 StorageDriver: Test case for #1698
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Richard Scothern
c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem
Add regulator to filesystem
2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown
c9c62380ff Don't wrap thead limits when using a negative int
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 16:03:44 -07:00
Tony Holdstock-Brown
33c448f147 Implement regulator in filesystem driver
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:22 -07:00
Richard Scothern
5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments
[Swift] wait for DLO segments to show up when Close()ing the writer
2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit
Clean uploads
2016-04-29 16:18:29 -07:00
Richard Scothern
ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0 When a blob upload is committed prevent writing out hashstate in the
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03 Add a test with a missing _manifests directory
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a Move garbage collect code into storage package
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Josh Hawn
e4dd3359cc Regulate filesystem driver to max of 100 calls
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-26 14:44:13 -07:00
Andrew Hsu
09a9b0cf90 separate the go/non-go imports and reorder
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-26 14:33:54 -07:00
Richard Scothern
c83afea0c9 Merge pull request #1660 from jhaohai/cn-north-1-fix
Add cn-north-1 to valid check
2016-04-25 16:07:54 -07:00
Richard Scothern
ef32134592 Merge pull request #1666 from sergeyfd/master
Add blobWrtiter.Close() call into blobWriter.Commit()
2016-04-25 16:02:48 -07:00
Andrew Hsu
c4df027d41 modify redirect test to include port
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:46 -07:00
Andrew Hsu
80248c3d3a scheme and host mandatory in baseurl
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:25 -07:00
Andrew Hsu
059bc5f5ef separate the go/non-go imports and reorder
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:03 -07:00
Anis Elleuch
987faca8a6 Sorting completed parts by part number for a better accordance with the S3 spec
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
2016-04-23 22:36:04 +01:00
Serge Dubrouski
21f38a74e6 Add blobWrtiter.Close() call into blobWriter.Commit()
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-22 19:23:17 -06:00
Andrew Hsu
4b217ccbf5 add middleware storage driver for redirect
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-21 16:02:52 -07:00
jhaohai
f76c622d8c add cn-north-1 to valid check
Signed-off-by: jhaohai <jhaohai@foxmail.com>
2016-04-21 11:51:34 +08:00
Stefan Majewsky
9a67520af7 wait for DLO segments to show up when Close()ing the writer
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-19 14:31:49 +02:00
Nikita Tarasov
b4f060599a docs + fix test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 20:05:51 +03:00
Nikita Tarasov
b51607f9f0 fix test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>

Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 19:52:40 +03:00
Nikita
bcb7989fca test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:38 +03:00
Nikita
e6f8d7c28e Update swift.go
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:25 +03:00
Richard Scothern
4c119524f1 Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check
Only check validity of S3 region if not using custom endpoint
2016-04-08 15:38:24 -07:00
Stefan Majewsky
67321cb622 detect outdated container listings during Stat() and getAllSegments()
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-06 15:21:27 +02:00
Arien Holthuizen
dbb6e28da2 Only check validity of S3 region if not using custom endpoint
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
2016-04-06 13:38:09 +02:00
Richard Scothern
04ff3c0359 Merge pull request #1578 from majewsky/do-not-swallow-error
don't swallow errors in Swift driver's GetContent()
2016-03-30 17:46:27 -07:00
Richard Scothern
4324b70c50 Fix signature handling with GC.
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist.  Handle this case.

If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist.  Disable the signature store in the garbage-collect command so
signatures are not fetched.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
80b310ca44 Add a --dry-run flag. If enabled this will print the mark and sweep process
with removing any files.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:10:27 -07:00
Stefan Majewsky
4c9bed2507 don't swallow errors in Swift driver's GetContent()
In 326c3a9c49, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.

I noticed this while investigating an error similar to #1539 aka
docker/docker#21290. It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.

This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-03-30 16:11:29 +02:00
Aaron Schlesinger
f4bdc6287a Remove the example
Instead, direct users to the one in the factory package

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:28 -07:00
Aaron Schlesinger
204ad474e4 Add documentation for how to register new StorageDrivers
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.

This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200.

cc/ @stevvooe

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:19 -07:00
姜继忠
92a9ee34d4 fix manifest revision search, closes #1535
Signed-off-by: 姜继忠 <jizhong.jiangjz@alibaba-inc.com>
2016-03-19 19:11:25 +08:00
Richard Scothern
2c635d1f7e Merge pull request #1532 from RichardScothern/azure-error-types
Update missing blob error checking with latest Azure API
2016-03-15 10:08:36 -07:00
Richard Scothern
772e6f4057 Update missing blob error checking with latest Azure API
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Matt Duch
fcb247dfce registry/storage/driver/s3-aws kms support
Signed-off-by: Matt Duch <matt@learnmetrics.com>
2016-03-11 17:19:01 -06:00
Brian Bland
c03b5fc5ee Merge pull request #1438 from BrianBland/newStorageDriverWriter
Adds new StorageDriver.FileWriter interface
2016-03-11 15:06:07 -08:00
Brian Bland
5967d33342 Removes ceph rados driver in favor of Swift API gateway support
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Keerthan Mala
2be1b4ef4f Added support to specifiy custom endpoint
Signed-off-by: Keerthan Reddy Mala <keerthan.mala@gmail.com>
2016-03-09 16:12:20 -07:00
Brian Bland
7fd1db9312 Updates Swift driver to support new storagedriver.FileWriter interface
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:38:39 -08:00
Li Yi
a9bf7a2aae Support FileWriter interface for OSS storage driver
Change-Id: Ie5533ad85f944800499ca1040fd67bf1378815e0
Signed-off-by: Li Yi <denverdino@gmail.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
307504713f Storagedriver: GCS: add chunksize parameter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
7162cb19c6 Storagedriver: GCS: implement resumable uploads
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
666273d9f6 StorageDriver: Testsuite: call Close before getting Size
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:38 -08:00
Brian Bland
ff03381d49 Adds new storagedriver.FileWriter interface
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Aaron Lehmann
87e34bd307 Fix two misspellings in source code comments
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-08 15:13:24 -08:00
Brian Bland
302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Richard Scothern
62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars
1593c2413f StorageDriver: GCS: remove support for directory Moves
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Andrew T Nguyen
feab4aafbc Implements garbage collection subcommand
- Includes a change in the command to run the registry. The registry
  server itself is now started up as a subcommand.
- Includes changes to the high level interfaces to support enumeration
  of various registry objects.

Signed-off-by: Andrew T Nguyen <andrew.nguyen@docker.com>
2016-02-29 14:15:21 -08:00
Stefan Weil
615c6dfced Fix some typos in comments and strings
All of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-23 22:33:38 +01:00
xiekeyang
d58188c4d7 compare error output in tagstore unit test
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-02-22 17:34:22 +08:00
Anton Tiurin
a048a4c8d5 Fix description of StorageDriver.WriteStream
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2016-02-17 13:57:20 +03:00
Brian Bland
c770f28f04 [driver/s3aws] Fix TestStorageClass
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-16 17:50:55 -08:00
Brian Bland
b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
2016-02-16 11:19:15 -08:00
liuchang0812
0758653b42 fix gofmt
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:42:09 +08:00
liuchang0812
dbba9cf299 closes #1461, enhance log message of oss driver
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:17:09 +08:00
Brian Bland
1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Aaron Lehmann
aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Aaron Lehmann
a3213ff331 Merge pull request #1420 from dmcgowan/configurable-trust-key
Add option to disable signatures
2016-02-10 16:15:59 -08:00
Derek McGowan
b34e571bff Add option to disable signatures
Add option for specifying trust key for signing schema1 manifests.
Since schema1 signature key identifiers are not verified anywhere and deprecated, storing signatures is no longer a requirement.
Furthermore in schema2 there is no signature, requiring the registry to already add signatures to generated schema1 manifests.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-10 15:20:39 -08:00
Brian Bland
52b4a328dd Improves flexibility of configuration handling for S3 driver
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 17:27:16 -08:00
Richard Scothern
4a1d36c22d Correct type for repo reference
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 17:32:55 -08:00
Richard Scothern
579981b979 Merge pull request #1350 from aibaars/storage-filewriter-pointer
Storage: remove bufferedFileWriter (dead code)
2016-02-04 16:47:52 -08:00
Richard Scothern
2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass
Adds "storageclass" configuration parameter for S3 driver.
2016-02-04 16:44:13 -08:00
Richard Scothern
b737de4856 Merge pull request #1408 from aaronlehmann/repository-interface
Rename Name method of Repository to Named
2016-02-04 16:43:35 -08:00
Aaron Lehmann
b0989446eb Rename Name method of Repository to Named
This makes code that gets the name as a string read like
repo.Named().Name() instead of repo.Name().Name().

Requested in
https://github.com/docker/docker/pull/19887#discussion_r51479753

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-04 09:55:36 -08:00
yuzou
44c0b9ae87 read the actual number of bytes according to the initial size.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-02-04 16:14:35 +08:00
Brian Bland
695bac7b0f Adds test for S3 storage class configuration option
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:55 -08:00
Brian Bland
4688d1adc2 Adds "storageclass" configuration parameter for S3 driver.
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:19 -08:00
Aaron Lehmann
2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString
Adds custom registry User-Agent header to s3 HTTP requests
2016-02-01 15:40:36 -08:00
Brian Bland
2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Richard Scothern
584c9b517c Correct test digest lengths and enable all unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:46 -08:00
Arthur Baars
3d67dae525 Storage: remove bufferedFileWriter (dead code)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:08 +00:00
Arthur Baars
26e7268a86 Storage: blobwriter.Write/Seek test case
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:07 +00:00
Aaron Lehmann
4441333912 Use reference package internally
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:47:05 -08:00
Arthur Baars
2a4345ca4b StorageDriver: GCS: retry all api calls
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Arthur Baars
6ee339464c StorageDriver: Test suite: improve cleanup
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Richard Scothern
fc7ee6d789 Merge pull request #1355 from hopkings2008/master
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
2016-01-18 12:38:26 -08:00
Arthur Baars
19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-15 11:47:47 +00:00
yuzou
cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-01-15 17:22:43 +08:00
Richard Scothern
7378e21678 Merge pull request #1332 from RichardScothern/gcs-params
Change the parameters to the GCS drivers to allow CircleCI testing.
2016-01-14 15:50:09 -08:00
Brian Bland
8c5a6c13c0 Splits up blob create options definitions to be package-specific
Redefines privately in both storage and client packages

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-14 10:47:33 -08:00
Richard Scothern
81dbebc7d3 Change the parameters to the GCS drivers to allow CircleCI testing.
Remove the requirement of file system access to run GCS unit tests.  Deconstruct
the input parameters to take the private key and email which can be specified on
the build system via environment variables.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-13 18:00:59 -08:00
Brian Bland
ce88d8a6f4 Adds functional options arguments to the Blobs Create method
Removes the Mount operation and instead implements this behavior as part
of Create a From option is provided, which in turn returns a rich
ErrBlobMounted indicating that a blob upload session was not initiated,
but instead the blob was mounted from another repository

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 16:42:59 -08:00
Richard Scothern
3a1220de01 Merge pull request #1319 from RichardScothern/update-tags
Remove tags referencing deleted manifests.
2016-01-11 11:33:22 -08:00
Richard Scothern
93d9070c8b Merge pull request #1269 from BrianBland/crossRepositoryPush
Adds cross-repository blob mounting behavior
2016-01-08 14:37:00 -08:00
Brian Bland
5df21570a7 Adds cross-repository blob mounting behavior
Extends blob upload POST endpoint to support mount and from query
parameters as described in #634

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:18 -08:00
Richard Scothern
cc0b6b86ea Merge pull request #1276 from denverdino/oss-test
Support large layer for OSS driver
2016-01-08 13:26:09 -08:00
Aaron Lehmann
6d17423a6d Move MediaType into manifest.Versioned
This makes content type sniffing cleaner. The document just needs to be
decoded into a manifest.Versioned structure. It's no longer a two-step
process.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
9c416f0e94 Add support for manifest list ("fat manifest")
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
befd4d6e3c Factor out schema-specific portions of manifestStore
Create signedManifestHandler and schema2ManifestHandler. Use these to
unmarshal and put the respective types of manifests from manifestStore.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Richard Scothern
9f83bfc071 Merge pull request #1321 from aibaars/gcs-fix-retry
GCS storage driver: fix retry function
2016-01-06 12:00:12 -08:00
Arthur Baars
bb1e9e0a82 GCS Storagedriver: fix test failure caused by #1187
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 18:36:28 +00:00
Arthur Baars
d3d40a8807 GCS driver: fix retry function
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 17:57:22 +00:00
Richard Scothern
94347c8611 Remove tags referencing deleted manifests.
When a manifest is deleted by digest, look up the referenced tags in the tag
store and remove all associations.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-05 13:47:52 -08:00
Stephen Day
00b983b837 Merge pull request #1304 from BrianBland/fsDriverRelaxedPermissions
Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
2015-12-29 19:06:45 -08:00
weiyuan.yl
75abeeca11 Replace 404 to http.StatusNotFound
Change-Id: Ia100975cb93c0a6d94ea5542b1c9ce386bc87649
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-29 12:09:04 +08:00
Brian Bland
f19deba809 Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
Leaves any further permissions restrictions to the process umask

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 15:22:28 -08:00
Brian Bland
cf487a7911 Serve blobs when a storage driver supports redirects but are disabled
Fixes issue where an error was returned instead of serving the blob

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 11:08:39 -08:00
weiyuan.yl
3e4b554f8d In HEAD request for missing resource, only 404 NOT FOUND is returned
Change-Id: I73caf67b59519e6f4f82f7d78f5d4fd4ad9affcd
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-28 11:28:42 +08:00
Richard Scothern
cb6f002350 Implementation of the Manifest Service API refactor.
Add a generic Manifest interface to represent manifests in the registry and
remove references to schema specific manifests.

Add a ManifestBuilder to construct Manifest objects. Concrete manifest builders
will exist for each manifest type and implementations will contain manifest
specific data used to build a manifest.

Remove Signatures() from Repository interface.

Signatures are relevant only to schema1 manifests.  Move access to the signature
store inside the schema1 manifestStore.  Add some API tests to verify
signature roundtripping.

schema1
-------

Change the way data is stored in schema1.Manifest to enable Payload() to be used
to return complete Manifest JSON from the HTTP handler without knowledge of the
schema1 protocol.

tags
----

Move tag functionality to a seperate TagService and update ManifestService
to use the new interfaces.  Implement a driver based tagService to be backward
compatible with the current tag service.

Add a proxyTagService to enable the registry to get a digest for remote manifests
from a tag.

manifest store
--------------

Remove revision store and move all signing functionality into the signed manifeststore.

manifest registration
---------------------

Add a mechanism to register manifest media types and to allow different manifest
types to be Unmarshalled correctly.

client
------

Add ManifestServiceOptions to client functions to allow tags to be passed into Put and
Get for building correct registry URLs.  Change functional arguments to be an interface type
to allow passing data without mutating shared state.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2015-12-17 17:09:14 -08:00
Aaron Lehmann
4c850e7165 Remove tarsum support for digest package
tarsum is not actually used by the registry. Remove support for it.

Convert numerous uses in unit tests to SHA256.

Update docs to remove mentions of tarsums (which were often inaccurate).

Remove tarsum dependency.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-15 17:22:18 -08:00
Aaron Lehmann
31047c8113 Simplify digest.FromBytes calling convention
The current implementation of digest.FromBytes returns an error. This
error can never be non-nil, but its presence in the function signature
means each call site needs error handling code for an error that is
always nil.

I verified that none of the hash.Hash implementations in the standard
library can return an error on Write. Nor can any of the hash.Hash
implementations vendored in distribution.

This commit changes digest.FromBytes not to return an error. If Write
returns an error, it will panic, but as discussed above, this should
never happen.

This commit also avoids using a bytes.Reader to feed data into the hash
function in FromBytes. This makes the hypothetical case that would panic
a bit more explicit, and should also be more performant.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-14 14:30:51 -08:00
Li Yi
54da47d636 Support large layer for OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-11 10:56:50 +08:00
Richard Scothern
796d6e7915 Merge pull request #1187 from stevvooe/check-storage-drivers-list-path-not-found
[WIP] registry/storage/driver: checking that non-existent path returns PathNotFoundError
2015-12-08 16:32:02 -08:00