TrueCloudLab/distribution
13
0
Fork 3
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions
2097 commits 2 branches 61 tags 129 MiB
Commit graph

365 commits

Author SHA1 Message Date
Edgar Lee
379312c148 Stop ErrFinishedWalk from escaping from Repositories walk 
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
 2016-08-05 17:21:48 -07:00
Michal Minář
7365003236 Provide stat descriptor for Create method during cross-repo mount (#1857) 
* Allow precomputed stats on cross-mounted blobs

Signed-off-by: Michal Minář <miminar@redhat.com>

* Extended cross-repo mount tests

Signed-off-by: Michal Minář <miminar@redhat.com>
 2016-08-01 19:07:11 -07:00
Frank Chen
87917f3052 Add 'objectAcl' Option to the S3 Storage Backend (#1867) 
* Add Object ACL Support to the S3 Storage Backend

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Made changes per @RichardScothern's comments

Signed-off-by: Frank Chen <frankchn@gmail.com>

* Fix Typos

Signed-off-by: Frank Chen <frankchn@gmail.com>
 2016-07-27 12:26:57 -07:00
Richard Scothern
c2a201dabf Merge pull request #1860 from Seb-Solon/master 
Fix #1854
 2016-07-22 10:04:01 -07:00
Richard Scothern
2287f1c42f Fix the build. 
Pass the manifestURL directly into the schema2 manifest handler instead of
accessing through the repository as it has since the reference is now an
interface.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-07-21 17:16:47 -07:00
Richard Scothern
2b72dd3927 Merge pull request #1829 from nwt/foreign-layer-host-whitelist 
Add a foreign layer URL host whitelist
 2016-07-21 16:02:20 -07:00
Aaron Lehmann
f56b143dc8 Merge pull request #1858 from RichardScothern/decouple 
Decouple storage components by redefining dependencies as interfaces
 2016-07-21 11:12:05 -07:00
Sebastien Coavoux
a9c9daf0f2 Fix: Compare path properly when list repository in catalog. #1854 
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
 2016-07-21 11:46:00 -04:00
Sebastien Coavoux
3290e2bef5 Enh: Test - Edit catalog_test to reproduce #1854 
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
 2016-07-21 11:23:19 -04:00
Richard Scothern
2a5fcacdf0 Decouple storage components by redefining dependencies as interfaces instead of concrete types 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-07-20 15:09:11 -07:00
Richard Scothern
857d0f15c0 Merge pull request #1757 from miminar/cross-mount-middleware-repository 
Allow to use repo middleware wrapper during cross-repo mounts
 2016-07-19 17:43:10 -07:00
Noah Treuhaft
61e5803b56 Add control over validation of URLs in pushed manifests 
Until we have some experience hosting foreign layer manifests, the Hub
operators wish to limit foreign layers on Hub. To that end, this change
adds registry configuration options to restrict the URLs that may appear
in pushed manifests.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
 2016-07-19 14:38:42 -07:00
Richard Scothern
f27ceb7ab5 Merge pull request #1710 from majewsky/swift/wait-for-dlo-segments-during-read 
[Swift] add simple heuristic to detect incomplete DLOs during read ops
 2016-07-19 09:07:44 -07:00
Derek McGowan
f8083b7ff3 Merge pull request #1836 from hinshun/catalog-walk-return-error 
Fix storage drivers dropping non EOF errors when listing repositories
 2016-07-15 14:29:50 -07:00
Derek McGowan
42a7e6e171 Merge pull request #1837 from RichardScothern/catalog-test 
Use registry APIs to generate catalog test environment
 2016-07-15 14:23:22 -07:00
Edgar Lee
3bfd03cbe6 Refactor errVal named parameter for catalog repositories to err 
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
 2016-07-14 13:28:08 -07:00
Richard Scothern
b126d6643e Use distribution APIs to generate catalog test environment 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-07-14 09:58:19 -07:00
Edgar Lee
5a0b35ca10 Fix storage drivers dropping non EOF errors when listing repositories 
This fixes errors other than io.EOF from being dropped when a storage driver
lists repositories. For example, filesystem driver may point to a missing
directory and errors, which then gets subsequently dropped.

Signed-off-by: Edgar Lee <edgar.lee@docker.com>
 2016-07-13 16:41:51 -07:00
Richard Scothern
2052f29be6 Merge pull request #1827 from docker/ec2-creds 
Allow EC2 IAM roles to be used when authorizing region endpoints
 2016-07-11 13:05:41 -07:00
Noah Treuhaft
042bc06175 Properly validate multi-URL foreign layers 
The existing code effectively ignored errors from all but the last of a
foreign layer's URLs.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
 2016-07-11 12:13:42 -07:00
Richard Scothern
3da5f9088d Allow EC2 IAM roles to be used when authorizing region endpoints 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-07-11 10:54:57 -07:00
Stefan Majewsky
1f03d4e77d [Swift] add simple heuristic to detect incomplete DLOs during read ops 
This is similar to waitForSegmentsToShowUp which is called during
Close/Commit. Intuitively, you wouldn't expect missing segments to be a
problem during read operations, since the previous Close/Commit
confirmed that all segments are there.

But due to the distributed nature of Swift, the read request could be
hitting a different storage node of the Swift cluster, where the
segments are still missing.

Load tests on my team's staging Swift cluster have shown this to occur
about once every 100-200 layer uploads when the Swift proxies are under
high load. The retry logic, borrowed from waitForSegmentsToShowUp, fixes
this temporary inconsistency.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
 2016-07-08 13:47:41 +02:00
Michal Minář
3f1434525b Export storage.CreateOptions in top-level package 
Let the options for `BlobStore.Create()` be modified in middleware
wrappers.

Signed-off-by: Michal Minar <miminar@redhat.com>
 2016-07-06 16:40:46 +02:00
Cezar Sa Espinola
e015cbadd6
registry: avoid formatting errors with %#v 
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
 2016-07-01 11:30:58 -03:00
Josh Chorlton
2d0a5ecc0e fixed s3 Delete bug due to read-after-delete inconsistency 
Signed-off-by: Josh Chorlton <josh.chorlton@docker.com>
 2016-06-28 14:22:15 -07:00
bin liu
913e12c8ff fix typos 
Signed-off-by: bin liu <liubin0329@gmail.com>
 2016-06-22 12:40:21 +08:00
Richard Scothern
edd7cb5249 Merge pull request #1739 from cezarsa/master 
[Swift] Expose EndpointType parameter in driver
 2016-06-15 10:33:48 -07:00
Richard Scothern
ccfa25cf00 If resumable digest support is disabled, detct this when closing the blobwriter 
and allow the close to continue.  Also update the name of the function.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-06-13 17:35:06 -07:00
Richard Scothern
1fc752c718 Merge pull request #1706 from aibaars/registry-size-close 
Blobwriter: call BlobWriter.Size after BlobWriter.Close
 2016-06-13 16:29:35 -07:00
Cezar Sa Espinola
7f72092940
Expose EndpointType parameter in swift storage driver 
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
 2016-06-13 19:28:45 -03:00
allencloud
db90724ab0 fix typos 
Signed-off-by: allencloud <allen.sun@daocloud.io>
 2016-06-02 23:03:27 +08:00
Richard Scothern
df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url 
Swift auth version param
 2016-06-01 11:23:23 -07:00
Richard Scothern
0c15ab6952 Remove signature store from registry. Return a generated signature for manifest 
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-05-27 13:19:26 -07:00
Richard Scothern
32e9779faf Merge pull request #1734 from legionus/blob-access-controller 
Add support for  blobAccessController middleware
 2016-05-24 10:34:36 -07:00
John Starks
f0052b8434 Add support for layers from foreign sources 
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
 2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6 Add support for blobAccessController middleware 
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
 2016-05-19 14:02:15 +02:00
Richard Scothern
4f2ee029a2 Add 'us-gov-west-1' to the valid region list. 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-05-09 16:38:16 +01:00
Arthur Baars
eca581cf36 StorageDriver: GCS: allow Cancel on a closed FileWriter 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Arthur Baars
0490ff450b Blobwriter: call BlobWriter.Size after BlobWriter.Close 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Arthur Baars
1d782c38f2 StorageDriver: Test case for #1698 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Richard Scothern
c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem 
Add regulator to filesystem
 2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown
c9c62380ff Don't wrap thead limits when using a negative int 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-03 16:03:44 -07:00
Tony Holdstock-Brown
33c448f147 Implement regulator in filesystem driver 
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-03 09:33:22 -07:00
Richard Scothern
5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments 
[Swift] wait for DLO segments to show up when Close()ing the writer
 2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit 
Clean uploads
 2016-04-29 16:18:29 -07:00
Richard Scothern
ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error 
Move GC into storage package and add tests
 2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect 
add middleware storage driver for redirect
 2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0 When a blob upload is committed prevent writing out hashstate in the 
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03 Add a test with a missing _manifests directory 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a Move garbage collect code into storage package 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 13:34:25 -07:00