TrueCloudLab/distribution
13
0
Fork 3
Code Issues 3 Pull requests Projects Releases 2 Packages Wiki Activity Actions
2033 commits 2 branches 61 tags 129 MiB
Commit graph

194 commits

Author SHA1 Message Date
Anton Tiurin
a048a4c8d5 Fix description of StorageDriver.WriteStream 
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
 2016-02-17 13:57:20 +03:00
Brian Bland
c770f28f04 [driver/s3aws] Fix TestStorageClass 
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-02-16 17:50:55 -08:00
Brian Bland
b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary 
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
 2016-02-16 11:19:15 -08:00
liuchang0812
0758653b42 fix gofmt 
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
 2016-02-16 11:42:09 +08:00
liuchang0812
dbba9cf299 closes #1461, enhance log message of oss driver 
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
 2016-02-16 11:17:09 +08:00
Brian Bland
1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz 
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-02-12 15:59:26 -08:00
Aaron Lehmann
aa80478b64 Typo fixes in comments 
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2016-02-10 16:26:29 -08:00
Brian Bland
52b4a328dd Improves flexibility of configuration handling for S3 driver 
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-02-09 17:27:16 -08:00
Richard Scothern
2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass 
Adds "storageclass" configuration parameter for S3 driver.
 2016-02-04 16:44:13 -08:00
yuzou
44c0b9ae87 read the actual number of bytes according to the initial size. 
Signed-off-by: yuzou <zouyu7@huawei.com>
 2016-02-04 16:14:35 +08:00
Brian Bland
695bac7b0f Adds test for S3 storage class configuration option 
Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-02-01 16:01:55 -08:00
Brian Bland
4688d1adc2 Adds "storageclass" configuration parameter for S3 driver. 
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-02-01 16:01:19 -08:00
Aaron Lehmann
2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString 
Adds custom registry User-Agent header to s3 HTTP requests
 2016-02-01 15:40:36 -08:00
Brian Bland
2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests 
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-02-01 13:43:46 -08:00
Arthur Baars
2a4345ca4b StorageDriver: GCS: retry all api calls 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-01-20 13:24:09 +00:00
Arthur Baars
6ee339464c StorageDriver: Test suite: improve cleanup 
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-01-20 13:24:09 +00:00
Richard Scothern
fc7ee6d789 Merge pull request #1355 from hopkings2008/master 
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
 2016-01-18 12:38:26 -08:00
Arthur Baars
19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-01-15 11:47:47 +00:00
yuzou
cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads 
Signed-off-by: yuzou <zouyu7@huawei.com>
 2016-01-15 17:22:43 +08:00
Richard Scothern
81dbebc7d3 Change the parameters to the GCS drivers to allow CircleCI testing. 
Remove the requirement of file system access to run GCS unit tests.  Deconstruct
the input parameters to take the private key and email which can be specified on
the build system via environment variables.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-01-13 18:00:59 -08:00
Richard Scothern
cc0b6b86ea Merge pull request #1276 from denverdino/oss-test 
Support large layer for OSS driver
 2016-01-08 13:26:09 -08:00
Richard Scothern
9f83bfc071 Merge pull request #1321 from aibaars/gcs-fix-retry 
GCS storage driver: fix retry function
 2016-01-06 12:00:12 -08:00
Arthur Baars
bb1e9e0a82 GCS Storagedriver: fix test failure caused by #1187 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-01-06 18:36:28 +00:00
Arthur Baars
d3d40a8807 GCS driver: fix retry function 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-01-06 17:57:22 +00:00
weiyuan.yl
75abeeca11 Replace 404 to http.StatusNotFound 
Change-Id: Ia100975cb93c0a6d94ea5542b1c9ce386bc87649
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
 2015-12-29 12:09:04 +08:00
Brian Bland
f19deba809 Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files) 
Leaves any further permissions restrictions to the process umask

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2015-12-28 15:22:28 -08:00
weiyuan.yl
3e4b554f8d In HEAD request for missing resource, only 404 NOT FOUND is returned 
Change-Id: I73caf67b59519e6f4f82f7d78f5d4fd4ad9affcd
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
 2015-12-28 11:28:42 +08:00
Li Yi
54da47d636 Support large layer for OSS driver 
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-12-11 10:56:50 +08:00
Richard Scothern
796d6e7915 Merge pull request #1187 from stevvooe/check-storage-drivers-list-path-not-found 
[WIP] registry/storage/driver: checking that non-existent path returns PathNotFoundError
 2015-12-08 16:32:02 -08:00
Stephen J Day
66cd2bf950 storage/driver/s3: adjust s3 driver to return unmunged path 
This fixes both the s3 driver and the oss driver to return the unmunged path
when returning errors.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
 2015-12-08 15:48:55 -08:00
Stephen Day
e4208ad5a7 Merge pull request #1246 from stevvooe/decrease-driver-testsuite-allocations 
storage/driver: decrease memory allocation done during testsuite
 2015-12-08 14:52:59 -08:00
Li Yi
b6756a3d89 Fix the issue for listing root directory 
Change-Id: I1c6181fa4e5666bd2e6ec69cb608c4778ae0fe48
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-12-08 12:04:04 -08:00
Li Yi
7161fa0559 Fix for stevvooe:check-storage-drivers-list-path-not-found in OSS driver 
Change-Id: I5e96fe761d3833c962084fd2d597f47e8a72e7c2
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-12-08 12:04:04 -08:00
davidli
79d4d7f546 driver/swift: treat empty object list as a PathNotFoundError 
Swift returns an empty object list when trying to read a non-existing object path, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: David li <wenquan.li@hpe.com>
 2015-12-08 12:04:04 -08:00
Vincent Giersch
f25ccea279 driver/rados: treat OMAP EIO as a PathNotFoundError 
RADOS returns a -EIO when trying to read a non-existing OMAP, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
 2015-12-08 12:04:04 -08:00
Stephen J Day
a889f46223 storage/driver/s3: correct response on list of missing directory 
Signed-off-by: Stephen J Day <stephen.day@docker.com>
 2015-12-08 12:04:03 -08:00
Stephen J Day
b45078eb44 storage/driver/base: use correct error format style 
Signed-off-by: Stephen J Day <stephen.day@docker.com>
 2015-12-08 12:04:03 -08:00
Stephen J Day
a4eae0917e driver/filesystem: address filesystem driver on behavior of List 
Signed-off-by: Stephen J Day <stephen.day@docker.com>
 2015-12-08 12:04:03 -08:00
Stephen J Day
0322c3bf1f registry/storage/driver: checking that non-existent path returns PathNotFoundError 
Issue #1186 describes a condition where a null tags response is returned when
using the s3 driver. The issue seems to be related to a missing
PathNotFoundError in s3. This change adds a test for that to get an idea of the
lack of compliance across storage drivers. If the failures are manageable,
we'll add this test condition and fix the s3 driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
 2015-12-08 12:04:03 -08:00
Stephen Day
807a80ff06 Merge pull request #1238 from noxiouz/fix_comment 
Fix comment for PathRegexp
 2015-12-08 10:41:16 -08:00
Stephen J Day
9dc3c6df0e storage/driver: decrease memory allocation done during testsuite 
Signed-off-by: Stephen J Day <stephen.day@docker.com>
 2015-12-07 19:05:13 -08:00
Anton Tiurin
d4435b79d9 Fix comment for PathRegexp 
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
 2015-12-04 22:14:21 +03:00
Sylvain Baubeau
ef4ab7a885 Use bulk delete to remove segments in Swift driver 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-12-04 12:16:49 +01:00
Richard Scothern
a33095af80 Merge pull request #1141 from lebauce/swift-no-missing-segment 
Ensure read after write for segments
 2015-11-10 10:39:43 -08:00
Aaron Lehmann
6664a1384e Fix empty delete requests with Swift fs driver 
The Delete method lists objects under the given path and tries to delete
all of them with a bulk delete request. If the path has no objects
underneath it, the body of this request will be empty, which causes
HTTP-level issues. Specifically, Go's HTTP client senses the empty
request buffer and doesn't include a Content-Length, which causes the
Swift server to fail the request.

This commit fixes the problem by avoiding sending empty bulk delete
requests. This is the correct thing to do anyway, since there's no
reason to request deletion of zero objects.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2015-11-09 15:33:05 -08:00
Richard Scothern
bc6e4cdceb Add a generic error type to capture non-typed errors 
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2015-11-03 11:19:44 -08:00
amitshukla
9637cb40cd Fix for issue 664: https://github.com/docker/distribution/issues/664 
Errors thrown by storage drivers don't have the name of the driver, causing user
confusion about whether the error is coming from Docker or from a storage driver.
This change adds the storage driver name to each error message.

This required changing ErrUnsupportedDriver to a type, leading to code changes
whenever ErrUnsupportedDriver is used.  The tests check whether the driver name
appears in the error message.

Signed-off-by: Amit Shukla <amit.shukla@docker.com>
 2015-11-03 11:19:17 -08:00
Sylvain Baubeau
3ff8af326b Ensure read after write for segments 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-11-03 17:09:52 +01:00
Sylvain Baubeau
7c3281861f Add support for temporary URL for Swift driver 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-11-03 09:59:50 +01:00
Arthur Baars
59784ecdd0 Storage driver for: Google Cloud Storage (gcs) 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2015-10-28 20:21:56 +00:00
Richard Scothern
61a4a1362c Merge pull request #950 from jackpgriffin/master 
Skip creating swift container if already exists
 2015-10-01 10:15:54 -07:00
Nuutti Kotivuori
26d8344872 Remove initial access check from S3 driver 
In the S3 storage driver there is currently an initial access permission check by listing the bucket. If this check fails, registry will panic and exit.

However, this check is broken in two ways. First of all it strips the final slash from the root directory path, meaning that any access permissions which limit access to a single directory will fail, because S3 treats the path as strict prefix match. Secondly it fails to strip any leading slash that might be present, unlike the other access places, which means that the path used is different as a leading slash is allowed and significant in a filename in S3.

Since there is also a periodic health check which correctly checks access permissions and shows the error more cleanly, the best solution seems to be to just remove this initial access check.

Signed-off-by: Nuutti Kotivuori <nuutti.kotivuori@poplatek.fi>
 2015-09-19 12:47:12 +03:00
Jack Griffin
2749bcdc28 Skip creating swift container if already exists 
Signed-off-by: Jack Griffin <jackpg14@gmail.com>
 2015-09-04 18:53:17 +00:00
Hua Wang
ddb689ef8b Add TrustId parameter to swift driver 
github/ncw/swift has added support for trust, so let's add it.

Signed-off-by: Hua Wang <wanghua.humble@gmail.com>
 2015-09-02 15:05:29 +08:00
Vincent Giersch
6c39af6708 fix(rados): Create OMAP for root directory 
When using the RADOS driver, the hierarchy of the files is stored
in OMAPs, but the root OMAP was not created and a call to List("/")
was returning an error instead of returned the first level files
stored. This patches creates an OMAP for "/" and excludes the listed
directory from the list of files returned.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
 2015-08-10 23:46:33 +02:00
tgic
72c794a810 remove unused code and fix todo format 
Signed-off-by: tgic <farmer1992@gmail.com>
 2015-07-31 13:22:56 +08:00
tgic
e53d99a837 fix goimports 
Signed-off-by: tgic <farmer1992@gmail.com>
 2015-07-31 13:22:56 +08:00
tgic
0058b08eeb add include_oss build tag 
Signed-off-by: tgic <farmer1992@gmail.com>
 2015-07-31 13:22:56 +08:00
Li Yi
708ad28114 Update the comment for the consistency model 
Change-Id: Iee49afeda1c11d6af8c0f26c96d8ccc328c22757
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-07-31 13:22:55 +08:00
Li Yi
539c7f5311 Update the comments for consistence model 
Change-Id: I161522ee51f247fb17e42844b3699bd9031e34e8
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-07-31 13:22:55 +08:00
tgic
8e5c901a26 fix testcase TestReadStreamWithOffset incompatible with oss 
Signed-off-by: tgic <farmer1992@gmail.com>
 2015-07-31 13:22:55 +08:00
tgic
2a8535626f fix oss: got 403 in TestContinueStreamAppendLarge 
Signed-off-by: tgic <farmer1992@gmail.com>
 2015-07-31 13:22:55 +08:00
Li Yi
813543d6e3 Update the OSS test case for latest code change 
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-07-31 13:22:55 +08:00
tgic
3a240de22e check access key and secret before run 2015-07-31 13:22:55 +08:00
tgic
3d30cb38f6 add endpoint support 2015-07-31 13:22:54 +08:00
Li Yi
c3b42db014 Add the secure access with HTTPS 
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-07-31 13:22:54 +08:00
Li Yi
b8a276f2db Fix the warning of golint 
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-07-31 13:22:54 +08:00
Li Yi
9e4975d8ff Support OSS driver 
Signed-off-by: Li Yi <denverdino@gmail.com>
 2015-07-31 13:22:54 +08:00
Stephen Day
34e5b1849c Merge pull request #493 from nevermosby/storage-support-openstack-swift 
Storage support openstack swift
 2015-07-22 23:53:36 -07:00
davidli
d1b4bae078 Remove IPC support from test file 
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
 2015-07-21 23:55:10 +02:00
Sylvain Baubeau
cca15a76f1 Catch either missing containers or objects 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:10 +02:00
Sylvain Baubeau
b4cf6c053b Do not use Swift server side copy for manifests to handle >5G files 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:10 +02:00
Sylvain Baubeau
326c3a9c49 Inline Swift errors handling 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:10 +02:00
Sylvain Baubeau
15d567671b Retrieve all the objects using pagination 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
49582a6188 Do not create objects for directories 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
ed08d8d4e0 Refactor segment path concatenation code 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
91e17e10cd Use http.StatusRequestedRangeNotSatisfiable instead of error code 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f4ca709d70 Use file instead of filepath as it may cause troubles on Windows 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
78d722e708 Protect against deletion of objects with the same prefix 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
61e3bce794 Show distribution version in User-Agent 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f256797cac Do not use suite style testing for Swift specific tests 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
53e80944f1 Rename environment variables to run Swift testsuite 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
d0dd36125c Check file has been opened before closing it 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1a80ec340b Rename DriverParameters structure to Parameters 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
2846913552 Change folder mime type to application/vnc.swift.directory 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
33028c11a7 Add code documentation 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
f21cbcebc6 Do not read segment if no padding is necessary 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3242378e1b Improve 404 errors handling 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
57cef57e1b Use only one Swift container for both files and manifests 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
a40502ec02 Increase default chunk size to 20M 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
davidli
af99dbd6bf Add support for Openstack Identity v3 API 
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3fb42a1502 Use 'prefix' parameter instead of 'path' when listing files 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1c01ca4b39 Create full folder hierarchy instead of just the top level folder 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
c4d845cb6c Remove one level of indentation in swift path handling code 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
4497d6c973 Use mitchellh/mapstructure library to parse Swift parameters 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
2632fd9dc8 Move Dynamic Large Object handling to dedicated methods 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
d8d50fb0e0 Simplify code that handles non existing manifests 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
Sylvain Baubeau
5e650a820a Handle error during copy of original content 
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
 2015-07-21 23:55:09 +02:00
nevermosby
7372c13f29 Update the import path for swift driver test 
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
 2015-07-21 23:55:09 +02:00
davidli
6dc1596be4 Use gofmt to format the code of swift driver. 
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
 2015-07-21 23:55:09 +02:00