Commit graph

68 commits

Author SHA1 Message Date
yaoyao.xyy
a4a227e351 oss native large file copy consume too much time which will eventually lead to client timeout because of no data transmit throughout native copy. change maxCopySize to 128MB, ensure only sm all medium size file use oss native copy to avoid connection reset by peer. And fix Move function with CopyLargeFileInParallel to optimize oss upload copy
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-11-08 12:14:13 +08:00
yaoyao.xyy
f394e82d2b oss data copy optimize after upload finished
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-10-26 09:37:58 +08:00
Derek McGowan
acae5dcfff Vendor update for aws sdk
Updated to latest version of go aws sdk.
Use vendored sub pakages within aws sdk.
Adds missing vendor packages for letsencrypt

Fixes #1832

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-07-12 12:47:37 -07:00
Cezar Sa Espinola
5ad9d19ff6
Update vendored version of ncw/swift library
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:28:05 -03:00
Derek McGowan
1c99939221 Vendor letsencrypt packages
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-09 17:56:43 -07:00
Olivier Gambier
d1444b56e9 Fix GCS
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-22 10:50:52 -07:00
Olivier Gambier
53e3c1d7b2 Fix godeps
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-22 10:45:50 -07:00
Brian Bland
5967d33342 Removes ceph rados driver in favor of Swift API gateway support
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Brian Bland
ff03381d49 Adds new storagedriver.FileWriter interface
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Brian Bland
1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Brian Bland
9259bd7133 [driver/s3] Adds support for ap-northeast-2 region
Updates docker/goamz dependency

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 19:42:02 -08:00
Brian Bland
ee4e6e3691 Updates docker/goamz dependency to support AWS profile env
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-02 18:10:03 -08:00
Brian Bland
2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Richard Scothern
cc0b6b86ea Merge pull request #1276 from denverdino/oss-test
Support large layer for OSS driver
2016-01-08 13:26:09 -08:00
Liu Hua
5b550f919f update panicwrap package to fix building error on aarch64 platform
Linux on aarch64 lacks of dup2, uses dup3 instead

Signed-off-by: Liu Hua <sdu.liu@huawei.com>
2015-12-29 19:11:59 +00:00
weiyuan.yl
8c97a853d8 Update OSS client for more details in the error message
Change-Id: Ie2aa70dfeb69930ee5eda17d7e744e3f95a8b926
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-24 21:02:35 +08:00
Aaron Lehmann
4c850e7165 Remove tarsum support for digest package
tarsum is not actually used by the registry. Remove support for it.

Convert numerous uses in unit tests to SHA256.

Update docs to remove mentions of tarsums (which were often inaccurate).

Remove tarsum dependency.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-12-15 17:22:18 -08:00
Li Yi
54da47d636 Support large layer for OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-11 10:56:50 +08:00
Sylvain Baubeau
7759153f2f Update Swift bindings
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-11-03 09:57:39 +01:00
Arthur Baars
95db61e97f Google Cloud Storage (gcs) dependencies (1)
Add new dependencies to Godeps/Godeps.json

Signed-off-by: Arthur Baars <arthur@semmle.com>
2015-10-28 11:26:39 +00:00
Troels Thomsen
6a3496d9a5 Remove duplicate dependency definition
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-30 10:48:28 +02:00
Troels Thomsen
5fc0c2f0b7 Use revision of saved package
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-30 10:48:21 +02:00
Troels Thomsen
cca9f74eec Upgrade Amazon library to revision aa6e716d710a0c7941cb2075cfbb9661f16d21f1
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2015-09-30 00:29:17 +02:00
Aaron Lehmann
540e3f1433 Update godeps for addition of cobra dependency
codegangsta/cli is being removed since the "dist" tool which used it is
no longer in the repo.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-09-09 14:39:31 -07:00
Hua Wang
ddb689ef8b Add TrustId parameter to swift driver
github/ncw/swift has added support for trust, so let's add it.

Signed-off-by: Hua Wang <wanghua.humble@gmail.com>
2015-09-02 15:05:29 +08:00
Aaron Lehmann
3a414deddb Merge pull request #514 from denverdino/master
Storage Driver for Aliyun OSS
2015-08-04 12:01:02 -07:00
Richard Scothern
db12c889e1 Merge pull request #763 from aaronlehmann/close-notifier
Use CloseNotifier to supress spurious HTTP 400 errors on early disconnect
2015-08-03 13:57:32 -07:00
Stephen J Day
7fc9e2112a Update s3 bindings
The S3 library has made a few fixes to the retry logic. Updating the bindings
accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-08-03 11:50:48 -07:00
Aaron Lehmann
1788ae4870 Update github.com/gorilla/handlers dependency
Support for exposing the CloseNotifier interface was just recently added
to its logging handler wrappers. This is needed for #597.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-07-31 17:30:26 -07:00
tgic
2a8535626f fix oss: got 403 in TestContinueStreamAppendLarge
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
9627c6c5a5 godep update oss sdk
Signed-off-by: tgic <farmer1992@gmail.com>
2015-07-31 13:22:55 +08:00
tgic
ee9971a5af update godep 2015-07-31 13:22:54 +08:00
tgic
3d30cb38f6 add endpoint support 2015-07-31 13:22:54 +08:00
Li Yi
38b23c8dff Update Godeps for Aliyun OSS
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-07-31 13:22:54 +08:00
Chris Dillon
63dcebab71 fix json parse error on Godeps
Signed-off-by: Chris Dillon <squarism@gmail.com>
2015-07-24 14:14:16 -07:00
davidli
3ff9f9b9cc Update swift lib for adding the support of Openstack Identity v3 API
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
4497d6c973 Use mitchellh/mapstructure library to parse Swift parameters
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
nevermosby
7372c13f29 Update the import path for swift driver test
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
ea7c53df08 Add Openstack Swift storage driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:08 +02:00
Stephen Day
3ea67df373 Merge pull request #623 from ahmetalpbalkan/azure-vendor
storage/driver/azure: Update vendored Azure SDK
2015-06-16 17:41:38 -07:00
Ahmet Alp Balkan
daa22cacba storage/driver/azure: Update vendored Azure SDK
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.

I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...

This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.

Fixes #620, #175.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-06-16 17:13:44 -07:00
Dave Trombley
d4f2260e04 Added dependency to golang.org/x/crypto/bcrypt
Signed-off-by: Dave Trombley <dave.trombley@gmail.com>
2015-06-10 19:06:38 -07:00
Stephen J Day
d24bfb776b Update tarsum package
This PR addresses some issues contributors have been having with updating
dependencies. The tarsum package had required an external google package which
is no longer required. This was being picked up as an import rewrite and
getting removed. It is no longer present, so we can just update the package and
delete the tar package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-04 18:39:25 -07:00
Stephen Day
318af0b1ce Merge pull request #443 from gierschv/driver-rados
Storage Driver: Ceph Object Storage (RADOS)
2015-05-29 18:10:49 -07:00
Stephen Day
a3006e8e45 Merge pull request #556 from stevvooe/remove-uuid-dependency
Replace uuid dependency with internal library
2015-05-26 10:39:35 -07:00
Stephen J Day
36e34a55ad Replace uuid dependency with internal library
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:59:41 -07:00
Stephen J Day
44da954565 Refactor specification of supported digests
To make the definition of supported digests more clear, we have refactored the
digest package to have a special Algorithm type. This represents the digest's
prefix and we associated various supported hash implementations through
function calls.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 18:08:23 -07:00
Stephen J Day
eee6cad2cf Remove digest package's dependency on external sha implementation
The change relies on a refactor of the upstream resumable sha256/sha512 package
that opts to register implementations with the standard library. This allows
the resumable support to be detected where it matters, avoiding unnecessary and
complex code. It also ensures that consumers of the digest package don't need
to depend on the forked sha implementations.

We also get an optimization with this change. If the size of data written to a
digester is the same as the file size, we check to see if the digest has been
verified. This works if the blob is written and committed in a single request.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-20 23:44:08 -07:00
Vincent Giersch
394eea0231 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Stephen J Day
9bd1186654 Update goamz package dependency
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 12:32:59 -07:00