Commit graph

41 commits

Author SHA1 Message Date
Stephen J Day
f265a29f24 Move shared checks to base storage driver
This changeset moves common checks from driver implementations into base
package. The Base type can be embedded in a driver implementation, ensuring
that incoming arguments are checked for validity.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 17:05:50 -08:00
Brian Bland
fb71af75c8 Updates goamz dependency from crowdmob->AdRoll
Also includes goamz PR #331 for s3 v4 auth + IAM role support
2015-02-02 11:03:20 -08:00
Andrey Kostov
c05e4682f8 Add a rotating buffer functionality to the s3 driver 2015-01-27 13:24:17 -08:00
Andrey Kostov
8341f0ce5d Add the chunksize parameter
Also change default chunkSize to 10MB (up from 5).
2015-01-23 17:28:09 -08:00
Andrey Kostov
34f86b9ad6 Fixes an IAM role authentication bug
More specifically, the driver panics if initialized with
FromParameters with empty accesskey or secretkey.
2015-01-23 15:50:55 -08:00
Andrey Kostov
671c69e0df Fix regionName parameter so as to not have a panic when it is not a string 2015-01-19 14:23:28 -08:00
Andrey Kostov
bdd5d35622 Add functionality to make a url signed for a HEAD request to S4 driver 2015-01-14 13:37:39 -08:00
Brian Bland
14c8e2b5f0 Clarifies s3 driver "secure" flag documentation
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-01-13 11:38:19 -08:00
Brian Bland
16a258ce69 Enables s3 driver, limits permission check to scope of rootdirectory
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-01-12 13:34:49 -08:00
Stephen Day
df4441d03c Merge pull request #18 from AndreyKostov/ng-add-s3driver-features
Add features to the s3 driver
2015-01-12 13:10:33 -08:00
Brian Bland
f22ad79d36 Factors out resolveBlobPath, renames expires -> expiry 2015-01-08 17:56:45 -08:00
Brian Bland
abb901e4ab Adds options map for storagedriver URLFor() method 2015-01-08 17:10:32 -08:00
Andrey Kostov
bc9509d85f Consolidate the s3 driver parameters with a struct 2015-01-08 00:46:48 +02:00
Andrey Kostov
a2b294f444 Add the URLFor optional method to the storagedriver api
We now also have a storagedriver error variable for identifying
api calls that are not implemented by drivers (the URLFor method
is not implemented by either the filesystem or inmemory drivers)
2015-01-07 18:44:01 +02:00
Andrey Kostov
7c9112fc3c Make the encrpyt parameter default to false 2015-01-07 11:51:29 +02:00
Andrey Kostov
a0ef0d6aad Add the v4auth parameter
v4auth will default to true and if the frankfurt (eu-central-1) region
is selected with v4auth set to false explicitly, the driver will error
out upon initialization.
2015-01-07 11:50:32 +02:00
Andrey Kostov
031c388543 Add the secure flag for the s3 driver
The secure flag will be true by default and will change the
s3 endpoint of the region to http instead of https when selected as false.
The main benefits of running with secure being false is that it apparently
has a roughly 33% performance boost (even on pure data transfer, not only
connection setup which is what I would have expected).
2015-01-07 11:24:09 +02:00
Andrey Kostov
3178d8cfad Add a README file and make encrypt and rootdirectory optional parameters.
Note that the README currently contains details about the secure parameter which is part of a
separate pull request. I feel confident adding it here since I am certain we will eventually add
the secure parameter. Also note that encrypt now defaults to true and rootdirectory defaults to
the empty string.
2015-01-06 00:52:38 +02:00
Olivier Gambier
67ca9d10cf Move from docker-registry to distribution 2014-12-23 17:13:02 -08:00
Andrey Kostov
da6e2f96ec S3 driver input parameter fixes 2014-12-23 10:54:01 +02:00
Andrey Kostov
d296a3d2c0 First pass at cleanup for PR merge 2014-12-23 00:24:45 +02:00
Andrey Kostov
a32e6125e0 Zero fill bug fix 2014-12-21 20:34:35 +02:00
Andrey Kostov
fee9e9ed6e Make driver work with read only creds and remove multi cleanup on boot 2014-12-21 08:48:42 +02:00
Andrey Kostov
1ffb5db12b Replace ReadFull with Read and some cleanup 2014-12-20 10:32:48 +02:00
Andrey Kostov
576495ec3e Loop bug fix 2014-12-20 00:18:27 +02:00
Andrey Kostov
a952c77b4a Minor style change 2014-12-19 23:32:39 +02:00
Andrey Kostov
11ed0515d0 Implements zero fill behaviour for large offset in WriteStream
This requires a very intricate WriteStream test, which will be in the next commit.
2014-12-19 19:20:07 +02:00
Andrey Kostov
8ca960a0b5 S3 driver refactor
This requires some discussion of how we will handle errors due to network problems
and after further changes in that direction some more stress testing. There is also an
upcomming commit implementing zero fill on WriteStream when offset is greater than
the current size of the file.
2014-12-19 19:16:51 +02:00
Brian Bland
9e38ca2231 Runs s3 storagedriver tests against all regions (excluding gov) 2014-12-08 16:16:36 -08:00
Stephen J Day
8cb0e3398c Disable s3, azure and ipc packages and testing
The packages causing build errors are being disabled for now to let us split up
the work in the different driver implementations without blocking integration
into the main branch. The s3 and azure implementations need some effort to add
Stat support. The ipc package needs that work plus some care around hanging
send calls.
2014-12-05 14:05:37 -08:00
Stephen J Day
66107df1af Use int64 for ReadStream and WriteStream offsets
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
2014-12-02 19:01:00 -08:00
Brian Bland
a3481c5f1c Adds ability to unwrap ipc errors into their original type
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.

Also improves storagedriver tests to enforce proper error types are
returned
2014-11-18 17:41:48 -08:00
Brian Bland
88795e0a14 Lots of various golint fixes
Changes some names to match go conventions
Comments all exported methods
Removes dot imports
2014-11-17 15:46:06 -08:00
Brian Bland
c8ea224f9c Miscellaneous go vet fixes
Fixes some format strings and uses keyed fields for struct construction
2014-11-12 17:19:19 -08:00
Brian Bland
cb1bdacbe3 Renames ResumeWritePosition to CurrentSize in storage driver api 2014-11-07 12:58:48 -08:00
Brian Bland
43716a2850 Uses IsTruncated and NextMarker for S3 list internal pagination 2014-11-03 16:27:19 -08:00
Brian Bland
0e5d41ff9b Updates documentation to follow godoc conventions 2014-10-29 16:44:26 -07:00
Brian Bland
ca0084fad1 Adds StorageDriverFactory, unifying creation of StorageDrivers
Custom storage drivers can register a factory to create the driver by
name, similar to the database/sql package's Register and Open
factory.Create returns an in-process driver if registered or an IPC
driver if one can be found, erroring otherwise
This standardizes parameter passing for creation of storage drivers

Also adds documentation for storagedriver package and children
2014-10-28 18:21:06 -07:00
Brian Bland
ff81f3a719 Adds conditional SkipCheck for storage driver tests 2014-10-27 13:24:07 -07:00
Andrey Kostov
e3a5955cd2 Unify permissions settings 2014-10-26 10:00:53 -07:00
Andrey Kostov
1342873367 Add s3 driver for the new Storage Layer API 2014-10-24 16:37:25 -07:00