Commit graph

83 commits

Author SHA1 Message Date
Andrey Kostov
8341f0ce5d Add the chunksize parameter
Also change default chunkSize to 10MB (up from 5).
2015-01-23 17:28:09 -08:00
Andrey Kostov
34f86b9ad6 Fixes an IAM role authentication bug
More specifically, the driver panics if initialized with
FromParameters with empty accesskey or secretkey.
2015-01-23 15:50:55 -08:00
Andrey Kostov
671c69e0df Fix regionName parameter so as to not have a panic when it is not a string 2015-01-19 14:23:28 -08:00
Andrey Kostov
bdd5d35622 Add functionality to make a url signed for a HEAD request to S4 driver 2015-01-14 13:37:39 -08:00
Brian Bland
14c8e2b5f0 Clarifies s3 driver "secure" flag documentation
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-01-13 11:38:19 -08:00
Brian Bland
16a258ce69 Enables s3 driver, limits permission check to scope of rootdirectory
Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-01-12 13:34:49 -08:00
Stephen Day
df4441d03c Merge pull request #18 from AndreyKostov/ng-add-s3driver-features
Add features to the s3 driver
2015-01-12 13:10:33 -08:00
Stephen J Day
75c5916dde Release lock during WriteStream for inmemory driver
While reading from the input in WriteStream, the inmemory driver can deadlock
if the reader is from the same instance. To fix this, the write lock is
released before reading into a local buffer. The lock is re-acquired to
finish the actual write.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:49:06 -08:00
Brian Bland
f22ad79d36 Factors out resolveBlobPath, renames expires -> expiry 2015-01-08 17:56:45 -08:00
Brian Bland
abb901e4ab Adds options map for storagedriver URLFor() method 2015-01-08 17:10:32 -08:00
Andrey Kostov
bc9509d85f Consolidate the s3 driver parameters with a struct 2015-01-08 00:46:48 +02:00
Andrey Kostov
a2b294f444 Add the URLFor optional method to the storagedriver api
We now also have a storagedriver error variable for identifying
api calls that are not implemented by drivers (the URLFor method
is not implemented by either the filesystem or inmemory drivers)
2015-01-07 18:44:01 +02:00
Andrey Kostov
7c9112fc3c Make the encrpyt parameter default to false 2015-01-07 11:51:29 +02:00
Andrey Kostov
a0ef0d6aad Add the v4auth parameter
v4auth will default to true and if the frankfurt (eu-central-1) region
is selected with v4auth set to false explicitly, the driver will error
out upon initialization.
2015-01-07 11:50:32 +02:00
Andrey Kostov
031c388543 Add the secure flag for the s3 driver
The secure flag will be true by default and will change the
s3 endpoint of the region to http instead of https when selected as false.
The main benefits of running with secure being false is that it apparently
has a roughly 33% performance boost (even on pure data transfer, not only
connection setup which is what I would have expected).
2015-01-07 11:24:09 +02:00
Brian Bland
f6aadc2028 Relaxes storagedriver path constraints (#47)
No longer requires that file paths match the repository naming scheme,
but instead allows path components as short as a single character, as to
accommodate for single-character tag names.
2015-01-06 17:18:42 -08:00
Andrey Kostov
3178d8cfad Add a README file and make encrypt and rootdirectory optional parameters.
Note that the README currently contains details about the secure parameter which is part of a
separate pull request. I feel confident adding it here since I am certain we will eventually add
the secure parameter. Also note that encrypt now defaults to true and rootdirectory defaults to
the empty string.
2015-01-06 00:52:38 +02:00
Brian Bland
5880dcfdba Adds basic READMEs to inmemory and filesystem storage drivers 2014-12-29 13:59:40 -08:00
Olivier Gambier
67ca9d10cf Move from docker-registry to distribution 2014-12-23 17:13:02 -08:00
Andrey Kostov
da6e2f96ec S3 driver input parameter fixes 2014-12-23 10:54:01 +02:00
Andrey Kostov
d296a3d2c0 First pass at cleanup for PR merge 2014-12-23 00:24:45 +02:00
Andrey Kostov
a32e6125e0 Zero fill bug fix 2014-12-21 20:34:35 +02:00
Andrey Kostov
ea24d0f735 Add eventual consistency test 2014-12-21 17:46:52 +02:00
Andrey Kostov
fee9e9ed6e Make driver work with read only creds and remove multi cleanup on boot 2014-12-21 08:48:42 +02:00
Andrey Kostov
1ffb5db12b Replace ReadFull with Read and some cleanup 2014-12-20 10:32:48 +02:00
Andrey Kostov
576495ec3e Loop bug fix 2014-12-20 00:18:27 +02:00
Andrey Kostov
a952c77b4a Minor style change 2014-12-19 23:32:39 +02:00
Andrey Kostov
11ed0515d0 Implements zero fill behaviour for large offset in WriteStream
This requires a very intricate WriteStream test, which will be in the next commit.
2014-12-19 19:20:07 +02:00
Andrey Kostov
8ca960a0b5 S3 driver refactor
This requires some discussion of how we will handle errors due to network problems
and after further changes in that direction some more stress testing. There is also an
upcomming commit implementing zero fill on WriteStream when offset is greater than
the current size of the file.
2014-12-19 19:16:51 +02:00
Stephen Day
3b8847f489 Merge pull request #863 from BrianBland/ng-storagedriver
Fixes filesystem storage driver List semantics for nonexistent directory
2014-12-18 10:47:31 -08:00
Brian Bland
030b0ff310 Allows storagedriver parameter values to be of type interface{}
This enables use of nil, booleans, numeric types, and even complex structures for parameter values, assuming they can be parsed from yaml.
2014-12-17 19:06:55 -08:00
Brian Bland
bc8ab9b392 Fixes filesystem storage driver List semantics for nonexistent directory
Now returns a PathNotFoundError
2014-12-17 17:00:18 -08:00
Brian Bland
a7b8e4fda0 Adds benchmarks for Put/Get, WriteStream/ReadStream, List, and Delete
Also fixes an open fd leak in the filesystem driver that these
benchmarks revealed.
2014-12-16 13:00:29 -08:00
Brian Bland
5fbf08bab5 Adds tests for storage driver handling of valid and invalid paths 2014-12-11 14:54:04 -08:00
Brian Bland
8a1889efeb Enforces a path format for storage drivers (#819)
Requires all paths in the inmemory and filesystem drivers to begin with
a slash, and then contain only valid path components (2+ alphanumeric
characters with optional period, hyphen, and underscore separators)
delimited by slashes.

Also updates the storage driver test suites to construct paths of this
format, and causes the suite to abort if files are not cleaned up after
the test run.
2014-12-11 14:19:41 -08:00
Brian Bland
f57e544919 Increases stress test factor of TestConcurrentFileStreams
Also makes this test respect the Short flag, reducing the number of
threads by a factor of 4 and space usage by a factor of 16

Note: this test is probably unreasonable to run on the inmemory driver
without the Short flag
2014-12-10 16:20:14 -08:00
Brian Bland
9297693675 Improves storagedriver concurrency testing
Creates trees instead of flat files for TestConcurrentFileStreams

Adds TestConcurrentStreamReads, which writes a large file (smaller in
Short mode), and then ensures that several concurrent readers properly
read their portions of the file with random offsets
2014-12-10 10:57:47 -08:00
Brian Bland
cb25cc65bf Fixes storagedriver Stat test
Checks Stat on the directory before creating the file to make sure that
it does not exist
Properly cleans up after the test.
2014-12-10 10:55:33 -08:00
Brian Bland
14a072cd5f Improves storagedriver tests for Move command semantics
Ensures that Move will properly overwrite the file at the destination
location.
Also checks that Move of a nonexistent source file will NOT delete the
file at the destination.
2014-12-10 10:53:51 -08:00
Brian Bland
bbf288a808 Adds another error test case for reading nonexistent files 2014-12-10 10:51:07 -08:00
Stephen Day
043c81bea2 Merge pull request #832 from BrianBland/ng-storagedriver-tests
Updates storagedriver tests to better test directory trees
2014-12-09 21:01:40 -08:00
Brian Bland
d375e264e1 Makes circle.yml run tests with the "-test.short" flag
This is due to longer tests which require more storage/memory than is
granted to us by the circle test runner vms.
2014-12-09 18:04:05 -08:00
Brian Bland
94052ea213 Fixes normalization of inmemory file paths
A normalized path always begins with "/" and never has a trailing slash
unless it is the root directory.
2014-12-09 17:20:10 -08:00
Stephen J Day
45c29be442 Address bug in inmemory filesystem WriteAt method 2014-12-08 21:08:07 -08:00
Brian Bland
cacf33ab62 Uses random paths with slashes for storagedriver tests, adds edge cases
This replaces only using flat filenames, to better test nested file
behaviors.
Fixed inmemory/mfs.go and filesystem/driver.go after finding bugs with
the new tests and test behavior.
2014-12-08 19:26:29 -08:00
Brian Bland
9e38ca2231 Runs s3 storagedriver tests against all regions (excluding gov) 2014-12-08 16:16:36 -08:00
Stephen J Day
14e7adb3a0 Add documentation for (*DriverSuite).TestStatCall 2014-12-05 19:20:42 -08:00
Stephen J Day
8cb0e3398c Disable s3, azure and ipc packages and testing
The packages causing build errors are being disabled for now to let us split up
the work in the different driver implementations without blocking integration
into the main branch. The s3 and azure implementations need some effort to add
Stat support. The ipc package needs that work plus some care around hanging
send calls.
2014-12-05 14:05:37 -08:00
Stephen J Day
d703a86a64 Add checks for ReadStream offset boundary conditions
Several checks for ReadStream with offset around boundary conditions were
missing. The new checks ensure negative offsets are detected and io.EOF is
returned properly when trying to read past the end of a file. The filesystem
and inmemory driver have been updated accordingly.

An outline of missing checks for List are also part of this commit. Action will
be taken here based on discussion in issue #819.
2014-12-05 11:46:41 -08:00
Stephen J Day
2ebc373d91 Refactor inmemory driver for Stat and WriteStream methods
This change started out as simply updating the existing inmemory driver to
implement the new Stat call. After struggling with the map based
implementation, it has been refactored to be a tree-based implementation.

This process has exposed a few missing error cases in the StorageDriver API
that should be addressed in the coming weeks.
2014-12-04 20:25:14 -08:00