Commit graph

31 commits

Author SHA1 Message Date
Andrey Kostov
d296a3d2c0 First pass at cleanup for PR merge 2014-12-23 00:24:45 +02:00
Andrey Kostov
a32e6125e0 Zero fill bug fix 2014-12-21 20:34:35 +02:00
Andrey Kostov
ea24d0f735 Add eventual consistency test 2014-12-21 17:46:52 +02:00
Andrey Kostov
11ed0515d0 Implements zero fill behaviour for large offset in WriteStream
This requires a very intricate WriteStream test, which will be in the next commit.
2014-12-19 19:20:07 +02:00
Andrey Kostov
8ca960a0b5 S3 driver refactor
This requires some discussion of how we will handle errors due to network problems
and after further changes in that direction some more stress testing. There is also an
upcomming commit implementing zero fill on WriteStream when offset is greater than
the current size of the file.
2014-12-19 19:16:51 +02:00
Brian Bland
a7b8e4fda0 Adds benchmarks for Put/Get, WriteStream/ReadStream, List, and Delete
Also fixes an open fd leak in the filesystem driver that these
benchmarks revealed.
2014-12-16 13:00:29 -08:00
Brian Bland
5fbf08bab5 Adds tests for storage driver handling of valid and invalid paths 2014-12-11 14:54:04 -08:00
Brian Bland
8a1889efeb Enforces a path format for storage drivers (#819)
Requires all paths in the inmemory and filesystem drivers to begin with
a slash, and then contain only valid path components (2+ alphanumeric
characters with optional period, hyphen, and underscore separators)
delimited by slashes.

Also updates the storage driver test suites to construct paths of this
format, and causes the suite to abort if files are not cleaned up after
the test run.
2014-12-11 14:19:41 -08:00
Brian Bland
f57e544919 Increases stress test factor of TestConcurrentFileStreams
Also makes this test respect the Short flag, reducing the number of
threads by a factor of 4 and space usage by a factor of 16

Note: this test is probably unreasonable to run on the inmemory driver
without the Short flag
2014-12-10 16:20:14 -08:00
Brian Bland
9297693675 Improves storagedriver concurrency testing
Creates trees instead of flat files for TestConcurrentFileStreams

Adds TestConcurrentStreamReads, which writes a large file (smaller in
Short mode), and then ensures that several concurrent readers properly
read their portions of the file with random offsets
2014-12-10 10:57:47 -08:00
Brian Bland
cb25cc65bf Fixes storagedriver Stat test
Checks Stat on the directory before creating the file to make sure that
it does not exist
Properly cleans up after the test.
2014-12-10 10:55:33 -08:00
Brian Bland
14a072cd5f Improves storagedriver tests for Move command semantics
Ensures that Move will properly overwrite the file at the destination
location.
Also checks that Move of a nonexistent source file will NOT delete the
file at the destination.
2014-12-10 10:53:51 -08:00
Brian Bland
bbf288a808 Adds another error test case for reading nonexistent files 2014-12-10 10:51:07 -08:00
Brian Bland
d375e264e1 Makes circle.yml run tests with the "-test.short" flag
This is due to longer tests which require more storage/memory than is
granted to us by the circle test runner vms.
2014-12-09 18:04:05 -08:00
Brian Bland
cacf33ab62 Uses random paths with slashes for storagedriver tests, adds edge cases
This replaces only using flat filenames, to better test nested file
behaviors.
Fixed inmemory/mfs.go and filesystem/driver.go after finding bugs with
the new tests and test behavior.
2014-12-08 19:26:29 -08:00
Stephen J Day
14e7adb3a0 Add documentation for (*DriverSuite).TestStatCall 2014-12-05 19:20:42 -08:00
Stephen J Day
8cb0e3398c Disable s3, azure and ipc packages and testing
The packages causing build errors are being disabled for now to let us split up
the work in the different driver implementations without blocking integration
into the main branch. The s3 and azure implementations need some effort to add
Stat support. The ipc package needs that work plus some care around hanging
send calls.
2014-12-05 14:05:37 -08:00
Stephen J Day
d703a86a64 Add checks for ReadStream offset boundary conditions
Several checks for ReadStream with offset around boundary conditions were
missing. The new checks ensure negative offsets are detected and io.EOF is
returned properly when trying to read past the end of a file. The filesystem
and inmemory driver have been updated accordingly.

An outline of missing checks for List are also part of this commit. Action will
be taken here based on discussion in issue #819.
2014-12-05 11:46:41 -08:00
Stephen J Day
2037b1d6bf Update testsuite with storagedriver interface changes
This change updates the testsuite to migrate to the new driver interface. This
includes the new Stat call, changes to int64 over uint64 and the changes to the
WriteStream signature. Several test cases have been added to vet
implementations against various assumptions.
2014-12-03 16:37:46 -08:00
Stephen J Day
b047c92e1c Use sync.WaitGroup to control concurrent tests 2014-12-02 20:43:31 -08:00
Stephen J Day
66107df1af Use int64 for ReadStream and WriteStream offsets
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
2014-12-02 19:01:00 -08:00
Brian Bland
18eac89506 Adds a test for concurrent storagedriver Write/Read Stream operations
This test is currently failing and Skipped for IPC drivers
2014-11-20 16:15:55 -08:00
Brian Bland
68fd15b688 Miscellaneous storagedriver+ipc fixes
Fixes/tests listing for keys beginning with "/"
No longer extraneously wraps Closers in ioutil.NopClosers
Uses omitempty for all ipc struct type fields
2014-11-20 14:14:55 -08:00
Brian Bland
a3481c5f1c Adds ability to unwrap ipc errors into their original type
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.

Also improves storagedriver tests to enforce proper error types are
returned
2014-11-18 17:41:48 -08:00
Brian Bland
88795e0a14 Lots of various golint fixes
Changes some names to match go conventions
Comments all exported methods
Removes dot imports
2014-11-17 15:46:06 -08:00
Brian Bland
cb1bdacbe3 Renames ResumeWritePosition to CurrentSize in storage driver api 2014-11-07 12:58:48 -08:00
Brian Bland
0e5d41ff9b Updates documentation to follow godoc conventions 2014-10-29 16:44:26 -07:00
Brian Bland
ca0084fad1 Adds StorageDriverFactory, unifying creation of StorageDrivers
Custom storage drivers can register a factory to create the driver by
name, similar to the database/sql package's Register and Open
factory.Create returns an in-process driver if registered or an IPC
driver if one can be found, erroring otherwise
This standardizes parameter passing for creation of storage drivers

Also adds documentation for storagedriver package and children
2014-10-28 18:21:06 -07:00
Brian Bland
ff81f3a719 Adds conditional SkipCheck for storage driver tests 2014-10-27 13:24:07 -07:00
Andrey Kostov
47ca8be42f Slight additions/modifications to the test suite 2014-10-24 16:36:17 -07:00
Brian Bland
3f95694180 Adds storage driver interface, tests, and two basic implementations 2014-10-22 10:50:43 -07:00