The secure flag will be true by default and will change the
s3 endpoint of the region to http instead of https when selected as false.
The main benefits of running with secure being false is that it apparently
has a roughly 33% performance boost (even on pure data transfer, not only
connection setup which is what I would have expected).
This requires some discussion of how we will handle errors due to network problems
and after further changes in that direction some more stress testing. There is also an
upcomming commit implementing zero fill on WriteStream when offset is greater than
the current size of the file.
Requires all paths in the inmemory and filesystem drivers to begin with
a slash, and then contain only valid path components (2+ alphanumeric
characters with optional period, hyphen, and underscore separators)
delimited by slashes.
Also updates the storage driver test suites to construct paths of this
format, and causes the suite to abort if files are not cleaned up after
the test run.
Also makes this test respect the Short flag, reducing the number of
threads by a factor of 4 and space usage by a factor of 16
Note: this test is probably unreasonable to run on the inmemory driver
without the Short flag
Creates trees instead of flat files for TestConcurrentFileStreams
Adds TestConcurrentStreamReads, which writes a large file (smaller in
Short mode), and then ensures that several concurrent readers properly
read their portions of the file with random offsets
Ensures that Move will properly overwrite the file at the destination
location.
Also checks that Move of a nonexistent source file will NOT delete the
file at the destination.
This replaces only using flat filenames, to better test nested file
behaviors.
Fixed inmemory/mfs.go and filesystem/driver.go after finding bugs with
the new tests and test behavior.
The packages causing build errors are being disabled for now to let us split up
the work in the different driver implementations without blocking integration
into the main branch. The s3 and azure implementations need some effort to add
Stat support. The ipc package needs that work plus some care around hanging
send calls.
Several checks for ReadStream with offset around boundary conditions were
missing. The new checks ensure negative offsets are detected and io.EOF is
returned properly when trying to read past the end of a file. The filesystem
and inmemory driver have been updated accordingly.
An outline of missing checks for List are also part of this commit. Action will
be taken here based on discussion in issue #819.
This change started out as simply updating the existing inmemory driver to
implement the new Stat call. After struggling with the map based
implementation, it has been refactored to be a tree-based implementation.
This process has exposed a few missing error cases in the StorageDriver API
that should be addressed in the coming weeks.
The filesystem driver has been migrated to impleemnt the storagedriver
interface changes. Most interetingly, this provides a filesystem-based
implementation of the Stat driver call. With this comes some refactoring of
Reads and Write to be much simpler and more robust.
The IPC tests have been disabled to stability problems that we'll have to
troubleshoot at a later date.
This change updates the testsuite to migrate to the new driver interface. This
includes the new Stat call, changes to int64 over uint64 and the changes to the
WriteStream signature. Several test cases have been added to vet
implementations against various assumptions.
We are change the the rpc call for WriteStream to not require the size
argument, opting to drive the process with io.Reader. The main issue was that
io.Reader may return io.EOF before reaching size, making the error handling
around this condition for callers more complex. To complement this, WriteStream
now returns the number of successfully written bytes.
The method no longer requires an io.ReadCloser, opting to require just an
io.Reader. This keeps the reader under the control of the caller, which
provides more flexibility.
This also begins to address some of the problems described in #791.
To support single-flight Size and ModTime queries against backend storage file,
we are replacing the CurrentSize call with a Stat call. A FileInfo interface is
provided for backends to provide a type, with a default implementation called
FileInfoInternal, for use by driver implementations.
More work needs to follow this change to update all the driver implementations.
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
Fixes/tests listing for keys beginning with "/"
No longer extraneously wraps Closers in ioutil.NopClosers
Uses omitempty for all ipc struct type fields
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.
Also improves storagedriver tests to enforce proper error types are
returned
