Creates trees instead of flat files for TestConcurrentFileStreams
Adds TestConcurrentStreamReads, which writes a large file (smaller in
Short mode), and then ensures that several concurrent readers properly
read their portions of the file with random offsets
Ensures that Move will properly overwrite the file at the destination
location.
Also checks that Move of a nonexistent source file will NOT delete the
file at the destination.
This replaces only using flat filenames, to better test nested file
behaviors.
Fixed inmemory/mfs.go and filesystem/driver.go after finding bugs with
the new tests and test behavior.
The tests are using way too much memory with the race detector enabled causing
the build machines to fall over. Cursory profiling shows no leaks but it may
need a closer look. For now, it will be disabled but this cannot be permanent.
The packages causing build errors are being disabled for now to let us split up
the work in the different driver implementations without blocking integration
into the main branch. The s3 and azure implementations need some effort to add
Stat support. The ipc package needs that work plus some care around hanging
send calls.
Several checks for ReadStream with offset around boundary conditions were
missing. The new checks ensure negative offsets are detected and io.EOF is
returned properly when trying to read past the end of a file. The filesystem
and inmemory driver have been updated accordingly.
An outline of missing checks for List are also part of this commit. Action will
be taken here based on discussion in issue #819.
This change updates the backend storage package that consumes StorageDriver to
use the new Stat call, over CurrentSize. It also makes minor updates for using
WriteStream and ReadStream.
This change started out as simply updating the existing inmemory driver to
implement the new Stat call. After struggling with the map based
implementation, it has been refactored to be a tree-based implementation.
This process has exposed a few missing error cases in the StorageDriver API
that should be addressed in the coming weeks.
The filesystem driver has been migrated to impleemnt the storagedriver
interface changes. Most interetingly, this provides a filesystem-based
implementation of the Stat driver call. With this comes some refactoring of
Reads and Write to be much simpler and more robust.
The IPC tests have been disabled to stability problems that we'll have to
troubleshoot at a later date.
This change updates the testsuite to migrate to the new driver interface. This
includes the new Stat call, changes to int64 over uint64 and the changes to the
WriteStream signature. Several test cases have been added to vet
implementations against various assumptions.
We are change the the rpc call for WriteStream to not require the size
argument, opting to drive the process with io.Reader. The main issue was that
io.Reader may return io.EOF before reaching size, making the error handling
around this condition for callers more complex. To complement this, WriteStream
now returns the number of successfully written bytes.
The method no longer requires an io.ReadCloser, opting to require just an
io.Reader. This keeps the reader under the control of the caller, which
provides more flexibility.
This also begins to address some of the problems described in #791.
To support single-flight Size and ModTime queries against backend storage file,
we are replacing the CurrentSize call with a Stat call. A FileInfo interface is
provided for backends to provide a type, with a default implementation called
FileInfoInternal, for use by driver implementations.
More work needs to follow this change to update all the driver implementations.
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
