Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it ( #2094 )
...
* fs: rename DirChangeNotify to ChangeNotify
* cache: switch to ChangeNotify
* ChangeNotify: keep order of notifications
2018-03-08 22:03:34 +02:00
Nick Craig-Wood
0175332987
vfs: fix applying modtime for an open Write Handle
...
The symptom of this was that the time set when the file was open was
lost. This was causing one of the mount tests to fail too.
2018-03-06 21:58:11 +00:00
Nick Craig-Wood
5795bd7db6
vfs: update cached copy if we know it has changed even if pending opens
...
This fixes a problem with Caddy serving corrupted files out of the VFS
cache when the file on the remote changed.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
9b011ce7e4
vfs: keep track of number of open RWHandles
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
5e334eedd2
vfs: re-use the File objects when re-reading the directory
...
Make it so that d.items is never nil to simplify the code
This should help with inconsistent reads when the source object changes.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
7fb53a031c
vfs: don't cache the object in read and read/write handles
...
This should help with inconsistent reads when the source object changes.
2018-03-04 11:20:22 +00:00
Nick Craig-Wood
b91bd32489
vfs: Fix TestWriteFileDoubleClose with --vfs-cache-mode >= writes
...
This was causing the file to be closed on Flush() instead of Release()
when the file was opened with O_TRUNC.
2018-02-26 21:26:32 +00:00
Nick Craig-Wood
c3d0f68923
vfs: fix truncation work-around on Windows
...
This no longer needs to deal with O_RDONLY and O_TRUNC since we
disallow this earlier. This also fixes the code to just do it for
O_APPEND, not for everything.
2018-02-26 19:46:38 +00:00
Nick Craig-Wood
f57e92b9a5
vfs: fix creation of files when truncating #2083
...
As spotted by @B4dM4n
2018-02-26 19:37:58 +00:00
Nick Craig-Wood
baf9ee5cf7
vfs: update cached copy if we know it has changed before using it
...
Before this change we would have to wait for the --vfs-cache-max-age
to expire before getting an update.
2018-02-26 18:00:51 +00:00
Nick Craig-Wood
354f1ad722
vfs: Use operations.Copy instead of CopyFile for efficiency
2018-02-26 17:54:18 +00:00
Nick Craig-Wood
54deb01f00
vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC
...
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file. This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead. This preserves the invariant O_RDONLY => file is not
changed.
2018-02-26 17:04:27 +00:00
Nick Craig-Wood
3282fd26af
vfs: clean path names before using them in the cache
...
This avoids inconsistent cache behaviour on open("potato/")
close("potato").
The tests were also adjusted to make them more comprehensive.
2018-02-26 16:59:14 +00:00
Nick Craig-Wood
88d830c7b7
vfs: create cache.opens and use it in place of cache.get to avoid potential race
2018-02-26 16:58:02 +00:00
Nick Craig-Wood
9252224d82
vfs: don't open the file when using a RW handle for a null Seek
...
Background: cmd/mount/file.go Open() function does a Seek(0, 1) to see
if the file handle is seekable to set a FUSE hint. Before this change
the file was downloaded before it needed to be which was inefficient
(and broke beta.rclone.org because HEAD requests caused downloads!).
2018-02-22 17:28:21 +00:00
Fabian Möller
62e72801be
vfs: fix race between multiple RWFileHandle ( #2052 )
...
Fixes #2034
2018-02-18 14:12:26 +01:00
Nick Craig-Wood
442334ba61
vfs: disable cache cleaner if --vfs-cache-poll-interval=0
...
And use this to disable the cleaner in the cache tests to make them
more reliable
2018-02-16 14:12:46 +00:00
Nick Craig-Wood
8a9ed57951
vfs: fix another race in cache tests
2018-02-16 12:05:59 +00:00
Nick Craig-Wood
9b800d7184
vfs: fix race in cache tests
2018-02-15 21:34:37 +00:00
Nick Craig-Wood
cd7fd51119
vfs: fix docs - fixes #2067
2018-02-12 11:29:32 +00:00
Stefan Breunig
846bbef1e9
vfs: write 0 bytes when flushing unwritten handles to avoid race conditions in FUSE - fixes #1181
2018-02-11 17:59:13 +00:00
Nick Craig-Wood
04a0a7406b
vfs: downgrade "poll-interval is not supported" message to Info
...
...to save confusion as it isn't very important
2018-02-09 07:57:50 +00:00
Nick Craig-Wood
295c3fabec
vfs: fill and clean the cache immediately on startup
2018-02-02 12:19:53 +00:00
Nick Craig-Wood
3f8d286a75
vfs: fix cache cleaning on startup
...
Previous to this fix the vfs cache wasn't being cleaned properly on
startup as the atimes of the existing files were being ignored.
2018-02-02 12:06:42 +00:00
Nick Craig-Wood
44276db454
vfs: make -ve sized files appear as 0 size. #320
...
This means that Google docs will no longer appear as huge files in
`rclone mount`. They will not be downloadable, though sometimes
trying twice will work.
2018-01-31 16:22:05 +00:00
Nick Craig-Wood
dbabb18b0c
vfs: Make error messages more informative #2009
2018-01-25 10:33:00 +00:00
Jody Frankowski
5e73acd40a
Clean up mount.go and vfs/help.go docs
...
* Title cleanups
* Typos
* `rclone mount vs rclone sync/copy` update with `File Caching`
2018-01-20 18:14:20 +00:00
Nick Craig-Wood
a80d8a21dc
vfs: add flags parameter to Dir.Create
2018-01-19 13:18:40 +00:00
Nick Craig-Wood
517bdc719b
vfs: make specialized file Open functions private
2018-01-19 11:46:01 +00:00
Nick Craig-Wood
11da2a6c9b
Break the fs package up into smaller parts.
...
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
2018-01-15 17:51:14 +00:00
Nick Craig-Wood
b8b620f5c2
Move all backends into backend directory
2018-01-12 17:07:38 +00:00
Nick Craig-Wood
29d34426bc
vfs: fix deletion of in use directories #1860
...
This was causing errors if the cache cleaner was called between the
Open and the pendingOpen of a RW file.
The fix was to move the cache open to the Open from the openPending.
2017-12-15 15:42:49 +00:00
Nick Craig-Wood
38b85e94ea
vfs: rename --cache-* options to --vfs-cache-* to save confusion
...
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
2017-12-03 12:14:15 +00:00
Nick Craig-Wood
c16ac697a9
vfs: keep track of directories in the cache also #1860
...
This makes managing empty directories more reliable.
2017-11-28 20:39:23 +00:00
Nick Craig-Wood
0978957a2e
vfs: make sure all 96 combinations of flags for Open work
2017-11-28 20:39:23 +00:00
Nick Craig-Wood
d1b19f975d
vfs: remove items from cache when deleted #1860
...
Also fixes Error message when items have been deleted from the cache
(eg when Moved) when the cache reaper comes to delete them.
2017-11-28 16:13:58 +00:00
Nick Craig-Wood
3fb4fe31d2
vfs: make sure write only handles never truncate files they shouldn't
2017-11-20 19:42:25 +00:00
Nick Craig-Wood
76b151984c
vfs: cache the size of the object in the read handle
2017-11-20 17:57:13 +00:00
Nick Craig-Wood
60945d0a37
vfs: remove misleading comment
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
b4083b4371
vfs: rename Fsync to Sync and implement Sync on Node and Handle
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9738f8532b
vfs: Add FlushDirCache method
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
a5b034a992
vfs: add WaitForWriters to wait until all writers have finished
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
321b6da7af
vfs: don't remove file from writers until it is transferred
...
This means that the list of active writers is up to date
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
1b22ee5b93
vfs: fix error handling in openPending so it returns the correct error
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eab55ce882
vfs: add open files to directories
2017-11-20 12:46:23 +00:00
Nick Craig-Wood
c560017934
vfs: add Path method to Node and use it to stop reading nil DirEntry
...
All DirEntry calls now have been checked for nil or converted to use Path.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
992647b157
vfs: Don't error a r/w file open without cache; delay error until Read called
...
If we open a file for r/w without the cache we now always return a
handle and return an error if the file is ever read from. This fixes
incompatibility with cmount under windows.
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63
vfs, cmount: make truncate work properly in the presence or otherwise of open files
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
e7f2935333
vfs: decode flags in Open/OpenFile for debug
2017-11-20 12:46:22 +00:00
Nick Craig-Wood
8951875c21
vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables
2017-11-20 12:46:21 +00:00