TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
6803 commits 181 branches 114 tags 335 MiB
Commit graph

34 commits

Author SHA1 Message Date
Nick Craig-Wood
617c5d5e1b rcat: preserve metadata when Copy falls back to Rcat 
Before this change if we copied files of unknown size, then they lost
their metadata.

This was particularly noticeable using --s3-decompress.

This change adds metadata to Rcat and RcatSized and changes Copy to
pass the metadata in when it calls Rcat for an unknown sized input.

Fixes #6546
 2022-11-10 12:04:35 +00:00
Nick Craig-Wood
a07d376fb1 vfs: reduce memory usage by re-ordering commonly used structures 2022-08-04 10:19:47 +01:00
Nick Craig-Wood
e749bc58f4 vfs: reduce memory use by embedding sync.Cond 2022-08-04 10:19:47 +01:00
albertony
ec117593f1 Fix lint issues reported by staticcheck 
Used staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
 2022-06-13 21:13:50 +02:00
Josh Soref
3e1cb8302a docs: spelling: etc. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood
4441e012cf vfs: fix saving from chrome without --vfs-cache-mode writes #4293 
Due to Chrome's rather complicated use of file handles when saving
files from the download windows, rclone was attempting to truncate a
closed file.

The file appeared closed due to the handling of 0 length files.

This patch removes the check for the file being closed in the
WriteFileHandle.Truncate call. This is safe because the only action
this method takes is to emit an error message if the file is the wrong
size.

See: https://forum.rclone.org/t/google-drive-cannot-save-files-directly-from-browser-to-gdrive-mounted-path/17992/
 2020-07-28 17:18:31 +01:00
Nick Craig-Wood
917cb4acb3 vfs: implement partial reads in --vfs-cache-mode full 
This allows reads to only read part of the file and it keeps on disk a
cache of what parts of each file have been loaded.

File data itself is kept in sparse files.
 2020-06-30 12:01:36 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood
0f9267d5fc vfs: factor waiting code from read and writes into common function 2020-05-18 18:09:23 +01:00
Nick Craig-Wood
238f26cc90 vfs: stop reading File members from outside file.go 
This also fixes locking for ReadFileHandle and WriteFileHandle
accessing File members
 2020-04-19 15:16:43 +01:00
Nick Craig-Wood
393b94bb70 vfs: add --vfs-read-wait and --vfs-write-wait flags 
--vfs-read-wait duration    Time to wait for in-sequence read before seeking. (default 5ms)
    --vfs-write-wait duration   Time to wait for in-sequence write before giving error. (default 1s)

See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
 2020-02-27 16:12:33 +00:00
Nick Craig-Wood
d377842395 vfs: make write without cache more efficient 
This updates the out of sequence write code to be more efficient using
a conditional lock with a timeout.
 2019-08-08 12:37:50 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic
f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Nick Craig-Wood
af030f74f5 vfs: make WriteAt for non cached files work with non-sequential writes 
This makes WriteAt for non cached files wait a short time if it gets
an out of order write (which would normally cause an error) to see if
the gap will be filled with an in order write.

The makes the SFTP backend work fine with --vfs-cache-mode off
 2019-05-11 23:39:04 +01:00
Nick Craig-Wood
0175332987 vfs: fix applying modtime for an open Write Handle 
The symptom of this was that the time set when the file was open was
lost.  This was causing one of the mount tests to fail too.
 2018-03-06 21:58:11 +00:00
Fabian Möller
62e72801be
vfs: fix race between multiple RWFileHandle (#2052) 
Fixes #2034
 2018-02-18 14:12:26 +01:00
Stefan Breunig
846bbef1e9 vfs: write 0 bytes when flushing unwritten handles to avoid race conditions in FUSE - fixes #1181 2018-02-11 17:59:13 +00:00
Nick Craig-Wood
dbabb18b0c vfs: Make error messages more informative #2009 2018-01-25 10:33:00 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
38b85e94ea vfs: rename --cache-* options to --vfs-cache-* to save confusion 
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
 2017-12-03 12:14:15 +00:00
Nick Craig-Wood
3fb4fe31d2 vfs: make sure write only handles never truncate files they shouldn't 2017-11-20 19:42:25 +00:00
Nick Craig-Wood
b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
321b6da7af vfs: don't remove file from writers until it is transferred 
This means that the list of active writers is up to date
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eab55ce882 vfs: add open files to directories 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
992647b157 vfs: Don't error a r/w file open without cache; delay error until Read called 
If we open a file for r/w without the cache we now always return a
handle and return an error if the file is ever read from.  This fixes
incompatibility with cmount under windows.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63 vfs, cmount: make truncate work properly in the presence or otherwise of open files 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
a0cfa0929b vfs: remove un-needed (after introduction of rcat) createInfo struct 2017-11-20 12:00:32 +00:00
Nick Craig-Wood
c31556c6d1 vfs: Make sure all public methods are locked in Read and Write Handle 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
2083ac6e2a vfs: add ECLOSED and tidy errors 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
e18122e88b vfs: add tests and subsequent fixes 
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
 2017-11-04 10:24:10 +00:00
Nick Craig-Wood
a5dc62f6c1 vfs: Make file handles compatible with OS 
* Implement directory handles
  * Unify OpenFile
  * Add all the methods to match *os.File
  * Add StatParent and Rename methods to VFS
 2017-11-03 13:00:00 +00:00
Nick Craig-Wood
c1aaff220d Factor new vfs module out of cmd/mountlib 
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
 2017-11-03 12:59:59 +00:00
Renamed from cmd/mountlib/write.go (Browse further)