TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
2398 commits 181 branches 114 tags 335 MiB
Commit graph

348 commits

Author SHA1 Message Date
Nick Craig-Wood
06e3fa3aba mounttest: reduce duplicated code and improve test output #2154 
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood
be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154 
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
 2018-04-07 11:42:55 +01:00
Nick Craig-Wood
e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood
80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Matt Holt
e13f65b953 serve restic: Print actual listener address 2018-04-04 14:56:26 +01:00
Alexander Neumann
5b8977a053 serve restic: Disallow overwriting files in append-only mode - Fixes #2195 
* Disallow overwriting files in append-only mode
* Add tests for append-only mode
 2018-04-04 14:49:13 +01:00
Steve Kriss
2c2bb0f750 cmd/serve/restic: add append-only mode 2018-03-30 19:54:52 +01:00
Stefan
a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Keith Goldfarb
6c10312c75 ncdu: added a "refresh" key - for #2174 
Added Control+L key to refresh screen. Not sure if this is the
best choice, but it appears to be somewhat common.
 2018-03-26 21:02:39 +01:00
Nick Craig-Wood
98a924602f mount, cmount: set --attr-timeout default to 1s - fixes #2157 
This  works around these 3 problems:

  * rclone using too much memory #2157
  * rclone not serving files to samba
    * https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112
  * excessive time listing directories #2095
 2018-03-23 22:42:51 +00:00
Nick Craig-Wood
4f6f07c074 cmount: fix error handling for Open/OpenDir 2018-03-21 19:44:30 +00:00
Nick Craig-Wood
1afac32d80 serve restic: script for running integration test against all remotes 2018-03-18 19:15:39 +00:00
Nick Craig-Wood
26fbd00b4f serve restic: don't buffer the JSON output in memory for the list command 2018-03-18 16:26:58 +00:00
Nick Craig-Wood
1313b529ff serve restic: use ListR (--fast-list) if available 
For Restic's use case, --fast-list will use less transactions and
calling ListR directly means we can avoid the usual memory overhead.
 2018-03-18 16:22:05 +00:00
Nick Craig-Wood
82e835d6fc serve restic: make it easy to run integration tests against any remote 
Just `cd cmd/serve/restic` then `go test -v -remote TestRemote:`
 2018-03-18 14:23:56 +00:00
Nick Craig-Wood
fa867a9a4c serve restic: implement accounting for uploads and downloads 
This means the bandwidth stats will be correct and the bandwidth
throttling will work correctly.  This was forgotten as a previous
iteration of the code was using the higher level operations.Rcat which
took care of this.
 2018-03-18 14:19:43 +00:00
Stefan
86892467d9
config: load config file only on first access (closes #1659, closes #2096) (#2147) 2018-03-17 12:36:30 +01:00
Nick Craig-Wood
34c45a7c04 mount, cmount: remove addition of O_CREATE to flags on file open #2141 
Previously this was adding it in to all file opens which was causing
inefficiencies under Windows where it stats the file using
open/fstat/close.

This change will make stat operations run much quicker under Windows
as they won't have to open the underlying file

This problem was introduced in61b6159a05336bd7ba105766de2d2ff171f7fb81
where we added O_CREATE to all file opens and creates.
 2018-03-15 20:48:56 +00:00
Alexander Neumann
84776c4e43 serve/restic: Remove log message on Close 2018-03-14 21:50:33 +00:00
Nick Craig-Wood
c1a3e363a6 mount: return ENOSYS rather than EIO on attempted link 
This fixes FileZilla accessing an rclone mount served over sftp.

See: https://forum.rclone.org/t/moving-files-on-rclone-mount-with-filezilla/5029
 2018-03-14 21:10:20 +00:00
Nick Craig-Wood
7ccc6080b0 serve restic: add more info to GET request error 2018-03-14 21:09:47 +00:00
remusb
97b48cf988 rc: add support for Go 1.6 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
86e5a35491 Implement Remote Control for rclone #2111 
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.

Still to do
  * docs - need finishing
  * tests
 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
8bb2854fe4 httplib: allow the flags to be prefixed when instantiating a server 2018-03-14 22:58:20 +02:00
Nick Craig-Wood
4e90ad04d5 serve restic: only accept v2 API requests for list 2018-03-11 17:35:01 +00:00
Alexander Neumann
fa003e89b6 serve restic: When listing return empty list instead of 'null' 2018-03-11 14:48:46 +00:00
Alexander Neumann
5114b11d6f serve restic: add http2 server on stdin/stdout 2018-03-11 14:48:46 +00:00
Alexander Neumann
f832433fa5 serve restic: Return empty list for non-existing dirs 2018-03-11 14:48:43 +00:00
Nick Craig-Wood
d073efdc6c serve restic: serves a remote in restic REST API format 2018-03-11 14:43:03 +00:00
Nick Craig-Wood
9e48748182 httplib: Note that authentication is a good idea for non localhost 2018-03-11 14:38:54 +00:00
Nick Craig-Wood
66c69fe620 mount: wait longer for consistency after rm in tests 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
7713acf23d mount: skip failing test TestFileModTimeWithOpenWriters on Windows 2018-03-09 23:15:38 +00:00
Nick Craig-Wood
473a388f6d mount: disable failing test TestWriteFileDoubleClose on OSX 2018-03-09 23:15:37 +00:00
Nick Craig-Wood
09c14af6d1 cmd: Fix go routines at exit message to make it less confusing 2018-03-09 17:15:48 +00:00
Jakub Tasiemski
acae10cd6f lsjson: add --encrypted to show encrypted name #1765 2018-03-09 08:44:02 +00:00
Remus Bunduc
70f07fd3ac
fs: add ChangeNotify and backend support for it (#2094) 
* fs: rename DirChangeNotify to ChangeNotify

* cache: switch to ChangeNotify

* ChangeNotify: keep order of notifications
 2018-03-08 22:03:34 +02:00
Nick Craig-Wood
fc32fee4ad mount, cmount: add --attr-timeout to control attribute caching in kernel 
This flag allows the attribute caching in the kernel to be controlled.
The default is 0s - no caching - which is recommended for filesystems
which can change outside the control of the kernel.

Previously this was at the default meaning it was 60s for mount and 1s
for cmount.  This showed strange effects when files changed on the
remote not via the kernel.  For instance Caddy would serve corrupted
files for a while when serving from an rclone mount when a file
changed on the remote.
 2018-03-04 11:20:22 +00:00
ishuah
ebfeec9fb4 mount: run rclone mount in the background - fixes #723 2018-03-04 14:06:07 +03:00
Nick Craig-Wood
54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC 
Before this change Open("name", os.O_RDONLY|os.O_TRUNC) would have
truncated the file.  This is what Linux does, but is counterintuitive.
POSIX states this is undefined, so return an error in this case
instead.  This preserves the invariant O_RDONLY => file is not
changed.
 2018-02-26 17:04:27 +00:00
Fabian Möller
00adf40f9f
cryptdecode: use Cipher instead of NewFs (#2087) 
* crypt: extract NewCipher out of NewFs
* cryptdecode: make use of crypt.NewCipher

Fixes #2075
 2018-02-25 12:57:14 +01:00
Nick Craig-Wood
334bf49d30 httplib: add Close() method to shut the server down and use it in tests 2018-02-19 15:45:24 +00:00
Nick Craig-Wood
d8f78a7266 serve http/webdav: update docs on SSL 2018-02-19 14:08:17 +00:00
Nick Craig-Wood
358c1fbac9 serve http/webdav: support SSL/TLS 2018-02-16 18:28:10 +00:00
Nick Craig-Wood
cc9d7156e4 serve http/webdav: add --user --pass authentication #1802 2018-02-16 18:28:10 +00:00
Nick Craig-Wood
221a8a9c5d serve http/webdav: add --htpasswd option for authentication #1802 2018-02-16 18:28:10 +00:00
Nick Craig-Wood
5530662ccc serve http/webdav: factor common http server creation to httplib 2018-02-16 17:48:20 +00:00
Nick Craig-Wood
0f787e43b0 mount: link the nssm service manager for mount under Windows 2018-02-12 11:29:32 +00:00
Nick Craig-Wood
3a7bb7b2df mount: update docs showing --vfs-cache-mode to work around limitations 2018-02-12 11:29:32 +00:00
Jakub Tasiemski
1018e9bb27 cmd: rewrite touch tests #1934 2018-02-02 13:46:56 +00:00
Fabian Möller
3c7a755631 lsjson: explain the Path value in the docs 2018-01-31 20:06:01 +00:00
Nick Craig-Wood
c277a4096c mount: don't set modtime twice #2021 2018-01-29 20:49:13 +00:00
Fabian Möller
1f5e23aedb scripts: make absolute paths consistent 
Change absolute binary paths in scripts to /usr/bin/env or make them
relative.
This allows the scripts to be used on linux distributions
like NixOS, where binaries are not located in /usr/ or /bin/.
 2018-01-26 09:39:05 +00:00
Nick Craig-Wood
fa500e6d21 lib/atexit: factor from cmd so it can be used by backend/cache #1946 2018-01-25 10:33:00 +00:00
Jody Frankowski
5e73acd40a Clean up mount.go and vfs/help.go docs 
* Title cleanups
* Typos
* `rclone mount vs rclone sync/copy` update with `File Caching`
 2018-01-20 18:14:20 +00:00
Nick Craig-Wood
a80d8a21dc vfs: add flags parameter to Dir.Create 2018-01-19 13:18:40 +00:00
Nick Craig-Wood
517bdc719b vfs: make specialized file Open functions private 2018-01-19 11:46:01 +00:00
Nick Craig-Wood
5ad226ab54 fs: Add dir option to fs.Purge #1891 
Purge optional interface needs fixing too.
 2018-01-19 11:45:50 +00:00
Nick Craig-Wood
97c414f025 config/hash: rename more symbols after factoring into own package 2018-01-18 20:27:52 +00:00
Nick Craig-Wood
71722b5b95 config: factor Obscure and Reveal into its own package 2018-01-18 20:19:55 +00:00
Nick Craig-Wood
821be5ebed ncdu: add link to asciinema demo of it in action 2018-01-18 14:22:43 +00:00
Ernest Borowski
5cce74d630 flags: remove --no-traverse flag because it is obsolete - fixes #1813 
Signed-off-by: Ernest Borowski <er.borowski@gmail.com>
 2018-01-18 11:00:25 +00:00
Nick Craig-Wood
11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood
60afda007b Move dircache, oauthutil, rest and pacer modules into lib 2018-01-12 17:07:38 +00:00
Nick Craig-Wood
b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
ishuah
0a7731cf0d cryptdecode: added option to return encrypted file names. Fixes #1923 2018-01-11 19:22:40 +03:00
Nick Craig-Wood
0eba37d8f3 lsf: add --files-only and --dirs-only flags 2018-01-06 18:04:24 +00:00
Nick Craig-Wood
c74c3b37da lsf: add option to print hashes 2018-01-06 17:53:37 +00:00
Nick Craig-Wood
ed20fa5ee7 ls* commands: update docs and add defaults into options for lsf 2018-01-06 17:00:20 +00:00
Nick Craig-Wood
54a9fdf421 ls2: remove in favour of lsf 2018-01-06 14:41:36 +00:00
Jakub Tasiemski
0d041602cf cmd: new command lsf 2018-01-06 14:39:31 +00:00
Nick Craig-Wood
65618afd8c serve/http: fix serving files with : in - fixes #1939 2018-01-05 17:25:05 +00:00
Nick Craig-Wood
2a01fa9fa0 moveto,copyto: clarify error message if source doesn't exist - fixes #1022 2017-12-15 11:37:31 +00:00
lewapm
9c242edc10 rmdirs: add --leave-root flag - fixes #1874 2017-12-13 10:23:54 +00:00
Nick Craig-Wood
2cf808c825 ncdu: fix crashes on empty directories - fixes #1910 
Up arrow or right arrow in an empty directory would crash ncdu
 2017-12-12 13:54:15 +00:00
remusb
b48b537325
cache: plex integration, refactor chunk storage and worker retries (#1899) 2017-12-09 23:54:26 +02:00
Laurence
e767c9ac9f Fix typo in dbhashsum description 2017-12-03 10:24:33 +00:00
Nick Craig-Wood
6d3df6f172 cmount: make tests more reliable on Windows 2017-11-28 20:39:24 +00:00
ishuah
aab8051f50 move: add --delete-empty-src-dirs flag - fixes #1854 2017-11-28 11:38:19 +03:00
Nick Craig-Wood
1248beb0b2 cachestats: Fix nil pointer if not a cache remote - fixes #1855 
Also don't retry or show stats
 2017-11-24 10:22:23 +00:00
Nick Craig-Wood
dc1c679c65 mount: support truncate properly 2017-11-20 19:42:35 +00:00
Nick Craig-Wood
af50f31f7d mounttest: wait for Release after every Read to stop using in use files under Windows 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
2b666187a6 cmount: disable tests on windows + race detector 
These either hang or produce incorrect results for reasons I haven't
worked out yet.
 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
00b46a8b96 mounttest: wait for files to disappear from the directory listing 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
b21f227bd3 mounttest: fix crash when FUSE not present 2017-11-20 12:46:24 +00:00
Nick Craig-Wood
e98e550021 mounttest: wait for all background Close/Release after writing a file 
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
eb3415db50 cmount: enable more tests for Windows 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
9fbd8a6419 mounttest: fixes for running under Windows 
* don't mount and unmount between cache runs - WinFSP doesn't suport it
  * use OS paths for opening things
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood
61b6159a05 mount, cmount: add O_CREATE to Open calls since fuse doesn't seem to supply it 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
c560017934 vfs: add Path method to Node and use it to stop reading nil DirEntry 
All DirEntry calls now have been checked for nil or converted to use Path.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
7c3584f4e6 mountlib: wait for mountpoint to disappear under Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
981cfb1bec mounttest: retry directory listings to account for slow updates on Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
dec21ccf63 vfs, cmount: make truncate work properly in the presence or otherwise of open files 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
94adf4f43b cmount: translate FUSE open flags into OS flags 
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations.  They are
mostly the same which is why this hasn't caused a problem before.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
f5f8c0c438 cmount: make Truncate call the correct Handle or Node method 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
60cdcf784c cmount: use -o atomic_o_trunc to make sure O_TRUNC is supplied to Open() 2017-11-20 12:46:22 +00:00
Nick Craig-Wood
57a5c67729 mounttest: run the tests for all 4 VFS cache modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
d7908c06c9 mountlib: ensure we don't open files with read and write intent 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00
Nick Craig-Wood
05a1e1532b vfs,mount,cmount,serve: Add documentation for vfs caching modes 2017-11-20 12:46:21 +00:00