Commit graph

4302 commits

Author SHA1 Message Date
Nick Craig-Wood
b91e01fd22 drive: strip trailing slashes in shortcut command #4098
This also fixes typo in the name of the function, and allows making
shortcuts from the root directory which are useful in cross drive
shortcut creation.

This also adds a basic suite of tests for creating listing, removing
shortcuts.
2020-05-12 17:08:05 +01:00
Nick Craig-Wood
177195aeeb accounting: fix race clearing stats
This race was introduced by

10a6a92e52 accounting: reset bytes read during copy retry
2020-05-12 17:02:32 +01:00
Nick Craig-Wood
cb5979a468 accounting: factor stats into its own structure
This makes it very obvious which mutex to take for accessing the
values.
2020-05-12 17:02:32 +01:00
Nick Craig-Wood
32507774de Add Caleb Case to contributors 2020-05-12 17:01:26 +01:00
Caleb Case
0ce662faad Tardigrade Backend 2020-05-12 15:56:50 +00:00
Caleb Case
03b629064a Tardigrade Backend: Dependencies 2020-05-12 15:56:50 +00:00
albertony
962fbc8257 jottacloud: update docs regarding cleanup, removed remains from old auth, and added warning about special mountpoints. 2020-05-11 11:41:17 +02:00
Nick Craig-Wood
bd4b91bd57 test_all: revert running fichier tests one at a time
This didn't achieve the objective of getting the tests to run clean
and makes them take 16 hours!

This reverts commit 97f6f8fe19.
2020-05-11 08:51:33 +01:00
Ankur Gupta
10a6a92e52 accounting: reset bytes read during copy retry - fixes #4178
During a copy/sync command, if an operation fails due to a network
issue and is retried, the underlying io.Reader is re-initialised,
but the stats for bytes already read are not reset, leading to incorrect
stats. THis was fixed by resetting the bytes read when an Account is
re-initialized.
2020-05-10 17:58:22 +00:00
Max Sum
54b16bd054 union: implement ListR 2020-05-10 17:57:03 +00:00
Max Sum
f21e97001b union: fix server-side copy 2020-05-10 17:56:18 +00:00
Nick Craig-Wood
1f005a82ad http: add missing comment to pacify linter 2020-05-10 18:53:38 +01:00
Nick Craig-Wood
bb65974e2f drive: implement backend shortcut command for creating shortcuts #4098 2020-05-09 15:16:15 +01:00
Nick Craig-Wood
bc0f487369 drive: look for dirs as well as files on NewObject
This means that we can return ErrorNotAFile when there is an object
with the same name as a directory rather than potentially creating a
duplicate name.
2020-05-09 15:16:15 +01:00
Nick Craig-Wood
e103c4c26a Add Maxime Suret to contributors 2020-05-09 15:16:15 +01:00
Maxime Suret
79d29bb41e
serve sftp: add support for multiple host keys by repeating --key flag 2020-05-09 14:43:17 +01:00
calisro
c80b6d96dd
http: improved directory listing with new template from Caddy project
This includes a new directory listing template which was originally
from the Caddy project (used with permission and copyright attribution).

This is used whenever we serve directory listings so `rclone serve
http`, `rclone serve webdav` and `rclone rcd --rc-serve`

This also modifies the tests so they work with the original template which
is easier to debug.
2020-05-08 16:15:21 +01:00
Nick Craig-Wood
97f6f8fe19 test_all: run fichier tests one at a time
This is in attempt to get the tests to run cleanly without hitting the
rate limiter.
2020-05-07 20:39:54 +01:00
Nick Craig-Wood
94920d39ae test_all: increase the test timeout to 60m from 30m
Some tests are failing at 30m - 60m doesn't seem unreasonable
2020-05-07 20:38:40 +01:00
Nick Craig-Wood
9403bd2990 test_all: allow -list-retries to be overridden on the command line 2020-05-07 20:36:45 +01:00
Nick Craig-Wood
e098924e61 fstest: change -list-retries default to 3 from 6
The integration tests have got much better about retrying the tests,
and we aren't testing ACD any more so we don't need it this high.
2020-05-07 20:28:41 +01:00
Nick Craig-Wood
437f9e2cef Add Sébastien Gross to contributors 2020-05-07 11:25:09 +01:00
Sébastien Gross
395f259978
cmd: when running --password-command allow use of stdin
Bind rclone standard input to password command's standard input. This
allows to provide password from a pipe and collect it using cat.

The typical use case is when rclone is on a remote server with an
encrypted configuration. This solved the environment variable
issue (#3368) and the password storage on remote host.

Now the following chain is allowed:

    echo 'secret' | ssh host.example.com \
       sudo -u rclone \
       rclone --config /path/to/rclone.conf \
       --password-command 'cat' ls remote:

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>

Co-authored-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2020-05-07 11:02:52 +01:00
Nick Craig-Wood
b78adc9f03 Add Fred @creativeproject as the seafile backend maintainer 2020-05-07 10:18:10 +01:00
Nick Craig-Wood
b7c21310b4 Add Fred to contributors 2020-05-06 18:33:44 +01:00
Fred
c754e89906 seafile: New backend for seafile server 2020-05-06 17:33:22 +00:00
Fred
62cfe3f384 vendor: add github.com/coreos/go-semver 2020-05-06 17:33:22 +00:00
Nick Craig-Wood
afde340c9e gcs: fix --header-upload - #59
Before this code we were settig the headers on the PUT request. However this isn't where GCS needs them.

After this fix we set the headers in the object upload request itself.

This means that we only support a limited range of headers

- Cache-Control
- Content-Disposition
- Content-Encoding
- Content-Language
- Content-Type
- X-Goog-Meta-

Note for the last of those are for setting custom metadata in the form
"X-Goog-Meta-Key: value".
2020-05-06 17:34:23 +01:00
Nick Craig-Wood
d0ad83de4b Add ElonH to contributors 2020-05-06 17:34:23 +01:00
ElonH
d119bfd934 rcd: disable duplicate log
if running `rclone rcd --rc-user=admin --rc-pass=admin
--rc-allow-origin="*"`, lots of duplicate warnings apperent in log

Warning: Allow origin set to *. This can cause serious security problems.
Warning: Allow origin set to *. This can cause serious security problems.
....

This is not conducive to analyzing debugging info.

Therefore, let's show it only once.
2020-05-05 13:47:25 +00:00
Nick Craig-Wood
bdc91eda0f serve http: add Last-Modified headers to files and directories
This means that using `rclone serve http` preserves modification times
when used with the http backend.

Fixes #4201
2020-05-05 09:41:08 +01:00
Nick Craig-Wood
ef9e6794c2 docs: make serve http/webdav template docs into a table 2020-05-04 17:36:31 +00:00
calistri
4362ca7bb9 serve http, serve webdav: Added a --template flag for user defined markup 2020-05-04 17:36:31 +00:00
Nick Craig-Wood
dcf945ed58 docs: add bin/.ignored-emails for removing email addresses from authors.md
Remove an email as requested for Anagh Kumar Baranwal
2020-05-04 17:38:25 +01:00
Anagh Kumar Baranwal
a86196a156 drive: Added command to change service_account_file and chunk_size
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-05-04 16:23:33 +00:00
Anagh Kumar Baranwal
856c2b565f crypt: Added decode/encode commands to replicate functionality of cryptdecode
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-05-04 16:23:33 +00:00
Nick Craig-Wood
1a8c5708c5 vfs: ignore file not found errors from Hash in Read.Release
There is nothing we can do about this at this point and this error can
happen when moving files so we ignore it to clean the logs up.
2020-05-04 12:18:28 +01:00
Nick Craig-Wood
14cab0fff0 local: fix "file not found" errors on post transfer Hash calculation
Before this change the local backend was returning file not found
errors for post transfer hashes for files which were moved. This was
caused by the routine which checks for the object being changed.

After this change we ignore file not found errors while checking to
see if the object has changed. If the hash has to be computed then a
file not found error will be thrown when it is opened, otherwise the
cached hash will be returned.
2020-05-04 12:17:46 +01:00
Nick Craig-Wood
69888bf966 cmount: send a hint as to whether the filesystem is case insensitive or not 2020-05-04 11:38:07 +01:00
Nick Craig-Wood
d260238f99 cmount: use ReaddirPlus on Windows to improve directory listing performance
Before this change Windows would read a directory then immedately stat
every item in the directory.

After this change we return the stat information along with the
directory which stops so many callbacks.
2020-05-04 11:38:07 +01:00
Nick Craig-Wood
6ca7198f57 mount: fix disappearing cwd problem - fixes #4104
Before this change, the current working directory could disappear
according to the Linux kernel.

This was caused by mount returning different nodes with the same
information in.

This change uses vfs.Node.SetSys to cache the information so we always
return the same node.
2020-05-04 11:38:07 +01:00
Nick Craig-Wood
cfcdc85b26 vfs: Add SetSys() methods to Node to allow caching stuff on a node 2020-05-04 11:38:07 +01:00
Nick Craig-Wood
7f8d74e903 docs: refresh rclone authorize help 2020-05-04 10:40:26 +01:00
Nick Craig-Wood
f2b1fedc4f drive: follow shortcuts by default, skip with --drive-skip-shortcuts
Before this change rclone would skip all shortcuts with a message

    Ignoring unknown document type "application/vnd.google-apps.shortcut"

After this message rclone resolves the shortcuts by default to the
actual files that they point to. See the docs for more info.

The --drive-skip-shortcuts flag can be used to skip shortcuts.
2020-05-02 18:28:38 +01:00
Nick Craig-Wood
b03cad3cf6 vendor: update google.golang.org/api/drive to pull in shortcuts definition 2020-05-02 18:28:38 +01:00
Nick Craig-Wood
70db13e6e8 vfs: pin the Fs in use in the Fs cache
This means we can reliably look up the Fs from the cache when using
`backend/command`.
2020-05-01 17:11:45 +01:00
Nick Craig-Wood
4c98360356 fs/cache: Add Pin and Unpin and canonicalised lookup
Before this change we stored cached Fs under the config string the
user gave us. As the result of fs.ConfigString() can often be
different after the backend has canonicalised the paths this meant
that we could not look up backends in the cache reliably.

After this change we store cached Fs under their config string as
returned from fs.ConfigString(f) after the Fs has been created. We
also store a map of user to canonical names (where they are different)
so the users can look up Fs under the names they passed to rclone too.

This change along with Pin and Unpin is necessary so we can look up
the Fs in use reliably in the `backend/command` remote control
interface.
2020-05-01 17:11:45 +01:00
Nick Craig-Wood
42f9f7fb5d lib/cache: add Rename, Pin and Unpin 2020-05-01 17:11:45 +01:00
Nick Craig-Wood
ca1856724c fs: add ConfigString function to return a canonical config string 2020-05-01 17:11:45 +01:00
Nick Craig-Wood
b52a39a84e drive: fix merge breakage
In 2f5a2d3c48 an incorrect merge caused compilation to fail
2020-05-01 13:02:32 +01:00