TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
5083 commits 181 branches 114 tags 335 MiB
Commit graph

1225 commits

Author SHA1 Message Date
Aleksandar Jankovic
38e70f1797 rc/jobs: add listener for finished jobs 
Add jobs.OnFinish method to register listener that will trigger when job
is finished.

Includes fix for stopping listeners.
 2021-01-21 13:43:41 +00:00
Nick Craig-Wood
a7689d7023 rcserver: fix 500 error when marshalling errors from core/command 
Before this change attempting to return an error from core/command
failed with a 500 error and a message about unmarshable types.

This is because it was attempting to marshal the input parameters
which get _response added to them which contains an unmarshalable
field.

This was fixed by using the original parameters in the error response
rather than the one modified during the error handling.

This also adds end to end tests for the streaming facilities as used
in core/command.
 2021-01-10 16:34:46 +00:00
Nick Craig-Wood
847a44e7ad fs/rc: add Copy method to rc.Params 2021-01-10 16:34:46 +00:00
Nick Craig-Wood
b3710c962e rc: fix core/command giving 500 internal error - fixes #4914 
Before this change calling core/command gave the error

    error: response object is required expecting *http.ResponseWriter value for key "_response" (was *http.response)

This was because the http.ResponseWriter is an interface not an object.

Removing the `*` fixes the problem.

This also indicates that this bit of code wasn't properly tested.
 2021-01-10 16:34:46 +00:00
Nick Craig-Wood
01adee7554 build: raise minimum go version to go1.12 2021-01-08 12:17:09 +00:00
Nick Craig-Wood
50344e7792 accounting: factor --tpslimit code into accounting from fshttp 2021-01-08 10:29:57 +00:00
Nick Craig-Wood
dd67a3d5f5 operations: add size if known to skipped items and JSON log - fixes #4624 2021-01-05 19:44:25 +00:00
Nick Craig-Wood
e972f2c98a log: make it easier to add parameters to JSON logging 2021-01-05 19:44:25 +00:00
Nick Craig-Wood
35da38e93f operations: fix --immutable error message 
This was accidentally changed in

53a1a0e3ef accounting: add reference to completed transfers
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood
963c0f28b9 sync: Only print "There was nothing to transfer" if no errors 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood
b3815dc0c2 sync: fix --immutable errors retrying many times 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood
8053fc4e16 fs: correct default implementation of fs.CountError 2020-12-31 18:16:51 +00:00
Nick Craig-Wood
f425950a52 fs: Always show stats when using --dry-run or --interactive #4624 2020-12-29 21:11:12 +00:00
Ingo Weiss
1d40bc1901 fs: Accumulate stats when using --dry-run 
Fixes #4624
 2020-12-29 21:11:12 +00:00
Benjamin Gustin
4d54454900
fs/log: don't compile systemd log integration for non unix systems 2020-12-28 23:07:12 +00:00
Nick Craig-Wood
060642ad14 flags: improve error message when reading environment vars #4888 
The message now includes the flag name to help the user work out what
is happening.

    Invalid value for environment variable "RCLONE_VERSION" when setting default
    for --version: strconv.ParseBool: parsing "yes": invalid syntax
 2020-12-28 12:26:23 +00:00
lostheli
c8cfa43ccc Add a download flag to hashsum and related commands to force rclone to download and hash files locally 
This commit modifies the operations.hashSum function by adding an alternate code path. This code path is triggered by passing downloadFlag = True. When activated, rclone will download files from the remote and hash them locally. downloadFlag = False preserves the existing behavior of using the remote to retrieve the hash.

This commit modifies HashLister to support the new hashSum method as well as consolidating the roles of HashLister, HashListerBase64, Md5sum, and Sha1sum.  The printing of hashes from the function defined in HashLister has been revised to work with --progress.  There are light changes to operations.syncFprintf and cmd.startProgress for this.

The unit test operations_test.TestHashSums is modified to support this change and test the download functionality.

The command functions hashsum, md5sum, sha1sum, and dbhashsum are modified to support this change.  A download flag has been added and an output-file flag has been added.  The output-file flag writes hashes to a file instead of stdout to avoid the need to redirect stdout.
 2020-12-27 15:40:44 +00:00
negative0
ed7af3f370 plugins: Create plugins files only if webui is enabled. Fixes #4592. May fix #4600. 2020-12-27 15:05:41 +00:00
Nathan Collins
be19d6a403 fshttp: prevent overlap of HTTP headers in logs 2020-12-27 12:44:46 +00:00
Nick Craig-Wood
ea8d13d841 fs: Fix parsing of .. when joining remotes - Fixes #4862 
Before this fix setting an alias of `s3:bucket` then using `alias:..`
would use the current working directory!

This fix corrects the path parsing. This parsing is also used in
wrapping backends like crypt, chunker, union etc.

It does not allow looking above the root of the alias, so `alias:..`
now lists `s3:bucket` as you might expect if you did `cd /` then
`ls ..`.
 2020-12-18 13:06:39 +00:00
Nick Craig-Wood
c98dd8755c log: fix enabling systemd logging when using --log-file 
This also moves all the systemd logging decisions to fs/log
 2020-12-17 11:55:27 +00:00
Nick Craig-Wood
86014cebd7 dedupe: add --dedupe-mode list to just list dupes, changing nothing 2020-12-02 16:52:12 +00:00
Nick Craig-Wood
507f861c67 dedupe: add --by-hash to dedupe on hash not file name - fixes #1674 2020-12-02 16:52:12 +00:00
zhucan
a223b78872 fs: support multi-threads to head dst object 
Signed-off-by: zhuc <zhucan.k8s@gmail.com>
 2020-12-02 16:26:37 +00:00
Nick Craig-Wood
584523672c dropbox: test file name length before upload to fix upload loop 
Before this change rclone would upload the whole of multipart files
before receiving a message from dropbox that the path was too long.

This change hard codes the 255 rune limit and checks that before
uploading any files.

Fixes #4805
 2020-12-01 17:56:36 +00:00
Nick Craig-Wood
47aada16a0 fs: add Shutdown optional method for backends 2020-11-27 17:35:01 +00:00
Nick Craig-Wood
c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood
354b4f19ec rc: add context to flag Reload function #4685 2020-11-27 17:28:42 +00:00
Maciej Zimnoch
70e8b11805
accounting: fix data race in Transferred() 
startedTransfers is accessed by multiple threads, and it wasn't
protected by the mutex call in Transferred() func.

Fixes #4799
 2020-11-26 22:54:38 +00:00
Nick Craig-Wood
9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Ankur Gupta
33ea55efed
fs: parseduration: fixed tests to use UTC time 2020-11-23 17:11:56 +00:00
buengese
2812816142 hash: add MultiHasher.Sum() to retrieve a single specific hash 2020-11-13 02:31:59 +01:00
Nick Craig-Wood
f7efce594b config: add context.Context #3257 #4685 
This add config to the Config callback in the backends and the related
config functions.
 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
e3fe31f7cb fs: add context.Context to fs.GetModifyWindow #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d69b96a94c test: Add context to mockfs.NewFs #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Anagh Kumar Baranwal
30c8b1b84f
fs: Fix nil pointer on copy & move operations directly to remote 
Fix the copy and move operations that broke in 127f0fc when copying directly
to a remote without a specific destination.

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-11-09 12:36:10 +00:00
Nick Craig-Wood
af55a74bd2 stats: add counter for deleted directories - fixes #4676 2020-11-03 11:47:00 +00:00
Nick Craig-Wood
d00c126cef operations: fix --cutof-mode hard not cutting off immediately 
This failure was noted on the integration tests server.

The fix was to be more careful about which error message was emitted
with which --cutoff-mode
 2020-11-02 17:13:19 +00:00
Adam Plánský
127f0fc64c operations: move and copy log name of the destination object in verbose 
If the object is moved or copied rclone in verbose mode prints name of the
destination object into the info log.
 2020-10-30 15:31:54 +00:00
Josh Soref
3e1cb8302a docs: spelling: etc. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Nick Craig-Wood
4e2b5389d7 check: make the error count match up in the log message 
Before this change we counted the final summary error as an error,
producing confusing log messages like:

    Failed to check with 54 errors: last error was: 53 differences found

This change marks the summary error as already being counted, so the
error message becomes:

    Failed to check with 53 errors: last error was: 53 differences found

This change also returns a listing failure in preference to a summary error.

See: https://forum.rclone.org/t/slow-checksum-validation/19763/22
 2020-10-15 12:57:48 +01:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood
51a230d7fd fs: Implement UnWrapObjectInfo for getting original object out of src objects 2020-10-14 15:20:06 +01:00
Nick Craig-Wood
72083c65ad cmd: make backend env vars show in help as the defaults for backend flags 
Before this change

    RCLONE_DRIVE_CHUNK_SIZE=111M rclone help flags | grep drive-chunk-size

Would show the default value, not the setting of RCLONE_DRIVE_CHUNK_SIZE
as the non backend flags do.

This change makes it work as expected by setting the default of the
option to the environment variable.

Fixes #4659
 2020-10-13 15:43:58 +01:00