Commit graph

129 commits

Author SHA1 Message Date
Nick Craig-Wood
9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
2020-11-26 16:40:12 +00:00
Nick Craig-Wood
1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
e3fe31f7cb fs: add context.Context to fs.GetModifyWindow #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs #3257 #4685
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
2020-11-09 18:05:54 +00:00
Anagh Kumar Baranwal
30c8b1b84f
fs: Fix nil pointer on copy & move operations directly to remote
Fix the copy and move operations that broke in 127f0fc when copying directly
to a remote without a specific destination.

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-11-09 12:36:10 +00:00
Nick Craig-Wood
af55a74bd2 stats: add counter for deleted directories - fixes #4676 2020-11-03 11:47:00 +00:00
Nick Craig-Wood
d00c126cef operations: fix --cutof-mode hard not cutting off immediately
This failure was noted on the integration tests server.

The fix was to be more careful about which error message was emitted
with which --cutoff-mode
2020-11-02 17:13:19 +00:00
Adam Plánský
127f0fc64c operations: move and copy log name of the destination object in verbose
If the object is moved or copied rclone in verbose mode prints name of the
destination object into the info log.
2020-10-30 15:31:54 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Nick Craig-Wood
20b00db390 operations: fix spurious "--checksum is in use but the source and destination have no hashes in common"
Before this change rclone would emit the message

    --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

When the source or destination hash was missing as well as when the
source and destination had no hashes in common.

This first case is very confusing for users when the source and
destination do have a hash in common.

This change fixes that and makes sure the error message is not emitted
on missing hashes even when there is a hash in common.

See: https://forum.rclone.org/t/source-and-destination-have-no-hashes-in-common-for-unencrypted-drive-to-local-sync/19531
2020-10-05 16:07:05 +01:00
Nick Craig-Wood
db4bbf9521 operations: fix use of --suffix without --backup-dir
As part of the original work adding this feature it was overlooked
that this didn't actually work for full rclone copy/sync.

This commit fixes the problem and adds a test to make sure it stays
working.

See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526
2020-10-04 16:40:20 +01:00
Nick Craig-Wood
2b7994e739 operations: add logs when need to upload files to set mod times #1505 2020-09-29 17:04:29 +01:00
Nick Craig-Wood
75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
2020-09-09 12:53:21 +01:00
Nick Craig-Wood
a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
2020-07-31 17:43:17 +01:00
Nick Craig-Wood
421585dd72 accounting: add context to Account and propagate changes #3257
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
2020-07-28 16:41:17 +01:00
Nick Craig-Wood
ff84351655 operations: factor Check and related functions into its own files 2020-07-21 22:08:13 +01:00
Nick Craig-Wood
8b6f2bbb4b check,cryptcheck: add reporting of filenames for same/missing/changed #3264
See: https://forum.rclone.org/t/rclone-check-v-doesnt-show-once-per-minute-update-counts/17402
2020-07-21 22:08:13 +01:00
Nick Craig-Wood
ce394426b0 check: fix successful retries with --download counting errors
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017
2020-06-20 14:52:04 +01:00
Nick Craig-Wood
40611fc4fc check: retry downloads if they fail when using the --download flag
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7
2020-06-18 16:16:19 +01:00
Nick Craig-Wood
7c4ba9fcb2 check: Fix misleading message which printed errors instead of differences
See: https://forum.rclone.org/t/tons-of-data-corruption-after-rclone-copy-to-mega-can-rclone-correct-it/17017/7
2020-06-18 16:16:19 +01:00
Nick Craig-Wood
3efdf5e095 operations: implement --refresh-times flag to set modtimes on hashless backends 2020-06-17 10:48:13 +01:00
Nick Craig-Wood
d55053098f check: make check do --checkers files concurrently - fixes #4318 2020-06-10 17:20:54 +01:00
Nick Craig-Wood
63cf0b1cdd check: make check command obey --dry-run/-i/--interactive - fixes #4325 2020-06-10 17:20:54 +01:00
Nick Craig-Wood
b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
2020-06-10 12:33:53 +01:00
fishbullet
ba5eb230fb operations: interactive mode -i/--interactive for destructive operations #3886 2020-06-10 12:33:53 +01:00
Roman Kredentser
55ad1354b6 link: Add --expire and --unlink flags
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
2020-06-05 14:51:05 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Nick Craig-Wood
1dced3b3c4 rcat: add support for --header-upload #59 2020-04-23 11:34:31 +01:00
Nick Craig-Wood
087bf1d584 cat: add support for --header-download #59 2020-04-23 11:34:24 +01:00
Tim Gallant
9bf3d3da4c fs: add UploadHeaders, DownloadHeaders to Update/Put/Open options 2020-04-23 11:07:21 +01:00
Nick Craig-Wood
7f15cc9556 operations: make ReOpen and NewReOpen public for re-use elsewhere 2020-04-19 15:18:49 +01:00
Denis
31a1cc46b7
copyurl: add no-clobber flag - fixes #3950 2020-04-19 12:40:17 +01:00
Nick Craig-Wood
b5f1bebc52 fs: add renames statistic for file and directory renames 2020-03-30 18:22:28 +01:00
Anuar Serdaliyev
f14871caf7
accounting: Correct exitcode on Transfer Limit Exceeded flag. Fixes #3203
Before this change the exit code for transfer limit exceeded was
incorrect. This was because the `resolveExitCode` function unwraps the
error thus reading the underlying error which is not the same as the
error it was comparing to (`ErrorMaxTransferLimitReached`).

This change fixes it by splitting the error definition in two so that
when the Fatal error is unwrapped we match against
`ErrorMaxTransferLimitReached` however when we return the error we
return `ErrorMaxTransferLimitReachedFatal`.
2020-03-10 12:00:10 +00:00
Franklyn Tackitt
2b3d13a841 fs: Use --cutoff-mode hard,soft,catious instead of 3 --max-transfer-mode flags
Fixes #2672
2020-03-01 09:49:55 +00:00
Shing Kit Chan
6f1766dd9e fs: Add support for --max-transfer-cutoff modes #2672
This also adds max transfer cut off check for server side copies too
2020-03-01 09:49:55 +00:00
Nick Craig-Wood
3893c14889 operations: make rcat obey --ignore-checksum 2020-02-14 12:47:11 +00:00
Nick Craig-Wood
f45425e5a9 operations: factor CommonHash out of Copy for re-use elsewhere 2020-02-14 12:12:10 +00:00
Tim Gallant
1d73f071f6 fs: improve log output when no changes are made - fixes #3454
- changes a few log messages to debug level
- adds a log message for when 0 bytes are transferred
2020-02-11 12:16:15 +00:00
landall
77e55b8265 hashsum: Add flag --base64 flag - fixes #3663
This flag can be used to output QuickXorHash in the same format as MS
Graph API.
2020-01-20 17:29:58 +00:00
Nick Craig-Wood
ef7ac088c0 operations: make NewOverrideObjectInfo public and factor uses 2020-01-18 11:41:33 +00:00
Nick Craig-Wood
422ad38e5b copyurl: add --stdout flag to write to stdout 2020-01-18 11:15:51 +00:00
Ole Schütt
f263828edc operations: write debug message when hashes could not be checked 2020-01-09 10:35:31 +00:00
Maciej Zimnoch
b7019a91c2 fs/operations: Clear accounting before low level retry
Statistics of ransfers which were interrupted are not cleared before
retry iteration. These transfers completed with over 100 percentage.

This change clears transfer accounting before next retry iteration is
done in order to keep numbers in track.

Fixes #3861
2020-01-09 10:32:49 +00:00
Nick Craig-Wood
207474abab sync: add --no-check-dest flag - fixes #3616 2019-12-29 16:47:57 +00:00
Nick Craig-Wood
50bb9b7bdd check: fix --one-way recursing more directories than it needs to
Before this change rclone traversed all directories in the destination.

After this change rclone doesn't traverse directories in the
destination that don't exist in the source if the `--one-way` flag is
set.

See: https://forum.rclone.org/t/check-with-one-way-flag-should-not-traverses-all-destination-directories/13263
2019-12-07 13:26:55 +00:00
Ankur Gupta
75a6c49f87 Fix error counter - fixes #3650
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
2019-11-18 14:13:02 +00:00
Nick Craig-Wood
f3b0f8a9f0 sync: --update/-u not transfer files that haven't changed - fixes #3232
Before this change --update would transfer any file which was newer
than the destination regardless of whether it had changed or not.
This is needlessly wasteful of bandwidth.

After this change --update will only transfer files if they are newer
**and** they are different (checked with checksum and size).
2019-10-12 11:54:56 +01:00
Vighnesh SK
ec09de1628 Change the Debug message in NeedTranser (#3608)
'Couldn't find file - Need to Transfer' changed to 'Need to transfer -
File Not Found at Destination' because while reading the debug logs, it
confuses with failure in operation.
2019-10-10 13:44:05 +01:00