Statistics of ransfers which were interrupted are not cleared before
retry iteration. These transfers completed with over 100 percentage.
This change clears transfer accounting before next retry iteration is
done in order to keep numbers in track.
Fixes#3861
Without the fix we can have a race, example:
```
Write at 0x00c000432039 by goroutine 187:
github.com/rclone/rclone/fs/accounting.(*StatsInfo).Error()
fs/accounting/stats.go:495 +0x3f1
github.com/rclone/rclone/fs/accounting.(*StatsInfo).Error-fm()
fs/accounting/stats.go:477 +0x55
github.com/rclone/rclone/fs/walk.listRwalk.func1()
fs/walk/walk.go:162 +0xd2
github.com/rclone/rclone/fs/walk.walk.func2()
fs/walk/walk.go:402 +0x30f
Previous read at 0x00c000432039 by goroutine 184:
github.com/rclone/rclone/fs/accounting.(*statsGroups).sum()
fs/accounting/stats_groups.go:351 +0xcae
github.com/rclone/rclone/fs/accounting.rcTransferredStats()
fs/accounting/stats_groups.go:132 +0x1f4
```
Fixes#3844
Before this change if an operation was retried on operations.Copy and
the operation was large enough to use an async buffer then an async
buffer was leaked on the retry. This leaked memory, a file handle and
a go routine.
After this change if Account.WithBuffer is called and there is already
a buffer, then a new one won't be allocated.
rclone library users might be intrested in changing default value to
other, or even disabling it. With current version it's impossible which
leads to races when number of uploaded objects exceeds default limit.
Fixes#3732
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
from rsync manual:
--compare-dest=DIR
This option instructs rsync to use DIR on the destination machine as an
additional hierarchy to compare destination files against doing transfers
(if the files are missing in the destination directory). If a file is found
in DIR that is identical to the sender's file, the file will NOT be
transferred to the destination directory. This is useful for creating
a sparse backup of just files that have changed from an earlier backup.
--copy-dest=DIR
This option behaves like --compare-dest, but rsync will also copy unchanged
files found in DIR to the destination directory using a local copy.
This is useful for doing transfers to a new destination while leaving
existing files intact, and then doing a flash-cutover when all files
have been successfully transferred.
On google fs (drive, google photos, and google cloud storage), if
headless is selected, do not open browser.
This also supplies a new option "auth-no-open-browser" for authorize
if the user does not want it.
This should fix#3323.
This was broken in e337cae0c5 when we deleted the transfers
immediately.
This is fixed by keeping a merged slice of time ranges of completed
transfers and adding those to the current transfers.
In a28239f005 we made --files-from obey --no-traverse. In the
process this caused --files-from without --no-traverse to do a
complete recursive scan unecessarily.
This was only noticeable in users of fs/march, so sync/copy/move/etc
not in ls/lsf/etc.
This fix makes sure that we use conventional directory listings in
fs/march unless `--files-from` and `--no-traverse` is set or
`--fast-list` is active.
Fixes#3619
It was reported that v1.49.4 which was accidentally compiled with
go1.13 instead of go1.12 produced errors like this:
Failed to get StartPageToken: Get https://www.googleapis.com/drive/v3/changes/startPageToken?XXX: stream error: stream ID 1789; INTERNAL_ERROR
IO error: open file failed: Get https://www.googleapis.com/drive/v3/files/XXX?alt=media: stream error: stream ID 1781; INTERNAL_ERROR
These are errors from the http2 library. It appears that go1.13 when
communicating with google drive defaults to http2 whereas with go1.12
it doesn't.
It is unclear what is causing these errors, but retrying them since
they don't happen very often seems like a valid strategy.
This was fixed in v1.49.5 by compiling with go1.12 - this fix is
designed to work with go1.13
See: https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108/
Before this fix, attempting to set a non top level environment
variable would fail with "Couldn't find flag".
This fixes it by passing in the flags that the env var is being set
from.
Fixes#3615
Before this change --update would transfer any file which was newer
than the destination regardless of whether it had changed or not.
This is needlessly wasteful of bandwidth.
After this change --update will only transfer files if they are newer
**and** they are different (checked with checksum and size).
'Couldn't find file - Need to Transfer' changed to 'Need to transfer -
File Not Found at Destination' because while reading the debug logs, it
confuses with failure in operation.
