TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions 2
5553 commits 181 branches 114 tags 335 MiB
Commit graph

127 commits

Author SHA1 Message Date
Chris Macklin
35a86193b7 accounting: deglobalize startTime/elapsedTime - fixes #5282 2021-05-01 14:51:21 +01:00
albertony
2925e1384c Use binary prefixes for size and rate units 
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
 2021-04-27 02:25:52 +03:00
albertony
2ec0c8d45f stats: correct spelling of data rate units 2021-04-27 02:25:52 +03:00
albertony
98579608ec docs: cleanup spelling of size and rate units 2021-04-27 02:25:52 +03:00
Nick Craig-Wood
268a7ff7b8 rc: add a full set of stats to core/stats 
This patch adds the missing stats to the output of core/stats

- totalChecks
- totalTransfers
- totalBytes
- eta

This now includes enough information to rebuild the normal stats
output from rclone including percentage completions and ETAs.

Fixes #5116
 2021-03-22 10:10:36 +00:00
albertony
97fc3b9046
rc: avoid +Inf value for speed in core/stats (#5134) 
Fixes #5132
 2021-03-18 10:02:30 +01:00
Nick Craig-Wood
a7fd65bf2d config: move account initialisation out into accounting 
Before this change the initialisation for the accounting package was
done in the config package for some strange historical reason.
 2021-03-11 17:29:26 +00:00
Nick Craig-Wood
2aebeb6061 accounting: fix --bwlimit when up or down is off - fixes #5019 
Before this change the core bandwidth limit was limited to upload or
download value if the other value was off.

This fix only applies a core bandwidth limit when both values are set.
 2021-02-13 12:45:12 +00:00
Nick Craig-Wood
cdd99a6f39 fs/accounting: fix occasionally failing test on macOS 2021-02-07 14:59:53 +00:00
Nick Craig-Wood
0d8bcc08da Revert "fs/accounting: make edge bandwidth limiters have smaller bursts to make smoother" #4987 
This reverts commit 463a18aa07.

In practice this caused Windows to have a max bandwidth limit of 8
MiB/s. This is because of the limited resolution (1ms) of the Windows
timer.

This needs fixing in a different way.

See: https://forum.rclone.org/t/bug-report-for-v1-54-0-beta-5133/22015
 2021-02-01 22:23:50 +00:00
Nick Craig-Wood
463a18aa07 fs/accounting: make edge bandwidth limiters have smaller bursts to make smoother 
This change decreases the edge limiter burst size which dramatically
increases the smoothness of the bandwidth limiting.

The core bandwidth limiter remains with a large burst so it isn't
affected by double rate limiting on the edge limiters.

See: #4395
See: https://forum.rclone.org/t/bwlimit-is-not-really-smooth/20947
 2021-01-30 17:39:30 +00:00
Nick Craig-Wood
0a932dc1f2 Add --bwlimit for upload and download #1873 2021-01-30 17:39:30 +00:00
Nick Craig-Wood
8856e0e559 fshttp: Obey bwlimit in http Transport - fixes #4395 
This change uses the bwlimit code to apply limits to the receive and
transmit data functions in the HTTP Transport.

This means that all HTTP transactions will have limiting applied -
this includes listings for example.

For HTTP based transorts this makes the limiting in Accounting
redundant and possibly counter productive
 2021-01-30 17:39:30 +00:00
Nick Craig-Wood
3b6df71838 accounting: refactor bwlimit code to allow for multiple slots 2021-01-30 17:39:30 +00:00
Nick Craig-Wood
50344e7792 accounting: factor --tpslimit code into accounting from fshttp 2021-01-08 10:29:57 +00:00
Nick Craig-Wood
e972f2c98a log: make it easier to add parameters to JSON logging 2021-01-05 19:44:25 +00:00
Ingo Weiss
1d40bc1901 fs: Accumulate stats when using --dry-run 
Fixes #4624
 2020-12-29 21:11:12 +00:00
Maciej Zimnoch
70e8b11805
accounting: fix data race in Transferred() 
startedTransfers is accessed by multiple threads, and it wasn't
protected by the mutex call in Transferred() func.

Fixes #4799
 2020-11-26 22:54:38 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
af55a74bd2 stats: add counter for deleted directories - fixes #4676 2020-11-03 11:47:00 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
LaSombra
aac2406e19
cmd: add --progress-terminal-title to print ETA to terminal title 
Adds a flag, --progress-terminal-title, that when used with --progress,
will print the string `ETA: %s` to the terminal title.

This also adds WriteTerminalTitle to lib/terminal
 2020-10-06 16:34:26 +01:00
Nick Craig-Wood
433b73a5a8 accounting: stabilize display order of transfers on Windows 
Before this change we sorted transfers in the stats list solely on
time started. However if --check-first was in use then lots of
transfers could be started in the same millisecond. Because Windows
time resolution is only 1mS this caused the entries to sort equal and
bounce around in the list.

This change fixes the sort so that if the time is equal it uses the
name which should stabilize the order.

Fixes #4599
 2020-09-24 19:10:37 +01:00
Nick Craig-Wood
5c49096e11 acounting: fix incorrect speed and transferTime in core/stats 
Before this change the code which summed up the existing transfers
over all the stats groups forgot to add the old transfer time and old
transfers in.

This meant that the speed and elapsedTime got increasingly inaccurate
over time due to the transfers being culled from the list but their
time not being accounted for.

This change adds the old transfers into the sum which fixes the
problem.

This was only a problem over the rc.

Fixes #4569
 2020-09-15 12:01:18 +01:00
Nick Craig-Wood
75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early 
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
 2020-09-09 12:53:21 +01:00
Nick Craig-Wood
05bea46c3e accounting: remove new line from end of --stats-one-line display 2020-09-08 16:10:52 +01:00
Nick Craig-Wood
f5439ddc54 accounting: fix deadlock in stats printing 
The deadlock was caused in transfermap.go by calling mu.RLock() in one
function then calling it again in a sub function. Normally this is
fine, however this leaves a window where mu.Lock() can be called. When
mu.Lock() is called it doesn't allow the second mu.RLock() and
deadlocks.

    Thead 1                    Thread 2
    String():mu.RLock()
                               del():mu.Lock()
    sortedSlice():mu.RLock()                     - DEADLOCK

Lesson learnt: don't try using locks recursively ever!

This patch fixes the problem by removing the second mu.RLock(). This
was done by factoring the code that was calling it into the
transfermap.go file so all the locking can be seen at once which was
ultimately the cause of the problem - the code which used the locks
was too far away from the rest of the code using the lock.

This problem was introduced in:

bfa5715017 fs/accounting: sort transfers by start time

Which hasn't been released in a stable version yet
 2020-08-05 17:13:00 +01:00
Nick Craig-Wood
122a47fba6 accounting: Allow transfers to be canceled with context #3257 
This makes all transfers cancelable even if the backend doesn't
support context as all transfers are done using the Accounting
framework.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood
421585dd72 accounting: add context to Account and propagate changes #3257 
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood
26001d520a fs: add --bwlimit-file flag to limit speeds of individual file transfers 2020-07-28 11:46:24 +01:00
Chaitanya Bankanhal
8913679d88
accounting: Fix elapsed time not show actual time since beginning 
This fixes the elapsed time display in the statistics output in the rc and in the log messages.
 2020-07-26 11:59:50 +01:00
Nick Craig-Wood
bcd362fcd5 accounting: fix documentation for speed/speedAvg 
Fix the documentation for the very confusingly names speed and
speedAvg stats items.

See: https://github.com/rclone/rclone-webui-react/issues/99
 2020-07-17 15:35:15 +01:00
Max Sum
bfa5715017 fs/accounting: sort transfers by start time 2020-07-07 15:42:06 +01:00
Max Sum
e2183ad661 fs/accounting: use transferMap instead of stringSet 2020-07-07 15:42:06 +01:00
Nick Craig-Wood
05ddef117a accounting: add HasBuffer method to Account 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
b58bb03e95 test: Don't run unreliable tests on CI #4171 2020-06-15 21:34:37 +01:00
Gary Kim
763944f673
rcd: fix incorrect prometheus metrics - fixes #4341 
This was caused by using the stats group from the context passed in by the rcd
rather than the global stats group.

Signed-off-by: Gary Kim <gary@garykim.dev>
 2020-06-14 10:09:24 +01:00
Nick Craig-Wood
3c14a893fb asyncreader: Make StopBuffer as well as Abandon and fix confusion in callers 2020-06-09 18:05:12 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Martin Michlmayr
fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
Nick Craig-Wood
177195aeeb accounting: fix race clearing stats 
This race was introduced by

10a6a92e52 accounting: reset bytes read during copy retry
 2020-05-12 17:02:32 +01:00
Nick Craig-Wood
cb5979a468 accounting: factor stats into its own structure 
This makes it very obvious which mutex to take for accessing the
values.
 2020-05-12 17:02:32 +01:00
Ankur Gupta
10a6a92e52 accounting: reset bytes read during copy retry - fixes #4178 
During a copy/sync command, if an operation fails due to a network
issue and is retried, the underlying io.Reader is re-initialised,
but the stats for bytes already read are not reset, leading to incorrect
stats. THis was fixed by resetting the bytes read when an Account is
re-initialized.
 2020-05-10 17:58:22 +00:00
Nick Craig-Wood
8bf6ab2c52 accounting: fix race condition in tests 2020-04-24 12:32:09 +01:00
Nick Craig-Wood
f5455d865b accounting: check for max transfer in WriteTo 
Before this change the max transfer tests were failing for remotes
which were using WriterTo.
 2020-04-23 11:13:13 +01:00
Nick Craig-Wood
fdada79ebf accounting: support WriterTo for less memory copying 
This should reduce memory copying when the async buffer is in use and
improve speeds.
 2020-04-19 15:18:49 +01:00
Nick Craig-Wood
424554bc85 fs: generalise machinery for putting extra values when using --use-json-log 2020-04-11 18:16:21 +01:00