Nick Craig-Wood
1557287c64
fs: Make Option.GetValue() public #2541
2018-10-06 11:47:45 +01:00
Nick Craig-Wood
e7e467fb3a
cmd: factor FlagName into fs.Option #2541
2018-10-06 11:47:45 +01:00
Nick Craig-Wood
59e14c25df
vfs: enable rename for nearly all remotes using server side Move or Copy
...
Before this change remotes without server side Move (eg swift, s3,
gcs) would not be able to rename files.
After it means nearly all remotes will be able to rename files on
rclone mount with the notable exceptions of b2 and yandex.
This changes checks to see if the remote can do Move or Copy then
calls `operations.Move` to do the actual move. This will do a server
side Move or Copy but won't download and re-upload the file.
It also checks to see if the destination exists first which avoids
conflicts or duplicates.
Fixes #1965
Fixes #2569
2018-09-29 14:56:20 +01:00
Fabian Möller
1eec59e091
fs: update ChangeNotifier interface
...
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
2018-09-25 14:01:13 +02:00
ssaqua
ae63e4b4f0
list: change debug logs for excluded items
2018-09-24 17:08:35 +01:00
frenos
382a6863b5
rc: add support for OPTIONS and basic CORS - #2575
2018-09-24 17:04:47 +01:00
sandeepkru
e0c5f7ff1b
fs - Remove unreferenced ListTierer optional interface
2018-09-18 13:56:09 +01:00
Joanna Marek
e022ffce0f
accounting: change too long names cutting mechanism - fixes #2490
2018-09-18 10:27:23 +01:00
sandeepkru
7accd30da8
cmd and fs: Added new command settier which performs storage tier changes on
...
supported remotes
2018-09-12 21:09:08 +01:00
sandeepkru
5716a58413
fs: Added new optional interfaces SetTierer, GetTierer and ListTierer, these are to
...
perform object tier changes on supported remotes
2018-09-12 21:09:08 +01:00
Cédric Connes
1b95ca2852
stats: handle FatalError and NoRetryError when reported to stats
2018-09-07 14:44:50 +01:00
albertony
09d7bd2d40
config: don't create default config dir when user supplies --config
...
Avoid creating empty default configuration directory when user supplies path to config file.
Fixes #2514 .
2018-09-03 15:30:53 +01:00
dcpu
f4aaec9ce5
log: Add --log-format flag - fixes #2424
2018-09-02 18:11:09 +01:00
Nick Craig-Wood
543d75723b
Start v1.43-DEV development
2018-09-01 18:37:48 +01:00
Nick Craig-Wood
20c55a6829
Version v1.43
2018-09-01 12:58:00 +01:00
Nick Craig-Wood
7d74686698
fs/accounting: increase maximum burst size of token bucket
...
This stops occasional errors when using --bwlimit which look like this
Token bucket error: rate: Wait(n=2255475) exceeds limiter's burst 2097152
2018-08-30 17:24:08 +01:00
Denis
1c578ced1c
cmd: add copyurl command - Fixes #1320
2018-08-30 16:45:41 +01:00
Nick Craig-Wood
de6ec8056f
fs/accounting: fix moving average speed for file stats
...
Before this change the moving average for the individual file stats
would start at 0 and only converge to the correct value over 15-30
seconds.
This change starts the weighting period as 1 and moves it up once per
sample which gets the average to a better value instantly.
2018-08-28 22:55:51 +01:00
Nick Craig-Wood
66fe4a2523
fs/accounting: fix stats display which was missing transferring data
...
Before this change the total stats were ignoring the in flight
checking, transferring and bytes.
2018-08-28 22:21:17 +01:00
Nick Craig-Wood
f5617dadf3
fs/accounting: factor out eta and percent calculations and write tests
2018-08-28 22:21:17 +01:00
Nick Craig-Wood
ca44fb1fba
accounting: fix time to completion estimates
...
Previous to this change package used for this
github.com/VividCortex/ewma took a 0 average to mean reset the
statistics. This happens quite often when transferring files though a
buffer.
Replace that implementation with a simple home grown one (with about
the same constant), without that feature.
2018-08-26 22:00:48 +01:00
Nick Craig-Wood
94950258a4
fs: allow backends to be named using their Name or Prefix #2449
...
This means that, for example Google Cloud Storage can be known as
`:gcs:bucket` on the command line, as well as `:google cloud
storage:bucket`.
2018-08-26 17:59:31 +01:00
Nick Craig-Wood
8656bd2bb0
fs: Allow on the fly remotes with :backend: syntax - fixes #2449
...
This change allows remotes to be created on the fly without a config
file by using the remote type prefixed with a : as the remote name, Eg
:s3: to make an s3 remote.
This assumes the user is supplying the backend config via command line
flags or environment variables.
2018-08-26 17:59:31 +01:00
Nick Craig-Wood
b6db90cc32
cmd: add --progress/-P flag to show progress
...
Fixes #2347
Fixes #1210
2018-08-26 17:20:38 +01:00
Nick Craig-Wood
9f671c5dd0
fs: fix tests for *SepList
2018-08-21 10:58:59 +01:00
Alex Chen
c6c74cb869
mountlib: fix mount --daemon not working with encrypted config - fixes #2473
...
This passes the configKey to the child process as an Obscured temporary file with an environment variable to the
2018-08-21 09:41:16 +01:00
Nick Craig-Wood
1b8f4b616c
fs: move CommaSepList and SpaceSepList here from config
...
fs can't import config so having them there means they are not usable
by rclone core.
2018-08-20 17:52:05 +01:00
Fabian Möller
f818df52b8
config: add List type
2018-08-20 17:38:51 +01:00
Nick Craig-Wood
7712a0e111
fs/asyncreader: skip some tests to work around race detector bug
...
The race detector currently detects a race with len(chan) against
close(chan).
See: https://github.com/golang/go/issues/27070
Skip the tests which trip this bug under the race detector.
2018-08-20 12:34:29 +01:00
Fabian Möller
6dc5aa7454
docs: clearify buffer-size is per transfer/filehandle
2018-08-17 18:11:40 +01:00
Fabian Möller
552eb8e06b
vfs: try to seek buffer on read only files
2018-08-17 18:10:28 +01:00
Nick Craig-Wood
6390dec7db
fs/accounting: add --stats-one-line flag for single line stats
2018-08-17 17:58:00 +01:00
Nick Craig-Wood
80a3db34a8
fs/accounting: show the total progress of the sync in the stats #379
2018-08-17 17:58:00 +01:00
Nick Craig-Wood
cb7a461287
sync: add a buffer for checks, uploads and renames #379
...
--max-backlog controls the queue length.
Add statistics for the check/upload/rename queues.
This means that checking can complete before the uploads which will
give rclone the ability to show exactly what is outstanding.
2018-08-17 17:58:00 +01:00
Andres Alvarez
990919f268
Add disclaimer about generated passwords being stored in an obscured format
2018-08-11 15:07:50 +01:00
Sebastian Bünger
dd3e912731
fs/OpenOptions: Make FixRangeOption clamp range to filesize.
2018-08-10 11:14:34 +01:00
Nick Craig-Wood
e979cd62c1
rc: fix formatting in docs
2018-08-07 21:05:21 +01:00
reddi1
6bfdabab6d
rc: added core/stats to return the stats - fixes #2405
2018-08-07 20:56:40 +01:00
Andres Alvarez
c1f5add049
Add tests for reveal functions
2018-08-07 20:51:50 +01:00
Oleg Kovalov
06c9f76cd2
all: fix go-critic linter suggestions
2018-08-06 21:14:03 +01:00
Fabian Möller
9815b09d90
fs: add multipliers for SizeSuffix
2018-07-29 18:17:05 +01:00
Fabian Möller
9c90b5e77c
stats: use appropriate Lock func's
2018-07-22 11:33:19 +02:00
Nick Craig-Wood
ab78eb13e4
sync: correct help for --delete-during and --delete-after
2018-07-18 19:30:14 +01:00
ishuah
dcc74fa404
move: fix delete-empty-src-dirs flag to delete all empty dirs on move - fixes #2372
2018-07-17 10:34:34 +01:00
Nick Craig-Wood
4d7d240c12
config: Add advanced section to the config editor
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
d046402d80
config: Make sure Required values are entered
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
9bdf465c10
config: make config wizard understand types and defaults
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
f3f48d7d49
Implement new backend config system
...
This unifies the 3 methods of reading config
* command line
* environment variable
* config file
And allows them all to be configured in all places. This is done by
making the []fs.Option in the backend registration be the master
source of what the backend options are.
The backend changes are:
* Use the new configmap.Mapper parameter
* Use configstruct to parse it into an Options struct
* Add all config to []fs.Option including defaults and help
* Remove all uses of pflag
* Remove all uses of config.FileGet
2018-07-16 21:20:47 +01:00
Nick Craig-Wood
3c89406886
config: Make fs.ConfigFileGet return an exists flag
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
85d09729f2
fs: factor OptionToEnv and ConfigToEnv into fs
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
b3bd2d1c9e
config: add configstruct parser to parse maps into config structures
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
4c586a9264
config: add configmap package to manage config in a generic way
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
1c80e84f8a
fs: Implement Scan method for SizeSuffix and Duration
2018-07-16 08:50:52 +01:00
Nick Craig-Wood
dbb4b2c900
fs/config: don't print errors about --config if supplied - fixes #2397
...
Before this change if the rclone was running in an environment which
couldn't find the HOME directory, it would print a warning about
supplying a --config flag even if the user had done so.
2018-07-15 12:39:11 +01:00
Nick Craig-Wood
a3d9a38f51
fs/fserrors: make sure Cause never returns nil
2018-07-13 10:31:40 +01:00
Nick Craig-Wood
144c1a04d4
fs: Fix parsing of paths under Windows - fixes #2353
...
Before this copyto would parse windows paths incorrectly.
This change moves the parsing code into fspath and makes sure
fspath.Split calls fspath.Parse which does the parsing correctly for
This also renames fspath.RemoteParse to fspath.Parse for consistency
2018-07-06 23:16:43 +01:00
Nick Craig-Wood
8c0335a176
build: fix for goimports format change
...
See https://github.com/golang/go/issues/23709
2018-07-03 22:33:15 +01:00
Nick Craig-Wood
947e10eb2b
config: fix error reading password from piped input - fixes #1308
2018-06-28 11:54:15 +01:00
Nick Craig-Wood
935533e57f
filter: raise --include and --exclude warning to ERROR so it appears without -v
2018-06-22 22:18:55 +01:00
Mateusz
56e1e82005
fs: added weekday schedule into --bwlimit - fixes #1822
2018-06-17 18:38:09 +01:00
Nick Craig-Wood
7110349547
Start v1.42-DEV development
2018-06-16 21:25:58 +01:00
Nick Craig-Wood
a9adb43896
Version v1.42
2018-06-16 18:21:09 +01:00
Nick Craig-Wood
1c75581959
sync: fix TestCopyRedownload after ModifyWindow changes #2310
2018-06-10 17:34:00 +01:00
Filip Bartodziej
ce109ed9c0
log: password prompt output fixed for unix - partially fixes #2220
2018-06-10 12:57:45 +01:00
Nick Craig-Wood
ba43acb6aa
sync: fix TestCopyEmptyDirectories after ModifyWindow changes #2310
2018-06-04 21:41:25 +01:00
ishuah
d758e1908e
copy: create (pseudo copy) empty source directories to destination - fixes #1837
2018-06-04 11:01:14 +01:00
ishuah
737aed8412
Ensure items in srcEmptyDirs are actually empty
2018-06-04 11:01:14 +01:00
Stefan
4009fb67c8
fs: calculate ModifyWindow each time on the fly instead of relying on global state - see #2319 , #2328
2018-06-03 20:45:34 +02:00
Nick Craig-Wood
3ef938ebde
lsf: add --absolute flag to add a leading / onto path names
2018-06-03 10:42:34 +01:00
Kasper Byrdal Nielsen
aad75e6720
check: Add one-way argument
...
--one-way argument will check that all files on source matches the files on detination,
but not the other way. For example files present on destination but not on source will not
trigger an error.
Fixes : #1526
2018-05-31 09:42:16 +01:00
Nick Craig-Wood
3a2248aa5f
rc: add core/gc to run a garbage collection on demand
2018-05-24 15:10:28 +01:00
Nick Craig-Wood
573ef4c8ee
rc: enable go profiling by default on the --rc port
...
This means you can use the pprof tool on a running rclone, eg
go tool pprof http://localhost:5572/debug/pprof/heap
2018-05-24 15:10:28 +01:00
Nick Craig-Wood
512f4b4487
Update error checking on fmt.Fprint* after errcheck update
...
Now we need to check or ignore errors on fmt.Fprint* explicitly -
previously errcheck just ignored them for us.
2018-05-22 09:41:13 +01:00
Nick Craig-Wood
8e625e0bc3
config: add ConfirmWithDefault to change the default on AutoConfig
2018-05-16 09:09:41 +01:00
Nick Craig-Wood
e56be0dfd8
lsf: Add --csv flag for compliant CSV output
2018-05-13 12:18:21 +01:00
Nick Craig-Wood
2a32e2d838
operations: turn ListFormatted into a Format method on ListFormat
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
f77efc7649
lsf: Add 'm' format specifier to show the MimeType
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
aadbcce486
fs: Add MimeTypeDirEntry to return the MimeType of a DirEntry
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
909c3a92d6
lsf: implement 'i' format for showing object ID - fixes #1476
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
826975c341
fs: add Optional ID() method to Object and implement it in backends
...
ID() shows the internal ID of the Object if available.
2018-05-13 12:17:55 +01:00
Nick Craig-Wood
e80ae4e09c
build: remove unused struct fields spotted by structcheck
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
1320e84bc2
build: remove unused code spotted by the deadcode linter
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
cb5bd47e61
build: fix errors spotted by ineffassign linter
...
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Nick Craig-Wood
f95835d613
fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors.
...
Before this change fatal errors which were wrapped in a system error (eg a
URLError) were not recognised as fatal errors.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
be79b47a7a
sync: log when we abandon the sync due to a fatal error
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
be22735609
fs/accounting: fix deadlock on GetBytes
...
A deadlock could occur since we have now put a mutex on GetBytes from
StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu)
- GetBytes (s.mu).
Fix this by giving stringSet its own locking and excluding the call
which caused the deadlock from the mutex in StatsInfo.String.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
1b1b3c13cd
sync: add a test for aborting on max upload
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
5c128272fd
Implement --max-transfer flag to quit transferring at a limit #1655
2018-05-05 12:58:28 +01:00
Nick Craig-Wood
d178233e74
sync,march: check the cancel context on every channel send and receive
...
This fixes a deadlock on sync when all the copying channels receive a
Fatal Error.
2018-05-05 12:58:28 +01:00
Fabian Möller
9fdf273614
fs: improve ChunkedReader
...
- make Close permanent and return errors afterwards
- use RangeSeek from the wrapped reader if present
- add a limit to chunk growth
- correct RangeSeek interface behavior
- add tests
2018-05-05 12:49:42 +01:00
Nick Craig-Wood
55674c0bfc
Start v1.41-DEV development
2018-04-28 12:37:55 +01:00
Nick Craig-Wood
e4c380b2a8
Version v1.41
2018-04-28 11:46:27 +01:00
Nick Craig-Wood
74cbdea0ef
Revert "copy: create (pseudo copy) empty source directories to destination"
...
Unfortunately this commit attempts to create every directory rather
than just the empty ones, so will need re-working.
Removing this feature for the 1.41 release
This reverts commit 0daced29db
2018-04-28 10:02:32 +01:00
ishuah
0daced29db
copy: create (pseudo copy) empty source directories to destination - fixes #1837
2018-04-27 16:15:32 +01:00
Richard Yang
a81ec00a8c
dedupe: Add dedupe largest functionality - fixes #2269
2018-04-26 16:21:07 +01:00
Nick Craig-Wood
9f04ce282e
rc: fix setting bwlimit to unlimited
2018-04-26 12:21:29 +01:00
Nick Craig-Wood
764440068e
filter: fix --min-age and --max-age together check
...
Somehow in the code reorganisation of
11da2a6c9b
2018-04-26 09:17:22 +01:00
Nick Craig-Wood
a703216286
filter: take double negatives out of filter flag help
2018-04-26 09:17:13 +01:00
Nick Craig-Wood
9ab2521ef2
rc: autogenerate and tidy the docs and commands
...
* Rename rc/pid -> core/pid
* Sort the output of `rc list`
* Make a script to autogenerate the docs
* Tidy docs
2018-04-23 20:57:17 +01:00
Nick Craig-Wood
21a10e58c9
rc: implement core/memstats to print internal memory usage info
2018-04-23 20:49:36 +01:00
