Nick Craig-Wood
e08e35984c
Add help to remote chooser in rclone config - fixes #43
2016-02-21 13:40:16 +00:00
Nick Craig-Wood
700e47d6e2
Stub out ReadPassword on plan9 and solaris to fix compilation
2016-02-21 10:31:53 +00:00
klauspost
ef06371c93
Create separate interface for object information.
...
Take out read-only information about a Fs in a separate struct to limit access.
See discussion at #282 .
2016-02-19 13:31:09 +00:00
klauspost
84b00b362f
Change back to original goconfig package.
...
Add documentation for `--ask-password`.
2016-02-17 11:45:05 +01:00
klauspost
bfd7601cf9
Add configuration file encryption
...
See #317 for details.
Use `rclone config` to add/change/remove password.
Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.
Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
2016-02-16 16:32:05 +01:00
Nick Craig-Wood
a6320bbad3
Fix delete command to wait until all finished - fixes missing deletes.
...
This also could affect deletes at the end of the sync command.
2016-02-15 16:43:59 +00:00
Nick Craig-Wood
85d99de26b
Fix typo in error strings
2016-02-09 16:15:50 +00:00
Nick Craig-Wood
e62c0a58a7
Version 1.27
2016-01-31 17:50:13 +00:00
Nick Craig-Wood
0f73129ab7
dedupe command to deduplicate a remote. Useful with google drive - fixes #41
2016-01-31 16:09:42 +00:00
Nick Craig-Wood
1373efaa39
Delete command which does obey the filters - fixes #327
2016-01-31 16:06:04 +00:00
Nick Craig-Wood
5c37b777fc
Make the --dry-run warnings into logs so they appear without the -v flag
2016-01-31 16:06:04 +00:00
Nick Craig-Wood
8c211fc8df
Warn the user about files with same name but different case
...
Relates to #107 & #119 .
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
74a71f7824
Add tests for --delete-before, --delete-during and --delete-after
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
12b51c5eb8
Remove duplicate check for filter IncludeObject
2016-01-26 16:57:09 +00:00
klauspost
14069fd8e6
Implement --delete-before, --delete-during, --delete-after - fixes #252 .
2016-01-26 16:57:09 +00:00
Nick Craig-Wood
cd62f41606
Reduce number of logs and show hash type where appropriate
2016-01-24 18:06:57 +00:00
Nick Craig-Wood
109d4ee490
Prefix all test remotes with rclone-test-
and make names more pronouncable
2016-01-24 12:37:46 +00:00
Nick Craig-Wood
18ebec8276
Check remote is empty between integration tests
2016-01-24 12:37:19 +00:00
Nick Craig-Wood
eda4130703
Fix integration tests so they can be run independently and out of order - fixes #291
...
* Make all integration tests start with an empty remote
* Add an -individual flag so this can be a different bucket/container/directory
* Fix up tests after changing the hashers
* Add sha1sum test
* Make directory checking in tests sleep more to fix acd inconsistencies
* Factor integration tests to make more maintainable
* Ensure remote writes have a fstest.CheckItems() before use
* this fixes eventual consistency on the directory listings later
* Call fs.Stats.ResetCounters() before every fs.Sync()
Note that the tests shouldn't be run concurrently as fs.Config is global state.
2016-01-23 17:02:09 +00:00
Nick Craig-Wood
27f67edb1a
Fix formatting problem in sha1sum
2016-01-17 13:56:42 +00:00
Nick Craig-Wood
3ffea738e6
Make hash constants start from 1 not 2
2016-01-17 10:47:24 +00:00
Nick Craig-Wood
e8daab7971
Fix integration tests for remotes with unsupported hash schemes
2016-01-16 09:45:15 +00:00
klauspost
78c3a5ccfa
Add support for multiple hash types.
...
Add support for multiple hash types with negotiation of common hash types for comparison.
Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
Nick Craig-Wood
af5f4ee724
Make --include rules add their implict exclude * at the end of the filter list
...
This means you can mix `--include` and `--include-from` with the
other filters (eg `--exclude`) but you must include all the files you
want in the include statement.
Fixes #280
2016-01-10 11:42:53 +00:00
klauspost
25f22ec561
Add "--ignore-existing" flag.
...
Add option to completely ignore existing files and not consider them for transfer.
Fixes #274
2016-01-08 08:20:04 +00:00
Nick Craig-Wood
5189231a34
Tweaks to rclone authorize
...
* Document the headless / remote setup procedure
* Move Config constants into fs
* Parse arguments in main for Authorize
2016-01-07 20:31:23 +00:00
klauspost
bcbd30bb8a
Add easier headless configuration.
...
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.
This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.
To test token generation, use `rclone auth "fs type"`.
2016-01-07 20:31:23 +00:00
Nick Craig-Wood
1ce3673006
Add -clean flag to test_all.go to clean left over test directories
2016-01-03 21:49:26 +00:00
Nick Craig-Wood
3423de65fa
Make canonical place for all fs in fs/all/all.go
2016-01-03 14:12:45 +00:00
Nick Craig-Wood
77fb8ac240
Version 1.26
2016-01-02 12:04:32 +00:00
dibu28
3ac4407b88
Implement Yandex storage backend - fixes #234
2015-12-30 12:11:46 +00:00
Nick Craig-Wood
8ea0d5212f
Add -verbose flag to test_all and fix tries count
2015-12-30 11:34:22 +00:00
Nick Craig-Wood
33b3eea6ec
Implement Backblaze B2 - fixes #224
2015-12-30 10:05:07 +00:00
Nick Craig-Wood
113624691a
Add -dump-headers and -dump-bodies flags for operations test debugging
2015-12-30 09:35:35 +00:00
Nick Craig-Wood
ddf39f2d57
Replace test_all.sh with test_all.go which is cross platform and parallel
2015-12-30 09:26:34 +00:00
Nick Craig-Wood
e69e181090
Fix --min-age and --max-age when only one is present
2015-12-17 14:22:43 +00:00
Nick Craig-Wood
ac65d8369e
Make fs.CheckClose public to stop duplication
2015-12-17 13:30:58 +00:00
Nick Craig-Wood
8057d668bb
Fix crash in http logging - fixes #223
...
A nil-pointer exception was caused if the http transaction ever
resulted in a go error while using `--dump-bodies`. Now don't ignore
the error and log it instead of the http body.
2015-12-17 13:30:58 +00:00
Nick Craig-Wood
36f1bc4a8a
Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169
...
* run check directory listings concurrently
2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles
3cbd57d9ad
Add support to filter files based on their age
2015-12-17 09:52:38 -02:00
Nick Craig-Wood
7f0020a407
Version v1.25
2015-11-14 13:06:39 +00:00
Nick Craig-Wood
fcea3777c0
Implement Hubic storage system - fixes #200
2015-11-14 08:08:52 +00:00
Nick Craig-Wood
bed01a303f
Add UnWrapper interface and implement in LimitedFs
2015-11-14 08:08:51 +00:00
Nick Craig-Wood
2c2cb84ca7
Make it so optional interface Purge can fail so it can be wrapped
2015-11-14 08:08:51 +00:00
Nick Craig-Wood
e9dda25c60
Implement Move in limited fs
2015-11-14 08:08:51 +00:00
Nick Craig-Wood
80ffbade22
Fix deletion of some excluded files without --delete-excluded #205
...
This only happened if the destination file was present but the source
file was missing.
2015-11-12 11:46:04 +00:00
Nick Craig-Wood
4276abc58b
Version v1.24
2015-11-07 16:23:12 +00:00
Nick Craig-Wood
d42cb11b84
Fix tests to run all tests again and add onedrive
2015-11-07 11:21:15 +00:00
Nick Craig-Wood
365b4babae
Make filter test files pass errcheck
2015-11-07 10:27:47 +00:00
Nick Craig-Wood
2fcb8f5db7
Add support for Microsoft One Drive - fixes #10
...
* Still to do
* Copy
* Move
* MoveDir
2015-11-07 10:19:10 +00:00
Nick Craig-Wood
ff16e0f6df
Factor common error handling into fs module
2015-11-06 15:26:58 +00:00
Nick Craig-Wood
ed72c678f8
Protect accounting from being closed twice
2015-11-06 15:26:58 +00:00
klauspost
b85ddc4e4f
Extend CI tests to include formatting checks.
...
CI tests now tests 'go vet', 'go fmt' (via goimports) and golint.
Adds Travis experimental OSX support.
2015-11-03 13:50:29 +01:00
klauspost
b872ff0237
Add option to disable server certificate verification.
...
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.
Put in the "developers" section in documentation with proper warnings.
Fixes #168
2015-10-29 16:42:25 +01:00
Nick Craig-Wood
1b95718460
Fix typos in filter docs and unit test assertions
2015-10-20 09:16:47 +01:00
Nick Craig-Wood
4e952af614
Allow spaces in remotes and check remote names for validity at creation time - fixes #171
2015-10-12 17:54:09 +01:00
Klaus Post
6344c3051c
Add async readahead buffer
...
This adds an async read buffer of 4x4MB when copying files >10MB.
This fixes #164 and reduces the number of IO operations for copy/move.
2015-10-12 08:30:27 +01:00
Nick Craig-Wood
3a900e5bb7
Version v1.23
2015-10-03 16:24:07 +01:00
Nick Craig-Wood
b4d7741611
Improve output of --dump-headers
2015-10-03 16:04:51 +01:00
Nick Craig-Wood
177dbbc29a
Implement rclone size for measuring remotes - fixes #153
2015-10-03 16:04:21 +01:00
Nick Craig-Wood
57aad81b68
Version v1.22
2015-09-28 19:38:20 +01:00
Nick Craig-Wood
a91bcaaeb0
Implement rsync like include and exclude - fixes #27
...
* Implement include/exclude
* Implement rsync compatible file globbing
* Implement command line filtering flags
* --delete-excluded - Delete files on dest excluded from sync
* --filter - Add a file-filtering rule
* --filter-from - Read filtering patterns from a file
* --exclude - Exclude files matching pattern
* --exclude-from - Read exclude patterns from file
* --include - Include files matching pattern
* --include-from - Read include patterns from file
* --files-from - Read list of source-file nam
* --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
* --max-size - Don't transfer any file larger than this in k or suffix k|M|G
* Document
2015-09-28 19:18:21 +01:00
Nick Craig-Wood
5d4b149884
Version v1.21
2015-09-22 21:05:11 +01:00
Nick Craig-Wood
5f20ae707d
Make lsl output times in localtime and fix tests - fixes #141
2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe
Fix golint warnings
2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3
Fixes from go vet and errcheck
2015-09-22 07:31:12 +01:00
Nick Craig-Wood
e391311512
gofmt
2015-09-17 18:42:39 +01:00
klauspost
788ef76f1c
Show more of the filename and align output.
...
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
2015-09-17 15:53:45 +02:00
klauspost
0a5870208e
Display individual transfer progress
...
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:
2015/09/15 16:38:21
Transferred: 183599104 Bytes (4646.49 kByte/s)
Errors: 0
Checks: 1
Transferred: 0
Elapsed time: 38.5s
Transferring:
* 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
* 01_12_15.mp3: 33% done. avg: 1002.2, cur: 943.4 kByte/s. ETA: 1m17s
* 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
* 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
2015-09-16 19:42:46 +01:00
Nick Craig-Wood
34193fd8d9
Version v1.20
2015-09-15 07:42:31 +01:00
Nick Craig-Wood
a288c2b3a3
Make a retry error wrapper for a plain error
2015-09-09 23:22:41 +01:00
Nick Craig-Wood
fec16b0ac8
acd: Skip test on FS which don't support ModifiedTime
2015-09-09 23:21:50 +01:00
Nick Craig-Wood
dd8717797e
Implement --dump-headers and --dump-bodies debug flags
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
8c3df224ef
Implement Amazon Cloud Drive - fixes #45
...
* Optional interfaces Copier, Mover, DirMover not done
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
c98a51b26c
Lightly obscure secrets
2015-09-01 22:33:34 +01:00
Nick Craig-Wood
8d33ce0154
Check for source and dest being the same in sync/copy/move
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950
Add Root() to Fs interface
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
0168f55f3e
Switch to spf13 fork of pflag - fixes #116
...
This supports --long value as well as --long=value which is as
expected for a unix utility.
2015-08-29 18:14:24 +01:00
Nick Craig-Wood
9ed2de3d6e
Version v1.19
2015-08-28 09:47:13 +01:00
Nick Craig-Wood
59ba8f28c8
Implement move command - fixes #35
...
* Define Mover interface to move a single object
* Define DirMover interface to move a directory
* Implement DirMove operation
* Add `rclone move` command
* Tests for Dir Move
To Do
* Implement Move, DirMover in local, drive, dropbox
* unit test for Mover
* unit test for DirMover
2015-08-28 08:49:16 +01:00
Nick Craig-Wood
5c226e91c0
Ignore rmdir in limited fs rather than throwing error - fixes #112
2015-08-25 19:16:25 +01:00
Nick Craig-Wood
a96b522958
Implement server side copies if possible - fixes #99
...
Add optional fs.Copier interface
Implemented for
* swift
* s3
* drive
* dropbox
* google cloud storage
2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7
Add Name() to Fs interface to return name as passed to NewFs
2015-08-23 13:36:38 +01:00
Nick Craig-Wood
0c6f816a49
Implement --retries flag - fixes #109
2015-08-20 21:07:00 +01:00
Nick Craig-Wood
dfe771fb0c
Correct log messages for remotes which don't support modtime/md5sum
2015-08-20 20:48:58 +01:00
Nick Craig-Wood
2fcc18779b
Version v1.18
2015-08-17 17:59:37 +01:00
Nick Craig-Wood
1d9e76bb0f
dropbox: remove datastore - Fixes #55 #84
...
This means that dropbox no longer stores MD5SUMs and modified times.
Fixup the tests so that blank MD5SUMs are ignored, and that if
Precision is set to a fs.ModTimeNotSupported, ModTimes can be ignored too.
This opens the door for other FSs which don't support metadata easily.
2015-08-17 17:26:36 +01:00
Nick Craig-Wood
771e60bd07
Show errors when reading the config file
2015-08-15 17:15:02 +01:00
Leonid Shalupov
3fcff32524
do not print stats in quiet mode - fixes #70
...
...unless had some errors or stats interval requested.
Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
2015-08-10 11:17:54 +01:00
Nick Craig-Wood
9d9a17547a
Version v1.17
2015-06-14 15:36:16 +01:00
Leonid Shalupov
bd5f685d0a
fix TestLsLong on non-UTC timezones
2015-06-14 15:23:56 +01:00
Nick Craig-Wood
539ad44757
Version v1.16
2015-06-09 18:00:33 +01:00
Nick Craig-Wood
74994a2ec1
Fix uploading big files which was causing timeouts or panics
...
The symtom was one of these two on upload of files only
* panic: d.nx != 0 in crypto/md5.(*digest).checkSum
* read tcp: i/o timeout
It turned out to be a combination of two upstream bugs
* 5a2187309e
* https://groups.google.com/forum/#!topic/golang-dev/0Nl6k5Sj6UU
This commit contains a work-around for the second problem, I've fixed
the first and had the change accepted upstream.
2015-06-09 17:32:45 +01:00
Nick Craig-Wood
97dced6a0b
Don't check md5sum after download with --size-only - fixes #75
2015-06-09 13:18:40 +01:00
Nick Craig-Wood
e04acb09ce
Version v1.15
2015-06-06 15:45:00 +01:00
Nick Craig-Wood
90744301d3
Fix package docs so they appear in godoc correctly
2015-06-06 14:24:30 +01:00
Nick Craig-Wood
e22b445cff
Implement --size-only flag to sync on size not checksum & modtime - fixes #75
2015-06-06 08:49:01 +01:00
Alex Couper
7af1282375
Add --checksum flag to only discard transfers by MD5SUM - fixes #61
...
Useful for copying between backends where checksum fetching is fast,
ie any of s3, swift, drive or googlecloudstorage.
2015-06-05 19:46:03 +01:00
Nick Craig-Wood
d9fcc32f70
Version v1.14
2015-05-21 20:13:40 +01:00
Nick Craig-Wood
88293a4b8a
Version v1.13
2015-05-10 12:39:06 +01:00