TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
732 commits 181 branches 114 tags 335 MiB
Commit graph

235 commits

Author SHA1 Message Date
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
1752ee3c8b Retry errors which indicate the connection closed prematurely. 
See discussion in #442
 2016-04-29 17:29:34 +01:00
Nick Craig-Wood
5c2d8ffe33 Retry only the failing tests in the integration tests 2016-04-26 10:20:07 +01:00
Klaus Post
6c0c9abd57 Use "password:" instead of "password>" prompt 
Fixes #410
 2016-04-21 19:39:46 +01:00
Nick Craig-Wood
157d7d45f5 Version v1.29 2016-04-18 18:30:29 +01:00
Nick Craig-Wood
b5cba73cc3 Make test more reliable 2016-04-18 17:48:52 +01:00
Nick Craig-Wood
0f8e7c3843 Make rclone check obey the --size-only flag - fixes #419 2016-04-07 15:01:45 +01:00
Nick Craig-Wood
b835330714 Use "application/octet-stream" if mime.TypeByExtension returns invalid type 
Fixes #424
 2016-04-07 14:32:01 +01:00
Klaus Post
8f4d6973fb Fix missing "quit" option when there are no remotes. 2016-03-23 16:57:56 +00:00
Nick Craig-Wood
ced3a4bc19 Implement -I, --ignore-times for unconditional upload - fixes #311 2016-03-22 17:02:27 +00:00
Nick Craig-Wood
cb22583212 b2: Enable mod time syncing - fixes #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
8ef551bf9c Make dedupe remove identical copies without asking and add non interactive mode - fixes #338 
* Now removes identical copies without asking
  * Now obeys `--dry-run`
  * Implement `--dedupe-mode` for non interactive running
    * `--dedupe-mode interactive` - interactive the default.
    * `--dedupe-mode skip` - removes identical files then skips anything left.
    * `--dedupe-mode first` - removes identical files then keeps the first one.
    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
  * Add tests which will only run on Google Drive.
 2016-03-06 18:15:01 +00:00
Nick Craig-Wood
874a64e5f6 A script to make a directory heirarchy for testing 2016-03-05 20:26:15 +00:00
Nick Craig-Wood
3c7f80f58f Version v1.28 2016-03-01 09:00:01 +00:00
Nick Craig-Wood
229ea3f86c Stop --update tests running on remotes which don't do mod time 2016-03-01 07:26:33 +00:00
Nick Craig-Wood
41eb386063 Reset password/config path in config tests to fix other tests 2016-02-29 21:43:37 +00:00
Nick Craig-Wood
dfc7cd97a3 Optionally disable gzip compression on downloads with --no-gzip-encoding - fixes #353 2016-02-29 19:48:54 +00:00
Nick Craig-Wood
280ac26464 Implement -u/--update so creation times can be used on all remotes - #226 2016-02-29 17:46:40 +00:00
Nick Craig-Wood
88cca8a6eb Simplify literals (after running gofmt -s over the code) 2016-02-29 16:57:23 +00:00
Nick Craig-Wood
3343c1afa4 Don't make directories if --dry-run set - fixes #342 2016-02-28 19:56:50 +00:00
Nick Craig-Wood
867a26fe4f Implement --low-level-retries flag - fixes #266 2016-02-25 22:58:21 +00:00
Nick Craig-Wood
3890105cdc Add -run-only flag to run_all test 2016-02-25 22:05:57 +00:00
Nick Craig-Wood
d2219a800a Fix and document the move command - fixes #334 
* Don't attempt to use server side Move unless they are on the same Fs
  * Fix move in the presense of filters
 2016-02-25 20:05:34 +00:00
Nick Craig-Wood
ccb59480bd Add InActive method to Filter to detect when no fiters are in use. 2016-02-25 19:58:00 +00:00
Nick Craig-Wood
95e4d837ef Make config chooser easier to understand 2016-02-21 13:40:16 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
Nick Craig-Wood
700e47d6e2 Stub out ReadPassword on plan9 and solaris to fix compilation 2016-02-21 10:31:53 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
klauspost
84b00b362f Change back to original goconfig package. 
Add documentation for `--ask-password`.
 2016-02-17 11:45:05 +01:00
klauspost
bfd7601cf9 Add configuration file encryption 
See #317 for details.

Use `rclone config` to add/change/remove password.

Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.

Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
 2016-02-16 16:32:05 +01:00
Nick Craig-Wood
a6320bbad3 Fix delete command to wait until all finished - fixes missing deletes. 
This also could affect deletes at the end of the sync command.
 2016-02-15 16:43:59 +00:00
Nick Craig-Wood
85d99de26b Fix typo in error strings 2016-02-09 16:15:50 +00:00
Nick Craig-Wood
e62c0a58a7 Version 1.27 2016-01-31 17:50:13 +00:00
Nick Craig-Wood
0f73129ab7 dedupe command to deduplicate a remote. Useful with google drive - fixes #41 2016-01-31 16:09:42 +00:00
Nick Craig-Wood
1373efaa39 Delete command which does obey the filters - fixes #327 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
5c37b777fc Make the --dry-run warnings into logs so they appear without the -v flag 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
8c211fc8df Warn the user about files with same name but different case 
Relates to #107 & #119.
 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
74a71f7824 Add tests for --delete-before, --delete-during and --delete-after 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
12b51c5eb8 Remove duplicate check for filter IncludeObject 2016-01-26 16:57:09 +00:00
klauspost
14069fd8e6 Implement --delete-before, --delete-during, --delete-after - fixes #252. 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
cd62f41606 Reduce number of logs and show hash type where appropriate 2016-01-24 18:06:57 +00:00
Nick Craig-Wood
109d4ee490 Prefix all test remotes with rclone-test- and make names more pronouncable 2016-01-24 12:37:46 +00:00
Nick Craig-Wood
18ebec8276 Check remote is empty between integration tests 2016-01-24 12:37:19 +00:00
Nick Craig-Wood
eda4130703 Fix integration tests so they can be run independently and out of order - fixes #291 
* Make all integration tests start with an empty remote
  * Add an -individual flag so this can be a different bucket/container/directory
  * Fix up tests after changing the hashers
  * Add sha1sum test
  * Make directory checking in tests sleep more to fix acd inconsistencies
  * Factor integration tests to make more maintainable
  * Ensure remote writes have a fstest.CheckItems() before use
    * this fixes eventual consistency on the directory listings later
  * Call fs.Stats.ResetCounters() before every fs.Sync()

Note that the tests shouldn't be run concurrently as fs.Config is global state.
 2016-01-23 17:02:09 +00:00
Nick Craig-Wood
27f67edb1a Fix formatting problem in sha1sum 2016-01-17 13:56:42 +00:00
Nick Craig-Wood
3ffea738e6 Make hash constants start from 1 not 2 2016-01-17 10:47:24 +00:00
Nick Craig-Wood
e8daab7971 Fix integration tests for remotes with unsupported hash schemes 2016-01-16 09:45:15 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
Nick Craig-Wood
af5f4ee724 Make --include rules add their implict exclude * at the end of the filter list 
This means you can mix `--include` and `--include-from` with the
other filters (eg `--exclude`) but you must include all the files you
want in the include statement.

Fixes #280
 2016-01-10 11:42:53 +00:00
klauspost
25f22ec561 Add "--ignore-existing" flag. 
Add option to completely ignore existing files and not consider them for transfer.

Fixes #274
 2016-01-08 08:20:04 +00:00
Nick Craig-Wood
5189231a34 Tweaks to rclone authorize 
* Document the headless / remote setup procedure
  * Move Config constants into fs
  * Parse arguments in main for Authorize
 2016-01-07 20:31:23 +00:00
klauspost
bcbd30bb8a Add easier headless configuration. 
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.

This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.

To test token generation, use `rclone auth "fs type"`.
 2016-01-07 20:31:23 +00:00
Nick Craig-Wood
1ce3673006 Add -clean flag to test_all.go to clean left over test directories 2016-01-03 21:49:26 +00:00
Nick Craig-Wood
3423de65fa Make canonical place for all fs in fs/all/all.go 2016-01-03 14:12:45 +00:00
Nick Craig-Wood
77fb8ac240 Version 1.26 2016-01-02 12:04:32 +00:00
dibu28
3ac4407b88 Implement Yandex storage backend - fixes #234 2015-12-30 12:11:46 +00:00
Nick Craig-Wood
8ea0d5212f Add -verbose flag to test_all and fix tries count 2015-12-30 11:34:22 +00:00
Nick Craig-Wood
33b3eea6ec Implement Backblaze B2 - fixes #224 2015-12-30 10:05:07 +00:00
Nick Craig-Wood
113624691a Add -dump-headers and -dump-bodies flags for operations test debugging 2015-12-30 09:35:35 +00:00
Nick Craig-Wood
ddf39f2d57 Replace test_all.sh with test_all.go which is cross platform and parallel 2015-12-30 09:26:34 +00:00
Nick Craig-Wood
e69e181090 Fix --min-age and --max-age when only one is present 2015-12-17 14:22:43 +00:00
Nick Craig-Wood
ac65d8369e Make fs.CheckClose public to stop duplication 2015-12-17 13:30:58 +00:00
Nick Craig-Wood
8057d668bb Fix crash in http logging - fixes #223 
A nil-pointer exception was caused if the http transaction ever
resulted in a go error while using `--dump-bodies`.  Now don't ignore
the error and log it instead of the http body.
 2015-12-17 13:30:58 +00:00
Nick Craig-Wood
36f1bc4a8a Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169 
* run check directory listings concurrently
 2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles
3cbd57d9ad Add support to filter files based on their age 2015-12-17 09:52:38 -02:00
Nick Craig-Wood
7f0020a407 Version v1.25 2015-11-14 13:06:39 +00:00
Nick Craig-Wood
fcea3777c0 Implement Hubic storage system - fixes #200 2015-11-14 08:08:52 +00:00
Nick Craig-Wood
bed01a303f Add UnWrapper interface and implement in LimitedFs 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
2c2cb84ca7 Make it so optional interface Purge can fail so it can be wrapped 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
e9dda25c60 Implement Move in limited fs 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
80ffbade22 Fix deletion of some excluded files without --delete-excluded #205 
This only happened if the destination file was present but the source
file was missing.
 2015-11-12 11:46:04 +00:00
Nick Craig-Wood
4276abc58b Version v1.24 2015-11-07 16:23:12 +00:00
Nick Craig-Wood
d42cb11b84 Fix tests to run all tests again and add onedrive 2015-11-07 11:21:15 +00:00
Nick Craig-Wood
365b4babae Make filter test files pass errcheck 2015-11-07 10:27:47 +00:00
Nick Craig-Wood
2fcb8f5db7 Add support for Microsoft One Drive - fixes #10 
* Still to do
    * Copy
    * Move
    * MoveDir
 2015-11-07 10:19:10 +00:00
Nick Craig-Wood
ff16e0f6df Factor common error handling into fs module 2015-11-06 15:26:58 +00:00
Nick Craig-Wood
ed72c678f8 Protect accounting from being closed twice 2015-11-06 15:26:58 +00:00
klauspost
b85ddc4e4f Extend CI tests to include formatting checks. 
CI tests now tests 'go vet', 'go fmt' (via goimports) and golint.

Adds Travis experimental OSX support.
 2015-11-03 13:50:29 +01:00
klauspost
b872ff0237 Add option to disable server certificate verification. 
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.

Put in the "developers" section in documentation with proper warnings.

Fixes #168
 2015-10-29 16:42:25 +01:00
Nick Craig-Wood
1b95718460 Fix typos in filter docs and unit test assertions 2015-10-20 09:16:47 +01:00
Nick Craig-Wood
4e952af614 Allow spaces in remotes and check remote names for validity at creation time - fixes #171 2015-10-12 17:54:09 +01:00
Klaus Post
6344c3051c Add async readahead buffer 
This adds an async read buffer of 4x4MB when copying files >10MB.

This fixes #164 and reduces the number of IO operations for copy/move.
 2015-10-12 08:30:27 +01:00
Nick Craig-Wood
3a900e5bb7 Version v1.23 2015-10-03 16:24:07 +01:00
Nick Craig-Wood
b4d7741611 Improve output of --dump-headers 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
177dbbc29a Implement rclone size for measuring remotes - fixes #153 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
57aad81b68 Version v1.22 2015-09-28 19:38:20 +01:00
Nick Craig-Wood
a91bcaaeb0 Implement rsync like include and exclude - fixes #27 
* Implement include/exclude
  * Implement rsync compatible file globbing
  * Implement command line filtering flags
    * --delete-excluded - Delete files on dest excluded from sync
    * --filter - Add a file-filtering rule
    * --filter-from - Read filtering patterns from a file
    * --exclude - Exclude files matching pattern
    * --exclude-from - Read exclude patterns from file
    * --include - Include files matching pattern
    * --include-from - Read include patterns from file
    * --files-from - Read list of source-file nam
    * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
    * --max-size - Don't transfer any file larger than this in k or suffix k|M|G
  * Document
 2015-09-28 19:18:21 +01:00
Nick Craig-Wood
5d4b149884 Version v1.21 2015-09-22 21:05:11 +01:00
Nick Craig-Wood
5f20ae707d Make lsl output times in localtime and fix tests - fixes #141 2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3 Fixes from go vet and errcheck 2015-09-22 07:31:12 +01:00
Nick Craig-Wood
e391311512 gofmt 2015-09-17 18:42:39 +01:00
klauspost
788ef76f1c Show more of the filename and align output. 
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
 2015-09-17 15:53:45 +02:00
klauspost
0a5870208e Display individual transfer progress 
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:

2015/09/15 16:38:21
Transferred:    183599104 Bytes (4646.49 kByte/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:       38.5s
Transferring:
 * 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
 * 01_12_15.mp3: 33% done. avg: 1002.2, cur:  943.4 kByte/s. ETA: 1m17s
 * 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
 * 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
 2015-09-16 19:42:46 +01:00
Nick Craig-Wood
34193fd8d9 Version v1.20 2015-09-15 07:42:31 +01:00
Nick Craig-Wood
a288c2b3a3 Make a retry error wrapper for a plain error 2015-09-09 23:22:41 +01:00
Nick Craig-Wood
fec16b0ac8 acd: Skip test on FS which don't support ModifiedTime 2015-09-09 23:21:50 +01:00
Nick Craig-Wood
dd8717797e Implement --dump-headers and --dump-bodies debug flags 2015-09-08 21:02:48 +01:00
Nick Craig-Wood
8c3df224ef Implement Amazon Cloud Drive - fixes #45 
* Optional interfaces Copier, Mover, DirMover not done
 2015-09-08 21:02:48 +01:00