TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
2240 commits 181 branches 114 tags 335 MiB
Commit graph

2240 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nick Craig-Wood
05050d53ad Implement compliant pacing scheme for Amazon Cloud Drive 
* Implement switchable pacing algorithm
  * Add tests for pacer
 2015-09-21 21:49:54 +01:00
Nick Craig-Wood
e391311512 gofmt 2015-09-17 18:42:39 +01:00
klauspost
3234c28f7c Minor adjustments to Unc path conversion function. 2015-09-17 18:42:10 +01:00
klauspost
6fbd9cf24b Make Google Drive Directory reads concurrent for increased speed. 
If you use the -v flag then this will show the progress of directory
listing.  Fixes #143.
 2015-09-17 18:42:10 +01:00
klauspost
bc5b63ffef Move ACD Connection-limiter into Pacer. 2015-09-17 18:42:10 +01:00
klauspost
788ef76f1c Show more of the filename and align output. 
Print more of the file name, and make the output aligned, so it is nicer on frequent updates.
 2015-09-17 15:53:45 +02:00
Nick Craig-Wood
0872ec3204 Allow user to override credentials again in drive, gcs and acd - fixes #139 2015-09-16 20:08:40 +01:00
klauspost
0a5870208e Display individual transfer progress 
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:

2015/09/15 16:38:21
Transferred:    183599104 Bytes (4646.49 kByte/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:       38.5s
Transferring:
 * 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
 * 01_12_15.mp3: 33% done. avg: 1002.2, cur:  943.4 kByte/s. ETA: 1m17s
 * 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
 * 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
 2015-09-16 19:42:46 +01:00
klauspost
3219334c3e Add "-race" test to travis. 2015-09-16 12:26:28 +02:00
klauspost
79fd662676 Protect concurrent read/writes to pacer. 
Protects all variables in the pacer from concurrent modifications. It is now safe to modify pacer settings while it is running.

I decided to not go for an RWMutex, since all accesses are very short, so the overhead of an RWMutex isn't worth it.

Fixes #138.
 2015-09-16 12:25:55 +02:00
Nick Craig-Wood
34193fd8d9 Version v1.20 2015-09-15 07:42:31 +01:00
Nick Craig-Wood
2203766f77 Retry listing in tests to work around eventual consistency 2015-09-14 21:01:25 +01:00
Nick Craig-Wood
235cbe0e57 amazon cloud drive: retry 409 errors too 2015-09-14 21:00:44 +01:00
klauspost
f50f353b5d local: always use UNC paths on Windows - fixes #124, fixes #130, fixes #90 
* Convert all paths to UNC paths on Windows.
  * Update local filesystem to always use UNC paths.
  * Change tests, so they can work with Windows character replacements.
  * Remove "/" suffix on paths.
  * Always use path/filepath
 2015-09-14 19:58:03 +01:00
Klaus Post
00afe6cc96 ACD: Shorten minimum and maximum sleep. 2015-09-14 18:22:21 +01:00
Klaus Post
dd48e62b7e Limit concurrent directory listings. 
Never run more than "fs.Config.Checkers" directory listings at once.
 2015-09-14 18:11:59 +01:00
Klaus Post
a1a780e847 Read folders in separate goroutines. 
As proposed in the FIXME, read folders in parallel.
This appears to fix "Next token is expired" on very big directories.
The only downside is that this doesn't abort at once if an error is found.
I added some logging, so there is some output for "-v".
 2015-09-14 18:11:59 +01:00
Klaus Post
fa87077211 Retry if a timeout occurs. 
This happens more rarely than 500, but can still be encountered.
 2015-09-14 18:02:35 +01:00
Klaus Post
6ac7145d2d Retry when we get a "500 Internal server error". 
It seems like this happens randomly, and retrying works fine for that.
 2015-09-14 18:02:35 +01:00
Nick Craig-Wood
f1226f19b2 drive, googlecloudstorage: optionally use auto config for the oauth token 2015-09-12 14:17:39 +01:00
Klaus Post
3ecbf2af25 Fix missing link text. 2015-09-12 11:22:17 +02:00
Nick Craig-Wood
79f2e95bf9 Add Klaus Post to Contributors 2015-09-11 20:20:28 +01:00
klauspost
faee50b238 List multiple matched commands. 
* When multiple commands match, list them all.
* Change example to an existing command.
 2015-09-11 19:44:50 +01:00
klauspost
807d4a3c00 Improve OAUTH2 usability - fixes #131 
Shorten the URL to be used by the user and automatically use the
returned code by the server. The browser is opened on
`http://(bindaddress)/auth`, and redirected to the actual URL. When
the code is returned it is automatically inserted, instead of
requiring a copy+paste.

This is also a workaround for the "open" package, which escapes "&"
wrongly on Windows, so the opened URL's are invalid.
 2015-09-11 19:44:50 +01:00
Nick Craig-Wood
073d112204 Factor pacer module from Drive and use it in Amazon Cloud Drive for 
smooth API pacing and retry logic.
 2015-09-11 19:18:41 +01:00
Nick Craig-Wood
14f814b806 acd: Retry on 429 errors with backoff and fix upload of 0 length files 2015-09-09 23:23:37 +01:00
Nick Craig-Wood
a288c2b3a3 Make a retry error wrapper for a plain error 2015-09-09 23:22:41 +01:00
Nick Craig-Wood
fec16b0ac8 acd: Skip test on FS which don't support ModifiedTime 2015-09-09 23:21:50 +01:00
Nick Craig-Wood
dd8717797e Implement --dump-headers and --dump-bodies debug flags 2015-09-08 21:02:48 +01:00
Nick Craig-Wood
7e7c239f09 Through popular demand add a donations page 2015-09-08 21:02:48 +01:00
Nick Craig-Wood
edd0e8abb1 Add Colin Nicholson to contributors 2015-09-08 21:02:48 +01:00
Colin Nicholson
d2b537d9a1 Fix docs link typo for googlecloudstorage - fixes #120 2015-09-08 21:02:48 +01:00
Nick Craig-Wood
8c3df224ef Implement Amazon Cloud Drive - fixes #45 
* Optional interfaces Copier, Mover, DirMover not done
 2015-09-08 21:02:48 +01:00
Nick Craig-Wood
967fd2a778 dircache: implement FindRootParent and unify locking 2015-09-06 10:27:15 +01:00
Nick Craig-Wood
ea12e446ca Factor DirCache from drive into its own module 2015-09-05 11:58:54 +01:00
Nick Craig-Wood
c8cd2b510f More notes 2015-09-02 19:17:33 +01:00
Nick Craig-Wood
8b05a8322b FAQ entry about bidirectional sync - see #118 2015-09-02 08:33:53 +01:00
Nick Craig-Wood
c98a51b26c Lightly obscure secrets 2015-09-01 22:33:34 +01:00
Nick Craig-Wood
e2717a031e Implement Mover and DirMover interfaces fixes #115 
* unit tests
  * local
  * drive
  * dropbox
 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
8d33ce0154 Check for source and dest being the same in sync/copy/move 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950 Add Root() to Fs interface 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
cbc6bf6a89 FAQ entry on partial transfers / binary diffs - fixes #113 2015-08-31 12:47:07 +01:00
Nick Craig-Wood
f72575e75f Remove support for go 1.2 now that go 1.5 is out 2015-08-29 18:58:48 +01:00
Nick Craig-Wood
0168f55f3e Switch to spf13 fork of pflag - fixes #116 
This supports --long value as well as --long=value which is as
expected for a unix utility.
 2015-08-29 18:14:24 +01:00
Nick Craig-Wood
8b60ab86a1 dropbox: force use of our custom transport which makes timeouts work 2015-08-29 17:48:15 +01:00
Nick Craig-Wood
7463a7a509 Use "golang.org/x/oauth2" as oauth libary of choice - fixes #102 
* get rid of depreprecated "code.google.com/p/goauth2/oauth"
  * store tokens in config file as before
  * read old format tokens and write in new format seamlessly
  * set our own transport to enforce timeouts etc
 2015-08-29 17:47:23 +01:00
Nick Craig-Wood
9ed2de3d6e Version v1.19 2015-08-28 09:47:13 +01:00
Nick Craig-Wood
4f35fb59c8 Build for plan9/amd64 and solaris/amd64 too 2015-08-28 09:40:46 +01:00
Nick Craig-Wood
59ba8f28c8 Implement move command - fixes #35 
* Define Mover interface to move a single object
  * Define DirMover interface to move a directory
  * Implement DirMove operation
  * Add `rclone move` command
  * Tests for Dir Move

To Do
  * Implement Move, DirMover in local, drive, dropbox
  * unit test for Mover
  * unit test for DirMover
 2015-08-28 08:49:16 +01:00
Nick Craig-Wood
d298b578ab s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114 2015-08-28 08:47:41 +01:00