TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
1662 commits 181 branches 114 tags 335 MiB
Commit graph

141 commits

Author SHA1 Message Date
Nick Craig-Wood
e6a0521ca2 Make it possible to test Fs multiple times and use this with crypt 
We test both the filename encryption modes for crypt.
 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
6a4e424630 Re-implement Obscure/Reveal so they use AES-CTR encryption 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
ab43005422 Make NewObject return an error 
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files 
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
 2016-06-28 08:51:43 +01:00
Nick Craig-Wood
cc6a776034 drive, acd: Tweak logging after changing Fs.Put so that it must cope with existing files 2016-06-18 10:54:42 +01:00
Nick Craig-Wood
2cfb3834f2 Log errors with %v 2016-06-18 09:36:47 +01:00
Nick Craig-Wood
e4650eff58 drive: fix retry of multipart uploads - fixes #520 
Reset the reader on retry otherwise it is empty when read again.
 2016-06-15 21:48:30 +01:00
Nick Craig-Wood
df1092ef33 Change Fs.Put so that it must cope with existing files 
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
 2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood
661715733a Make sure we don't use conflicting content types on upload - fixes #513 2016-06-09 17:52:58 +01:00
Nick Craig-Wood
0a922ad1dc acd: Reauth on 401 errors 
Fixes #493
Fixes #501
 2016-05-28 16:49:26 +01:00
Nick Craig-Wood
c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
ddb47758f3 drive: increase default chunk size to 8 MB and document - fixes #397 2016-04-12 21:33:55 +01:00
Nick Craig-Wood
6cc9c09610 drive: preserve mime type on file update - fixes #417 2016-04-04 16:58:42 +01:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
fdd4b4ee22 drive: Add missing retries for Move and DirMove 2016-03-06 18:15:01 +00:00
Nick Craig-Wood
2119fb4314 drive: tweak pacer to speed up directory listings and make more reliable 2016-03-06 18:15:01 +00:00
Nick Craig-Wood
7d4e143dee Make it obvious that the client secrets are encrypted 2016-02-28 19:57:19 +00:00
Nick Craig-Wood
b279df2e67 Drive: disable copy and move for google docs - fixes #332 2016-02-28 09:35:28 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Nick Craig-Wood
1cd0d9a1f2 Fix listing drive docs at root - fixes #336 
* Remove full drive list code
    * it is slower and uses more data
    * having two directory listing routines is causing problems (including this one)
    * less code is more
  * Make sure we don't recurse into directories we don't own
  * Fix export extension handling and add tests
 2016-02-15 16:46:43 +00:00
Nick Craig-Wood
558bc2e132 drive: Export Google documents - fixes #49 
Rclone will download one format of a google doc. The choice of which
export format is controlled by the `--drive-formats` flag.
 2016-01-31 16:10:43 +00:00
Nick Craig-Wood
ccba859812 Test all available hashes for each remote 2016-01-23 09:10:36 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
Nick Craig-Wood
5189231a34 Tweaks to rclone authorize 
* Document the headless / remote setup procedure
  * Move Config constants into fs
  * Parse arguments in main for Authorize
 2016-01-07 20:31:23 +00:00
klauspost
bcbd30bb8a Add easier headless configuration. 
This will allow setting up a remote with copy&paste of values to a headless machine. It will allow copy+pasting a token into the configuration.

This requires rclone to be on a machine with a proper browser. Custom client id and secrets are supported.

To test token generation, use `rclone auth "fs type"`.
 2016-01-07 20:31:23 +00:00
Björn Harrtell
78edafcaac drive: add --drive-auth-owner-only to only consider files owned by the user. 2016-01-05 17:02:04 +00:00
Nick Craig-Wood
b257de4aba Be more constistent with naming in remotes 
* External objects are called Fs and Object
  * Object.fs always points to the Fs
 2015-11-07 11:14:46 +00:00
Nick Craig-Wood
ff16e0f6df Factor common error handling into fs module 2015-11-06 15:26:58 +00:00
Nick Craig-Wood
bf562d7373 Tweak wording on client/secret ids in acd, gcs and drive - fixes #155 2015-10-03 16:04:45 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
klauspost
6fbd9cf24b Make Google Drive Directory reads concurrent for increased speed. 
If you use the -v flag then this will show the progress of directory
listing.  Fixes #143.
 2015-09-17 18:42:10 +01:00
Nick Craig-Wood
0872ec3204 Allow user to override credentials again in drive, gcs and acd - fixes #139 2015-09-16 20:08:40 +01:00
Nick Craig-Wood
073d112204 Factor pacer module from Drive and use it in Amazon Cloud Drive for 
smooth API pacing and retry logic.
 2015-09-11 19:18:41 +01:00
Nick Craig-Wood
ea12e446ca Factor DirCache from drive into its own module 2015-09-05 11:58:54 +01:00
Nick Craig-Wood
c98a51b26c Lightly obscure secrets 2015-09-01 22:33:34 +01:00
Nick Craig-Wood
e2717a031e Implement Mover and DirMover interfaces fixes #115 
* unit tests
  * local
  * drive
  * dropbox
 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950 Add Root() to Fs interface 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
0168f55f3e Switch to spf13 fork of pflag - fixes #116 
This supports --long value as well as --long=value which is as
expected for a unix utility.
 2015-08-29 18:14:24 +01:00
Nick Craig-Wood
7463a7a509 Use "golang.org/x/oauth2" as oauth libary of choice - fixes #102 
* get rid of depreprecated "code.google.com/p/goauth2/oauth"
  * store tokens in config file as before
  * read old format tokens and write in new format seamlessly
  * set our own transport to enforce timeouts etc
 2015-08-29 17:47:23 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99 
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
 2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7 Add Name() to Fs interface to return name as passed to NewFs 2015-08-23 13:36:38 +01:00
Nick Craig-Wood
d8306938a1 drive: Add --drive-use-trash flag so rclone trashes instead of deletes - fixes #82 2015-08-16 15:51:39 +01:00
Nick Craig-Wood
88ea8b305d drive: Add "Forbidden to download" message for files with no downloadURL - fixes #95 2015-08-16 15:51:39 +01:00
Leonid Shalupov
3fcff32524 do not print stats in quiet mode - fixes #70 
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
 2015-08-10 11:17:54 +01:00
Nick Craig-Wood
7c120b8bc5 drive: add --drive-chunk-size and --drive-upload-cutoff parameters 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
81a933ae38 drive: Use chunked upload for files - fixes #33 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
ecb3c7bcef drive, googlecloudstorage: remove SeekWrapper after googleapi fix 2015-03-04 20:47:59 +00:00
Nick Craig-Wood
c03d6a1ec3 drive: fix crash on failed to update remote mtime - fixes #36 2015-03-02 09:25:33 +00:00
Nick Craig-Wood
46d39ebaf7 Factor Mime Type guessing into fs.MimeType() 2015-03-02 09:21:15 +00:00
Nick Craig-Wood
fe68737268 Fix niggles found by go vet 2015-02-28 15:35:54 +00:00
Nick Craig-Wood
85f8552c4d Tidy logging 2015-02-27 15:22:05 +00:00
Nick Craig-Wood
8e4d8d13b8 drive: rename internal api 2015-02-20 09:51:07 +00:00
Nick Craig-Wood
5fb6f94579 drive: fix rate limit exceeded errors - fixes #20 
This is done by pacing the requests to drive and backing them off
using an exponential increase.  Put and Modify operations can now be
retried also.
 2015-02-07 22:32:51 +00:00
Nick Craig-Wood
1758621a51 drive: fix subdirectory listing to not list entire drive - fixes #23 
This was causing inexplicably slow transfers on subdirectories of
drives with lots of files.
 2015-02-04 22:22:03 +00:00
Nick Craig-Wood
5710247bf6 drive: Fix SetModTime 2015-01-04 23:19:59 +00:00
Nick Craig-Wood
14567952b3 google cloud storage: Fix memory leak - fixes #17 
This was the same problem as issue #5 (which affected google drive)
 2014-12-23 11:03:34 +00:00
Nick Craig-Wood
0fb35f081a Use new location of Google API package - fixes #16 2014-12-12 20:02:08 +00:00
Nick Craig-Wood
d3c16608e4 Test Listing the Root of each Fs 2014-07-31 23:20:39 +01:00
Nick Craig-Wood
3e27ff1b95 Add Root List test and fs.Limited tests for single files 2014-07-31 21:35:29 +01:00
Nick Craig-Wood
c389616657 all: make private functions / variables / constant which shouldn't be public 2014-07-29 17:50:07 +01:00
Nick Craig-Wood
442578ca25 drive: reset root directory on Rmdir and Purge 2014-07-29 17:32:06 +01:00
Nick Craig-Wood
457ef2c190 Automatically generate the tests files for each Fs 2014-07-29 17:18:22 +01:00
Nick Craig-Wood
17ffb0855f Fixes after running errcheck 2014-07-25 18:19:49 +01:00
Nick Craig-Wood
a5b0d88608 Make tests for each Fs 
Factor tests out of rclonetest
 2014-07-24 23:13:32 +01:00
Nick Craig-Wood
f9b31591f9 drive: Flush directory cache on Purge 2014-07-19 13:05:06 +01:00
Nick Craig-Wood
b83441081c drive: factor common authentication code into googleauth module 
In preparation for Google Cloud Storage support
 2014-07-15 19:27:30 +01:00
Nick Craig-Wood
66cf2df780 drive: check errors in Open() better 2014-07-15 13:28:35 +01:00
Nick Craig-Wood
c1a245d1c8 Factor UserAgent to fs and move Version to fs 2014-07-13 19:19:58 +01:00
Nick Craig-Wood
e40b09fe61 drive: Fix comment 2014-07-13 10:54:35 +01:00
Nick Craig-Wood
91cfbd4146 drive: fix transfer of big files using up lots of memory - fixes #5 
This was done by making a seekWrapper which wraps an io.Reader with a
basic Seek for code.google.com/p/google-api-go-client/googleapi to
detect the length.  Without this the getReaderSize function reads the
entire file into memory to find its length.
 2014-07-04 17:17:21 +01:00
Nick Craig-Wood
93cb0a47e4 drive: fix whole second dates - fixes #4 2014-07-03 21:32:01 +01:00
Nick Craig-Wood
99695d57ab Implement single file operations for all file systems 2014-05-05 22:17:57 +01:00
Nick Craig-Wood
8fd59f2e7d drive: Use o.Update and fs.Put to optimise transfers 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
02afcb00e9 Factor Fs.Put into Object.Update and call Update rather than Put if possible 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
86b77f3ca8 drive: Fix multiple files of same name being created 
ModifiedDate seems to be set on Insert if set, so do that
 2014-04-17 22:27:33 +01:00
Nick Craig-Wood
92ec29fe3f Factor the generic code into fs and add some more intefaces 2014-03-28 22:26:42 +00:00
Nick Craig-Wood
bc221fb27e drive: Fix path parsing 2014-03-27 17:49:36 +00:00
Nick Craig-Wood
a628bef9c2 Use pflag for gnu style flags 2014-03-27 16:55:29 +00:00
Nick Craig-Wood
b4dd693d23 drive: Rework token aquisition into config framework and store token in config file 2014-03-27 16:52:39 +00:00
Nick Craig-Wood
2e5f0ef258 Rename filing systems modules (so I can tell them apart in emacs buffers!) 2014-03-16 10:09:55 +00:00
Nick Craig-Wood
0a108832e2 Remove filesystem flags and put in config file with editor 2014-03-15 16:06:11 +00:00
Nick Craig-Wood
a34bf2ae5e Make imports absolute and use own copy of goamz pending patch inclusion 2013-07-11 17:37:33 +01:00
Nick Craig-Wood
7153b05342 Note that drive goroutine leak now fixed upstream (by my patch :-) 2013-06-29 21:13:30 +01:00
Nick Craig-Wood
b98923a20f drive: Finish refactor 2013-06-29 12:15:31 +01:00
Nick Craig-Wood
b1f681b912 Rename fs.FsX to fs.X 2013-06-28 08:57:32 +01:00
Nick Craig-Wood
6085dc1b5a Refactor into sub filesystems 2013-06-27 22:17:51 +01:00