TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
946 commits 181 branches 114 tags 335 MiB
Commit graph

51 commits

Author SHA1 Message Date
Nick Craig-Wood
215fd2a11d b2: use new prefix and delimiter parameters in directory listings 
This makes --max-depth 1 directory listings much more efficient (it no
longer lists all the files) and simplifies the code, bringing it into
line with s3/swift/gcs

Fixes #944
 2016-12-14 17:37:26 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831 
This will enable rclone to manage directories properly in the future.
 2016-11-26 12:02:53 +00:00
Nick Craig-Wood
4105da206a b2: reauth the account while doing uploads too #825 
Originally it was thought the upload URL expiring would provide 401
errors so it was excluded from reauth when doing uploads, but on
re-reading the docs and looking at this issue it seems that 401 errors
are only caused by the account token expiring and not the upload token
expiring.

We will refresh both the upload token and account token on a 401 error
while uploading, and just the account token when we get a 401 at any
other time.
 2016-11-07 13:30:51 +00:00
Nick Craig-Wood
441951a93b Stop removing failed upload to cloud storage remotes - fixes #559 
We do remove a partially written file on local so we don't have
corrupted files lying around.
 2016-11-04 21:34:25 +00:00
Nick Craig-Wood
544ca6035a b2: Make sure each upload has at least one upload slot - fixes #731 2016-10-17 16:48:11 +01:00
Nick Craig-Wood
98804cb860 b2: Fix seek producing corrupted file errors 2016-10-07 12:16:25 +01:00
Nick Craig-Wood
aef2ac5c04 Add options for Open and implement Range for all remotes 2016-10-05 21:03:56 +01:00
Nick Craig-Wood
945f49ab5e Make ContentType be preserved for cloud -> cloud copies - fixes #733 2016-10-03 20:02:04 +01:00
Nick Craig-Wood
83ba59749f Make failed uploads not count as "Transferred" - fixes #708 2016-09-12 18:15:58 +01:00
Nick Craig-Wood
b3d0848d09 b2: Fix download of large files - fixes #678 
Large files were failing to download with an sha1 mismatch error.
Correct this by making sure we use the sha1 read from the info rather
than the header.
 2016-09-05 17:26:04 +01:00
Nick Craig-Wood
d9bba67d18 b2: return error when we try to create a bucket which someone else owns #645 2016-08-25 21:43:43 +01:00
Nick Craig-Wood
e6a0521ca2 Make it possible to test Fs multiple times and use this with crypt 
We test both the filename encryption modes for crypt.
 2016-08-23 17:45:37 +01:00
Nick Craig-Wood
037a000cc8 b2: fix stats accounting for upload - fixes #602 2016-08-22 21:19:38 +01:00
Nick Craig-Wood
b4f2ada820 b2: on cleanup delete hide marker if it is the current file #604 2016-08-18 18:36:00 +01:00
Per Cederberg
bb62c49489 New B2 API endpoint 
Backblaze will change the authentication API endpoint on August 16, 2016. The old endpoint will be removed Feb 2nd 2017.

See https://help.backblaze.com/hc/en-us/articles/224959187-B2-Domain-Migration-Plan
 2016-08-15 15:59:19 +02:00
Nick Craig-Wood
57f8f1ec92 b2: set maximum backoff to 5 Minutes #597 2016-08-01 22:57:52 +01:00
Nick Craig-Wood
41917eb1f2 b2: Fix upload of files large files not in root - fixes #582 2016-07-13 15:28:39 +01:00
Nick Craig-Wood
cc628717d8 b2: Add --b2-versions flag so old versions can be listed and retreived. #420 2016-07-05 11:27:04 +01:00
Nick Craig-Wood
606961f49d b2: Treat 403 errors (eg cap exceeded) as fatal #420 2016-07-04 13:45:53 +01:00
Nick Craig-Wood
018fe80bcb b2: cleanup old file versions - fixes #462 2016-07-02 17:03:08 +01:00
Nick Craig-Wood
c8e2531c8b b2: make error handling compliant 2016-07-01 16:23:23 +01:00
Nick Craig-Wood
cbebefebc4 b2: Fix handling of token expiry #420 
Found with --b2-test-mode expire_some_account_authorization_tokens
 2016-07-01 11:47:42 +01:00
Nick Craig-Wood
6f3897ce2c b2: implement --b2-test-mode to set X-Bz-Test-Mode header #420 2016-07-01 11:30:09 +01:00
Nick Craig-Wood
ea5878f590 b2: set cutoff for chunked upload to 200MB #420 
This is the value recommended in the b2 integration checklist:

https://www.backblaze.com/b2/docs/integration_checklist.html
 2016-07-01 10:08:09 +01:00
Nick Craig-Wood
46f8e50614 b2: Make upload multi-threaded - fixes #531 2016-07-01 10:04:52 +01:00
Nick Craig-Wood
ab43005422 Make NewObject return an error 
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files 
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
 2016-06-28 08:51:43 +01:00
Nick Craig-Wood
2a46be8cf3 b2: implement large file uploading - fixes #456 2016-06-18 13:38:05 +01:00
Nick Craig-Wood
df1092ef33 Change Fs.Put so that it must cope with existing files 
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
 2016-06-13 19:29:10 +01:00
Nick Craig-Wood
4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood
c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
93c60c34e1 b2: Fix incorrect value of Precision - should be 1ms not 1s 2016-03-24 15:23:27 +00:00
Nick Craig-Wood
cb22583212 b2: Enable mod time syncing - fixes #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
20f4b2c91d b2: update API to new version - fixes #393 
* Make reading mod time and SHA1 much more efficient
    * removes an HTTP transaction to increase speed
  * Reduce memory usage of the objects
 2016-03-22 14:39:56 +00:00
Nick Craig-Wood
37543bd1d9 b2: Fix parsing of mod time when not in metadata 
This files this error `Failed to parse mod time string "":
"src_last_modified_millis" not found in metadata`.
 2016-03-22 10:26:37 +00:00
Nick Craig-Wood
6b6b43402b b2: Use one upload URL per go routine 
This fixes `more than one upload using auth token` errors.
 2016-02-27 13:00:35 +00:00
Nick Craig-Wood
6f46270735 b2: Add pacing, retries and reauthentication - fixes #310 2016-02-27 12:04:45 +00:00
Nick Craig-Wood
ee5e34a19c b2: factor authorize account into its own method 2016-02-27 12:04:45 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost
758c7f2d84 Avoid b2 temporary file. 
If source can provide SHA1 hash we don't copy input to a temporary file.

Fixes #358
 2016-02-19 18:07:15 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Nick Craig-Wood
85a0f25b95 b2: Fix reading metadata for all files when using a subdir - fixes #356 
Also fix some confusion with Metadata prefix/root.
 2016-02-19 12:11:30 +00:00
Nick Craig-Wood
ccba859812 Test all available hashes for each remote 2016-01-23 09:10:36 +00:00
Nick Craig-Wood
ae3a0ec27e b2: Don't re-read the SHA1 if we already have it 2016-01-19 08:21:20 +00:00
Nick Craig-Wood
d9458fb4ee b2: return error in Hash from readFileMetadata operation 2016-01-19 08:21:10 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00