TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
612 commits 181 branches 114 tags 335 MiB
Commit graph

40 commits

Author SHA1 Message Date
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
a1323eb204 s3: Fix uploading files bigger than 50GB - fixes #386 2016-03-10 16:48:55 +00:00
Jakub Gedeon
694d390710 s3: Check if directory exists during Mkdir 
If you dont have privs to create a bucket in S3 but it exists, don't
fail with an auth error, but detect that the mkdir was not needed and
return successfully.
 2016-02-27 13:24:46 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Nick Craig-Wood
ce4c1d4f35 s3: Fix empty checks in auth 2016-02-09 17:19:33 +00:00
Brian Stengaard
ce05ef7110 Add IAM role and Env credentials 
This will make the s3 provider authentaction logic

  - Configured credentials if both key and secret available
  - Anonymous if key and secret missing and env_auth not set
  - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool)
    - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables
    - IAM role credentials as fallback
 2016-02-09 16:32:36 +00:00
Nick Craig-Wood
cae19df058 s3: URL escape CopySource 
This fixes metadata update and copy for files with `+` in

Fixes #315
 2016-01-27 17:39:33 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
Joseph Spurrier
fe0289f2f5 s3: Fix corrupting Content-Type on mod time update 
This fixes an issue where updating the modification time resets the
content-type to the S3 default of binary/octet-stream which breaks
static websites that expect an html file to have a content-type of
text/html.
 2016-01-02 11:47:52 +00:00
Nick Craig-Wood
a795d93bc3 swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172 2015-11-07 15:32:40 +00:00
Nick Craig-Wood
b257de4aba Be more constistent with naming in remotes 
* External objects are called Fs and Object
  * Object.fs always points to the Fs
 2015-11-07 11:14:46 +00:00
klauspost
345c98ed62 Update to AWS SDK 0.10.0 
Tested with S3 and Dreamhost

Here is a link to the release notes:

http://aws.amazon.com/releasenotes/5476699172355228
 2015-11-02 19:52:11 +00:00
Nick Craig-Wood
9f1daabb2c s3: allow anonymous access to public repositories - fixes #154 
When setting up the remote, leave both the access key and secret key
blank.
 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
92745aa950 Add Root() to Fs interface 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
d298b578ab s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114 2015-08-28 08:47:41 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99 
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
 2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7 Add Name() to Fs interface to return name as passed to NewFs 2015-08-23 13:36:38 +01:00
Nick Craig-Wood
337110b7a0 s3: remove verbose debug about invalid md5sums for multipart upload 2015-08-16 18:14:22 +01:00
Nick Craig-Wood
6a8de87116 s3: make v2 signatures work for ceph 2015-08-16 15:51:38 +01:00
Nick Craig-Wood
0da6f24221 s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101 2015-08-16 15:51:04 +01:00
Leonid Shalupov
3fcff32524 do not print stats in quiet mode - fixes #70 
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
 2015-08-10 11:17:54 +01:00
Nick Craig-Wood
1d254a3674 Implement --timeout and --conntimeout - fixes #54 
NB dropbox still to do
 2015-05-10 11:29:55 +01:00
Nick Craig-Wood
f88d171afd s3: ignore etags from multipart uploads which aren't md5sums - fixes #56 2015-05-10 11:29:55 +01:00
Nick Craig-Wood
46d39ebaf7 Factor Mime Type guessing into fs.MimeType() 2015-03-02 09:21:15 +00:00
Nick Craig-Wood
85f8552c4d Tidy logging 2015-02-27 15:22:05 +00:00
Nick Craig-Wood
0faed16899 s3: list an unlimited number of items - fixes #22 2015-02-10 17:58:29 +00:00
Nick Craig-Wood
c389616657 all: make private functions / variables / constant which shouldn't be public 2014-07-29 17:50:07 +01:00
Nick Craig-Wood
0b51d6221a s3: make reading metadata more reliable to work around eventual consistency problems 2014-07-29 17:32:06 +01:00
Nick Craig-Wood
125fc8f1f0 s3: strip trailing / from ListDir() 2014-07-24 23:13:33 +01:00
Nick Craig-Wood
282cba20a0 swift, s3, dropbox: fix metadata read on Update() 
This was causing changed files to be marked as corrupted on upload
 2014-07-20 11:23:05 +01:00
Nick Craig-Wood
1c472348b6 s3: Read metadata after Update or Put 2014-07-19 13:05:07 +01:00
Nick Craig-Wood
def9adac4e s3: Treat missing Content-Length as 0 for some ceph installations 2014-05-16 16:27:53 +01:00
Nick Craig-Wood
99695d57ab Implement single file operations for all file systems 2014-05-05 22:17:57 +01:00
Nick Craig-Wood
ca3752f824 s3: support sub-bucket paths 2014-05-05 18:26:37 +01:00
Nick Craig-Wood
02afcb00e9 Factor Fs.Put into Object.Update and call Update rather than Put if possible 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
92ec29fe3f Factor the generic code into fs and add some more intefaces 2014-03-28 22:26:42 +00:00
Nick Craig-Wood
2e5f0ef258 Rename filing systems modules (so I can tell them apart in emacs buffers!) 2014-03-16 10:09:55 +00:00
Renamed from s3/fs.go (Browse further)