TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
1696 commits 181 branches 114 tags 335 MiB
Commit graph

104 commits

Author SHA1 Message Date
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Fabian Ruff
640d7bd365 Add domain option for openstack (v3 auth) 2016-04-22 21:00:54 +01:00
Klaus Post
02c11dd4a7 Don't de-reference swift connection 
The connection object contains a mutex, so it is good practice not to dereference it to a value.

Reported by Go tip "go vet".
 2016-03-23 17:09:05 +00:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood
bd27473762 swift: Don't return an MD5SUM for static large objects - #392 
* rename isManifest to isDynamicLargeObject for clarity
 2016-03-17 17:36:20 +00:00
Nick Craig-Wood
9c263e3e2b Commit missing tests 2016-02-28 20:25:51 +00:00
Nick Craig-Wood
e6f340d245 swift: Fix uploading of chunked files with non ascii characters - fixes #350 2016-02-27 18:59:16 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Xavier Lucas
c2e8f06bfa Swift storageUrl overloading fixes #167 2016-02-09 22:17:13 +00:00
Nick Craig-Wood
c3a0c0c451 swift: Fix upload from unprivileged user - fixes #273 2016-01-23 20:32:53 +00:00
Nick Craig-Wood
ccba859812 Test all available hashes for each remote 2016-01-23 09:10:36 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
xlucas
e7b7432079 OVH Swift authentication enpoint 2016-01-08 08:30:13 +00:00
Nick Craig-Wood
8369b5209f swift: Make sure we read the size for 0 length files - Fixes #237 
This was causing a problem with sync for chunked files.  The directory
listing would read their size back as 0 and see that the size had
changed and immediately resync it.
 2015-12-17 13:30:58 +00:00
Nick Craig-Wood
fcea3777c0 Implement Hubic storage system - fixes #200 2015-11-14 08:08:52 +00:00
Nick Craig-Wood
e8ba43c479 swift: Use ContentType from Object to avoid lookups in listings - fixes #208 2015-11-11 17:19:57 +00:00
Nick Craig-Wood
5bb76cc35c Stop SetModTime losing metadata (eg X-Object-Manifest) - fixes #203 2015-11-11 17:00:08 +00:00
Nick Craig-Wood
a795d93bc3 swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172 2015-11-07 15:32:40 +00:00
Nick Craig-Wood
5df04cb763 swift: ignore directory marker objects where appropriate - fixes #190 
* When creating a LimitedFs
  * When calling List() to list files
  * In the Storable() method
  * Add a Purge() method to delete the directory marker objects too

This is a partial fix for #172
 2015-11-07 15:32:11 +00:00
Nick Craig-Wood
b257de4aba Be more constistent with naming in remotes 
* External objects are called Fs and Object
  * Object.fs always points to the Fs
 2015-11-07 11:14:46 +00:00
Nick Craig-Wood
95fd79faf9 swift: use Content-Length on uploads - fixes #125 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
b79dc01016 swift: stop chunked operations logging "Failed to read info: Object Not Found" 2015-10-03 16:04:51 +01:00
Nick Craig-Wood
4a0a42c2f1 swift: large file upload fixes 
* Read metadata in file listing for 0 length files to fix syncs
  * Ignore non-existent files in isManifestFile to fix errors on copy
  * remove nsToSwiftFloatString - experiments with the swift program
    indicate that it puts a variable number of points after the
    decimal, so might as well use the one in the swift library.
  * Make sure segments get deleted properly when move from segmented
    to non segmented and vice versa
  * Use internal list routine to detect errors on listing
  * Remove the _segments container if possible
  * Remove manifest first when deleting
 2015-09-26 17:58:04 +01:00
Sergey Tolmachev
cc7b9af50e swift: support files > 5GB - fixes #46 
* Write segments to ..._segments container
  * Choice of container and segment names compatible with swift tool
    * See http://docs.openstack.org/developer/swift/overview_large_objects.html
  * Controlled by command line flag --swift-chunk-size
  * Segments removed on delete
 2015-09-26 13:03:58 +01:00
Nick Craig-Wood
68fef49c55 swift: fetch headers as the only source of metadata 2015-09-26 13:03:58 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
e2717a031e Implement Mover and DirMover interfaces fixes #115 
* unit tests
  * local
  * drive
  * dropbox
 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
92745aa950 Add Root() to Fs interface 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99 
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
 2015-08-23 21:18:38 +01:00
Nick Craig-Wood
fedf81c2b7 Add Name() to Fs interface to return name as passed to NewFs 2015-08-23 13:36:38 +01:00
Leonid Shalupov
3fcff32524 do not print stats in quiet mode - fixes #70 
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
 2015-08-10 11:17:54 +01:00
Nick Craig-Wood
1d254a3674 Implement --timeout and --conntimeout - fixes #54 
NB dropbox still to do
 2015-05-10 11:29:55 +01:00
Nick Craig-Wood
c47c9cd440 swift: add region parameter - fixes #38 2015-03-04 17:09:53 +00:00
Nick Craig-Wood
ee6b39aa6c Add tenant parameter for swift - fixes #13 2014-12-12 15:26:08 +00:00
Nick Craig-Wood
d3c16608e4 Test Listing the Root of each Fs 2014-07-31 23:20:39 +01:00
Nick Craig-Wood
3e27ff1b95 Add Root List test and fs.Limited tests for single files 2014-07-31 21:35:29 +01:00
Nick Craig-Wood
c389616657 all: make private functions / variables / constant which shouldn't be public 2014-07-29 17:50:07 +01:00
Nick Craig-Wood
457ef2c190 Automatically generate the tests files for each Fs 2014-07-29 17:18:22 +01:00
Nick Craig-Wood
b013c58537 swift: return directories without / in ListDir 2014-07-24 23:13:33 +01:00
Nick Craig-Wood
a5b0d88608 Make tests for each Fs 
Factor tests out of rclonetest
 2014-07-24 23:13:32 +01:00
Nick Craig-Wood
282cba20a0 swift, s3, dropbox: fix metadata read on Update() 
This was causing changed files to be marked as corrupted on upload
 2014-07-20 11:23:05 +01:00
Nick Craig-Wood
5a8bce6353 swift: Read metadata after Update or Put 2014-07-19 13:05:06 +01:00
Nick Craig-Wood
c1a245d1c8 Factor UserAgent to fs and move Version to fs 2014-07-13 19:19:58 +01:00
Nick Craig-Wood
99695d57ab Implement single file operations for all file systems 2014-05-05 22:17:57 +01:00
Nick Craig-Wood
d0ca58bbb1 swift: Support sub container paths 2014-05-05 18:26:37 +01:00
Nick Craig-Wood
02afcb00e9 Factor Fs.Put into Object.Update and call Update rather than Put if possible 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
92ec29fe3f Factor the generic code into fs and add some more intefaces 2014-03-28 22:26:42 +00:00
Nick Craig-Wood
2e5f0ef258 Rename filing systems modules (so I can tell them apart in emacs buffers!) 2014-03-16 10:09:55 +00:00
Nick Craig-Wood
0a108832e2 Remove filesystem flags and put in config file with editor 2014-03-15 16:06:11 +00:00
Nick Craig-Wood
a34bf2ae5e Make imports absolute and use own copy of goamz pending patch inclusion 2013-07-11 17:37:33 +01:00
Nick Craig-Wood
f1a08ad6bf Namespace the swift options 2013-06-28 18:24:06 +01:00
Nick Craig-Wood
b1f681b912 Rename fs.FsX to fs.X 2013-06-28 08:57:32 +01:00
Nick Craig-Wood
6085dc1b5a Refactor into sub filesystems 2013-06-27 22:17:51 +01:00