TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
1345 commits 181 branches 114 tags 335 MiB
Commit graph

71 commits

Author SHA1 Message Date
Nick Craig-Wood
8a6a8b9623 Change List interface and add ListR optional interface 
This simplifies the implementation of remotes.  The only required
interface is now `List` which is a simple one level directory list.

Optionally remotes may implement `ListR` if they have an efficient way
of doing a recursive list.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood
20da3e6352 Add options to Put, PutUnchecked and Update, add HashOption and speed up local 
* Add options to Put, PutUnchecked and Update for all Fses
  * Use these to create HashOption
  * Implement this in local
  * Pass the option in fs.Copy

This has the effect that we only calculate hashes we need to in the
local Fs which speeds up transfers significantly.
 2017-05-29 12:04:52 +01:00
Bob Potter
525220b14e Add --local-no-unicode-normalization flag 
Fixes #1411
 2017-05-24 07:36:06 +01:00
Nick Craig-Wood
5355881332 local: fix unnormalised unicode causing problems reading directories #1212 2017-03-16 22:37:56 +00:00
Nick Craig-Wood
fdaac6df67 local: open files in write only mode so they can write to an rclone mount 
Fixes #1195
 2017-03-02 22:03:07 +00:00
Nick Craig-Wood
5fba913207 local: fix detection of directories in new object creation 
This stops the local listing erroring on all symlinks
 2017-02-27 17:03:31 +00:00
Nick Craig-Wood
79e3c67bbd local, yandex, dropbox: fix NewObject suceeding on a directory #1079 
Add tests to make it consistent across all remotes
 2017-02-25 11:09:57 +00:00
Nick Craig-Wood
f3c5745468 Add srcRemote and dstRemote parameters to DirMove #954 2017-02-16 17:42:37 +00:00
Nick Craig-Wood
370f242fa2 local: Fix interaction between -x flag and --max-depth - fixes #1126 
This was causing the by directory sync to ignore the -x flag because
it was putting directories into the listing which should have been
excluded.
 2017-02-13 09:24:29 +00:00
Nick Craig-Wood
006227baed Replace -v with -vv where necessary or change Debugf to Logf 2017-02-11 20:27:46 +00:00
Nick Craig-Wood
80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood
94947f2523 Implement -L, --copy-links flag to allow rclone to follow symlinks 
Fixes #40
 2017-01-29 13:43:20 +00:00
Nick Craig-Wood
1fa258c2b4 Define a new Features() method for Fs 
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
 2017-01-16 17:33:25 +00:00
Nick Craig-Wood
f73ee5eade Make all config file variables be settable in the environment 
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
 2017-01-03 22:42:47 +00:00
Nick Craig-Wood
e79a5de7df local: fix Mkdir/Rmdir with a dir on Windows 2016-12-05 18:09:45 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831 
This will enable rclone to manage directories properly in the future.
 2016-11-26 12:02:53 +00:00
Nick Craig-Wood
441951a93b Stop removing failed upload to cloud storage remotes - fixes #559 
We do remove a partially written file on local so we don't have
corrupted files lying around.
 2016-11-04 21:34:25 +00:00
Nick Craig-Wood
b35123ba48 Make -x/--one-file-system compile under Windows and add docs 2016-11-03 11:53:49 +00:00
Durval Menezes
15c9fed60f local: Implement -x/--one-file-system to stay on a single file system 2016-11-03 11:52:40 +00:00
Nick Craig-Wood
368cce93ff Ignore files with control characters in the names - fixes #689 2016-10-18 15:24:29 +01:00
Nick Craig-Wood
aef2ac5c04 Add options for Open and implement Range for all remotes 2016-10-05 21:03:56 +01:00
Nick Craig-Wood
3e43ff7414 local: windows - ignore the symlink bit on files 
This allows files with reparse points to be backed up.

Fixes #614
 2016-09-19 17:29:22 +01:00
Nick Craig-Wood
bb21cf6f0e local: ignore directory based junction points on windows 
These are a kind of symlink and rclone doesn't follow symlinks.

Fixes #692
 2016-09-19 17:29:22 +01:00
Nick Craig-Wood
6089f443b9 Fix windows build - fixes #628 
Try to make clearer the distinction between OS paths and rclone paths
(remotes) so it is harder to muddle them up.
 2016-08-20 12:29:54 +01:00
Nick Craig-Wood
9eeed25418 local: fix filenames with invalid UTF-8 not being uploaded #568 2016-07-15 14:18:09 +01:00
Nick Craig-Wood
ab43005422 Make NewObject return an error 
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood
1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files 
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
 2016-06-28 08:51:43 +01:00
Klaus Post
e0aa4bb492 Fix incomplete local hashes. 
Fixes #533
 2016-06-19 16:51:49 +02:00
Nick Craig-Wood
4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood
c6a79ff72d Fix remaining places in listing where we were logging errors not returning them 2016-05-30 19:51:15 +01:00
Nick Craig-Wood
b6f1391da3 Fix new style directory listing on windows 2016-05-30 19:44:15 +01:00
Nick Craig-Wood
58befe280c Fix directory name normalisation on OS X 2016-05-28 20:23:37 +01:00
Nick Craig-Wood
c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood
68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
d205dc23e9 Fix oddities using a file in the root - fixes #471 
* Check return from NewFsObject which caused nil ptr deref
  * Correct root directory from "" to string(os.PathSeparator) in getDirFile
 2016-05-06 13:52:50 +01:00
Nick Craig-Wood
7c01bbddf8 Normalise path names for OSX local filesystem 
Fixes #194 Fixes #451 Fixes #463
 2016-05-01 13:13:20 +01:00
Nick Craig-Wood
414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Klaus Post
e57c4406f3 Add mutex to "warned" map. 
Fixes #385
 2016-03-10 15:51:56 +01:00
Nick Craig-Wood
88cca8a6eb Simplify literals (after running gofmt -s over the code) 2016-02-29 16:57:23 +00:00
Nick Craig-Wood
e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Nick Craig-Wood
d0678bc3e5 local: report error on stat in Hash in case file disappeared 2016-01-17 10:46:19 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
klauspost
3c31d711b3 Add local file system option to disable UNC on Windows. 
This will add an option to disable UNC conversion on Windows to deal with buggy file system implementations like EncFS.

Fixes #261
 2016-01-05 17:08:11 +00:00
Nick Craig-Wood
b257de4aba Be more constistent with naming in remotes 
* External objects are called Fs and Object
  * Object.fs always points to the Fs
 2015-11-07 11:14:46 +00:00
klauspost
16c9fba5de Fix tests failing on Windows. 
* ":" is kept when part of a drive.
  * Create tests.
  * Fix test framework.
 2015-10-19 17:36:15 +01:00
klauspost
ab9f521cbd Allow '&' and disallow ':' in Windows filenames. 
Fixes #161
 2015-10-05 11:04:25 +02:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00