Commit graph

82 commits

Author SHA1 Message Date
Nick Craig-Wood
db1995e63a Add MergeDirs optional interface and implement it for drive 2017-08-07 15:32:47 +01:00
Stefan Breunig
28a18303f3 implement rcat – fixes #230, fixes #1001 2017-08-03 21:42:35 +02:00
Nick Craig-Wood
e01741b557 fs: Cleaning up directories in test is no longer needed
..as it is done in the finalise method.
2017-08-03 20:01:05 +01:00
Nick Craig-Wood
cd4895690a fstest: Factor test initialisation into Initialise() 2017-07-26 16:38:33 +01:00
Nick Craig-Wood
e7d04fc103 Create fs.Directory interface and use it everywhere 2017-06-30 14:26:59 +01:00
Nick Craig-Wood
e2d7d413ef fs: rename BasicInfo to DirEntry 2017-06-30 14:26:58 +01:00
Nick Craig-Wood
6fc88ff32e Use --fast-list flag for sync/copy/move - fixes #1277
Redo test framework to take a -fast-list flag and test remotes with that flag.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
0edb025257 Fixup tests with dirs vs bucket based fs 2017-06-14 16:49:40 +01:00
Nick Craig-Wood
7e20e16cff core: Implement Walk directory listing and use in place of Lister
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
2017-06-14 16:49:40 +01:00
Nick Craig-Wood
9a11d3efd9 Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1"
Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is
fixed, this can be reverted.

This reverts commit 320c53eab0.
2017-06-10 13:48:00 +01:00
Nick Craig-Wood
b49821956a Fix copyto/moveto test error (see #1261) 2017-06-07 14:08:46 +01:00
Nick Craig-Wood
74a3dfc4e1 Fix TestHashSums 2017-06-06 23:21:47 +01:00
Stefan Breunig
decd960867 make moveto/copyto no-ops when source and destination are the same (fixes #1261) 2017-05-30 23:01:19 +01:00
Nick Craig-Wood
6381959850 dropbox: support Dropbox content hashing scheme - fixes #1302
* add support to hashing module
  * add dbhashsum to list the hashes
  * add support to dropbox module

This means objects up and downloaded to/from Dropbox will have their
hashes checked.

Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
2017-05-29 12:04:44 +01:00
Nick Craig-Wood
9b07d32c02 onedrive, drive, amazonclouddrive: make sure we find the root
This fixes copyto copying things to the wrong place - fixes #1231
2017-03-16 09:42:49 +00:00
Nick Craig-Wood
320c53eab0 Start Cat tests from 2 as onedrive doesn't support ranging from 1
This has been reported here: https://github.com/OneDrive/onedrive-api-docs/issues/543
2017-03-12 14:24:33 +00:00
Nick Craig-Wood
4d5b73df85 Fix TestListDirSorted eventual consistency listing problems 2017-03-12 14:00:22 +00:00
Nick Craig-Wood
541929258b check: Add --download flag to check all the data, not just hashes 2017-02-13 10:48:26 +00:00
Nick Craig-Wood
ac1c041377 Redo log level flags
* -vv or --log-level DEBUG
  * -v or --log-level INFO
  * --log-level NOTICE (default)
  * -q --log-level ERROR

Replace Config.Verbose and Config.Quiet with Config.LogLevel

Fixes #739 Fixes #1108 Fixes #1000
2017-02-11 20:22:42 +00:00
Nick Craig-Wood
80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood
d091d4a8bb rclone cat: add --head, --tail, --offset, --count and --discard
Fixes #819
2017-02-08 08:09:41 +00:00
Nick Craig-Wood
c3b2b89473 Add ListDirSorted function to list a directory
* fix error return of readFilesFn also
2017-02-04 10:30:25 +00:00
Nick Craig-Wood
85f05c57d1 Clean empty directories between test runs 2017-02-04 10:30:25 +00:00
Nick Craig-Wood
288302c2cf Make fallback purge delete empty directories too.
This was implemented to make the SFTP unit tests pass.
2017-02-04 10:29:46 +00:00
Nick Craig-Wood
1fa258c2b4 Define a new Features() method for Fs
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
2017-01-16 17:33:25 +00:00
Nick Craig-Wood
c123c702ab Fix fs.Overlapping and factor fs.SameConfig 2017-01-14 09:55:53 +00:00
Bjørn Erik Pedersen
47d3a450a4 sync: Track and perform server-side renames
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.

Fixes #888
2017-01-03 20:35:05 +00:00
Nick Craig-Wood
4dd5428b13 Fix rmdirs test and integration tests which depend on each other 2017-01-02 14:15:07 +00:00
Nick Craig-Wood
c265f451f2 Implement moveto and copyto commands for choosing a destination name on copy/move
Fixes #227
Fixes #476
2016-12-03 23:43:52 +00:00
Nick Craig-Wood
539853df36 Fix rmdirs test 2016-11-28 12:23:24 +00:00
Nick Craig-Wood
f3365dd251 Make rclone rmdirs command to delete empty directories - fixes #831 2016-11-27 11:49:31 +00:00
Nick Craig-Wood
aaa1370a36 Add directory parameter to Rmdir and Mkdir #100 #831
This will enable rclone to manage directories properly in the future.
2016-11-26 12:02:53 +00:00
Nick Craig-Wood
84eb7031bb Implement the rclone cat command 2016-08-18 22:45:32 +01:00
Nick Craig-Wood
ac3c83f966 Fix integration tests for drive 2016-07-12 21:38:15 +01:00
Nick Craig-Wood
a6056408dd Fix move command - stop it running for overlapping fses - fixes #577
* Make move command check for overlapping remotes and refuse to run
  * Do copy/delete rather than all the copies then all the deletes
  * Doesn't purge the source - this was unexpected behaviour see #512 and #416
  * Add -list-retries flag to test suite to control retries

This changes the semantics of `move` slightly.  However it now errs on
the side of not deleting stuff.
2016-07-12 10:49:37 +01:00
Nick Craig-Wood
13591c7c00 Redo error handling for sync/copy/move
* Factor sync/copy/move into its own file
  * Make fatal errors abort the sync
  * Make Copy return errors
  * Make Sync/Copy/Move return the last Copy error if there was one
  * Prioritise returning Fatal errors
  * NoRetry errors are returned if no other types of errors
2016-07-04 13:45:53 +01:00
Nick Craig-Wood
70dc97231e Convert more tests to use assert/require 2016-06-30 15:45:30 +01:00
Nick Craig-Wood
af4ef8ad8d Implement --no-traverse flag to stop copy traversing the destination remote.
Refactor sync/copy/move
  * Don't load the src listing unless doing a sync and --delete-before
  * Don't load the dst listing if doing copy/move and --no-traverse is set

`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.

This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.

Fixes #8
Fixes #544
Fixes #546
2016-06-29 17:38:50 +01:00
Nick Craig-Wood
46135d830e Add --ignore-size flag - fixes #399 2016-06-17 17:20:08 +01:00
Nick Craig-Wood
df1092ef33 Change Fs.Put so that it must cope with existing files
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
2016-06-13 19:29:10 +01:00
Nick Craig-Wood
a21cc161de Make 0 size files specifiable with --max-size 0b - fixes #450 2016-06-03 21:54:27 +01:00
Nick Craig-Wood
ccdd1ea6c4 Add --max-depth parameter
This will apply to ls/lsd/sync/copy etc

Fixes #412
Fixes #213
2016-06-03 17:05:39 +01:00
Nick Craig-Wood
68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
b5cba73cc3 Make test more reliable 2016-04-18 17:48:52 +01:00
Nick Craig-Wood
0f8e7c3843 Make rclone check obey the --size-only flag - fixes #419 2016-04-07 15:01:45 +01:00
Nick Craig-Wood
ced3a4bc19 Implement -I, --ignore-times for unconditional upload - fixes #311 2016-03-22 17:02:27 +00:00
Nick Craig-Wood
8ef551bf9c Make dedupe remove identical copies without asking and add non interactive mode - fixes #338
* Now removes identical copies without asking
  * Now obeys `--dry-run`
  * Implement `--dedupe-mode` for non interactive running
    * `--dedupe-mode interactive` - interactive the default.
    * `--dedupe-mode skip` - removes identical files then skips anything left.
    * `--dedupe-mode first` - removes identical files then keeps the first one.
    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
  * Add tests which will only run on Google Drive.
2016-03-06 18:15:01 +00:00
Nick Craig-Wood
229ea3f86c Stop --update tests running on remotes which don't do mod time 2016-03-01 07:26:33 +00:00
Nick Craig-Wood
280ac26464 Implement -u/--update so creation times can be used on all remotes - #226 2016-02-29 17:46:40 +00:00