Commit graph

391 commits

Author SHA1 Message Date
Nick Craig-Wood
753b0717be Refactor the List and ListDir interface
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
2016-05-06 16:52:34 +01:00
Nick Craig-Wood
cb22583212 b2: Enable mod time syncing - fixes #348 2016-03-22 15:56:44 +00:00
klauspost
ef06371c93 Create separate interface for object information.
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
2016-02-19 13:31:09 +00:00
klauspost
bfd7601cf9 Add configuration file encryption
See #317 for details.

Use `rclone config` to add/change/remove password.

Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.

Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
2016-02-16 16:32:05 +01:00
Nick Craig-Wood
109d4ee490 Prefix all test remotes with rclone-test- and make names more pronouncable 2016-01-24 12:37:46 +00:00
Nick Craig-Wood
eda4130703 Fix integration tests so they can be run independently and out of order - fixes #291
* Make all integration tests start with an empty remote
  * Add an -individual flag so this can be a different bucket/container/directory
  * Fix up tests after changing the hashers
  * Add sha1sum test
  * Make directory checking in tests sleep more to fix acd inconsistencies
  * Factor integration tests to make more maintainable
  * Ensure remote writes have a fstest.CheckItems() before use
    * this fixes eventual consistency on the directory listings later
  * Call fs.Stats.ResetCounters() before every fs.Sync()

Note that the tests shouldn't be run concurrently as fs.Config is global state.
2016-01-23 17:02:09 +00:00
Nick Craig-Wood
ccba859812 Test all available hashes for each remote 2016-01-23 09:10:36 +00:00
Nick Craig-Wood
de3cf5e8d7 Add -verbose flag to unit tests and add some more eventual consistency retries 2016-01-20 20:06:05 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types.
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
2016-01-11 13:39:33 +01:00
dibu28
3ac4407b88 Implement Yandex storage backend - fixes #234 2015-12-30 12:11:46 +00:00
Nick Craig-Wood
acd350d833 Add retry for eventual consistency in findObject test 2015-12-30 10:46:04 +00:00
Nick Craig-Wood
33b3eea6ec Implement Backblaze B2 - fixes #224 2015-12-30 10:05:07 +00:00
Nick Craig-Wood
afaec1a2e9 Use test logger instead of log for test output 2015-12-30 09:35:25 +00:00
Nick Craig-Wood
c270c1c80c Increase retries for eventual consistency in tests 2015-11-14 12:57:17 +00:00
Nick Craig-Wood
fcea3777c0 Implement Hubic storage system - fixes #200 2015-11-14 08:08:52 +00:00
Nick Craig-Wood
b257de4aba Be more constistent with naming in remotes
* External objects are called Fs and Object
  * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Nick Craig-Wood
6d48dffa2f Add -dump-headers and -dump-bodies flags for remote tests 2015-11-07 10:27:47 +00:00
Nick Craig-Wood
be6115fbfa Fix nil pointer exception on test failure 2015-11-07 10:19:10 +00:00
Nick Craig-Wood
2fcb8f5db7 Add support for Microsoft One Drive - fixes #10
* Still to do
    * Copy
    * Move
    * MoveDir
2015-11-07 10:19:10 +00:00
klauspost
16c9fba5de Fix tests failing on Windows.
* ":" is kept when part of a drive.
  * Create tests.
  * Fix test framework.
2015-10-19 17:36:15 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2203766f77 Retry listing in tests to work around eventual consistency 2015-09-14 21:01:25 +01:00
klauspost
f50f353b5d local: always use UNC paths on Windows - fixes #124, fixes #130, fixes #90
* Convert all paths to UNC paths on Windows.
  * Update local filesystem to always use UNC paths.
  * Change tests, so they can work with Windows character replacements.
  * Remove "/" suffix on paths.
  * Always use path/filepath
2015-09-14 19:58:03 +01:00
Nick Craig-Wood
8c3df224ef Implement Amazon Cloud Drive - fixes #45
* Optional interfaces Copier, Mover, DirMover not done
2015-09-08 21:02:48 +01:00
Nick Craig-Wood
e2717a031e Implement Mover and DirMover interfaces fixes #115
* unit tests
  * local
  * drive
  * dropbox
2015-09-01 21:49:13 +01:00
Nick Craig-Wood
59ba8f28c8 Implement move command - fixes #35
* Define Mover interface to move a single object
  * Define DirMover interface to move a directory
  * Implement DirMove operation
  * Add `rclone move` command
  * Tests for Dir Move

To Do
  * Implement Move, DirMover in local, drive, dropbox
  * unit test for Mover
  * unit test for DirMover
2015-08-28 08:49:16 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
2015-08-23 21:18:38 +01:00
Nick Craig-Wood
1d9e76bb0f dropbox: remove datastore - Fixes #55 #84
This means that dropbox no longer stores MD5SUMs and modified times.

Fixup the tests so that blank MD5SUMs are ignored, and that if
Precision is set to a fs.ModTimeNotSupported, ModTimes can be ignored too.

This opens the door for other FSs which don't support metadata easily.
2015-08-17 17:26:36 +01:00
Nick Craig-Wood
40b3c4883f testing: add -remote flag to allow unit tests to run on other remotes
Use this on the individual fs unit tests, eg

    cd s3
    go run -v -remote mytestS3:
2015-08-15 15:40:18 +01:00
Nick Craig-Wood
90744301d3 Fix package docs so they appear in godoc correctly 2015-06-06 14:24:30 +01:00
Nick Craig-Wood
fe68737268 Fix niggles found by go vet 2015-02-28 15:35:54 +00:00
Nick Craig-Wood
b4e19cfd62 windows: make tests work properly 2015-02-07 22:32:51 +00:00
Nick Craig-Wood
78b03929b7 Fix ModTime test 2015-01-04 16:57:55 +00:00
Nick Craig-Wood
492362ec7d Catch nil from List() in tests 2015-01-04 16:23:22 +00:00
Nick Craig-Wood
18439cf2d7 Move rclonetest into go tests for fs module 2014-08-03 11:18:25 +01:00
Nick Craig-Wood
d3c16608e4 Test Listing the Root of each Fs 2014-07-31 23:20:39 +01:00
Nick Craig-Wood
3e27ff1b95 Add Root List test and fs.Limited tests for single files 2014-07-31 21:35:29 +01:00
Nick Craig-Wood
ff91698fb5 Skip tests if test remote not configured 2014-07-31 08:51:39 +01:00
Nick Craig-Wood
2f9f9afac2 fs: Document that Purger returns error on empty directory, test and fix 2014-07-29 17:18:22 +01:00
Nick Craig-Wood
457ef2c190 Automatically generate the tests files for each Fs 2014-07-29 17:18:22 +01:00
Nick Craig-Wood
a5b0d88608 Make tests for each Fs
Factor tests out of rclonetest
2014-07-24 23:13:32 +01:00