TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
1678 commits 181 branches 114 tags 335 MiB
Commit graph

151 commits

Author SHA1 Message Date
Nick Craig-Wood
d2219a800a Fix and document the move command - fixes #334 
* Don't attempt to use server side Move unless they are on the same Fs
  * Fix move in the presense of filters
 2016-02-25 20:05:34 +00:00
klauspost
ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Nick Craig-Wood
a6320bbad3 Fix delete command to wait until all finished - fixes missing deletes. 
This also could affect deletes at the end of the sync command.
 2016-02-15 16:43:59 +00:00
Nick Craig-Wood
0f73129ab7 dedupe command to deduplicate a remote. Useful with google drive - fixes #41 2016-01-31 16:09:42 +00:00
Nick Craig-Wood
1373efaa39 Delete command which does obey the filters - fixes #327 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
5c37b777fc Make the --dry-run warnings into logs so they appear without the -v flag 2016-01-31 16:06:04 +00:00
Nick Craig-Wood
8c211fc8df Warn the user about files with same name but different case 
Relates to #107 & #119.
 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
12b51c5eb8 Remove duplicate check for filter IncludeObject 2016-01-26 16:57:09 +00:00
klauspost
14069fd8e6 Implement --delete-before, --delete-during, --delete-after - fixes #252. 2016-01-26 16:57:09 +00:00
Nick Craig-Wood
cd62f41606 Reduce number of logs and show hash type where appropriate 2016-01-24 18:06:57 +00:00
Nick Craig-Wood
eda4130703 Fix integration tests so they can be run independently and out of order - fixes #291 
* Make all integration tests start with an empty remote
  * Add an -individual flag so this can be a different bucket/container/directory
  * Fix up tests after changing the hashers
  * Add sha1sum test
  * Make directory checking in tests sleep more to fix acd inconsistencies
  * Factor integration tests to make more maintainable
  * Ensure remote writes have a fstest.CheckItems() before use
    * this fixes eventual consistency on the directory listings later
  * Call fs.Stats.ResetCounters() before every fs.Sync()

Note that the tests shouldn't be run concurrently as fs.Config is global state.
 2016-01-23 17:02:09 +00:00
Nick Craig-Wood
27f67edb1a Fix formatting problem in sha1sum 2016-01-17 13:56:42 +00:00
klauspost
78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
klauspost
25f22ec561 Add "--ignore-existing" flag. 
Add option to completely ignore existing files and not consider them for transfer.

Fixes #274
 2016-01-08 08:20:04 +00:00
Nick Craig-Wood
36f1bc4a8a Make ls/lsl/md5sum/size/check obey includes and excludes - fixes #169 
* run check directory listings concurrently
 2015-12-17 13:30:58 +00:00
Adriano Aurélio Meirelles
3cbd57d9ad Add support to filter files based on their age 2015-12-17 09:52:38 -02:00
Nick Craig-Wood
2c2cb84ca7 Make it so optional interface Purge can fail so it can be wrapped 2015-11-14 08:08:51 +00:00
Nick Craig-Wood
80ffbade22 Fix deletion of some excluded files without --delete-excluded #205 
This only happened if the destination file was present but the source
file was missing.
 2015-11-12 11:46:04 +00:00
Klaus Post
6344c3051c Add async readahead buffer 
This adds an async read buffer of 4x4MB when copying files >10MB.

This fixes #164 and reduces the number of IO operations for copy/move.
 2015-10-12 08:30:27 +01:00
Nick Craig-Wood
177dbbc29a Implement rclone size for measuring remotes - fixes #153 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
a91bcaaeb0 Implement rsync like include and exclude - fixes #27 
* Implement include/exclude
  * Implement rsync compatible file globbing
  * Implement command line filtering flags
    * --delete-excluded - Delete files on dest excluded from sync
    * --filter - Add a file-filtering rule
    * --filter-from - Read filtering patterns from a file
    * --exclude - Exclude files matching pattern
    * --exclude-from - Read exclude patterns from file
    * --include - Include files matching pattern
    * --include-from - Read include patterns from file
    * --files-from - Read list of source-file nam
    * --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
    * --max-size - Don't transfer any file larger than this in k or suffix k|M|G
  * Document
 2015-09-28 19:18:21 +01:00
Nick Craig-Wood
5f20ae707d Make lsl output times in localtime and fix tests - fixes #141 2015-09-22 19:04:12 +01:00
Nick Craig-Wood
e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood
2ed158aba3 Fixes from go vet and errcheck 2015-09-22 07:31:12 +01:00
klauspost
0a5870208e Display individual transfer progress 
Improve progress printing by displaying individual file progress, as well
as a moving average speed with ETA. Example output:

2015/09/15 16:38:21
Transferred:    183599104 Bytes (4646.49 kByte/s)
Errors:                 0
Checks:                 1
Transferred:            0
Elapsed time:       38.5s
Transferring:
 * 01_06_14.mp3: 33% done. avg: 1280.5, cur: 1288.8 kByte/s. ETA: 1m12s
 * 01_12_15.mp3: 33% done. avg: 1002.2, cur:  943.4 kByte/s. ETA: 1m17s
 * 01_13_14.mp3: 48% done. avg: 1456.8, cur: 1425.2 kByte/s. ETA: 39s
 * 01_19_15.mp3: 28% done. avg: 1226.9, cur: 1114.4 kByte/s. ETA: 1m37s
 2015-09-16 19:42:46 +01:00
Nick Craig-Wood
8d33ce0154 Check for source and dest being the same in sync/copy/move 2015-09-01 21:49:13 +01:00
Nick Craig-Wood
59ba8f28c8 Implement move command - fixes #35 
* Define Mover interface to move a single object
  * Define DirMover interface to move a directory
  * Implement DirMove operation
  * Add `rclone move` command
  * Tests for Dir Move

To Do
  * Implement Move, DirMover in local, drive, dropbox
  * unit test for Mover
  * unit test for DirMover
 2015-08-28 08:49:16 +01:00
Nick Craig-Wood
a96b522958 Implement server side copies if possible - fixes #99 
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
 2015-08-23 21:18:38 +01:00
Nick Craig-Wood
dfe771fb0c Correct log messages for remotes which don't support modtime/md5sum 2015-08-20 20:48:58 +01:00
Nick Craig-Wood
1d9e76bb0f dropbox: remove datastore - Fixes #55 #84 
This means that dropbox no longer stores MD5SUMs and modified times.

Fixup the tests so that blank MD5SUMs are ignored, and that if
Precision is set to a fs.ModTimeNotSupported, ModTimes can be ignored too.

This opens the door for other FSs which don't support metadata easily.
 2015-08-17 17:26:36 +01:00
Leonid Shalupov
3fcff32524 do not print stats in quiet mode - fixes #70 
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
 2015-08-10 11:17:54 +01:00
Nick Craig-Wood
97dced6a0b Don't check md5sum after download with --size-only - fixes #75 2015-06-09 13:18:40 +01:00
Nick Craig-Wood
e22b445cff Implement --size-only flag to sync on size not checksum & modtime - fixes #75 2015-06-06 08:49:01 +01:00
Alex Couper
7af1282375 Add --checksum flag to only discard transfers by MD5SUM - fixes #61 
Useful for copying between backends where checksum fetching is fast,
ie any of s3, swift, drive or googlecloudstorage.
 2015-06-05 19:46:03 +01:00
Nick Craig-Wood
5cc5429f99 drive: switch to insert from update when a failed copy deletes the upload 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
c643e4585e core: Log duplicate files if they are detected 2015-03-15 15:27:55 +00:00
Nick Craig-Wood
46d39ebaf7 Factor Mime Type guessing into fs.MimeType() 2015-03-02 09:21:15 +00:00
Nick Craig-Wood
2360bf907a Add synchronization to list output to stop corruptions - fixes #29 2015-02-28 15:30:40 +00:00
Nick Craig-Wood
5fb6f94579 drive: fix rate limit exceeded errors - fixes #20 
This is done by pacing the requests to drive and backing them off
using an exponential increase.  Put and Modify operations can now be
retried also.
 2015-02-07 22:32:51 +00:00
Nick Craig-Wood
18439cf2d7 Move rclonetest into go tests for fs module 2014-08-03 11:18:25 +01:00
Nick Craig-Wood
17ffb0855f Fixes after running errcheck 2014-07-25 18:19:49 +01:00
Nick Craig-Wood
e09ef62d5b core: Fix race detected by go race detector 2014-07-22 23:03:14 +01:00
Nick Craig-Wood
e559194fb2 fs: Verify sizes are the same after transfer in Copy() 2014-07-19 13:05:07 +01:00
Nick Craig-Wood
8a76568ea8 core: Verify MD5 sums after each transfer 2014-07-15 19:27:05 +01:00
Nick Craig-Wood
eb2b4ea8aa rclone: Don't purge if --dry-run set 2014-07-13 10:54:30 +01:00
Nick Craig-Wood
e055ed0489 rclone: change "ls" and add "lsl" and "md5sum" commands 
Changed "ls" command not to show modification time by default only
size and path.  That is because it is slow for nearly all the remotes
as it requires extra metadata lookup.  All remotes can look up files
and sizes without extra operations.

Added "lsl" which does what "ls" used to - namely show modification
time, size and path.

Added "md5sum" which produces the same output as the md5sum command -
md5sums and paths that is.
 2014-07-12 12:09:20 +01:00
Nick Craig-Wood
7ebf48ef42 Fix --dry-run not working and add tests for it - fixes #3 2014-06-26 15:33:06 +01:00
Nick Craig-Wood
8fd59f2e7d drive: Use o.Update and fs.Put to optimise transfers 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
02afcb00e9 Factor Fs.Put into Object.Update and call Update rather than Put if possible 2014-04-18 17:49:01 +01:00
Nick Craig-Wood
d6a5bfe2d4 Get rid of fs.CopyFs and replace with fs.Sync in preparation for Object.Update 2014-04-18 17:48:46 +01:00
Nick Craig-Wood
92ec29fe3f Factor the generic code into fs and add some more intefaces 2014-03-28 22:26:42 +00:00