rclone/README.md

378 lines
11 KiB
Markdown
Raw Normal View History

% rclone(1) User Manual
% Nick Craig-Wood
2014-07-03 20:56:54 +00:00
% Jul 7, 2014
2013-06-27 19:00:01 +00:00
Rclone
======
[![Logo](http://rclone.org/img/rclone-120x120.png)](http://rclone.org/)
2014-04-26 16:43:41 +00:00
Rclone is a command line program to sync files and directories to and from
2013-06-27 19:00:01 +00:00
* Google Drive
* Amazon S3
* Openstack Swift / Rackspace cloud files / Memset Memstore
* Dropbox
* Google Cloud Storage
2013-06-27 19:00:01 +00:00
* The local filesystem
2014-03-15 17:39:56 +00:00
Features
2014-03-15 17:39:56 +00:00
* MD5SUMs checked at all times for file integrity
* Timestamps preserved on files
* Partial syncs supported on a whole file basis
* Copy mode to just copy new/changed files
* Sync mode to make a directory identical
* Check mode to check all MD5SUMs
* Can sync to and from network, eg two different Drive accounts
See the Home page for more documentation and configuration walkthroughs.
* http://rclone.org/
Install
-------
2013-06-27 19:00:01 +00:00
Rclone is a Go program and comes as a single binary file.
Download the binary for your OS from
* http://rclone.org/downloads/
Or alternatively if you have Go installed use
go install github.com/ncw/rclone
and this will build the binary in `$GOPATH/bin`.
2014-03-15 17:39:56 +00:00
Configure
---------
First you'll need to configure rclone. As the object storage systems
have quite complicated authentication these are kept in a config file
`.rclone.conf` in your home directory by default. (You can use the
2014-04-26 16:43:41 +00:00
`--config` option to choose a different config file.)
2014-03-15 17:39:56 +00:00
The easiest way to make the config is to run rclone with the config
option, Eg
2014-03-16 00:28:32 +00:00
rclone config
2014-03-15 17:39:56 +00:00
Usage
-----
2014-03-15 17:39:56 +00:00
Rclone syncs a directory tree from local to remote.
2014-04-26 16:43:41 +00:00
Its basic syntax is
2014-03-15 17:39:56 +00:00
Syntax: [options] subcommand <parameters> <parameters...>
2014-03-16 00:28:32 +00:00
See below for how to specify the source and destination paths.
Subcommands
-----------
2014-03-15 17:39:56 +00:00
2014-03-27 17:31:57 +00:00
rclone copy source:path dest:path
2014-03-15 17:39:56 +00:00
Copy the source to the destination. Doesn't transfer
unchanged files, testing by size and modification time or
2014-03-15 17:39:56 +00:00
MD5SUM. Doesn't delete files from the destination.
2014-03-27 17:31:57 +00:00
rclone sync source:path dest:path
2014-03-15 17:39:56 +00:00
Sync the source to the destination, changing the destination
only. Doesn't transfer unchanged files, testing by size and
modification time or MD5SUM. Destination is updated to match
source, including deleting files if necessary. Since this can
cause data loss, test first with the `--dry-run` flag.
2014-03-15 17:39:56 +00:00
2014-03-27 17:31:57 +00:00
rclone ls [remote:path]
2014-03-15 17:39:56 +00:00
2015-02-28 13:08:18 +00:00
List all the objects in the the path with size and path.
rclone lsd [remote:path]
List all directories/containers/buckets in the the path.
rclone lsl [remote:path]
List all the objects in the the path with modification time,
size and path.
2014-03-15 17:39:56 +00:00
2015-02-28 13:08:18 +00:00
rclone md5sum [remote:path]
2014-03-15 17:39:56 +00:00
2015-02-28 13:08:18 +00:00
Produces an md5sum file for all the objects in the path. This
is in the same format as the standard md5sum tool produces.
2014-03-15 17:39:56 +00:00
2014-03-27 17:31:57 +00:00
rclone mkdir remote:path
2014-03-15 17:39:56 +00:00
Make the path if it doesn't already exist
2014-03-27 17:31:57 +00:00
rclone rmdir remote:path
2014-03-15 17:39:56 +00:00
Remove the path. Note that you can't remove a path with
objects in it, use purge for that.
2014-03-27 17:31:57 +00:00
rclone purge remote:path
2014-03-15 17:39:56 +00:00
Remove the path and all of its contents.
2014-03-27 17:31:57 +00:00
rclone check source:path dest:path
2014-03-15 17:39:56 +00:00
Checks the files in the source and destination match. It
compares sizes and MD5SUMs and prints a report of files which
don't match. It doesn't alter the source or destination.
2015-02-28 13:08:18 +00:00
rclone config
Enter an interactive configuration session.
rclone help
2015-02-28 13:08:18 +00:00
This help.
2014-03-15 17:39:56 +00:00
General options:
2014-04-26 16:43:41 +00:00
```
--bwlimit=0: Bandwidth limit in kBytes/s, or use suffix k|M|G
2014-04-26 16:43:41 +00:00
--checkers=8: Number of checkers to run in parallel.
--config="~/.rclone.conf": Config file.
--contimeout=1m0s: Connect timeout
2014-04-26 16:43:41 +00:00
-n, --dry-run=false: Do a trial run with no permanent changes
--log-file="": Log everything to this file
2014-04-26 16:43:41 +00:00
--modify-window=1ns: Max time diff to be considered the same
-q, --quiet=false: Print as little stuff as possible
--stats=1m0s: Interval to print stats (0 to disable)
--timeout=5m0s: IO idle timeout
2014-04-26 16:43:41 +00:00
--transfers=4: Number of file transfers to run in parallel.
-v, --verbose=false: Print lots more stuff
-V, --version=false: Print the version number
2014-04-26 16:43:41 +00:00
```
2014-03-15 17:39:56 +00:00
Developer options:
2014-04-26 16:43:41 +00:00
```
--cpuprofile="": Write cpu profile to file
```
2014-03-15 17:39:56 +00:00
Local Filesystem
----------------
Paths are specified as normal filesystem paths, so
rclone sync /home/source /tmp/destination
2014-04-26 16:43:41 +00:00
Will sync `/home/source` to `/tmp/destination`
2014-03-15 17:39:56 +00:00
Swift / Rackspace cloudfiles / Memset Memstore
----------------------------------------------
2014-03-27 17:31:57 +00:00
Paths are specified as remote:container (or remote: for the `lsd`
command.) You may put subdirectories in too, eg
`remote:container/path/to/dir`.
2014-03-15 17:39:56 +00:00
2014-03-16 00:28:32 +00:00
So to copy a local directory to a swift container called backup:
2014-03-15 17:39:56 +00:00
2014-03-27 17:31:57 +00:00
rclone sync /home/source swift:backup
2014-03-15 17:39:56 +00:00
For more help see the [online docs on Openstack Swift](http://rclone.org/swift).
2014-03-15 17:39:56 +00:00
Amazon S3
---------
Paths are specified as remote:bucket. You may put subdirectories in
too, eg `remote:bucket/path/to/dir`.
2014-03-15 17:39:56 +00:00
So to copy a local directory to a s3 container called backup
2014-03-27 17:31:57 +00:00
rclone sync /home/source s3:backup
2014-03-15 17:39:56 +00:00
For more help see the [online docs on Amazon S3](http://rclone.org/s3).
2014-03-15 17:39:56 +00:00
Google drive
------------
Paths are specified as remote:path Drive paths may be as deep as required.
2014-03-15 17:39:56 +00:00
The initial setup for drive involves getting a token from Google drive
2014-04-26 16:43:41 +00:00
which you need to do in your browser. `rclone config` walks you
through it.
2014-03-15 17:39:56 +00:00
To copy a local directory to a drive directory called backup
rclone copy /home/source remote:backup
For more help see the [online docs on Google Drive](http://rclone.org/drive).
Dropbox
-------
Paths are specified as remote:path Dropbox paths may be as deep as required.
The initial setup for dropbox involves getting a token from Dropbox
which you need to do in your browser. `rclone config` walks you
through it.
To copy a local directory to a drive directory called backup
rclone copy /home/source dropbox:backup
For more help see the [online docs on Dropbox](http://rclone.org/dropbox).
Google Cloud Storage
--------------------
Paths are specified as remote:path Google Cloud Storage paths may be
as deep as required.
The initial setup for Google Cloud Storage involves getting a token
from Google which you need to do in your browser. `rclone config`
walks you through it.
To copy a local directory to a google cloud storage directory called backup
rclone copy /home/source remote:backup
For more help see the [online docs on Google Cloud Storage](http://rclone.org/googlecloudstorage/).
Single file copies
------------------
Rclone can copy single files
rclone src:path/to/file dst:path/dir
Or
rclone src:path/to/file dst:path/to/file
Note that you can't rename the file if you are copying from one file to another.
License
-------
This is free software under the terms of MIT the license (check the
COPYING file included in this package).
2014-03-15 17:39:56 +00:00
Bugs
----
* Empty directories left behind with Local and Drive
* eg purging a local directory with subdirectories doesn't work
2014-03-15 17:39:56 +00:00
Changelog
---------
2015-05-10 11:39:06 +00:00
* v1.13 - 2015-05-10
* Revise documentation (especially sync)
* Implement --timeout and --conntimeout
* s3: ignore etags from multipart uploads which aren't md5sums
2015-03-15 15:55:38 +00:00
* v1.12 - 2015-03-15
* drive: Use chunked upload for files above a certain size
* drive: add --drive-chunk-size and --drive-upload-cutoff parameters
* drive: switch to insert from update when a failed copy deletes the upload
* core: Log duplicate files if they are detected
2015-03-04 17:59:31 +00:00
* v1.11 - 2015-03-04
* swift: add region parameter
* drive: fix crash on failed to update remote mtime
* In remote paths, change native directory separators to /
* Add synchronization to ls/lsl/lsd output to stop corruptions
* Ensure all stats/log messages to go stderr
* Add --log-file flag to log everything (including panics) to file
* Make it possible to disable stats printing with --stats=0
* Implement --bwlimit to limit data transfer bandwidth
2015-02-12 18:00:20 +00:00
* v1.10 - 2015-02-12
* s3: list an unlimited number of items
* Fix getting stuck in the configurator
2015-02-07 22:44:23 +00:00
* v1.09 - 2015-02-07
* windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:)
* local: Fix directory separators on Windows
* drive: fix rate limit exceeded errors
2015-02-04 22:31:56 +00:00
* v1.08 - 2015-02-04
* drive: fix subdirectory listing to not list entire drive
* drive: Fix SetModTime
* dropbox: adapt code to recent library changes
2014-12-23 11:26:32 +00:00
* v1.07 - 2014-12-23
* google cloud storage: fix memory leak
2014-12-12 20:13:03 +00:00
* v1.06 - 2014-12-12
* Fix "Couldn't find home directory" on OSX
2014-12-23 11:26:32 +00:00
* swift: Add tenant parameter
2014-12-12 20:13:03 +00:00
* Use new location of Google API packages
2014-08-09 16:22:17 +00:00
* v1.05 - 2014-08-09
* Improved tests and consequently lots of minor fixes
* core: Fix race detected by go race detector
* core: Fixes after running errcheck
* drive: reset root directory on Rmdir and Purge
* fs: Document that Purger returns error on empty directory, test and fix
* google cloud storage: fix ListDir on subdirectory
* google cloud storage: re-read metadata in SetModTime
* s3: make reading metadata more reliable to work around eventual consistency problems
* s3: strip trailing / from ListDir()
* swift: return directories without / in ListDir
2014-07-21 20:32:37 +00:00
* v1.04 - 2014-07-21
* google cloud storage: Fix crash on Update
2014-07-20 10:28:50 +00:00
* v1.03 - 2014-07-20
2014-07-21 20:32:37 +00:00
* swift, s3, dropbox: fix updated files being marked as corrupted
* Make compile with go 1.1 again
2014-07-19 12:12:20 +00:00
* v1.02 - 2014-07-19
* Implement Dropbox remote
* Implement Google Cloud Storage remote
* Verify Md5sums and Sizes after copies
* Remove times from "ls" command - lists sizes only
* Add add "lsl" - lists times and sizes
* Add "md5sum" command
2014-07-04 20:15:27 +00:00
* v1.01 - 2014-07-04
* drive: fix transfer of big files using up lots of memory
2014-07-03 20:56:54 +00:00
* v1.00 - 2014-07-03
2014-07-04 20:15:27 +00:00
* drive: fix whole second dates
2014-06-26 16:49:26 +00:00
* v0.99 - 2014-06-26
* Fix --dry-run not working
* Make compatible with go 1.1
2014-05-30 16:29:20 +00:00
* v0.98 - 2014-05-30
* s3: Treat missing Content-Length as 0 for some ceph installations
* rclonetest: add file with a space in
2014-05-05 21:25:29 +00:00
* v0.97 - 2014-05-05
* Implement copying of single files
* s3 & swift: support paths inside containers/buckets
* v0.96 - 2014-04-24
* drive: Fix multiple files of same name being created
* drive: Use o.Update and fs.Put to optimise transfers
* Add version number, -V and --version
* v0.95 - 2014-03-28
* rclone.org: website, docs and graphics
* drive: fix path parsing
* v0.94 - 2014-03-27
* Change remote format one last time
* GNU style flags
* v0.93 - 2014-03-16
* drive: store token in config file
* cross compile other versions
* set strict permissions on config file
* v0.92 - 2014-03-15
* Config fixes and --config option
* v0.91 - 2014-03-15
* Make config file
* v0.90 - 2013-06-27
* Project named rclone
* v0.00 - 2012-11-18
* Project started
Contact and support
-------------------
The project website is at:
* https://github.com/ncw/rclone
2014-04-26 16:43:41 +00:00
There you can file bug reports, ask for help or send pull requests.
Authors
-------
* Nick Craig-Wood <nick@craig-wood.com>
Contributors
------------
* Your name goes here!