7198 lines
189 KiB
Groff
7198 lines
189 KiB
Groff
.\"t
|
|
.\" Automatically generated by Pandoc 1.16.0.2
|
|
.\"
|
|
.TH "rclone" "1" "Jan 02, 2017" "User Manual" ""
|
|
.hy
|
|
.SH Rclone
|
|
.PP
|
|
[IMAGE: Logo (http://rclone.org/img/rclone-120x120.png)] (http://rclone.org/)
|
|
.PP
|
|
Rclone is a command line program to sync files and directories to and
|
|
from
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Amazon S3
|
|
.IP \[bu] 2
|
|
Openstack Swift / Rackspace cloud files / Memset Memstore
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Microsoft One Drive
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Yandex Disk
|
|
.IP \[bu] 2
|
|
The local filesystem
|
|
.PP
|
|
Features
|
|
.IP \[bu] 2
|
|
MD5/SHA1 hashes checked at all times for file integrity
|
|
.IP \[bu] 2
|
|
Timestamps preserved on files
|
|
.IP \[bu] 2
|
|
Partial syncs supported on a whole file basis
|
|
.IP \[bu] 2
|
|
Copy (http://rclone.org/commands/rclone_copy/) mode to just copy
|
|
new/changed files
|
|
.IP \[bu] 2
|
|
Sync (http://rclone.org/commands/rclone_sync/) (one way) mode to make a
|
|
directory identical
|
|
.IP \[bu] 2
|
|
Check (http://rclone.org/commands/rclone_check/) mode to check for file
|
|
hash equality
|
|
.IP \[bu] 2
|
|
Can sync to and from network, eg two different cloud accounts
|
|
.IP \[bu] 2
|
|
Optional encryption (Crypt (http://rclone.org/crypt/))
|
|
.IP \[bu] 2
|
|
Optional FUSE mount (rclone
|
|
mount (http://rclone.org/commands/rclone_mount/))
|
|
.PP
|
|
Links
|
|
.IP \[bu] 2
|
|
Home page (http://rclone.org/)
|
|
.IP \[bu] 2
|
|
Github project page for source and bug
|
|
tracker (http://github.com/ncw/rclone)
|
|
.IP \[bu] 2
|
|
Rclone Forum (https://forum.rclone.org)
|
|
.IP \[bu] 2
|
|
Google+ page
|
|
.IP \[bu] 2
|
|
Downloads (http://rclone.org/downloads/)
|
|
.SH Install
|
|
.PP
|
|
Rclone is a Go program and comes as a single binary file.
|
|
.SS Quickstart
|
|
.IP \[bu] 2
|
|
Download (http://rclone.org/downloads/) the relevant binary.
|
|
.IP \[bu] 2
|
|
Unpack and the \f[C]rclone\f[] binary.
|
|
.IP \[bu] 2
|
|
Run \f[C]rclone\ config\f[] to setup.
|
|
See rclone config docs (http://rclone.org/docs/) for more details.
|
|
.PP
|
|
See below for some expanded Linux / macOS instructions.
|
|
.PP
|
|
See the Usage section (http://rclone.org/docs/) of the docs for how to
|
|
use rclone, or run \f[C]rclone\ \-h\f[].
|
|
.SS Linux installation from precompiled binary
|
|
.PP
|
|
Fetch and unpack
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-O\ http://downloads.rclone.org/rclone\-current\-linux\-amd64.zip
|
|
unzip\ rclone\-current\-linux\-amd64.zip
|
|
cd\ rclone\-*\-linux\-amd64
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Copy binary file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ cp\ rclone\ /usr/sbin/
|
|
sudo\ chown\ root:root\ /usr/sbin/rclone
|
|
sudo\ chmod\ 755\ /usr/sbin/rclone
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Install manpage
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ mkdir\ \-p\ /usr/local/share/man/man1
|
|
sudo\ cp\ rclone.1\ /usr/local/share/man/man1/
|
|
sudo\ mandb\
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone\ config\f[] to setup.
|
|
See rclone config docs (http://rclone.org/docs/) for more details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.SS macOS installation from precompiled binary
|
|
.PP
|
|
Download the latest version of rclone.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cd\ &&\ curl\ \-O\ http://downloads.rclone.org/rclone\-current\-osx\-amd64.zip
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Unzip the download and cd to the extracted folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
unzip\ \-a\ rclone\-current\-osx\-amd64.zip\ &&\ cd\ rclone\-*\-osx\-amd64
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Move rclone to your $PATH.
|
|
You will be prompted for your password.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ mv\ rclone\ /usr/local/bin/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Remove the leftover files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cd\ ..\ &&\ rm\ \-rf\ rclone\-*\-osx\-amd64\ rclone\-current\-osx\-amd64.zip
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone\ config\f[] to setup.
|
|
See rclone config docs (http://rclone.org/docs/) for more details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.SS Install from source
|
|
.PP
|
|
Make sure you have at least Go (https://golang.org/) 1.5 installed.
|
|
Make sure your \f[C]GOPATH\f[] is set, then:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go\ get\ \-u\ \-v\ github.com/ncw/rclone
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
and this will build the binary in \f[C]$GOPATH/bin\f[].
|
|
If you have built rclone before then you will want to update its
|
|
dependencies first with this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go\ get\ \-u\ \-v\ github.com/ncw/rclone/...
|
|
\f[]
|
|
.fi
|
|
.SS Installation with Ansible
|
|
.PP
|
|
This can be done with Stefan Weichinger\[aq]s ansible
|
|
role (https://github.com/stefangweichinger/ansible-rclone).
|
|
.PP
|
|
Instructions
|
|
.IP "1." 3
|
|
\f[C]git\ clone\ https://github.com/stefangweichinger/ansible\-rclone.git\f[]
|
|
into your local roles\-directory
|
|
.IP "2." 3
|
|
add the role to the hosts you want rclone installed to:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \-\ hosts:\ rclone\-hosts
|
|
\ \ \ \ \ \ roles:
|
|
\ \ \ \ \ \ \ \ \ \ \-\ rclone
|
|
\f[]
|
|
.fi
|
|
.SS Configure
|
|
.PP
|
|
First you\[aq]ll need to configure rclone.
|
|
As the object storage systems have quite complicated authentication
|
|
these are kept in a config file \f[C]\&.rclone.conf\f[] in your home
|
|
directory by default.
|
|
(You can use the \f[C]\-\-config\f[] option to choose a different config
|
|
file.)
|
|
.PP
|
|
The easiest way to make the config is to run rclone with the config
|
|
option:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the following for detailed instructions for
|
|
.IP \[bu] 2
|
|
Google drive (http://rclone.org/drive/)
|
|
.IP \[bu] 2
|
|
Amazon S3 (http://rclone.org/s3/)
|
|
.IP \[bu] 2
|
|
Swift / Rackspace Cloudfiles / Memset
|
|
Memstore (http://rclone.org/swift/)
|
|
.IP \[bu] 2
|
|
Dropbox (http://rclone.org/dropbox/)
|
|
.IP \[bu] 2
|
|
Google Cloud Storage (http://rclone.org/googlecloudstorage/)
|
|
.IP \[bu] 2
|
|
Local filesystem (http://rclone.org/local/)
|
|
.IP \[bu] 2
|
|
Amazon Drive (http://rclone.org/amazonclouddrive/)
|
|
.IP \[bu] 2
|
|
Backblaze B2 (http://rclone.org/b2/)
|
|
.IP \[bu] 2
|
|
Hubic (http://rclone.org/hubic/)
|
|
.IP \[bu] 2
|
|
Microsoft One Drive (http://rclone.org/onedrive/)
|
|
.IP \[bu] 2
|
|
Yandex Disk (http://rclone.org/yandex/)
|
|
.IP \[bu] 2
|
|
Crypt (http://rclone.org/crypt/) \- to encrypt other remotes
|
|
.SS Usage
|
|
.PP
|
|
Rclone syncs a directory tree from one storage system to another.
|
|
.PP
|
|
Its syntax is like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Syntax:\ [options]\ subcommand\ <parameters>\ <parameters...>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Source and destination paths are specified by the name you gave the
|
|
storage system in the config file then the sub path, eg "drive:myfolder"
|
|
to look at "myfolder" in Google drive.
|
|
.PP
|
|
You can define as many storage paths as you like in the config file.
|
|
.SS Subcommands
|
|
.PP
|
|
rclone uses a system of subcommands.
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:path\ #\ lists\ a\ re
|
|
rclone\ copy\ /local/path\ remote:path\ #\ copies\ /local/path\ to\ the\ remote
|
|
rclone\ sync\ /local/path\ remote:path\ #\ syncs\ /local/path\ to\ the\ remote
|
|
\f[]
|
|
.fi
|
|
.SS rclone config
|
|
.PP
|
|
Enter an interactive configuration session.
|
|
.SS Synopsis
|
|
.PP
|
|
Enter an interactive configuration session.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.SS rclone copy
|
|
.PP
|
|
Copy files from source to dest, skipping already copied
|
|
.SS Synopsis
|
|
.PP
|
|
Copy the source to the destination.
|
|
Doesn\[aq]t transfer unchanged files, testing by size and modification
|
|
time or MD5SUM.
|
|
Doesn\[aq]t delete files from the destination.
|
|
.PP
|
|
Note that it is always the contents of the directory that is synced, not
|
|
the directory so when source:path is a directory, it\[aq]s the contents
|
|
of source:path that are copied, not the directory name and contents.
|
|
.PP
|
|
If dest:path doesn\[aq]t exist, it is created and the source:path
|
|
contents go there.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ source:sourcepath\ dest:destpath
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Let\[aq]s say there are two files in sourcepath
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sourcepath/one.txt
|
|
sourcepath/two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This copies them to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
destpath/one.txt
|
|
destpath/two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Not to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
destpath/sourcepath/one.txt
|
|
destpath/sourcepath/two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you are familiar with \f[C]rsync\f[], rclone always works as if you
|
|
had written a trailing / \- meaning "copy the contents of this
|
|
directory".
|
|
This applies to all commands and whether you are talking about the
|
|
source or destination.
|
|
.PP
|
|
See the \f[C]\-\-no\-traverse\f[] option for controlling whether rclone
|
|
lists the destination directory or not.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ source:path\ dest:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone sync
|
|
.PP
|
|
Make source and dest identical, modifying destination only.
|
|
.SS Synopsis
|
|
.PP
|
|
Sync the source to the destination, changing the destination only.
|
|
Doesn\[aq]t transfer unchanged files, testing by size and modification
|
|
time or MD5SUM.
|
|
Destination is updated to match source, including deleting files if
|
|
necessary.
|
|
.PP
|
|
\f[B]Important\f[]: Since this can cause data loss, test first with the
|
|
\f[C]\-\-dry\-run\f[] flag to see exactly what would be copied and
|
|
deleted.
|
|
.PP
|
|
Note that files in the destination won\[aq]t be deleted if there were
|
|
any errors at any point.
|
|
.PP
|
|
It is always the contents of the directory that is synced, not the
|
|
directory so when source:path is a directory, it\[aq]s the contents of
|
|
source:path that are copied, not the directory name and contents.
|
|
See extended explanation in the \f[C]copy\f[] command above if unsure.
|
|
.PP
|
|
If dest:path doesn\[aq]t exist, it is created and the source:path
|
|
contents go there.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ source:path\ dest:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone move
|
|
.PP
|
|
Move files from source to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
Moves the contents of the source directory to the destination directory.
|
|
Rclone will error if the source and destination overlap and the remote
|
|
does not support a server side directory move operation.
|
|
.PP
|
|
If no filters are in use and if possible this will server side move
|
|
\f[C]source:path\f[] into \f[C]dest:path\f[].
|
|
After this \f[C]source:path\f[] will no longer longer exist.
|
|
.PP
|
|
Otherwise for each file in \f[C]source:path\f[] selected by the filters
|
|
(if any) this will move it into \f[C]dest:path\f[].
|
|
If possible a server side move will be used, otherwise it will copy it
|
|
(server side if possible) into \f[C]dest:path\f[] then delete the
|
|
original (if no errors on copy) in \f[C]source:path\f[].
|
|
.PP
|
|
\f[B]Important\f[]: Since this can cause data loss, test first with the
|
|
\-\-dry\-run flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ move\ source:path\ dest:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone delete
|
|
.PP
|
|
Remove the contents of path.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the contents of path.
|
|
Unlike \f[C]purge\f[] it obeys include/exclude filters so can be used to
|
|
selectively delete files.
|
|
.PP
|
|
Eg delete all files bigger than 100MBytes
|
|
.PP
|
|
Check what would be deleted first (use either)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-min\-size\ 100M\ lsl\ remote:path
|
|
rclone\ \-\-dry\-run\ \-\-min\-size\ 100M\ delete\ remote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then delete
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-min\-size\ 100M\ delete\ remote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
That reads "delete everything with a minimum size of 100 MB", hence
|
|
delete all files bigger than 100MBytes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ delete\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone purge
|
|
.PP
|
|
Remove the path and all of its contents.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the path and all of its contents.
|
|
Note that this does not obey include/exclude filters \- everything will
|
|
be removed.
|
|
Use \f[C]delete\f[] if you want to selectively delete files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ purge\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone mkdir
|
|
.PP
|
|
Make the path if it doesn\[aq]t already exist.
|
|
.SS Synopsis
|
|
.PP
|
|
Make the path if it doesn\[aq]t already exist.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone rmdir
|
|
.PP
|
|
Remove the path if empty.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the path.
|
|
Note that you can\[aq]t remove a path with objects in it, use purge for
|
|
that.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rmdir\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone check
|
|
.PP
|
|
Checks the files in the source and destination match.
|
|
.SS Synopsis
|
|
.PP
|
|
Checks the files in the source and destination match.
|
|
It compares sizes and MD5SUMs and prints a report of files which
|
|
don\[aq]t match.
|
|
It doesn\[aq]t alter the source or destination.
|
|
.PP
|
|
\f[C]\-\-size\-only\f[] may be used to only compare the sizes, not the
|
|
MD5SUMs.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ check\ source:path\ dest:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone ls
|
|
.PP
|
|
List all the objects in the path with size and path.
|
|
.SS Synopsis
|
|
.PP
|
|
List all the objects in the path with size and path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone lsd
|
|
.PP
|
|
List all directories/containers/buckets in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
List all directories/containers/buckets in the path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone lsl
|
|
.PP
|
|
List all the objects path with modification time, size and path.
|
|
.SS Synopsis
|
|
.PP
|
|
List all the objects path with modification time, size and path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsl\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone md5sum
|
|
.PP
|
|
Produces an md5sum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces an md5sum file for all the objects in the path.
|
|
This is in the same format as the standard md5sum tool produces.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ md5sum\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone sha1sum
|
|
.PP
|
|
Produces an sha1sum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces an sha1sum file for all the objects in the path.
|
|
This is in the same format as the standard sha1sum tool produces.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sha1sum\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone size
|
|
.PP
|
|
Prints the total size and number of objects in remote:path.
|
|
.SS Synopsis
|
|
.PP
|
|
Prints the total size and number of objects in remote:path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ size\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone version
|
|
.PP
|
|
Show the version number.
|
|
.SS Synopsis
|
|
.PP
|
|
Show the version number.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ version
|
|
\f[]
|
|
.fi
|
|
.SS rclone cleanup
|
|
.PP
|
|
Clean up the remote if possible
|
|
.SS Synopsis
|
|
.PP
|
|
Clean up the remote if possible.
|
|
Empty the trash or delete old file versions.
|
|
Not supported by all remotes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cleanup\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone dedupe
|
|
.PP
|
|
Interactively find duplicate files delete/rename them.
|
|
.SS Synopsis
|
|
.PP
|
|
By default \f[C]dedup\f[] interactively finds duplicate files and offers
|
|
to delete all but one or rename them to be different.
|
|
Only useful with Google Drive which can have duplicate file names.
|
|
.PP
|
|
The \f[C]dedupe\f[] command will delete all but one of any identical
|
|
(same md5sum) files it finds without confirmation.
|
|
This means that for most duplicated files the \f[C]dedupe\f[] command
|
|
will not be interactive.
|
|
You can use \f[C]\-\-dry\-run\f[] to see what would happen without doing
|
|
anything.
|
|
.PP
|
|
Here is an example run.
|
|
.PP
|
|
Before \- with duplicates
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsl\ drive:dupes
|
|
\ \ 6048320\ 2016\-03\-05\ 16:23:16.798000000\ one.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:23:11.775000000\ one.txt
|
|
\ \ \ 564374\ 2016\-03\-05\ 16:23:06.731000000\ one.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:18:26.092000000\ one.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:22:46.185000000\ two.txt
|
|
\ \ 1744073\ 2016\-03\-05\ 16:22:38.104000000\ two.txt
|
|
\ \ \ 564374\ 2016\-03\-05\ 16:22:52.118000000\ two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Now the \f[C]dedupe\f[] session
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ dedupe\ drive:dupes
|
|
2016/03/05\ 16:24:37\ Google\ drive\ root\ \[aq]dupes\[aq]:\ Looking\ for\ duplicates\ using\ interactive\ mode.
|
|
one.txt:\ Found\ 4\ duplicates\ \-\ deleting\ identical\ copies
|
|
one.txt:\ Deleting\ 2/3\ identical\ duplicates\ (md5sum\ "1eedaa9fe86fd4b8632e2ac549403b36")
|
|
one.txt:\ 2\ duplicates\ remain
|
|
\ \ 1:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-03\-05\ 16:23:16.798000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
|
|
\ \ 2:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-03\-05\ 16:23:06.731000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
|
|
s)\ Skip\ and\ do\ nothing
|
|
k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
|
|
r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
|
|
s/k/r>\ k
|
|
Enter\ the\ number\ of\ the\ file\ to\ keep>\ 1
|
|
one.txt:\ Deleted\ 1\ extra\ copies
|
|
two.txt:\ Found\ 3\ duplicates\ \-\ deleting\ identical\ copies
|
|
two.txt:\ 3\ duplicates\ remain
|
|
\ \ 1:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-03\-05\ 16:22:52.118000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
|
|
\ \ 2:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-03\-05\ 16:22:46.185000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
|
|
\ \ 3:\ \ \ \ \ \ 1744073\ bytes,\ 2016\-03\-05\ 16:22:38.104000000,\ md5sum\ 851957f7fb6f0bc4ce76be966d336802
|
|
s)\ Skip\ and\ do\ nothing
|
|
k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
|
|
r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
|
|
s/k/r>\ r
|
|
two\-1.txt:\ renamed\ from:\ two.txt
|
|
two\-2.txt:\ renamed\ from:\ two.txt
|
|
two\-3.txt:\ renamed\ from:\ two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The result being
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsl\ drive:dupes
|
|
\ \ 6048320\ 2016\-03\-05\ 16:23:16.798000000\ one.txt
|
|
\ \ \ 564374\ 2016\-03\-05\ 16:22:52.118000000\ two\-1.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:22:46.185000000\ two\-2.txt
|
|
\ \ 1744073\ 2016\-03\-05\ 16:22:38.104000000\ two\-3.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Dedupe can be run non interactively using the \f[C]\-\-dedupe\-mode\f[]
|
|
flag or by using an extra parameter with the same value
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ interactive\f[] \- interactive as above.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ skip\f[] \- removes identical files then skips
|
|
anything left.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ first\f[] \- removes identical files then keeps
|
|
the first one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ newest\f[] \- removes identical files then keeps
|
|
the newest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ oldest\f[] \- removes identical files then keeps
|
|
the oldest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ rename\f[] \- removes identical files then
|
|
renames the rest to be different.
|
|
.PP
|
|
For example to rename all the identically named photos in your Google
|
|
Photos directory, do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dedupe\ \-\-dedupe\-mode\ rename\ "drive:Google\ Photos"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dedupe\ rename\ "drive:Google\ Photos"
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dedupe\ [mode]\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-dedupe\-mode\ string\ \ \ Dedupe\ mode\ interactive|skip|first|newest|oldest|rename.\ (default\ "interactive")
|
|
\f[]
|
|
.fi
|
|
.SS rclone authorize
|
|
.PP
|
|
Remote authorization.
|
|
.SS Synopsis
|
|
.PP
|
|
Remote authorization.
|
|
Used to authorize a remote or headless rclone from a machine with a
|
|
browser \- use as instructed by rclone config.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ authorize
|
|
\f[]
|
|
.fi
|
|
.SS rclone cat
|
|
.PP
|
|
Concatenates any files and sends them to stdout.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cat sends any files to standard output.
|
|
.PP
|
|
You can use it like this to output a single file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cat\ remote:path/to/file
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or like this to output any file in dir or subdirectories.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cat\ remote:path/to/dir
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or like this to output any .txt files in dir or subdirectories.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-include\ "*.txt"\ cat\ remote:path/to/dir
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cat\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone copyto
|
|
.PP
|
|
Copy files from source to dest, skipping already copied
|
|
.SS Synopsis
|
|
.PP
|
|
If source:path is a file or directory then it copies it to a file or
|
|
directory named dest:path.
|
|
.PP
|
|
This can be used to upload single files to other than their current
|
|
name.
|
|
If the source is a directory then it acts exactly like the copy command.
|
|
.PP
|
|
So
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copyto\ src\ dst
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
where src and dst are rclone paths, either remote:path or /path/to/local
|
|
or C:.
|
|
.PP
|
|
This will:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
if\ src\ is\ file
|
|
\ \ \ \ copy\ it\ to\ dst,\ overwriting\ an\ existing\ file\ if\ it\ exists
|
|
if\ src\ is\ directory
|
|
\ \ \ \ copy\ it\ to\ dst,\ overwriting\ existing\ files\ if\ they\ exist
|
|
\ \ \ \ see\ copy\ command\ for\ full\ details
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This doesn\[aq]t transfer unchanged files, testing by size and
|
|
modification time or MD5SUM.
|
|
It doesn\[aq]t delete files from the destination.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copyto\ source:path\ dest:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone genautocomplete
|
|
.PP
|
|
Output bash completion script for rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a bash shell autocompletion script for rclone.
|
|
.PP
|
|
This writes to /etc/bash_completion.d/rclone by default so will probably
|
|
need to be run with sudo or as root, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ rclone\ genautocomplete
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Logout and login again to use the autocompletion scripts, or source them
|
|
directly
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&.\ /etc/bash_completion
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you supply a command line argument the script will be written there.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ genautocomplete\ [output_file]
|
|
\f[]
|
|
.fi
|
|
.SS rclone gendocs
|
|
.PP
|
|
Output markdown docs for rclone to the directory supplied.
|
|
.SS Synopsis
|
|
.PP
|
|
This produces markdown docs for the rclone commands to the directory
|
|
supplied.
|
|
These are in a format suitable for hugo to render into the rclone.org
|
|
website.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ gendocs\ output_directory
|
|
\f[]
|
|
.fi
|
|
.SS rclone listremotes
|
|
.PP
|
|
List all the remotes in the config file.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone listremotes lists all the available remotes from the config file.
|
|
.PP
|
|
When uses with the \-l flag it lists the types too.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ listremotes
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-l,\ \-\-long\ \ \ Show\ the\ type\ as\ well\ as\ names.
|
|
\f[]
|
|
.fi
|
|
.SS rclone mount
|
|
.PP
|
|
Mount the remote as a mountpoint.
|
|
\f[B]EXPERIMENTAL\f[]
|
|
.SS Synopsis
|
|
.PP
|
|
rclone mount allows Linux, FreeBSD and macOS to mount any of
|
|
Rclone\[aq]s cloud storage systems as a file system with FUSE.
|
|
.PP
|
|
This is \f[B]EXPERIMENTAL\f[] \- use with care.
|
|
.PP
|
|
First set up your remote using \f[C]rclone\ config\f[].
|
|
Check it works with \f[C]rclone\ ls\f[] etc.
|
|
.PP
|
|
Start the mount like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mount\ remote:path/to/files\ /path/to/local/mount\ &
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Stop the mount with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
fusermount\ \-u\ /path/to/local/mount
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or with OS X
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
umount\ \-u\ /path/to/local/mount
|
|
\f[]
|
|
.fi
|
|
.SS Limitations
|
|
.PP
|
|
This can only write files seqentially, it can only seek when reading.
|
|
.PP
|
|
Rclone mount inherits rclone\[aq]s directory handling.
|
|
In rclone\[aq]s world directories don\[aq]t really exist.
|
|
This means that empty directories will have a tendency to disappear once
|
|
they fall out of the directory cache.
|
|
.PP
|
|
The bucket based FSes (eg swift, s3, google compute storage, b2)
|
|
won\[aq]t work from the root \- you will need to specify a bucket, or a
|
|
path within the bucket.
|
|
So \f[C]swift:\f[] won\[aq]t work whereas \f[C]swift:bucket\f[] will as
|
|
will \f[C]swift:bucket/path\f[].
|
|
.PP
|
|
Only supported on Linux, FreeBSD and OS X at the moment.
|
|
.SS rclone mount vs rclone sync/copy
|
|
.PP
|
|
File systems expect things to be 100% reliable, whereas cloud storage
|
|
systems are a long way from 100% reliable.
|
|
The rclone sync/copy commands cope with this with lots of retries.
|
|
However rclone mount can\[aq]t use retries in the same way without
|
|
making local copies of the uploads.
|
|
This might happen in the future, but for the moment rclone mount
|
|
won\[aq]t do that, so will be less reliable than the rclone command.
|
|
.SS Bugs
|
|
.IP \[bu] 2
|
|
All the remotes should work for read, but some may not for write
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
those which need to know the size in advance won\[aq]t \- eg B2
|
|
.IP \[bu] 2
|
|
maybe should pass in size as \-1 to mean work it out
|
|
.IP \[bu] 2
|
|
Or put in an an upload cache to cache the files on disk first
|
|
.RE
|
|
.SS TODO
|
|
.IP \[bu] 2
|
|
Check hashes on upload/download
|
|
.IP \[bu] 2
|
|
Preserve timestamps
|
|
.IP \[bu] 2
|
|
Move directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mount\ remote:path\ /path/to/mountpoint
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-allow\-non\-empty\ \ \ \ \ \ \ \ \ \ \ Allow\ mounting\ over\ a\ non\-empty\ directory.
|
|
\ \ \ \ \ \ \-\-allow\-other\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Allow\ access\ to\ other\ users.
|
|
\ \ \ \ \ \ \-\-allow\-root\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Allow\ access\ to\ root\ user.
|
|
\ \ \ \ \ \ \-\-debug\-fuse\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Debug\ the\ FUSE\ internals\ \-\ needs\ \-v.
|
|
\ \ \ \ \ \ \-\-default\-permissions\ \ \ \ \ \ \ Makes\ kernel\ enforce\ access\ control\ based\ on\ the\ file\ mode.
|
|
\ \ \ \ \ \ \-\-dir\-cache\-time\ duration\ \ \ Time\ to\ cache\ directory\ entries\ for.\ (default\ 5m0s)
|
|
\ \ \ \ \ \ \-\-gid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ gid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \ \ \ \ \-\-max\-read\-ahead\ int\ \ \ \ \ \ \ \ The\ number\ of\ bytes\ that\ can\ be\ prefetched\ for\ sequential\ reads.\ (default\ 128k)
|
|
\ \ \ \ \ \ \-\-no\-modtime\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ read\ the\ modification\ time\ (can\ speed\ things\ up).
|
|
\ \ \ \ \ \ \-\-no\-seek\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ allow\ seeking\ in\ files.
|
|
\ \ \ \ \ \ \-\-read\-only\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Mount\ read\-only.
|
|
\ \ \ \ \ \ \-\-uid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ uid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \ \ \ \ \-\-umask\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ permission\ bits\ set\ by\ the\ filesystem.\ (default\ 2)
|
|
\ \ \ \ \ \ \-\-write\-back\-cache\ \ \ \ \ \ \ \ \ \ Makes\ kernel\ buffer\ writes\ before\ sending\ them\ to\ rclone.\ Without\ this,\ writethrough\ caching\ is\ used.
|
|
\f[]
|
|
.fi
|
|
.SS rclone moveto
|
|
.PP
|
|
Move file or directory from source to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
If source:path is a file or directory then it moves it to a file or
|
|
directory named dest:path.
|
|
.PP
|
|
This can be used to rename files or upload single files to other than
|
|
their existing name.
|
|
If the source is a directory then it acts exacty like the move command.
|
|
.PP
|
|
So
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ moveto\ src\ dst
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
where src and dst are rclone paths, either remote:path or /path/to/local
|
|
or C:.
|
|
.PP
|
|
This will:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
if\ src\ is\ file
|
|
\ \ \ \ move\ it\ to\ dst,\ overwriting\ an\ existing\ file\ if\ it\ exists
|
|
if\ src\ is\ directory
|
|
\ \ \ \ move\ it\ to\ dst,\ overwriting\ existing\ files\ if\ they\ exist
|
|
\ \ \ \ see\ move\ command\ for\ full\ details
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This doesn\[aq]t transfer unchanged files, testing by size and
|
|
modification time or MD5SUM.
|
|
src will be deleted on successful transfer.
|
|
.PP
|
|
\f[B]Important\f[]: Since this can cause data loss, test first with the
|
|
\-\-dry\-run flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ moveto\ source:path\ dest:path
|
|
\f[]
|
|
.fi
|
|
.SS rclone rmdirs
|
|
.PP
|
|
Remove any empty directoryies under the path.
|
|
.SS Synopsis
|
|
.PP
|
|
This removes any empty directories (or directories that only contain
|
|
empty directories) under the path that it finds, including the path if
|
|
it has nothing in.
|
|
.PP
|
|
This is useful for tidying up remotes that rclone has left a lot of
|
|
empty directories in.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rmdirs\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS Copying single files
|
|
.PP
|
|
rclone normally syncs or copies directories.
|
|
However if the source remote points to a file, rclone will just copy
|
|
that file.
|
|
The destination remote must point to a directory \- rclone will give the
|
|
error
|
|
\f[C]Failed\ to\ create\ file\ system\ for\ "remote:file":\ is\ a\ file\ not\ a\ directory\f[]
|
|
if it isn\[aq]t.
|
|
.PP
|
|
For example, suppose you have a remote with a file in called
|
|
\f[C]test.jpg\f[], then you could copy just that file like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ remote:test.jpg\ /tmp/download
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The file \f[C]test.jpg\f[] will be placed inside \f[C]/tmp/download\f[].
|
|
.PP
|
|
This is equivalent to specifying
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-no\-traverse\ \-\-files\-from\ /tmp/files\ remote:\ /tmp/download
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Where \f[C]/tmp/files\f[] contains the single line
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
test.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
It is recommended to use \f[C]copy\f[] when copying single files not
|
|
\f[C]sync\f[].
|
|
They have pretty much the same effect but \f[C]copy\f[] will use a lot
|
|
less memory.
|
|
.SS Quoting and the shell
|
|
.PP
|
|
When you are typing commands to your computer you are using something
|
|
called the command line shell.
|
|
This interprets various characters in an OS specific way.
|
|
.PP
|
|
Here are some gotchas which may help users unfamiliar with the shell
|
|
rules
|
|
.SS Linux / OSX
|
|
.PP
|
|
If your names have spaces or shell metacharacters (eg \f[C]*\f[],
|
|
\f[C]?\f[], \f[C]$\f[], \f[C]\[aq]\f[], \f[C]"\f[] etc) then you must
|
|
quote them.
|
|
Use single quotes \f[C]\[aq]\f[] by default.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \[aq]Important\ files?\[aq]\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you want to send a \f[C]\[aq]\f[] you will need to use \f[C]"\f[], eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ "O\[aq]Reilly\ Reviews"\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The rules for quoting metacharacters are complicated and if you want the
|
|
full details you\[aq]ll have to consult the manual page for your shell.
|
|
.SS Windows
|
|
.PP
|
|
If your names have spaces in you need to put them in \f[C]"\f[], eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ "E:\\folder\ name\\folder\ name\\folder\ name"\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you are using the root directory on its own then don\[aq]t quote it
|
|
(see #464 (https://github.com/ncw/rclone/issues/464) for why), eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ E:\\\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Server Side Copy
|
|
.PP
|
|
Drive, S3, Dropbox, Swift and Google Cloud Storage support server side
|
|
copy.
|
|
.PP
|
|
This means if you want to copy one folder to another then rclone
|
|
won\[aq]t download all the files and re\-upload them; it will instruct
|
|
the server to copy them in place.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ s3:oldbucket\ s3:newbucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Will copy the contents of \f[C]oldbucket\f[] to \f[C]newbucket\f[]
|
|
without downloading and re\-uploading.
|
|
.PP
|
|
Remotes which don\[aq]t support server side copy (eg local)
|
|
\f[B]will\f[] download and re\-upload in this case.
|
|
.PP
|
|
Server side copies are used with \f[C]sync\f[] and \f[C]copy\f[] and
|
|
will be identified in the log when using the \f[C]\-v\f[] flag.
|
|
.PP
|
|
Server side copies will only be attempted if the remote names are the
|
|
same.
|
|
.PP
|
|
This can be used when scripting to make aged backups efficiently, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ remote:current\-backup\ remote:previous\-backup
|
|
rclone\ sync\ /path/to/files\ remote:current\-backup
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.PP
|
|
Rclone has a number of options to control its behaviour.
|
|
.PP
|
|
Options which use TIME use the go time parser.
|
|
A duration string is a possibly signed sequence of decimal numbers, each
|
|
with optional fraction and a unit suffix, such as "300ms", "\-1.5h" or
|
|
"2h45m".
|
|
Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
|
|
.PP
|
|
Options which use SIZE use kByte by default.
|
|
However a suffix of \f[C]b\f[] for bytes, \f[C]k\f[] for kBytes,
|
|
\f[C]M\f[] for MBytes and \f[C]G\f[] for GBytes may be used.
|
|
These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively.
|
|
.SS \-\-bwlimit=SIZE
|
|
.PP
|
|
Bandwidth limit in kBytes/s, or use suffix b|k|M|G.
|
|
The default is \f[C]0\f[] which means to not limit bandwidth.
|
|
.PP
|
|
For example to limit bandwidth usage to 10 MBytes/s use
|
|
\f[C]\-\-bwlimit\ 10M\f[]
|
|
.PP
|
|
This only limits the bandwidth of the data transfer, it doesn\[aq]t
|
|
limit the bandwith of the directory listings etc.
|
|
.PP
|
|
Note that the units are Bytes/s not Bits/s.
|
|
Typically connections are measured in Bits/s \- to convert divide by 8.
|
|
For example let\[aq]s say you have a 10 Mbit/s connection and you wish
|
|
rclone to use half of it \- 5 Mbit/s.
|
|
This is 5/8 = 0.625MByte/s so you would use a
|
|
\f[C]\-\-bwlimit\ 0.625M\f[] parameter for rclone.
|
|
.SS \-\-checkers=N
|
|
.PP
|
|
The number of checkers to run in parallel.
|
|
Checkers do the equality checking of files during a sync.
|
|
For some storage systems (eg s3, swift, dropbox) this can take a
|
|
significant amount of time so they are run in parallel.
|
|
.PP
|
|
The default is to run 8 checkers in parallel.
|
|
.SS \-c, \-\-checksum
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check the file hash and size to
|
|
determine if files are equal.
|
|
.PP
|
|
This is useful when the remote doesn\[aq]t support setting modified time
|
|
and a more accurate sync is desired than just checking the file size.
|
|
.PP
|
|
This is very useful when transferring between remotes which store the
|
|
same hash type on the object, eg Drive and Swift.
|
|
For details of which remotes support which hash type see the table in
|
|
the overview section (http://rclone.org/overview/).
|
|
.PP
|
|
Eg \f[C]rclone\ \-\-checksum\ sync\ s3:/bucket\ swift:/bucket\f[] would
|
|
run much quicker than without the \f[C]\-\-checksum\f[] flag.
|
|
.PP
|
|
When using this flag, rclone won\[aq]t update mtimes of remote files if
|
|
they are incorrect as it would normally.
|
|
.SS \-\-config=CONFIG_FILE
|
|
.PP
|
|
Specify the location of the rclone config file.
|
|
Normally this is in your home directory as a file called
|
|
\f[C]\&.rclone.conf\f[].
|
|
If you run \f[C]rclone\ \-h\f[] and look at the help for the
|
|
\f[C]\-\-config\f[] option you will see where the default location is
|
|
for you.
|
|
Use this flag to override the config location, eg
|
|
\f[C]rclone\ \-\-config=".myconfig"\ .config\f[].
|
|
.SS \-\-contimeout=TIME
|
|
.PP
|
|
Set the connection timeout.
|
|
This should be in go time format which looks like \f[C]5s\f[] for 5
|
|
seconds, \f[C]10m\f[] for 10 minutes, or \f[C]3h30m\f[].
|
|
.PP
|
|
The connection timeout is the amount of time rclone will wait for a
|
|
connection to go through to a remote object storage system.
|
|
It is \f[C]1m\f[] by default.
|
|
.SS \-\-dedupe\-mode MODE
|
|
.PP
|
|
Mode to run dedupe command in.
|
|
One of \f[C]interactive\f[], \f[C]skip\f[], \f[C]first\f[],
|
|
\f[C]newest\f[], \f[C]oldest\f[], \f[C]rename\f[].
|
|
The default is \f[C]interactive\f[].
|
|
See the dedupe command for more information as to what these options
|
|
mean.
|
|
.SS \-n, \-\-dry\-run
|
|
.PP
|
|
Do a trial run with no permanent changes.
|
|
Use this to see what rclone would do without actually doing it.
|
|
Useful when setting up the \f[C]sync\f[] command which deletes files in
|
|
the destination.
|
|
.SS \-\-ignore\-existing
|
|
.PP
|
|
Using this option will make rclone unconditionally skip all files that
|
|
exist on the destination, no matter the content of these files.
|
|
.PP
|
|
While this isn\[aq]t a generally recommended option, it can be useful in
|
|
cases where your files change due to encryption.
|
|
However, it cannot correct partial transfers in case a transfer was
|
|
interrupted.
|
|
.SS \-\-ignore\-size
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check only the modification time.
|
|
If \f[C]\-\-checksum\f[] is set then it only checks the checksum.
|
|
.PP
|
|
It will also cause rclone to skip verifying the sizes are the same after
|
|
transfer.
|
|
.PP
|
|
This can be useful for transferring files to and from onedrive which
|
|
occasionally misreports the size of image files (see
|
|
#399 (https://github.com/ncw/rclone/issues/399) for more info).
|
|
.SS \-I, \-\-ignore\-times
|
|
.PP
|
|
Using this option will cause rclone to unconditionally upload all files
|
|
regardless of the state of files on the destination.
|
|
.PP
|
|
Normally rclone would skip any files that have the same modification
|
|
time and are the same size (or have the same checksum if using
|
|
\f[C]\-\-checksum\f[]).
|
|
.SS \-\-log\-file=FILE
|
|
.PP
|
|
Log all of rclone\[aq]s output to FILE.
|
|
This is not active by default.
|
|
This can be useful for tracking down problems with syncs in combination
|
|
with the \f[C]\-v\f[] flag.
|
|
See the Logging section for more info.
|
|
.SS \-\-low\-level\-retries NUMBER
|
|
.PP
|
|
This controls the number of low level retries rclone does.
|
|
.PP
|
|
A low level retry is used to retry a failing operation \- typically one
|
|
HTTP request.
|
|
This might be uploading a chunk of a big file for example.
|
|
You will see low level retries in the log with the \f[C]\-v\f[] flag.
|
|
.PP
|
|
This shouldn\[aq]t need to be changed from the default in normal
|
|
operations, however if you get a lot of low level retries you may wish
|
|
to reduce the value so rclone moves on to a high level retry (see the
|
|
\f[C]\-\-retries\f[] flag) quicker.
|
|
.PP
|
|
Disable low level retries with \f[C]\-\-low\-level\-retries\ 1\f[].
|
|
.SS \-\-max\-depth=N
|
|
.PP
|
|
This modifies the recursion depth for all the commands except purge.
|
|
.PP
|
|
So if you do \f[C]rclone\ \-\-max\-depth\ 1\ ls\ remote:path\f[] you
|
|
will see only the files in the top level directory.
|
|
Using \f[C]\-\-max\-depth\ 2\f[] means you will see all the files in
|
|
first two directory levels and so on.
|
|
.PP
|
|
For historical reasons the \f[C]lsd\f[] command defaults to using a
|
|
\f[C]\-\-max\-depth\f[] of 1 \- you can override this with the command
|
|
line flag.
|
|
.PP
|
|
You can use this command to disable recursion (with
|
|
\f[C]\-\-max\-depth\ 1\f[]).
|
|
.PP
|
|
Note that if you use this with \f[C]sync\f[] and
|
|
\f[C]\-\-delete\-excluded\f[] the files not recursed through are
|
|
considered excluded and will be deleted on the destination.
|
|
Test first with \f[C]\-\-dry\-run\f[] if you are not sure what will
|
|
happen.
|
|
.SS \-\-modify\-window=TIME
|
|
.PP
|
|
When checking whether a file has been modified, this is the maximum
|
|
allowed time difference that a file can have and still be considered
|
|
equivalent.
|
|
.PP
|
|
The default is \f[C]1ns\f[] unless this is overridden by a remote.
|
|
For example OS X only stores modification times to the nearest second so
|
|
if you are reading and writing to an OS X filing system this will be
|
|
\f[C]1s\f[] by default.
|
|
.PP
|
|
This command line flag allows you to override that computed default.
|
|
.SS \-\-no\-gzip\-encoding
|
|
.PP
|
|
Don\[aq]t set \f[C]Accept\-Encoding:\ gzip\f[].
|
|
This means that rclone won\[aq]t ask the server for compressed files
|
|
automatically.
|
|
Useful if you\[aq]ve set the server to return files with
|
|
\f[C]Content\-Encoding:\ gzip\f[] but you uploaded compressed files.
|
|
.PP
|
|
There is no need to set this in normal operation, and doing so will
|
|
decrease the network transfer efficiency of rclone.
|
|
.SS \-\-no\-update\-modtime
|
|
.PP
|
|
When using this flag, rclone won\[aq]t update modification times of
|
|
remote files if they are incorrect as it would normally.
|
|
.PP
|
|
This can be used if the remote is being synced with another tool also
|
|
(eg the Google Drive client).
|
|
.SS \-q, \-\-quiet
|
|
.PP
|
|
Normally rclone outputs stats and a completion message.
|
|
If you set this flag it will make as little output as possible.
|
|
.SS \-\-retries int
|
|
.PP
|
|
Retry the entire sync if it fails this many times it fails (default 3).
|
|
.PP
|
|
Some remotes can be unreliable and a few retries helps pick up the files
|
|
which didn\[aq]t get transferred because of errors.
|
|
.PP
|
|
Disable retries with \f[C]\-\-retries\ 1\f[].
|
|
.SS \-\-size\-only
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check only the size.
|
|
.PP
|
|
This can be useful transferring files from dropbox which have been
|
|
modified by the desktop sync client which doesn\[aq]t set checksums of
|
|
modification times in the same way as rclone.
|
|
.SS \-\-stats=TIME
|
|
.PP
|
|
Commands which transfer data (\f[C]sync\f[], \f[C]copy\f[],
|
|
\f[C]copyto\f[], \f[C]move\f[], \f[C]moveto\f[]) will print data
|
|
transfer stats at regular intervals to show their progress.
|
|
.PP
|
|
This sets the interval.
|
|
.PP
|
|
The default is \f[C]1m\f[].
|
|
Use 0 to disable.
|
|
.PP
|
|
If you set the stats interval then all command can show stats.
|
|
This can be useful when running other commands, \f[C]check\f[] or
|
|
\f[C]mount\f[] for example.
|
|
.SS \-\-stats\-unit=bits|bytes
|
|
.PP
|
|
By default data transfer rates will be printed in bytes/second.
|
|
.PP
|
|
This option allows the data rate to be printed in bits/second.
|
|
.PP
|
|
Data transfer volume will still be reported in bytes.
|
|
.PP
|
|
The rate is reported as a binary unit, not SI unit.
|
|
So 1 Mbit/s equals 1,048,576 bits/s and not 1,000,000 bits/s.
|
|
.PP
|
|
The default is \f[C]bytes\f[].
|
|
.SS \-\-delete\-(before,during,after)
|
|
.PP
|
|
This option allows you to specify when files on your destination are
|
|
deleted when you sync folders.
|
|
.PP
|
|
Specifying the value \f[C]\-\-delete\-before\f[] will delete all files
|
|
present on the destination, but not on the source \f[I]before\f[]
|
|
starting the transfer of any new or updated files.
|
|
This uses extra memory as it has to store the source listing before
|
|
proceeding.
|
|
.PP
|
|
Specifying \f[C]\-\-delete\-during\f[] (default value) will delete files
|
|
while checking and uploading files.
|
|
This is usually the fastest option.
|
|
Currently this works the same as \f[C]\-\-delete\-after\f[] but it may
|
|
change in the future.
|
|
.PP
|
|
Specifying \f[C]\-\-delete\-after\f[] will delay deletion of files until
|
|
all new/updated files have been successfully transfered.
|
|
.SS \-\-timeout=TIME
|
|
.PP
|
|
This sets the IO idle timeout.
|
|
If a transfer has started but then becomes idle for this long it is
|
|
considered broken and disconnected.
|
|
.PP
|
|
The default is \f[C]5m\f[].
|
|
Set to 0 to disable.
|
|
.SS \-\-transfers=N
|
|
.PP
|
|
The number of file transfers to run in parallel.
|
|
It can sometimes be useful to set this to a smaller number if the remote
|
|
is giving a lot of timeouts or bigger if you have lots of bandwidth and
|
|
a fast remote.
|
|
.PP
|
|
The default is to run 4 file transfers in parallel.
|
|
.SS \-u, \-\-update
|
|
.PP
|
|
This forces rclone to skip any files which exist on the destination and
|
|
have a modified time that is newer than the source file.
|
|
.PP
|
|
If an existing destination file has a modification time equal (within
|
|
the computed modify window precision) to the source file\[aq]s, it will
|
|
be updated if the sizes are different.
|
|
.PP
|
|
On remotes which don\[aq]t support mod time directly the time checked
|
|
will be the uploaded time.
|
|
This means that if uploading to one of these remoes, rclone will skip
|
|
any files which exist on the destination and have an uploaded time that
|
|
is newer than the modification time of the source file.
|
|
.PP
|
|
This can be useful when transferring to a remote which doesn\[aq]t
|
|
support mod times directly as it is more accurate than a
|
|
\f[C]\-\-size\-only\f[] check and faster than using
|
|
\f[C]\-\-checksum\f[].
|
|
.SS \-v, \-\-verbose
|
|
.PP
|
|
If you set this flag, rclone will become very verbose telling you about
|
|
every file it considers and transfers.
|
|
.PP
|
|
Very useful for debugging.
|
|
.SS \-V, \-\-version
|
|
.PP
|
|
Prints the version number
|
|
.SS Configuration Encryption
|
|
.PP
|
|
Your configuration file contains information for logging in to your
|
|
cloud services.
|
|
This means that you should keep your \f[C]\&.rclone.conf\f[] file in a
|
|
secure location.
|
|
.PP
|
|
If you are in an environment where that isn\[aq]t possible, you can add
|
|
a password to your configuration.
|
|
This means that you will have to enter the password every time you start
|
|
rclone.
|
|
.PP
|
|
To add a password to your rclone configuration, execute
|
|
\f[C]rclone\ config\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
>rclone\ config
|
|
Current\ remotes:
|
|
|
|
e)\ Edit\ existing\ remote
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
e/n/d/s/q>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Go into \f[C]s\f[], Set configuration password:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
e/n/d/s/q>\ s
|
|
Your\ configuration\ is\ not\ encrypted.
|
|
If\ you\ add\ a\ password,\ you\ will\ protect\ your\ login\ information\ to\ cloud\ services.
|
|
a)\ Add\ Password
|
|
q)\ Quit\ to\ main\ menu
|
|
a/q>\ a
|
|
Enter\ NEW\ configuration\ password:
|
|
password:
|
|
Confirm\ NEW\ password:
|
|
password:
|
|
Password\ set
|
|
Your\ configuration\ is\ encrypted.
|
|
c)\ Change\ Password
|
|
u)\ Unencrypt\ configuration
|
|
q)\ Quit\ to\ main\ menu
|
|
c/u/q>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Your configuration is now encrypted, and every time you start rclone you
|
|
will now be asked for the password.
|
|
In the same menu you can change the password or completely remove
|
|
encryption from your configuration.
|
|
.PP
|
|
There is no way to recover the configuration if you lose your password.
|
|
.PP
|
|
rclone uses nacl
|
|
secretbox (https://godoc.org/golang.org/x/crypto/nacl/secretbox) which
|
|
in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your
|
|
configuration with secret\-key cryptography.
|
|
The password is SHA\-256 hashed, which produces the key for secretbox.
|
|
The hashed password is not stored.
|
|
.PP
|
|
While this provides very good security, we do not recommend storing your
|
|
encrypted rclone configuration in public if it contains sensitive
|
|
information, maybe except if you use a very strong password.
|
|
.PP
|
|
If it is safe in your environment, you can set the
|
|
\f[C]RCLONE_CONFIG_PASS\f[] environment variable to contain your
|
|
password, in which case it will be used for decrypting the
|
|
configuration.
|
|
.PP
|
|
You can set this for a session from a script.
|
|
For unix like systems save this to a file called
|
|
\f[C]set\-rclone\-password\f[]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#!/bin/echo\ Source\ this\ file\ don\[aq]t\ run\ it
|
|
|
|
read\ \-s\ RCLONE_CONFIG_PASS
|
|
export\ RCLONE_CONFIG_PASS
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then source the file when you want to use it.
|
|
From the shell you would do \f[C]source\ set\-rclone\-password\f[].
|
|
It will then ask you for the password and set it in the envonment
|
|
variable.
|
|
.PP
|
|
If you are running rclone inside a script, you might want to disable
|
|
password prompts.
|
|
To do that, pass the parameter \f[C]\-\-ask\-password=false\f[] to
|
|
rclone.
|
|
This will make rclone fail instead of asking for a password if
|
|
\f[C]RCLONE_CONFIG_PASS\f[] doesn\[aq]t contain a valid password.
|
|
.SS Developer options
|
|
.PP
|
|
These options are useful when developing or debugging rclone.
|
|
There are also some more remote specific options which aren\[aq]t
|
|
documented here which are used for testing.
|
|
These start with remote name eg \f[C]\-\-drive\-test\-option\f[] \- see
|
|
the docs for the remote in question.
|
|
.SS \-\-cpuprofile=FILE
|
|
.PP
|
|
Write CPU profile to file.
|
|
This can be analysed with \f[C]go\ tool\ pprof\f[].
|
|
.SS \-\-dump\-auth
|
|
.PP
|
|
Dump HTTP headers \- will contain sensitive info such as
|
|
\f[C]Authorization:\f[] headers \- use \f[C]\-\-dump\-headers\f[] to
|
|
dump without \f[C]Authorization:\f[] headers.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.SS \-\-dump\-bodies
|
|
.PP
|
|
Dump HTTP headers and bodies \- may contain sensitive info.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.SS \-\-dump\-filters
|
|
.PP
|
|
Dump the filters to the output.
|
|
Useful to see exactly what include and exclude options are filtering on.
|
|
.SS \-\-dump\-headers
|
|
.PP
|
|
Dump HTTP headers with \f[C]Authorization:\f[] lines removed.
|
|
May still contain sensitive info.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.PP
|
|
Use \f[C]\-\-dump\-auth\f[] if you do want the \f[C]Authorization:\f[]
|
|
headers.
|
|
.SS \-\-memprofile=FILE
|
|
.PP
|
|
Write memory profile to file.
|
|
This can be analysed with \f[C]go\ tool\ pprof\f[].
|
|
.SS \-\-no\-check\-certificate=true/false
|
|
.PP
|
|
\f[C]\-\-no\-check\-certificate\f[] controls whether a client verifies
|
|
the server\[aq]s certificate chain and host name.
|
|
If \f[C]\-\-no\-check\-certificate\f[] is true, TLS accepts any
|
|
certificate presented by the server and any host name in that
|
|
certificate.
|
|
In this mode, TLS is susceptible to man\-in\-the\-middle attacks.
|
|
.PP
|
|
This option defaults to \f[C]false\f[].
|
|
.PP
|
|
\f[B]This should be used only for testing.\f[]
|
|
.SS \-\-no\-traverse
|
|
.PP
|
|
The \f[C]\-\-no\-traverse\f[] flag controls whether the destination file
|
|
system is traversed when using the \f[C]copy\f[] or \f[C]move\f[]
|
|
commands.
|
|
.PP
|
|
If you are only copying a small number of files and/or have a large
|
|
number of files on the destination then \f[C]\-\-no\-traverse\f[] will
|
|
stop rclone listing the destination and save time.
|
|
.PP
|
|
However if you are copying a large number of files, escpecially if you
|
|
are doing a copy where lots of the files haven\[aq]t changed and
|
|
won\[aq]t need copying then you shouldn\[aq]t use
|
|
\f[C]\-\-no\-traverse\f[].
|
|
.PP
|
|
It can also be used to reduce the memory usage of rclone when copying \-
|
|
\f[C]rclone\ \-\-no\-traverse\ copy\ src\ dst\f[] won\[aq]t load either
|
|
the source or destination listings into memory so will use the minimum
|
|
amount of memory.
|
|
.SS Filtering
|
|
.PP
|
|
For the filtering options
|
|
.IP \[bu] 2
|
|
\f[C]\-\-delete\-excluded\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-files\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-min\-size\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-max\-size\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-min\-age\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-max\-age\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dump\-filters\f[]
|
|
.PP
|
|
See the filtering section (http://rclone.org/filtering/).
|
|
.SS Logging
|
|
.PP
|
|
rclone has 3 levels of logging, \f[C]Error\f[], \f[C]Info\f[] and
|
|
\f[C]Debug\f[].
|
|
.PP
|
|
By default rclone logs \f[C]Error\f[] and \f[C]Info\f[] to standard
|
|
error and \f[C]Debug\f[] to standard output.
|
|
This means you can redirect standard output and standard error to
|
|
different places.
|
|
.PP
|
|
By default rclone will produce \f[C]Error\f[] and \f[C]Info\f[] level
|
|
messages.
|
|
.PP
|
|
If you use the \f[C]\-q\f[] flag, rclone will only produce
|
|
\f[C]Error\f[] messages.
|
|
.PP
|
|
If you use the \f[C]\-v\f[] flag, rclone will produce \f[C]Error\f[],
|
|
\f[C]Info\f[] and \f[C]Debug\f[] messages.
|
|
.PP
|
|
If you use the \f[C]\-\-log\-file=FILE\f[] option, rclone will redirect
|
|
\f[C]Error\f[], \f[C]Info\f[] and \f[C]Debug\f[] messages along with
|
|
standard error to FILE.
|
|
.SS Exit Code
|
|
.PP
|
|
If any errors occurred during the command, rclone with an exit code of
|
|
\f[C]1\f[].
|
|
This allows scripts to detect when rclone operations have failed.
|
|
.PP
|
|
During the startup phase rclone will exit immediately if an error is
|
|
detected in the configuration.
|
|
There will always be a log message immediately before exiting.
|
|
.PP
|
|
When rclone is running it will accumulate errors as it goes along, and
|
|
only exit with an non\-zero exit code if (after retries) there were no
|
|
transfers with errors remaining.
|
|
For every error counted there will be a high priority log message
|
|
(visibile with \f[C]\-q\f[]) showing the message and which file caused
|
|
the problem.
|
|
A high priority message is also shown when starting a retry so the user
|
|
can see that any previous error messages may not be valid after the
|
|
retry.
|
|
If rclone has done a retry it will log a high priority message if the
|
|
retry was successful.
|
|
.SH Configuring rclone on a remote / headless machine
|
|
.PP
|
|
Some of the configurations (those involving oauth2) require an Internet
|
|
connected web browser.
|
|
.PP
|
|
If you are trying to set rclone up on a remote or headless box with no
|
|
browser available on it (eg a NAS or a server in a datacenter) then you
|
|
will need to use an alternative means of configuration.
|
|
There are two ways of doing it, described below.
|
|
.SS Configuring using rclone authorize
|
|
.PP
|
|
On the headless box
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&...
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ n
|
|
For\ this\ to\ work,\ you\ will\ need\ rclone\ available\ on\ a\ machine\ that\ has\ a\ web\ browser\ available.
|
|
Execute\ the\ following\ on\ your\ machine:
|
|
\ \ \ \ rclone\ authorize\ "amazon\ cloud\ drive"
|
|
Then\ paste\ the\ result\ below:
|
|
result>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then on your main desktop machine
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ authorize\ "amazon\ cloud\ drive"
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
Paste\ the\ following\ into\ your\ remote\ machine\ \-\-\->
|
|
SECRET_TOKEN
|
|
<\-\-\-End\ paste
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then back to the headless box, paste in the code
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
result>\ SECRET_TOKEN
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[acd12]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ SECRET_TOKEN
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>
|
|
\f[]
|
|
.fi
|
|
.SS Configuring by copying the config file
|
|
.PP
|
|
Rclone stores all of its config in a single configuration file.
|
|
This can easily be copied to configure a remote rclone.
|
|
.PP
|
|
So first configure rclone on your desktop machine
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
to set up the config file.
|
|
.PP
|
|
Find the config file by running \f[C]rclone\ \-h\f[] and looking for the
|
|
help for the \f[C]\-\-config\f[] option
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-h
|
|
[snip]
|
|
\ \ \ \ \ \ \-\-config="/home/user/.rclone.conf":\ Config\ file.
|
|
[snip]
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Now transfer it to the remote box (scp, cut paste, ftp, sftp etc) and
|
|
place it in the correct place (use \f[C]rclone\ \-h\f[] on the remote
|
|
box to find out where).
|
|
.SH Filtering, includes and excludes
|
|
.PP
|
|
Rclone has a sophisticated set of include and exclude rules.
|
|
Some of these are based on patterns and some on other things like file
|
|
size.
|
|
.PP
|
|
The filters are applied for the \f[C]copy\f[], \f[C]sync\f[],
|
|
\f[C]move\f[], \f[C]ls\f[], \f[C]lsl\f[], \f[C]md5sum\f[],
|
|
\f[C]sha1sum\f[], \f[C]size\f[], \f[C]delete\f[] and \f[C]check\f[]
|
|
operations.
|
|
Note that \f[C]purge\f[] does not obey the filters.
|
|
.PP
|
|
Each path as it passes through rclone is matched against the include and
|
|
exclude rules like \f[C]\-\-include\f[], \f[C]\-\-exclude\f[],
|
|
\f[C]\-\-include\-from\f[], \f[C]\-\-exclude\-from\f[],
|
|
\f[C]\-\-filter\f[], or \f[C]\-\-filter\-from\f[].
|
|
The simplest way to try them out is using the \f[C]ls\f[] command, or
|
|
\f[C]\-\-dry\-run\f[] together with \f[C]\-v\f[].
|
|
.SS Patterns
|
|
.PP
|
|
The patterns used to match files for inclusion or exclusion are based on
|
|
"file globs" as used by the unix shell.
|
|
.PP
|
|
If the pattern starts with a \f[C]/\f[] then it only matches at the top
|
|
level of the directory tree, \f[B]relative to the root of the remote\f[]
|
|
(not necessarily the root of the local drive).
|
|
If it doesn\[aq]t start with \f[C]/\f[] then it is matched starting at
|
|
the \f[B]end of the path\f[], but it will only match a complete path
|
|
element:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
file.jpg\ \ \-\ matches\ "file.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ matches\ "directory/file.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "afile.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "directory/afile.jpg"
|
|
/file.jpg\ \-\ matches\ "file.jpg"\ in\ the\ root\ directory\ of\ the\ remote
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "afile.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "directory/file.jpg"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
\f[B]Important\f[] Note that you must use \f[C]/\f[] in patterns and not
|
|
\f[C]\\\f[] even if running on Windows.
|
|
.PP
|
|
A \f[C]*\f[] matches anything but not a \f[C]/\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
*.jpg\ \ \-\ matches\ "file.jpg"
|
|
\ \ \ \ \ \ \ \-\ matches\ "directory/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "file.jpg/something"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Use \f[C]**\f[] to match anything, including slashes (\f[C]/\f[]).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
dir/**\ \-\ matches\ "dir/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ matches\ "dir/dir1/dir2/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "directory/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "adir/file.jpg"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
A \f[C]?\f[] matches any character except a slash \f[C]/\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
l?ss\ \ \-\ matches\ "less"
|
|
\ \ \ \ \ \ \-\ matches\ "lass"
|
|
\ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "floss"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
A \f[C][\f[] and \f[C]]\f[] together make a a character class, such as
|
|
\f[C][a\-z]\f[] or \f[C][aeiou]\f[] or \f[C][[:alpha:]]\f[].
|
|
See the go regexp docs (https://golang.org/pkg/regexp/syntax/) for more
|
|
info on these.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
h[ae]llo\ \-\ matches\ "hello"
|
|
\ \ \ \ \ \ \ \ \ \-\ matches\ "hallo"
|
|
\ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "hullo"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
A \f[C]{\f[] and \f[C]}\f[] define a choice between elements.
|
|
It should contain a comma seperated list of patterns, any of which might
|
|
match.
|
|
These patterns can contain wildcards.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{one,two}_potato\ \-\ matches\ "one_potato"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \-\ matches\ "two_potato"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "three_potato"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "_potato"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Special characters can be escaped with a \f[C]\\\f[] before them.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\\*.jpg\ \ \ \ \ \ \ \-\ matches\ "*.jpg"
|
|
\\\\.jpg\ \ \ \ \ \ \ \-\ matches\ "\\.jpg"
|
|
\\[one\\].jpg\ \ \-\ matches\ "[one].jpg"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note also that rclone filter globs can only be used in one of the filter
|
|
command line flags, not in the specification of the remote, so
|
|
\f[C]rclone\ copy\ "remote:dir*.jpg"\ /path/to/dir\f[] won\[aq]t work \-
|
|
what is required is
|
|
\f[C]rclone\ \-\-include\ "*.jpg"\ copy\ remote:dir\ /path/to/dir\f[]
|
|
.SS Directories
|
|
.PP
|
|
Rclone keeps track of directories that could match any file patterns.
|
|
.PP
|
|
Eg if you add the include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/a/*.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Rclone will synthesize the directory include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/a/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you put any rules which end in \f[C]/\f[] then it will only match
|
|
directories.
|
|
.PP
|
|
Directory matches are \f[B]only\f[] used to optimise directory access
|
|
patterns \- you must still match the files that you want to match.
|
|
Directory matches won\[aq]t optimise anything on bucket based remotes
|
|
(eg s3, swift, google compute storage, b2) which don\[aq]t have a
|
|
concept of directory.
|
|
.SS Differences between rsync and rclone patterns
|
|
.PP
|
|
Rclone implements bash style \f[C]{a,b,c}\f[] glob matching which rsync
|
|
doesn\[aq]t.
|
|
.PP
|
|
Rclone always does a wildcard match so \f[C]\\\f[] must always escape a
|
|
\f[C]\\\f[].
|
|
.SS How the rules are used
|
|
.PP
|
|
Rclone maintains a combined list of include rules and exclude rules.
|
|
.PP
|
|
Each file is matched in order, starting from the top, against the rule
|
|
in the list until it finds a match.
|
|
The file is then included or excluded according to the rule type.
|
|
.PP
|
|
If the matcher fails to find a match after testing against all the
|
|
entries in the list then the path is included.
|
|
.PP
|
|
For example given the following rules, \f[C]+\f[] being include,
|
|
\f[C]\-\f[] being exclude,
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\ secret*.jpg
|
|
+\ *.jpg
|
|
+\ *.png
|
|
+\ file2.avi
|
|
\-\ *
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This would include
|
|
.IP \[bu] 2
|
|
\f[C]file1.jpg\f[]
|
|
.IP \[bu] 2
|
|
\f[C]file3.png\f[]
|
|
.IP \[bu] 2
|
|
\f[C]file2.avi\f[]
|
|
.PP
|
|
This would exclude
|
|
.IP \[bu] 2
|
|
\f[C]secret17.jpg\f[]
|
|
.IP \[bu] 2
|
|
non \f[C]*.jpg\f[] and \f[C]*.png\f[]
|
|
.PP
|
|
A similar process is done on directory entries before recursing into
|
|
them.
|
|
This only works on remotes which have a concept of directory (Eg local,
|
|
google drive, onedrive, amazon drive) and not on bucket based remotes
|
|
(eg s3, swift, google compute storage, b2).
|
|
.SS Adding filtering rules
|
|
.PP
|
|
Filtering rules are added with the following command line flags.
|
|
.SS Repeating options
|
|
.PP
|
|
You can repeat the following options to add more than one rule of that
|
|
type.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\-from\f[]
|
|
.PP
|
|
Note that all the options of the same type are processed together in the
|
|
order above, regardless of what order they were placed on the command
|
|
line.
|
|
.PP
|
|
So all \f[C]\-\-include\f[] options are processed first in the order
|
|
they appeared on the command line, then all \f[C]\-\-include\-from\f[]
|
|
options etc.
|
|
.PP
|
|
To mix up the order includes and excludes, the \f[C]\-\-filter\f[] flag
|
|
can be used.
|
|
.SS \f[C]\-\-exclude\f[] \- Exclude files matching pattern
|
|
.PP
|
|
Add a single exclude rule with \f[C]\-\-exclude\f[].
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Eg \f[C]\-\-exclude\ *.bak\f[] to exclude all bak files from the sync.
|
|
.SS \f[C]\-\-exclude\-from\f[] \- Read exclude patterns from file
|
|
.PP
|
|
Add exclude rules from a file.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Prepare a file like this \f[C]exclude\-file.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ a\ sample\ exclude\ rule\ file
|
|
*.bak
|
|
file2.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-exclude\-from\ exclude\-file.txt\f[].
|
|
This will sync all files except those ending in \f[C]bak\f[] and
|
|
\f[C]file2.jpg\f[].
|
|
.PP
|
|
This is useful if you have a lot of rules.
|
|
.SS \f[C]\-\-include\f[] \- Include files matching pattern
|
|
.PP
|
|
Add a single include rule with \f[C]\-\-include\f[].
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Eg \f[C]\-\-include\ *.{png,jpg}\f[] to include all \f[C]png\f[] and
|
|
\f[C]jpg\f[] files in the backup and no others.
|
|
.PP
|
|
This adds an implicit \f[C]\-\-exclude\ *\f[] at the very end of the
|
|
filter list.
|
|
This means you can mix \f[C]\-\-include\f[] and
|
|
\f[C]\-\-include\-from\f[] with the other filters (eg
|
|
\f[C]\-\-exclude\f[]) but you must include all the files you want in the
|
|
include statement.
|
|
If this doesn\[aq]t provide enough flexibility then you must use
|
|
\f[C]\-\-filter\-from\f[].
|
|
.SS \f[C]\-\-include\-from\f[] \- Read include patterns from file
|
|
.PP
|
|
Add include rules from a file.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Prepare a file like this \f[C]include\-file.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ a\ sample\ include\ rule\ file
|
|
*.jpg
|
|
*.png
|
|
file2.avi
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-include\-from\ include\-file.txt\f[].
|
|
This will sync all \f[C]jpg\f[], \f[C]png\f[] files and
|
|
\f[C]file2.avi\f[].
|
|
.PP
|
|
This is useful if you have a lot of rules.
|
|
.PP
|
|
This adds an implicit \f[C]\-\-exclude\ *\f[] at the very end of the
|
|
filter list.
|
|
This means you can mix \f[C]\-\-include\f[] and
|
|
\f[C]\-\-include\-from\f[] with the other filters (eg
|
|
\f[C]\-\-exclude\f[]) but you must include all the files you want in the
|
|
include statement.
|
|
If this doesn\[aq]t provide enough flexibility then you must use
|
|
\f[C]\-\-filter\-from\f[].
|
|
.SS \f[C]\-\-filter\f[] \- Add a file\-filtering rule
|
|
.PP
|
|
This can be used to add a single include or exclude rule.
|
|
Include rules start with \f[C]+\f[] and exclude rules start with
|
|
\f[C]\-\f[].
|
|
A special rule called \f[C]!\f[] can be used to clear the existing
|
|
rules.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Eg \f[C]\-\-filter\ "\-\ *.bak"\f[] to exclude all bak files from the
|
|
sync.
|
|
.SS \f[C]\-\-filter\-from\f[] \- Read filtering patterns from a file
|
|
.PP
|
|
Add include/exclude rules from a file.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Prepare a file like this \f[C]filter\-file.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ a\ sample\ exclude\ rule\ file
|
|
\-\ secret*.jpg
|
|
+\ *.jpg
|
|
+\ *.png
|
|
+\ file2.avi
|
|
#\ exclude\ everything\ else
|
|
\-\ *
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-filter\-from\ filter\-file.txt\f[].
|
|
The rules are processed in the order that they are defined.
|
|
.PP
|
|
This example will include all \f[C]jpg\f[] and \f[C]png\f[] files,
|
|
exclude any files matching \f[C]secret*.jpg\f[] and include
|
|
\f[C]file2.avi\f[].
|
|
Everything else will be excluded from the sync.
|
|
.SS \f[C]\-\-files\-from\f[] \- Read list of source\-file names
|
|
.PP
|
|
This reads a list of file names from the file passed in and
|
|
\f[B]only\f[] these files are transferred.
|
|
The filtering rules are ignored completely if you use this option.
|
|
.PP
|
|
This option can be repeated to read from more than one file.
|
|
These are read in the order that they are placed on the command line.
|
|
.PP
|
|
Prepare a file like this \f[C]files\-from.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ comment
|
|
file1.jpg
|
|
file2.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-files\-from\ files\-from.txt\f[].
|
|
This will only transfer \f[C]file1.jpg\f[] and \f[C]file2.jpg\f[]
|
|
providing they exist.
|
|
.PP
|
|
For example, let\[aq]s say you had a few files you want to back up
|
|
regularly with these absolute paths:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/important
|
|
/home/user1/dir/file
|
|
/home/user2/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy these you\[aq]d find a common subdirectory \- in this case
|
|
\f[C]/home\f[] and put the remaining files in \f[C]files\-from.txt\f[]
|
|
with or without leading \f[C]/\f[], eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
user1/important
|
|
user1/dir/file
|
|
user2/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You could then copy these to a remote like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /home\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The 3 files will arrive in \f[C]remote:backup\f[] with the paths as in
|
|
the \f[C]files\-from.txt\f[].
|
|
.PP
|
|
You could of course choose \f[C]/\f[] as the root too in which case your
|
|
\f[C]files\-from.txt\f[] might look like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/important
|
|
/home/user1/dir/file
|
|
/home/user2/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And you would transfer it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
In this case there will be an extra \f[C]home\f[] directory on the
|
|
remote.
|
|
.SS \f[C]\-\-min\-size\f[] \- Don\[aq]t transfer any file smaller than
|
|
this
|
|
.PP
|
|
This option controls the minimum size file which will be transferred.
|
|
This defaults to \f[C]kBytes\f[] but a suffix of \f[C]k\f[], \f[C]M\f[],
|
|
or \f[C]G\f[] can be used.
|
|
.PP
|
|
For example \f[C]\-\-min\-size\ 50k\f[] means no files smaller than
|
|
50kByte will be transferred.
|
|
.SS \f[C]\-\-max\-size\f[] \- Don\[aq]t transfer any file larger than
|
|
this
|
|
.PP
|
|
This option controls the maximum size file which will be transferred.
|
|
This defaults to \f[C]kBytes\f[] but a suffix of \f[C]k\f[], \f[C]M\f[],
|
|
or \f[C]G\f[] can be used.
|
|
.PP
|
|
For example \f[C]\-\-max\-size\ 1G\f[] means no files larger than 1GByte
|
|
will be transferred.
|
|
.SS \f[C]\-\-max\-age\f[] \- Don\[aq]t transfer any file older than this
|
|
.PP
|
|
This option controls the maximum age of files to transfer.
|
|
Give in seconds or with a suffix of:
|
|
.IP \[bu] 2
|
|
\f[C]ms\f[] \- Milliseconds
|
|
.IP \[bu] 2
|
|
\f[C]s\f[] \- Seconds
|
|
.IP \[bu] 2
|
|
\f[C]m\f[] \- Minutes
|
|
.IP \[bu] 2
|
|
\f[C]h\f[] \- Hours
|
|
.IP \[bu] 2
|
|
\f[C]d\f[] \- Days
|
|
.IP \[bu] 2
|
|
\f[C]w\f[] \- Weeks
|
|
.IP \[bu] 2
|
|
\f[C]M\f[] \- Months
|
|
.IP \[bu] 2
|
|
\f[C]y\f[] \- Years
|
|
.PP
|
|
For example \f[C]\-\-max\-age\ 2d\f[] means no files older than 2 days
|
|
will be transferred.
|
|
.SS \f[C]\-\-min\-age\f[] \- Don\[aq]t transfer any file younger than
|
|
this
|
|
.PP
|
|
This option controls the minimum age of files to transfer.
|
|
Give in seconds or with a suffix (see \f[C]\-\-max\-age\f[] for list of
|
|
suffixes)
|
|
.PP
|
|
For example \f[C]\-\-min\-age\ 2d\f[] means no files younger than 2 days
|
|
will be transferred.
|
|
.SS \f[C]\-\-delete\-excluded\f[] \- Delete files on dest excluded from
|
|
sync
|
|
.PP
|
|
\f[B]Important\f[] this flag is dangerous \- use with
|
|
\f[C]\-\-dry\-run\f[] and \f[C]\-v\f[] first.
|
|
.PP
|
|
When doing \f[C]rclone\ sync\f[] this will delete any files which are
|
|
excluded from the sync on the destination.
|
|
.PP
|
|
If for example you did a sync from \f[C]A\f[] to \f[C]B\f[] without the
|
|
\f[C]\-\-min\-size\ 50k\f[] flag
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ A:\ B:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then you repeated it like this with the \f[C]\-\-delete\-excluded\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-min\-size\ 50k\ \-\-delete\-excluded\ sync\ A:\ B:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This would delete all files on \f[C]B\f[] which are less than 50 kBytes
|
|
as these are now excluded from the sync.
|
|
.PP
|
|
Always test first with \f[C]\-\-dry\-run\f[] and \f[C]\-v\f[] before
|
|
using this flag.
|
|
.SS \f[C]\-\-dump\-filters\f[] \- dump the filters to the output
|
|
.PP
|
|
This dumps the defined filters to the output as regular expressions.
|
|
.PP
|
|
Useful for debugging.
|
|
.SS Quoting shell metacharacters
|
|
.PP
|
|
The examples above may not work verbatim in your shell as they have
|
|
shell metacharacters in them (eg \f[C]*\f[]), and may require quoting.
|
|
.PP
|
|
Eg linux, OSX
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\ \\*.jpg\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\ \[aq]*.jpg\[aq]\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include=\[aq]*.jpg\[aq]\f[]
|
|
.PP
|
|
In Windows the expansion is done by the command not the shell so this
|
|
should work fine
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\ *.jpg\f[]
|
|
.SH Overview of cloud storage systems
|
|
.PP
|
|
Each cloud storage system is slighly different.
|
|
Rclone attempts to provide a unified interface to them, but some
|
|
underlying differences show through.
|
|
.SS Features
|
|
.PP
|
|
Here is an overview of the major features of each cloud storage system.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c c c c.
|
|
T{
|
|
Name
|
|
T}@T{
|
|
Hash
|
|
T}@T{
|
|
ModTime
|
|
T}@T{
|
|
Case Insensitive
|
|
T}@T{
|
|
Duplicate Files
|
|
T}@T{
|
|
MIME Type
|
|
T}
|
|
_
|
|
T{
|
|
Google Drive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Amazon S3
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Openstack Swift
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Dropbox
|
|
T}@T{
|
|
\-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Google Cloud Storage
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Amazon Drive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Microsoft One Drive
|
|
T}@T{
|
|
SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Hubic
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Backblaze B2
|
|
T}@T{
|
|
SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Yandex Disk
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
The local filesystem
|
|
T}@T{
|
|
All
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
.TE
|
|
.SS Hash
|
|
.PP
|
|
The cloud storage system supports various hash types of the objects.
|
|
.PD 0
|
|
.P
|
|
.PD
|
|
The hashes are used when transferring data as an integrity check and can
|
|
be specifically used with the \f[C]\-\-checksum\f[] flag in syncs and in
|
|
the \f[C]check\f[] command.
|
|
.PP
|
|
To use the checksum checks between filesystems they must support a
|
|
common hash type.
|
|
.SS ModTime
|
|
.PP
|
|
The cloud storage system supports setting modification times on objects.
|
|
If it does then this enables a using the modification times as part of
|
|
the sync.
|
|
If not then only the size will be checked by default, though the MD5SUM
|
|
can be checked with the \f[C]\-\-checksum\f[] flag.
|
|
.PP
|
|
All cloud storage systems support some kind of date on the object and
|
|
these will be set when transferring from the cloud storage system.
|
|
.SS Case Insensitive
|
|
.PP
|
|
If a cloud storage systems is case sensitive then it is possible to have
|
|
two files which differ only in case, eg \f[C]file.txt\f[] and
|
|
\f[C]FILE.txt\f[].
|
|
If a cloud storage system is case insensitive then that isn\[aq]t
|
|
possible.
|
|
.PP
|
|
This can cause problems when syncing between a case insensitive system
|
|
and a case sensitive system.
|
|
The symptom of this is that no matter how many times you run the sync it
|
|
never completes fully.
|
|
.PP
|
|
The local filesystem may or may not be case sensitive depending on OS.
|
|
.IP \[bu] 2
|
|
Windows \- usually case insensitive, though case is preserved
|
|
.IP \[bu] 2
|
|
OSX \- usually case insensitive, though it is possible to format case
|
|
sensitive
|
|
.IP \[bu] 2
|
|
Linux \- usually case sensitive, but there are case insensitive file
|
|
systems (eg FAT formatted USB keys)
|
|
.PP
|
|
Most of the time this doesn\[aq]t cause any problems as people tend to
|
|
avoid files whose name differs only by case even on case sensitive
|
|
systems.
|
|
.SS Duplicate files
|
|
.PP
|
|
If a cloud storage system allows duplicate files then it can have two
|
|
objects with the same name.
|
|
.PP
|
|
This confuses rclone greatly when syncing \- use the
|
|
\f[C]rclone\ dedupe\f[] command to rename or remove duplicates.
|
|
.SS MIME Type
|
|
.PP
|
|
MIME types (also known as media types) classify types of documents using
|
|
a simple text classification, eg \f[C]text/html\f[] or
|
|
\f[C]application/pdf\f[].
|
|
.PP
|
|
Some cloud storage systems support reading (\f[C]R\f[]) the MIME type of
|
|
objects and some support writing (\f[C]W\f[]) the MIME type of objects.
|
|
.PP
|
|
The MIME type can be important if you are serving files directly to HTTP
|
|
from the storage system.
|
|
.PP
|
|
If you are copying from a remote which supports reading (\f[C]R\f[]) to
|
|
a remote which supports writing (\f[C]W\f[]) then rclone will preserve
|
|
the MIME types.
|
|
Otherwise they will be guessed from the extension, or the remote itself
|
|
may assign the MIME type.
|
|
.SS Optional Features
|
|
.PP
|
|
All the remotes support a basic set of features, but there are some
|
|
optional features supported by some remotes used to make some operations
|
|
more efficient.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c c c c.
|
|
T{
|
|
Name
|
|
T}@T{
|
|
Purge
|
|
T}@T{
|
|
Copy
|
|
T}@T{
|
|
Move
|
|
T}@T{
|
|
DirMove
|
|
T}@T{
|
|
CleanUp
|
|
T}
|
|
_
|
|
T{
|
|
Google Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}
|
|
T{
|
|
Amazon S3
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Openstack Swift
|
|
T}@T{
|
|
Yes †
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Dropbox
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}
|
|
T{
|
|
Google Cloud Storage
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Amazon Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}
|
|
T{
|
|
Microsoft One Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #197 (https://github.com/ncw/rclone/issues/197)
|
|
T}@T{
|
|
No #197 (https://github.com/ncw/rclone/issues/197)
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}
|
|
T{
|
|
Hubic
|
|
T}@T{
|
|
Yes †
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Backblaze B2
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Yandex Disk
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}
|
|
T{
|
|
The local filesystem
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}
|
|
.TE
|
|
.SS Purge
|
|
.PP
|
|
This deletes a directory quicker than just deleting all the files in the
|
|
directory.
|
|
.PP
|
|
† Note Swift and Hubic implement this in order to delete directory
|
|
markers but they don\[aq]t actually have a quicker way of deleting files
|
|
other than deleting them individually.
|
|
.SS Copy
|
|
.PP
|
|
Used when copying an object to and from the same remote.
|
|
This known as a server side copy so you can copy a file without
|
|
downloading it and uploading it again.
|
|
It is used if you use \f[C]rclone\ copy\f[] or \f[C]rclone\ move\f[] if
|
|
the remote doesn\[aq]t support \f[C]Move\f[] directly.
|
|
.PP
|
|
If the server doesn\[aq]t support \f[C]Copy\f[] directly then for copy
|
|
operations the file is downloaded then re\-uploaded.
|
|
.SS Move
|
|
.PP
|
|
Used when moving/renaming an object on the same remote.
|
|
This is known as a server side move of a file.
|
|
This is used in \f[C]rclone\ move\f[] if the server doesn\[aq]t support
|
|
\f[C]DirMove\f[].
|
|
.PP
|
|
If the server isn\[aq]t capable of \f[C]Move\f[] then rclone simulates
|
|
it with \f[C]Copy\f[] then delete.
|
|
If the server doesn\[aq]t support \f[C]Copy\f[] then rclone will
|
|
download the file and re\-upload it.
|
|
.SS DirMove
|
|
.PP
|
|
This is used to implement \f[C]rclone\ move\f[] to move a directory if
|
|
possible.
|
|
If it isn\[aq]t then it will use \f[C]Move\f[] on each file (which falls
|
|
back to \f[C]Copy\f[] then download and upload \- see \f[C]Move\f[]
|
|
section).
|
|
.SS CleanUp
|
|
.PP
|
|
This is used for emptying the trash for a remote by
|
|
\f[C]rclone\ cleanup\f[].
|
|
.PP
|
|
If the server can\[aq]t do \f[C]CleanUp\f[] then
|
|
\f[C]rclone\ cleanup\f[] will return an error.
|
|
.SS Google Drive
|
|
.PP
|
|
Paths are specified as \f[C]drive:path\f[]
|
|
.PP
|
|
Drive paths may be as deep as required, eg
|
|
\f[C]drive:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for drive involves getting a token from Google drive
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 6
|
|
Google\ Application\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Google\ Application\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine\ or\ Y\ didn\[aq]t\ work
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014\-03\-16T13:57:58.955387075Z","Extra":null}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall, or use
|
|
manual mode.
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
Google drive stores modification times accurate to 1 ms.
|
|
.SS Revisions
|
|
.PP
|
|
Google drive stores revisions of files.
|
|
When you upload a change to an existing file to google drive using
|
|
rclone it will create a new revision of that file.
|
|
.PP
|
|
Revisions follow the standard google policy which at time of writing was
|
|
.IP \[bu] 2
|
|
They are deleted after 30 days or 100 revisions (whatever comes first).
|
|
.IP \[bu] 2
|
|
They do not count towards a user storage quota.
|
|
.SS Deleting files
|
|
.PP
|
|
By default rclone will delete files permanently when requested.
|
|
If sending them to the trash is required instead then use the
|
|
\f[C]\-\-drive\-use\-trash\f[] flag.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-drive\-chunk\-size=SIZE
|
|
.PP
|
|
Upload chunk size.
|
|
Must a power of 2 >= 256k.
|
|
Default value is 8 MB.
|
|
.PP
|
|
Making this larger will improve performance, but note that each chunk is
|
|
buffered in memory one per transfer.
|
|
.PP
|
|
Reducing this will reduce memory usage but decrease performance.
|
|
.SS \-\-drive\-full\-list
|
|
.PP
|
|
No longer does anything \- kept for backwards compatibility.
|
|
.SS \-\-drive\-upload\-cutoff=SIZE
|
|
.PP
|
|
File size cutoff for switching to chunked upload.
|
|
Default is 8 MB.
|
|
.SS \-\-drive\-use\-trash
|
|
.PP
|
|
Send files to the trash instead of deleting permanently.
|
|
Defaults to off, namely deleting files permanently.
|
|
.SS \-\-drive\-auth\-owner\-only
|
|
.PP
|
|
Only consider files owned by the authenticated user.
|
|
Requires that \-\-drive\-full\-list=true (default).
|
|
.SS \-\-drive\-formats
|
|
.PP
|
|
Google documents can only be exported from Google drive.
|
|
When rclone downloads a Google doc it chooses a format to download
|
|
depending upon this setting.
|
|
.PP
|
|
By default the formats are \f[C]docx,xlsx,pptx,svg\f[] which are a
|
|
sensible default for an editable document.
|
|
.PP
|
|
When choosing a format, rclone runs down the list provided in order and
|
|
chooses the first file format the doc can be exported as from the list.
|
|
If the file can\[aq]t be exported to a format on the formats list, then
|
|
rclone will choose a format from the default list.
|
|
.PP
|
|
If you prefer an archive copy then you might use
|
|
\f[C]\-\-drive\-formats\ pdf\f[], or if you prefer
|
|
openoffice/libreoffice formats you might use
|
|
\f[C]\-\-drive\-formats\ ods,odt,odp\f[].
|
|
.PP
|
|
Note that rclone adds the extension to the google doc, so if it is
|
|
calles \f[C]My\ Spreadsheet\f[] on google docs, it will be exported as
|
|
\f[C]My\ Spreadsheet.xlsx\f[] or \f[C]My\ Spreadsheet.pdf\f[] etc.
|
|
.PP
|
|
Here are the possible extensions with their corresponding mime types.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(9.7n) lw(11.7n) lw(12.6n).
|
|
T{
|
|
Extension
|
|
T}@T{
|
|
Mime Type
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
csv
|
|
T}@T{
|
|
text/csv
|
|
T}@T{
|
|
Standard CSV format for Spreadsheets
|
|
T}
|
|
T{
|
|
doc
|
|
T}@T{
|
|
application/msword
|
|
T}@T{
|
|
Micosoft Office Document
|
|
T}
|
|
T{
|
|
docx
|
|
T}@T{
|
|
application/vnd.openxmlformats\-officedocument.wordprocessingml.document
|
|
T}@T{
|
|
Microsoft Office Document
|
|
T}
|
|
T{
|
|
epub
|
|
T}@T{
|
|
application/epub+zip
|
|
T}@T{
|
|
E\-book format
|
|
T}
|
|
T{
|
|
html
|
|
T}@T{
|
|
text/html
|
|
T}@T{
|
|
An HTML Document
|
|
T}
|
|
T{
|
|
jpg
|
|
T}@T{
|
|
image/jpeg
|
|
T}@T{
|
|
A JPEG Image File
|
|
T}
|
|
T{
|
|
odp
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.presentation
|
|
T}@T{
|
|
Openoffice Presentation
|
|
T}
|
|
T{
|
|
ods
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.spreadsheet
|
|
T}@T{
|
|
Openoffice Spreadsheet
|
|
T}
|
|
T{
|
|
ods
|
|
T}@T{
|
|
application/x\-vnd.oasis.opendocument.spreadsheet
|
|
T}@T{
|
|
Openoffice Spreadsheet
|
|
T}
|
|
T{
|
|
odt
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.text
|
|
T}@T{
|
|
Openoffice Document
|
|
T}
|
|
T{
|
|
pdf
|
|
T}@T{
|
|
application/pdf
|
|
T}@T{
|
|
Adobe PDF Format
|
|
T}
|
|
T{
|
|
png
|
|
T}@T{
|
|
image/png
|
|
T}@T{
|
|
PNG Image Format
|
|
T}
|
|
T{
|
|
pptx
|
|
T}@T{
|
|
application/vnd.openxmlformats\-officedocument.presentationml.presentation
|
|
T}@T{
|
|
Microsoft Office Powerpoint
|
|
T}
|
|
T{
|
|
rtf
|
|
T}@T{
|
|
application/rtf
|
|
T}@T{
|
|
Rich Text Format
|
|
T}
|
|
T{
|
|
svg
|
|
T}@T{
|
|
image/svg+xml
|
|
T}@T{
|
|
Scalable Vector Graphics Format
|
|
T}
|
|
T{
|
|
tsv
|
|
T}@T{
|
|
text/tab\-separated\-values
|
|
T}@T{
|
|
Standard TSV format for spreadsheets
|
|
T}
|
|
T{
|
|
txt
|
|
T}@T{
|
|
text/plain
|
|
T}@T{
|
|
Plain Text
|
|
T}
|
|
T{
|
|
xls
|
|
T}@T{
|
|
application/vnd.ms\-excel
|
|
T}@T{
|
|
Microsoft Office Spreadsheet
|
|
T}
|
|
T{
|
|
xlsx
|
|
T}@T{
|
|
application/vnd.openxmlformats\-officedocument.spreadsheetml.sheet
|
|
T}@T{
|
|
Microsoft Office Spreadsheet
|
|
T}
|
|
T{
|
|
zip
|
|
T}@T{
|
|
application/zip
|
|
T}@T{
|
|
A ZIP file of HTML, Images CSS
|
|
T}
|
|
.TE
|
|
.SS Limitations
|
|
.PP
|
|
Drive has quite a lot of rate limiting.
|
|
This causes rclone to be limited to transferring about 2 files per
|
|
second only.
|
|
Individual files may be transferred much faster at 100s of MBytes/s but
|
|
lots of small files can take a long time.
|
|
.SS Making your own client_id
|
|
.PP
|
|
When you use rclone with Google drive in its default configuration you
|
|
are using rclone\[aq]s client_id.
|
|
This is shared between all the rclone users.
|
|
There is a global rate limit on the number of queries per second that
|
|
each client_id can do set by Google.
|
|
rclone already has a high quota and I will continue to make sure it is
|
|
high enough by contacting Google.
|
|
.PP
|
|
However you might find you get better performance making your own
|
|
client_id if you are a heavy user.
|
|
Or you may not depending on exactly how Google have been raising
|
|
rclone\[aq]s rate limit.
|
|
.PP
|
|
Here is how to create your own Google Drive client ID for rclone:
|
|
.IP "1." 3
|
|
Log into the Google API Console (https://console.developers.google.com/)
|
|
with your Google account.
|
|
It doesn\[aq]t matter what Google account you use.
|
|
(It need not be the same account as the Google Drive you want to access)
|
|
.IP "2." 3
|
|
Select a project or create a new project.
|
|
.IP "3." 3
|
|
Under Overview, Google APIs, Google Apps APIs, click "Drive API", then
|
|
"Enable".
|
|
.IP "4." 3
|
|
Click "Credentials" in the left\-side panel (not "Go to credentials",
|
|
which opens the wizard), then "Create credentials", then "OAuth client
|
|
ID".
|
|
It will prompt you to set the OAuth consent screen product name, if you
|
|
haven\[aq]t set one already.
|
|
.IP "5." 3
|
|
Choose an application type of "other", and click "Create".
|
|
(the default name is fine)
|
|
.IP "6." 3
|
|
It will show you a client ID and client secret.
|
|
Use these values in rclone config to add a new remote or edit an
|
|
existing remote.
|
|
.PP
|
|
(Thanks to \@balazer on github for these instructions.)
|
|
.SS Amazon S3
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making an s3 configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 2
|
|
Get\ AWS\ credentials\ from\ runtime\ (environment\ variables\ or\ EC2\ meta\ data\ if\ no\ env\ vars).\ Only\ applies\ if\ access_key_id\ and\ secret_access_key\ is\ blank.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Enter\ AWS\ credentials\ in\ the\ next\ step
|
|
\ \ \ \\\ "false"
|
|
\ 2\ /\ Get\ AWS\ credentials\ from\ the\ environment\ (env\ vars\ or\ IAM)
|
|
\ \ \ \\\ "true"
|
|
env_auth>\ 1
|
|
AWS\ Access\ Key\ ID\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
access_key_id>\ access_key
|
|
AWS\ Secret\ Access\ Key\ (password)\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
secret_access_key>\ secret_key
|
|
Region\ to\ connect\ to.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ \ \ /\ The\ default\ endpoint\ \-\ a\ good\ choice\ if\ you\ are\ unsure.
|
|
\ 1\ |\ US\ Region,\ Northern\ Virginia\ or\ Pacific\ Northwest.
|
|
\ \ \ |\ Leave\ location\ constraint\ empty.
|
|
\ \ \ \\\ "us\-east\-1"
|
|
\ \ \ /\ US\ West\ (Oregon)\ Region
|
|
\ 2\ |\ Needs\ location\ constraint\ us\-west\-2.
|
|
\ \ \ \\\ "us\-west\-2"
|
|
\ \ \ /\ US\ West\ (Northern\ California)\ Region
|
|
\ 3\ |\ Needs\ location\ constraint\ us\-west\-1.
|
|
\ \ \ \\\ "us\-west\-1"
|
|
\ \ \ /\ EU\ (Ireland)\ Region\ Region
|
|
\ 4\ |\ Needs\ location\ constraint\ EU\ or\ eu\-west\-1.
|
|
\ \ \ \\\ "eu\-west\-1"
|
|
\ \ \ /\ EU\ (Frankfurt)\ Region
|
|
\ 5\ |\ Needs\ location\ constraint\ eu\-central\-1.
|
|
\ \ \ \\\ "eu\-central\-1"
|
|
\ \ \ /\ Asia\ Pacific\ (Singapore)\ Region
|
|
\ 6\ |\ Needs\ location\ constraint\ ap\-southeast\-1.
|
|
\ \ \ \\\ "ap\-southeast\-1"
|
|
\ \ \ /\ Asia\ Pacific\ (Sydney)\ Region
|
|
\ 7\ |\ Needs\ location\ constraint\ ap\-southeast\-2.
|
|
\ \ \ \\\ "ap\-southeast\-2"
|
|
\ \ \ /\ Asia\ Pacific\ (Tokyo)\ Region
|
|
\ 8\ |\ Needs\ location\ constraint\ ap\-northeast\-1.
|
|
\ \ \ \\\ "ap\-northeast\-1"
|
|
\ \ \ /\ South\ America\ (Sao\ Paulo)\ Region
|
|
\ 9\ |\ Needs\ location\ constraint\ sa\-east\-1.
|
|
\ \ \ \\\ "sa\-east\-1"
|
|
\ \ \ /\ If\ using\ an\ S3\ clone\ that\ only\ understands\ v2\ signatures
|
|
10\ |\ eg\ Ceph/Dreamhost
|
|
\ \ \ |\ set\ this\ and\ make\ sure\ you\ set\ the\ endpoint.
|
|
\ \ \ \\\ "other\-v2\-signature"
|
|
\ \ \ /\ If\ using\ an\ S3\ clone\ that\ understands\ v4\ signatures\ set\ this
|
|
11\ |\ and\ make\ sure\ you\ set\ the\ endpoint.
|
|
\ \ \ \\\ "other\-v4\-signature"
|
|
region>\ 1
|
|
Endpoint\ for\ S3\ API.
|
|
Leave\ blank\ if\ using\ AWS\ to\ use\ the\ default\ endpoint\ for\ the\ region.
|
|
Specify\ if\ using\ an\ S3\ clone\ such\ as\ Ceph.
|
|
endpoint>\
|
|
Location\ constraint\ \-\ must\ be\ set\ to\ match\ the\ Region.\ Used\ when\ creating\ buckets\ only.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Empty\ for\ US\ Region,\ Northern\ Virginia\ or\ Pacific\ Northwest.
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ US\ West\ (Oregon)\ Region.
|
|
\ \ \ \\\ "us\-west\-2"
|
|
\ 3\ /\ US\ West\ (Northern\ California)\ Region.
|
|
\ \ \ \\\ "us\-west\-1"
|
|
\ 4\ /\ EU\ (Ireland)\ Region.
|
|
\ \ \ \\\ "eu\-west\-1"
|
|
\ 5\ /\ EU\ Region.
|
|
\ \ \ \\\ "EU"
|
|
\ 6\ /\ Asia\ Pacific\ (Singapore)\ Region.
|
|
\ \ \ \\\ "ap\-southeast\-1"
|
|
\ 7\ /\ Asia\ Pacific\ (Sydney)\ Region.
|
|
\ \ \ \\\ "ap\-southeast\-2"
|
|
\ 8\ /\ Asia\ Pacific\ (Tokyo)\ Region.
|
|
\ \ \ \\\ "ap\-northeast\-1"
|
|
\ 9\ /\ South\ America\ (Sao\ Paulo)\ Region.
|
|
\ \ \ \\\ "sa\-east\-1"
|
|
location_constraint>\ 1
|
|
Canned\ ACL\ used\ when\ creating\ buckets\ and/or\ storing\ objects\ in\ S3.
|
|
For\ more\ info\ visit\ http://docs.aws.amazon.com/AmazonS3/latest/dev/acl\-overview.html#canned\-acl
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Owner\ gets\ FULL_CONTROL.\ No\ one\ else\ has\ access\ rights\ (default).
|
|
\ \ \ \\\ "private"
|
|
\ 2\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ access.
|
|
\ \ \ \\\ "public\-read"
|
|
\ \ \ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ and\ WRITE\ access.
|
|
\ 3\ |\ Granting\ this\ on\ a\ bucket\ is\ generally\ not\ recommended.
|
|
\ \ \ \\\ "public\-read\-write"
|
|
\ 4\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AuthenticatedUsers\ group\ gets\ READ\ access.
|
|
\ \ \ \\\ "authenticated\-read"
|
|
\ \ \ /\ Object\ owner\ gets\ FULL_CONTROL.\ Bucket\ owner\ gets\ READ\ access.
|
|
\ 5\ |\ If\ you\ specify\ this\ canned\ ACL\ when\ creating\ a\ bucket,\ Amazon\ S3\ ignores\ it.
|
|
\ \ \ \\\ "bucket\-owner\-read"
|
|
\ \ \ /\ Both\ the\ object\ owner\ and\ the\ bucket\ owner\ get\ FULL_CONTROL\ over\ the\ object.
|
|
\ 6\ |\ If\ you\ specify\ this\ canned\ ACL\ when\ creating\ a\ bucket,\ Amazon\ S3\ ignores\ it.
|
|
\ \ \ \\\ "bucket\-owner\-full\-control"
|
|
acl>\ private
|
|
The\ server\-side\ encryption\ algorithm\ used\ when\ storing\ this\ object\ in\ S3.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ None
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ AES256
|
|
\ \ \ \\\ "AES256"
|
|
server_side_encryption>
|
|
The\ storage\ class\ to\ use\ when\ storing\ objects\ in\ S3.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Default
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ Standard\ storage\ class
|
|
\ \ \ \\\ "STANDARD"
|
|
\ 3\ /\ Reduced\ redundancy\ storage\ class
|
|
\ \ \ \\\ "REDUCED_REDUNDANCY"
|
|
\ 4\ /\ Standard\ Infrequent\ Access\ storage\ class
|
|
\ \ \ \\\ "STANDARD_IA"
|
|
storage_class>
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ access_key
|
|
secret_access_key\ =\ secret_key
|
|
region\ =\ us\-east\-1
|
|
endpoint\ =\
|
|
location_constraint\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Amz\-Meta\-Mtime\f[] as floating point since the epoch accurate
|
|
to 1 ns.
|
|
.SS Multipart uploads
|
|
.PP
|
|
rclone supports multipart uploads with S3 which means that it can upload
|
|
files bigger than 5GB.
|
|
Note that files uploaded with multipart upload don\[aq]t have an MD5SUM.
|
|
.SS Buckets and Regions
|
|
.PP
|
|
With Amazon S3 you can list buckets (\f[C]rclone\ lsd\f[]) using any
|
|
region, but you can only access the content of a bucket from the region
|
|
it was created in.
|
|
If you attempt to access a bucket from the wrong region, you will get an
|
|
error,
|
|
\f[C]incorrect\ region,\ the\ bucket\ is\ not\ in\ \[aq]XXX\[aq]\ region\f[].
|
|
.SS Authentication
|
|
.PP
|
|
There are two ways to supply \f[C]rclone\f[] with a set of AWS
|
|
credentials.
|
|
In order of precedence:
|
|
.IP \[bu] 2
|
|
Directly in the rclone configuration file (as configured by
|
|
\f[C]rclone\ config\f[])
|
|
.IP \[bu] 2
|
|
set \f[C]access_key_id\f[] and \f[C]secret_access_key\f[]
|
|
.IP \[bu] 2
|
|
Runtime configuration:
|
|
.IP \[bu] 2
|
|
set \f[C]env_auth\f[] to \f[C]true\f[] in the config file
|
|
.IP \[bu] 2
|
|
Exporting the following environment variables before running
|
|
\f[C]rclone\f[]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access Key ID: \f[C]AWS_ACCESS_KEY_ID\f[] or \f[C]AWS_ACCESS_KEY\f[]
|
|
.IP \[bu] 2
|
|
Secret Access Key: \f[C]AWS_SECRET_ACCESS_KEY\f[] or
|
|
\f[C]AWS_SECRET_KEY\f[]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Running \f[C]rclone\f[] on an EC2 instance with an IAM role
|
|
.PP
|
|
If none of these option actually end up providing \f[C]rclone\f[] with
|
|
AWS credentials then S3 interaction will be non\-authenticated (see
|
|
below).
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-s3\-acl=STRING
|
|
.PP
|
|
Canned ACL used when creating buckets and/or storing objects in S3.
|
|
.PP
|
|
For more info visit the canned ACL
|
|
docs (http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl).
|
|
.SS \-\-s3\-storage\-class=STRING
|
|
.PP
|
|
Storage class to upload new objects with.
|
|
.PP
|
|
Available options include:
|
|
.IP \[bu] 2
|
|
STANDARD \- default storage class
|
|
.IP \[bu] 2
|
|
STANDARD_IA \- for less frequently accessed data (e.g backups)
|
|
.IP \[bu] 2
|
|
REDUCED_REDUNDANCY (only for noncritical, reproducible data, has lower
|
|
redundancy)
|
|
.SS Anonymous access to public buckets
|
|
.PP
|
|
If you want to use rclone to access a public bucket, configure with a
|
|
blank \f[C]access_key_id\f[] and \f[C]secret_access_key\f[].
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
q)\ Quit\ config
|
|
n/q>\ n
|
|
name>\ anons3
|
|
What\ type\ of\ source\ is\ it?
|
|
Choose\ a\ number\ from\ below
|
|
\ 1)\ amazon\ cloud\ drive
|
|
\ 2)\ b2
|
|
\ 3)\ drive
|
|
\ 4)\ dropbox
|
|
\ 5)\ google\ cloud\ storage
|
|
\ 6)\ swift
|
|
\ 7)\ hubic
|
|
\ 8)\ local
|
|
\ 9)\ onedrive
|
|
10)\ s3
|
|
11)\ yandex
|
|
type>\ 10
|
|
Get\ AWS\ credentials\ from\ runtime\ (environment\ variables\ or\ EC2\ meta\ data\ if\ no\ env\ vars).\ Only\ applies\ if\ access_key_id\ and\ secret_access_key\ is\ blank.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ *\ Enter\ AWS\ credentials\ in\ the\ next\ step
|
|
\ 1)\ false
|
|
\ *\ Get\ AWS\ credentials\ from\ the\ environment\ (env\ vars\ or\ IAM)
|
|
\ 2)\ true
|
|
env_auth>\ 1
|
|
AWS\ Access\ Key\ ID\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
access_key_id>
|
|
AWS\ Secret\ Access\ Key\ (password)\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
secret_access_key>
|
|
\&...
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use it as normal with the name of the public bucket, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ anons3:1000genomes
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You will be able to list and copy data but not upload it.
|
|
.SS Ceph
|
|
.PP
|
|
Ceph is an object storage system which presents an Amazon S3 interface.
|
|
.PP
|
|
To use rclone with ceph, you need to set the following parameters in the
|
|
config.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
access_key_id\ =\ Whatever
|
|
secret_access_key\ =\ Whatever
|
|
endpoint\ =\ https://ceph.endpoint.goes.here/
|
|
region\ =\ other\-v2\-signature
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note also that Ceph sometimes puts \f[C]/\f[] in the passwords it gives
|
|
users.
|
|
If you read the secret access key using the command line tools you will
|
|
get a JSON blob with the \f[C]/\f[] escaped as \f[C]\\/\f[].
|
|
Make sure you only write \f[C]/\f[] in the secret access key.
|
|
.PP
|
|
Eg the dump from Ceph looks something like this (irrelevant keys
|
|
removed).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "user_id":\ "xxx",
|
|
\ \ \ \ "display_name":\ "xxxx",
|
|
\ \ \ \ "keys":\ [
|
|
\ \ \ \ \ \ \ \ {
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "user":\ "xxx",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "access_key":\ "xxxxxx",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "secret_key":\ "xxxxxx\\/xxxx"
|
|
\ \ \ \ \ \ \ \ }
|
|
\ \ \ \ ],
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Because this is a json dump, it is encoding the \f[C]/\f[] as
|
|
\f[C]\\/\f[], so if you use the secret key as \f[C]xxxxxx/xxxx\f[] it
|
|
will work fine.
|
|
.SS Minio
|
|
.PP
|
|
Minio (https://minio.io/) is an object storage server built for cloud
|
|
application developers and devops.
|
|
.PP
|
|
It is very easy to install and provides an S3 compatible server which
|
|
can be used by rclone.
|
|
.PP
|
|
To use it, install Minio following the instructions from the web site.
|
|
.PP
|
|
When it configures itself Minio will print something like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
AccessKey:\ WLGDGYAQYIGI833EV05A\ \ SecretKey:\ BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF\ Region:\ us\-east\-1
|
|
|
|
Minio\ Object\ Storage:
|
|
\ \ \ \ \ http://127.0.0.1:9000
|
|
\ \ \ \ \ http://10.0.0.3:9000
|
|
|
|
Minio\ Browser:
|
|
\ \ \ \ \ http://127.0.0.1:9000
|
|
\ \ \ \ \ http://10.0.0.3:9000
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
These details need to go into \f[C]rclone\ config\f[] like this.
|
|
Note that it is important to put the region in as stated above.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
env_auth>\ 1
|
|
access_key_id>\ WLGDGYAQYIGI833EV05A
|
|
secret_access_key>\ BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF\ \ \
|
|
region>\ us\-east\-1
|
|
endpoint>\ http://10.0.0.3:9000
|
|
location_constraint>\
|
|
server_side_encryption>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Which makes the config file look like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[minio]
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ WLGDGYAQYIGI833EV05A
|
|
secret_access_key\ =\ BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
|
|
region\ =\ us\-east\-1
|
|
endpoint\ =\ http://10.0.0.3:9000
|
|
location_constraint\ =\
|
|
server_side_encryption\ =\
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Minio doesn\[aq]t support all the features of S3 yet.
|
|
In particular it doesn\[aq]t support MD5 checksums (ETags) or metadata.
|
|
This means rclone can\[aq]t check MD5SUMs or store the modified date.
|
|
However you can work around this with the \f[C]\-\-size\-only\f[] flag
|
|
of rclone.
|
|
.PP
|
|
So once set up, for example to copy files into a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-size\-only\ copy\ /path/to/files\ minio:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Swift
|
|
.PP
|
|
Swift refers to Openstack Object
|
|
Storage (http://www.openstack.org/software/openstack-storage/).
|
|
Commercial implementations of that being:
|
|
.IP \[bu] 2
|
|
Rackspace Cloud Files (http://www.rackspace.com/cloud/files/)
|
|
.IP \[bu] 2
|
|
Memset Memstore (http://www.memset.com/cloud/storage/)
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[] (or \f[C]remote:\f[]
|
|
for the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:container/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making a swift configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 10
|
|
User\ name\ to\ log\ in.
|
|
user>\ user_name
|
|
API\ key\ or\ password.
|
|
key>\ password_or_api_key
|
|
Authentication\ URL\ for\ server.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Rackspace\ US
|
|
\ \ \ \\\ "https://auth.api.rackspacecloud.com/v1.0"
|
|
\ 2\ /\ Rackspace\ UK
|
|
\ \ \ \\\ "https://lon.auth.api.rackspacecloud.com/v1.0"
|
|
\ 3\ /\ Rackspace\ v2
|
|
\ \ \ \\\ "https://identity.api.rackspacecloud.com/v2.0"
|
|
\ 4\ /\ Memset\ Memstore\ UK
|
|
\ \ \ \\\ "https://auth.storage.memset.com/v1.0"
|
|
\ 5\ /\ Memset\ Memstore\ UK\ v2
|
|
\ \ \ \\\ "https://auth.storage.memset.com/v2.0"
|
|
\ 6\ /\ OVH
|
|
\ \ \ \\\ "https://auth.cloud.ovh.net/v2.0"
|
|
auth>\ 1
|
|
User\ domain\ \-\ optional\ (v3\ auth)
|
|
domain>\ Default
|
|
Tenant\ name\ \-\ optional
|
|
tenant>\
|
|
Tenant\ domain\ \-\ optional\ (v3\ auth)
|
|
tenant_domain>
|
|
Region\ name\ \-\ optional
|
|
region>\
|
|
Storage\ URL\ \-\ optional
|
|
storage_url>\
|
|
Remote\ config
|
|
AuthVersion\ \-\ optional\ \-\ set\ to\ (1,2,3)\ if\ your\ auth\ URL\ has\ no\ version
|
|
auth_version>\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
user\ =\ user_name
|
|
key\ =\ password_or_api_key
|
|
auth\ =\ https://auth.api.rackspacecloud.com/v1.0
|
|
tenant\ =\
|
|
region\ =\
|
|
storage_url\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all containers
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:container
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:container
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote container, deleting
|
|
any excess files in the container.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:container
|
|
\f[]
|
|
.fi
|
|
.SS Configuration from an Openstack credentials file
|
|
.PP
|
|
An Opentstack credentials file typically looks something something like
|
|
this (without the comments)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export\ OS_AUTH_URL=https://a.provider.net/v2.0
|
|
export\ OS_TENANT_ID=ffffffffffffffffffffffffffffffff
|
|
export\ OS_TENANT_NAME="1234567890123456"
|
|
export\ OS_USERNAME="123abc567xy"
|
|
echo\ "Please\ enter\ your\ OpenStack\ Password:\ "
|
|
read\ \-sr\ OS_PASSWORD_INPUT
|
|
export\ OS_PASSWORD=$OS_PASSWORD_INPUT
|
|
export\ OS_REGION_NAME="SBG1"
|
|
if\ [\ \-z\ "$OS_REGION_NAME"\ ];\ then\ unset\ OS_REGION_NAME;\ fi
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The config file needs to look something like this where
|
|
\f[C]$OS_USERNAME\f[] represents the value of the \f[C]OS_USERNAME\f[]
|
|
variable \- \f[C]123abc567xy\f[] in the example above.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type\ =\ swift
|
|
user\ =\ $OS_USERNAME
|
|
key\ =\ $OS_PASSWORD
|
|
auth\ =\ $OS_AUTH_URL
|
|
tenant\ =\ $OS_TENANT_NAME
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that you may (or may not) need to set \f[C]region\f[] too \- try
|
|
without first.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-swift\-chunk\-size=SIZE
|
|
.PP
|
|
Above this size files will be chunked into a _segments container.
|
|
The default for this is 5GB which is its maximum value.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Object\-Meta\-Mtime\f[] as floating point since the epoch
|
|
accurate to 1 ns.
|
|
.PP
|
|
This is a defacto standard (used in the official python\-swiftclient
|
|
amongst others) for storing the modification time for an object.
|
|
.SS Limitations
|
|
.PP
|
|
The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
|
|
(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
|
|
MD5SUM for these.
|
|
.SS Troubleshooting
|
|
.SS Rclone gives Failed to create file system for "remote:": Bad Request
|
|
.PP
|
|
Due to an oddity of the underlying swift library, it gives a "Bad
|
|
Request" error rather than a more sensible error when the authentication
|
|
fails for Swift.
|
|
.PP
|
|
So this most likely means your username / password is wrong.
|
|
You can investigate further with the \f[C]\-\-dump\-bodies\f[] flag.
|
|
.PP
|
|
This may also be caused by specifying the region when you shouldn\[aq]t
|
|
have (eg OVH).
|
|
.SS Rclone gives Failed to create file system: Response didn\[aq]t have
|
|
storage storage url and auth token
|
|
.PP
|
|
This is most likely caused by forgetting to specify your tenant when
|
|
setting up a swift remote.
|
|
.SS Dropbox
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Dropbox paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for dropbox involves getting a token from Dropbox
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 4
|
|
Dropbox\ App\ Key\ \-\ leave\ blank\ normally.
|
|
app_key>\
|
|
Dropbox\ App\ Secret\ \-\ leave\ blank\ normally.
|
|
app_secret>\
|
|
Remote\ config
|
|
Please\ visit:
|
|
https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code
|
|
Enter\ the\ code:\ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
app_key\ =\
|
|
app_secret\ =\
|
|
token\ =\ XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a dropbox directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and MD5SUMs
|
|
.PP
|
|
Dropbox doesn\[aq]t provide the ability to set modification times in the
|
|
V1 public API, so rclone can\[aq]t support modified time with Dropbox.
|
|
.PP
|
|
This may change in the future \- see these issues for details:
|
|
.IP \[bu] 2
|
|
Dropbox V2 API (https://github.com/ncw/rclone/issues/349)
|
|
.IP \[bu] 2
|
|
Allow syncs for remotes that can\[aq]t set modtime on existing
|
|
objects (https://github.com/ncw/rclone/issues/348)
|
|
.PP
|
|
Dropbox doesn\[aq]t return any sort of checksum (MD5 or SHA1).
|
|
.PP
|
|
Together that means that syncs to dropbox will effectively have the
|
|
\f[C]\-\-size\-only\f[] flag set.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-dropbox\-chunk\-size=SIZE
|
|
.PP
|
|
Upload chunk size.
|
|
Max 150M.
|
|
The default is 128MB.
|
|
Note that this isn\[aq]t buffered into memory.
|
|
.SS Limitations
|
|
.PP
|
|
Note that Dropbox is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
There are some file names such as \f[C]thumbs.db\f[] which Dropbox
|
|
can\[aq]t store.
|
|
There is a full list of them in the "Ignored Files" section of this
|
|
document (https://www.dropbox.com/en/help/145).
|
|
Rclone will issue an error message
|
|
\f[C]File\ name\ disallowed\ \-\ not\ uploading\f[] if it attempt to
|
|
upload one of those file names, but the sync won\[aq]t fail.
|
|
.PP
|
|
If you have more than 10,000 files in a directory then
|
|
\f[C]rclone\ purge\ dropbox:dir\f[] will return the error
|
|
\f[C]Failed\ to\ purge:\ There\ are\ too\ many\ files\ involved\ in\ this\ operation\f[].
|
|
As a work\-around do an \f[C]rclone\ delete\ dropbix:dir\f[] followed by
|
|
an \f[C]rclone\ rmdir\ dropbox:dir\f[].
|
|
.SS Google Cloud Storage
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
The initial setup for google cloud storage involves getting a token from
|
|
Google Cloud Storage which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 5
|
|
Google\ Application\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Google\ Application\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Project\ number\ optional\ \-\ needed\ only\ for\ list/create/delete\ buckets\ \-\ see\ your\ developer\ console.
|
|
project_number>\ 12345678
|
|
Service\ Account\ Credentials\ JSON\ file\ path\ \-\ needed\ only\ if\ you\ want\ use\ SA\ instead\ of\ interactive\ login.
|
|
service_account_file>\
|
|
Access\ Control\ List\ for\ new\ objects.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ *\ Object\ owner\ gets\ OWNER\ access,\ and\ all\ Authenticated\ Users\ get\ READER\ access.
|
|
\ 1)\ authenticatedRead
|
|
\ *\ Object\ owner\ gets\ OWNER\ access,\ and\ project\ team\ owners\ get\ OWNER\ access.
|
|
\ 2)\ bucketOwnerFullControl
|
|
\ *\ Object\ owner\ gets\ OWNER\ access,\ and\ project\ team\ owners\ get\ READER\ access.
|
|
\ 3)\ bucketOwnerRead
|
|
\ *\ Object\ owner\ gets\ OWNER\ access\ [default\ if\ left\ blank].
|
|
\ 4)\ private
|
|
\ *\ Object\ owner\ gets\ OWNER\ access,\ and\ project\ team\ members\ get\ access\ according\ to\ their\ roles.
|
|
\ 5)\ projectPrivate
|
|
\ *\ Object\ owner\ gets\ OWNER\ access,\ and\ all\ Users\ get\ READER\ access.
|
|
\ 6)\ publicRead
|
|
object_acl>\ 4
|
|
Access\ Control\ List\ for\ new\ buckets.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ *\ Project\ team\ owners\ get\ OWNER\ access,\ and\ all\ Authenticated\ Users\ get\ READER\ access.
|
|
\ 1)\ authenticatedRead
|
|
\ *\ Project\ team\ owners\ get\ OWNER\ access\ [default\ if\ left\ blank].
|
|
\ 2)\ private
|
|
\ *\ Project\ team\ members\ get\ access\ according\ to\ their\ roles.
|
|
\ 3)\ projectPrivate
|
|
\ *\ Project\ team\ owners\ get\ OWNER\ access,\ and\ all\ Users\ get\ READER\ access.
|
|
\ 4)\ publicRead
|
|
\ *\ Project\ team\ owners\ get\ OWNER\ access,\ and\ all\ Users\ get\ WRITER\ access.
|
|
\ 5)\ publicReadWrite
|
|
bucket_acl>\ 2
|
|
Remote\ config
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine\ or\ Y\ didn\[aq]t\ work
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
type\ =\ google\ cloud\ storage
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014\-07\-17T20:49:14.929208288+01:00","Extra":null}
|
|
project_number\ =\ 12345678
|
|
object_acl\ =\ private
|
|
bucket_acl\ =\ private
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall, or use
|
|
manual mode.
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all the buckets in your project
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Service Account support
|
|
.PP
|
|
You can set up rclone with Google Cloud Storage in an unattended mode,
|
|
i.e.
|
|
not tied to a specific end\-user Google account.
|
|
This is useful when you want to synchronise files onto machines that
|
|
don\[aq]t have actively logged\-in users, for example build machines.
|
|
.PP
|
|
To get credentials for Google Cloud Platform IAM Service
|
|
Accounts (https://cloud.google.com/iam/docs/service-accounts), please
|
|
head to the Service
|
|
Account (https://console.cloud.google.com/permissions/serviceaccounts)
|
|
section of the Google Developer Console.
|
|
Service Accounts behave just like normal \f[C]User\f[] permissions in
|
|
Google Cloud Storage
|
|
ACLs (https://cloud.google.com/storage/docs/access-control), so you can
|
|
limit their access (e.g.
|
|
make them read only).
|
|
After creating an account, a JSON file containing the Service
|
|
Account\[aq]s credentials will be downloaded onto your machines.
|
|
These credentials are what rclone will use for authentication.
|
|
.PP
|
|
To use a Service Account instead of OAuth2 token flow, enter the path to
|
|
your Service Account credentials at the \f[C]service_account_file\f[]
|
|
prompt and rclone won\[aq]t use the browser based authentication flow.
|
|
.SS Modified time
|
|
.PP
|
|
Google google cloud storage stores md5sums natively and rclone stores
|
|
modification times as metadata on the object, under the "mtime" key in
|
|
RFC3339 format accurate to 1ns.
|
|
.SS Amazon Drive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for Amazon Drive involves getting a token from Amazon
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 1
|
|
Amazon\ Application\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Amazon\ Application\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015\-09\-06T16:07:39.658438471+01:00"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (http://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Amazon.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your Amazon Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your Amazon Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Amazon Drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and MD5SUMs
|
|
.PP
|
|
Amazon Drive doesn\[aq]t allow modification times to be changed via the
|
|
API so these won\[aq]t be accurate or used for syncing.
|
|
.PP
|
|
It does store MD5SUMs so for a more accurate sync, you can use the
|
|
\f[C]\-\-checksum\f[] flag.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Amazon don\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Amazon\[aq]s
|
|
apps or via the Amazon Drive website.
|
|
.SS Using with non \f[C]\&.com\f[] Amazon accounts
|
|
.PP
|
|
Let\[aq]s say you usually use \f[C]amazon.co.uk\f[].
|
|
When you authenticate with rclone it will take you to an
|
|
\f[C]amazon.com\f[] page to log in.
|
|
Your \f[C]amazon.co.uk\f[] email and password should work here just
|
|
fine.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-acd\-templink\-threshold=SIZE
|
|
.PP
|
|
Files this size or more will be downloaded via their \f[C]tempLink\f[].
|
|
This is to work around a problem with Amazon Drive which blocks
|
|
downloads of files bigger than about 10GB.
|
|
The default for this is 9GB which shouldn\[aq]t need to be changed.
|
|
.PP
|
|
To download files above this threshold, rclone requests a
|
|
\f[C]tempLink\f[] which downloads the file through a temporary URL
|
|
directly from the underlying S3 storage.
|
|
.SS \-\-acd\-upload\-wait\-per\-gb=TIME
|
|
.PP
|
|
Sometimes Amazon Drive gives an error when a file has been fully
|
|
uploaded but the file appears anyway after a little while.
|
|
This happens sometimes for files over 1GB in size and nearly every time
|
|
for files bigger than 10GB.
|
|
This parameter controls the time rclone waits for the file to appear.
|
|
.PP
|
|
The default value for this parameter is 3 minutes per GB, so by default
|
|
it will wait 3 minutes for every GB uploaded to see if the file appears.
|
|
.PP
|
|
You can disable this feature by setting it to 0.
|
|
This may cause conflict errors as rclone retries the failed upload but
|
|
the file will most likely appear correctly eventually.
|
|
.PP
|
|
These values were determined empirically by observing lots of uploads of
|
|
big files for a range of file sizes.
|
|
.PP
|
|
Upload with the \f[C]\-v\f[] flag to see more info about what rclone is
|
|
doing in this situation.
|
|
.SS Limitations
|
|
.PP
|
|
Note that Amazon Drive is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
Amazon Drive has rate limiting so you may notice errors in the sync (429
|
|
errors).
|
|
rclone will automatically retry the sync up to 3 times by default (see
|
|
\f[C]\-\-retries\f[] flag) which should hopefully work around this
|
|
problem.
|
|
.PP
|
|
Amazon Drive has an internal limit of file sizes that can be uploaded to
|
|
the service.
|
|
This limit is not officially published, but all files larger than this
|
|
will fail.
|
|
.PP
|
|
At the time of writing (Jan 2016) is in the area of 50GB per file.
|
|
This means that larger files are likely to fail.
|
|
.PP
|
|
Unfortunatly there is no way for rclone to see that this failure is
|
|
because of file size, so it will retry the operation, as any other
|
|
failure.
|
|
To avoid this problem, use \f[C]\-\-max\-size\ 50000M\f[] option to
|
|
limit the maximum size of uploaded files.
|
|
Note that \f[C]\-\-max\-size\f[] does not split files into segments, it
|
|
only ignores files over this size.
|
|
.SS Microsoft One Drive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for One Drive involves getting a token from Microsoft
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 9
|
|
Microsoft\ App\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Microsoft\ App\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"access_token":"XXXXXX"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (http://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Microsoft.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your One Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your One Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an One Drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
One Drive allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
One drive supports SHA1 type hashes, so you can use
|
|
\f[C]\-\-checksum\f[] flag.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Microsoft doesn\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Microsoft\[aq]s
|
|
apps or via the One Drive website.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-onedrive\-chunk\-size=SIZE
|
|
.PP
|
|
Above this size files will be chunked \- must be multiple of 320k.
|
|
The default is 10MB.
|
|
Note that the chunks will be buffered into memory.
|
|
.SS \-\-onedrive\-upload\-cutoff=SIZE
|
|
.PP
|
|
Cutoff for switching to chunked upload \- must be <= 100MB.
|
|
The default is 10MB.
|
|
.SS Limitations
|
|
.PP
|
|
Note that One Drive is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
Rclone only supports your default One Drive, and doesn\[aq]t work with
|
|
One Drive for business.
|
|
Both these issues may be fixed at some point depending on user demand!
|
|
.PP
|
|
There are quite a few characters that can\[aq]t be in One Drive file
|
|
names.
|
|
These can\[aq]t occur on Windows platforms, but on non\-Windows
|
|
platforms they are common.
|
|
Rclone will map these names to and from an identical looking unicode
|
|
equivalent.
|
|
For example if a file has a \f[C]?\f[] in it will be mapped to
|
|
\f[C]？\f[] instead.
|
|
.PP
|
|
The largest allowed file size is 10GiB (10,737,418,240 bytes).
|
|
.SS Hubic
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[] (or \f[C]remote:\f[]
|
|
for the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:container/path/to/dir\f[].
|
|
.PP
|
|
The initial setup for Hubic involves getting a token from Hubic which
|
|
you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 7
|
|
Hubic\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Hubic\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"access_token":"XXXXXX"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (http://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Hubic.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List containers in the top level of your Hubic
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your Hubic
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Hubic directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you want the directory to be visible in the official \f[I]Hubic
|
|
browser\f[], you need to copy your files to the \f[C]default\f[]
|
|
directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:default/backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Object\-Meta\-Mtime\f[] as floating point since the epoch
|
|
accurate to 1 ns.
|
|
.PP
|
|
This is a defacto standard (used in the official python\-swiftclient
|
|
amongst others) for storing the modification time for an object.
|
|
.PP
|
|
Note that Hubic wraps the Swift backend, so most of the properties of
|
|
are the same.
|
|
.SS Limitations
|
|
.PP
|
|
This uses the normal OpenStack Swift mechanism to refresh the Swift API
|
|
credentials and ignores the expires field returned by the Hubic API.
|
|
.PP
|
|
The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
|
|
(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
|
|
MD5SUM for these.
|
|
.SS Backblaze B2
|
|
.PP
|
|
B2 is Backblaze\[aq]s cloud storage
|
|
system (https://www.backblaze.com/b2/).
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making a b2 configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
You will need your account number (a short hex number) and key (a long
|
|
hex number) which you can get from the b2 control panel.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
q)\ Quit\ config
|
|
n/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 3
|
|
Account\ ID
|
|
account>\ 123456789abc
|
|
Application\ Key
|
|
key>\ 0123456789abcdef0123456789abcdef0123456789
|
|
Endpoint\ for\ the\ service\ \-\ leave\ blank\ normally.
|
|
endpoint>\
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
account\ =\ 123456789abc
|
|
key\ =\ 0123456789abcdef0123456789abcdef0123456789
|
|
endpoint\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Bz\-Info\-src_last_modified_millis\f[] as milliseconds since
|
|
1970\-01\-01 in the Backblaze standard.
|
|
Other tools should be able to use this as a modified time.
|
|
.PP
|
|
Modified times are used in syncing and are fully supported except in the
|
|
case of updating a modification time on an existing object.
|
|
In this case the object will be uploaded again as B2 doesn\[aq]t have an
|
|
API method to set the modification time independent of doing an upload.
|
|
.SS SHA1 checksums
|
|
.PP
|
|
The SHA1 checksums of the files are checked on upload and download and
|
|
will be used in the syncing process.
|
|
.PP
|
|
Large files which are uploaded in chunks will store their SHA1 on the
|
|
object as \f[C]X\-Bz\-Info\-large_file_sha1\f[] as recommended by
|
|
Backblaze.
|
|
.SS Transfers
|
|
.PP
|
|
Backblaze recommends that you do lots of transfers simultaneously for
|
|
maximum speed.
|
|
In tests from my SSD equiped laptop the optimum setting is about
|
|
\f[C]\-\-transfers\ 32\f[] though higher numbers may be used for a
|
|
slight speed improvement.
|
|
The optimum number for you may vary depending on your hardware, how big
|
|
the files are, how much you want to load your computer, etc.
|
|
The default of \f[C]\-\-transfers\ 4\f[] is definitely too low for
|
|
Backblaze B2 though.
|
|
.PP
|
|
Note that uploading big files (bigger than 200 MB by default) will use a
|
|
96 MB RAM buffer by default.
|
|
There can be at most \f[C]\-\-transfers\f[] of these in use at any
|
|
moment, so this sets the upper limit on the memory used.
|
|
.SS Versions
|
|
.PP
|
|
When rclone uploads a new version of a file it creates a new version of
|
|
it (https://www.backblaze.com/b2/docs/file_versions.html).
|
|
Likewise when you delete a file, the old version will still be
|
|
available.
|
|
.PP
|
|
Old versions of files are visible using the \f[C]\-\-b2\-versions\f[]
|
|
flag.
|
|
.PP
|
|
If you wish to remove all the old versions then you can use the
|
|
\f[C]rclone\ cleanup\ remote:bucket\f[] command which will delete all
|
|
the old versions of files, leaving the current ones intact.
|
|
You can also supply a path and only old versions under that path will be
|
|
deleted, eg \f[C]rclone\ cleanup\ remote:bucket/path/to/stuff\f[].
|
|
.PP
|
|
When you \f[C]purge\f[] a bucket, the current and the old versions will
|
|
be deleted then the bucket will be deleted.
|
|
.PP
|
|
However \f[C]delete\f[] will cause the current versions of the files to
|
|
become hidden old versions.
|
|
.PP
|
|
Here is a session showing the listing and and retreival of an old
|
|
version followed by a \f[C]cleanup\f[] of the old versions.
|
|
.PP
|
|
Show current version and all the versions with \f[C]\-\-b2\-versions\f[]
|
|
flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\ \ \ \ \ \ \ \ 8\ one\-v2016\-07\-04\-141032\-000.txt
|
|
\ \ \ \ \ \ \ 16\ one\-v2016\-07\-04\-141003\-000.txt
|
|
\ \ \ \ \ \ \ 15\ one\-v2016\-07\-02\-155621\-000.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Retreive an old verson
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ copy\ b2:cleanup\-test/one\-v2016\-07\-04\-141003\-000.txt\ /tmp
|
|
|
|
$\ ls\ \-l\ /tmp/one\-v2016\-07\-04\-141003\-000.txt
|
|
\-rw\-rw\-r\-\-\ 1\ ncw\ ncw\ 16\ Jul\ \ 2\ 17:46\ /tmp/one\-v2016\-07\-04\-141003\-000.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Clean up all the old versions and show that they\[aq]ve gone.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ cleanup\ b2:cleanup\-test
|
|
|
|
$\ rclone\ \-q\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\f[]
|
|
.fi
|
|
.SS Data usage
|
|
.PP
|
|
It is useful to know how many requests are sent to the server in
|
|
different scenarios.
|
|
.PP
|
|
All copy commands send the following 4 requests:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_authorize_account
|
|
/b2api/v1/b2_create_bucket
|
|
/b2api/v1/b2_list_buckets
|
|
/b2api/v1/b2_list_file_names
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The \f[C]b2_list_file_names\f[] request will be sent once for every 1k
|
|
files in the remote path, providing the checksum and modification time
|
|
of the listed files.
|
|
As of version 1.33 issue #818 (https://github.com/ncw/rclone/issues/818)
|
|
causes extra requests to be sent when using B2 with Crypt.
|
|
When a copy operation does not require any files to be uploaded, no more
|
|
requests will be sent.
|
|
.PP
|
|
Uploading files that do not require chunking, will send 2 requests per
|
|
file upload:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_get_upload_url
|
|
/b2api/v1/b2_upload_file/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Uploading files requiring chunking, will send 2 requests (one each to
|
|
start and finish the upload) and another 2 requests for each chunk:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_start_large_file
|
|
/b2api/v1/b2_get_upload_part_url
|
|
/b2api/v1/b2_upload_part/
|
|
/b2api/v1/b2_finish_large_file
|
|
\f[]
|
|
.fi
|
|
.SS B2 with crypt
|
|
.PP
|
|
When using B2 with \f[C]crypt\f[] files are encrypted into a temporary
|
|
location and streamed from there.
|
|
This is required to calculate the encrypted file\[aq]s checksum before
|
|
beginning the upload.
|
|
On Windows the %TMPDIR% environment variable is used as the temporary
|
|
location.
|
|
If the file requires chunking, both the chunking and encryption will
|
|
take place in memory.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-b2\-chunk\-size valuee=SIZE
|
|
.PP
|
|
When uploading large files chunk the file into this size.
|
|
Note that these chunks are buffered in memory and there might a maximum
|
|
of \f[C]\-\-transfers\f[] chunks in progress at once.
|
|
100,000,000 Bytes is the minimim size (default 96M).
|
|
.SS \-\-b2\-upload\-cutoff=SIZE
|
|
.PP
|
|
Cutoff for switching to chunked upload (default 190.735 MiB == 200 MB).
|
|
Files above this size will be uploaded in chunks of
|
|
\f[C]\-\-b2\-chunk\-size\f[].
|
|
.PP
|
|
This value should be set no larger than 4.657GiB (== 5GB) as this is the
|
|
largest file size that can be uploaded.
|
|
.SS \-\-b2\-test\-mode=FLAG
|
|
.PP
|
|
This is for debugging purposes only.
|
|
.PP
|
|
Setting FLAG to one of the strings below will cause b2 to return
|
|
specific errors for debugging purposes.
|
|
.IP \[bu] 2
|
|
\f[C]fail_some_uploads\f[]
|
|
.IP \[bu] 2
|
|
\f[C]expire_some_account_authorization_tokens\f[]
|
|
.IP \[bu] 2
|
|
\f[C]force_cap_exceeded\f[]
|
|
.PP
|
|
These will be set in the \f[C]X\-Bz\-Test\-Mode\f[] header which is
|
|
documented in the b2 integrations
|
|
checklist (https://www.backblaze.com/b2/docs/integration_checklist.html).
|
|
.SS \-\-b2\-versions
|
|
.PP
|
|
When set rclone will show and act on older versions of files.
|
|
For example
|
|
.PP
|
|
Listing without \f[C]\-\-b2\-versions\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\ \ \ \ \ \ \ \ 8\ one\-v2016\-07\-04\-141032\-000.txt
|
|
\ \ \ \ \ \ \ 16\ one\-v2016\-07\-04\-141003\-000.txt
|
|
\ \ \ \ \ \ \ 15\ one\-v2016\-07\-02\-155621\-000.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Showing that the current version is unchanged but older versions can be
|
|
seen.
|
|
These have the UTC date that they were uploaded to the server to the
|
|
nearest millisecond appended to them.
|
|
.PP
|
|
Note that when using \f[C]\-\-b2\-versions\f[] no file write operations
|
|
are permitted, so you can\[aq]t upload files or delete them.
|
|
.SS Yandex Disk
|
|
.PP
|
|
Yandex Disk (https://disk.yandex.com) is a cloud storage solution
|
|
created by Yandex (http://yandex.com).
|
|
.PP
|
|
Yandex paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
Here is an example of making a yandex configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 6\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 7\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 8\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
\ 9\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
10\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
11\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 11
|
|
Yandex\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Yandex\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","expiry":"2016\-12\-29T12:27:11.362788025Z"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (http://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Yandex Disk.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
See top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote path, deleting any
|
|
excess files in the path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
Modified times are supported and are stored accurate to 1 ns in custom
|
|
metadata called \f[C]rclone_modified\f[] in RFC3339 with nanoseconds
|
|
format.
|
|
.SS MD5 checksums
|
|
.PP
|
|
MD5 checksums are natively supported by Yandex Disk.
|
|
.SS Crypt
|
|
.PP
|
|
The \f[C]crypt\f[] remote encrypts and decrypts another remote.
|
|
.PP
|
|
To use it first set up the underlying remote following the config
|
|
instructions for that remote.
|
|
You can also use a local pathname instead of a remote which will encrypt
|
|
and decrypt from that directory which might be useful for encrypting
|
|
onto a USB stick for example.
|
|
.PP
|
|
First check your chosen remote is working \- we\[aq]ll call it
|
|
\f[C]remote:path\f[] in these docs.
|
|
Note that anything inside \f[C]remote:path\f[] will be encrypted and
|
|
anything outside won\[aq]t.
|
|
This means that if you are using a bucket based remote (eg S3, B2,
|
|
swift) then you should probably put the bucket in the remote
|
|
\f[C]s3:bucket\f[].
|
|
If you just use \f[C]s3:\f[] then rclone will make encrypted bucket
|
|
names too (if using file name encryption) which may or may not be what
|
|
you want.
|
|
.PP
|
|
Now configure \f[C]crypt\f[] using \f[C]rclone\ config\f[].
|
|
We will call this one \f[C]secret\f[] to differentiate it from the
|
|
\f[C]remote\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n\ \ \
|
|
name>\ secret
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 5
|
|
Remote\ to\ encrypt/decrypt.
|
|
Normally\ should\ contain\ a\ \[aq]:\[aq]\ and\ a\ path,\ eg\ "myremote:path/to/dir",
|
|
"myremote:bucket"\ or\ "myremote:"
|
|
remote>\ remote:path
|
|
How\ to\ encrypt\ the\ filenames.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Don\[aq]t\ encrypt\ the\ file\ names.\ \ Adds\ a\ ".bin"\ extension\ only.
|
|
\ \ \ \\\ "off"
|
|
\ 2\ /\ Encrypt\ the\ filenames\ see\ the\ docs\ for\ the\ details.
|
|
\ \ \ \\\ "standard"
|
|
filename_encryption>\ 2
|
|
Password\ or\ pass\ phrase\ for\ encryption.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
y/g>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
Password\ or\ pass\ phrase\ for\ salt.\ Optional\ but\ recommended.
|
|
Should\ be\ different\ to\ the\ previous\ password.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
n)\ No\ leave\ this\ optional\ password\ blank
|
|
y/g/n>\ g
|
|
Password\ strength\ in\ bits.
|
|
64\ is\ just\ about\ memorable
|
|
128\ is\ secure
|
|
1024\ is\ the\ maximum
|
|
Bits>\ 128
|
|
Your\ password\ is:\ JAsJvRcgR\-_veXNfy_sGmQ
|
|
Use\ this\ password?
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[secret]
|
|
remote\ =\ remote:path
|
|
filename_encryption\ =\ standard
|
|
password\ =\ ***\ ENCRYPTED\ ***
|
|
password2\ =\ ***\ ENCRYPTED\ ***
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
\f[B]Important\f[] The password is stored in the config file is lightly
|
|
obscured so it isn\[aq]t immediately obvious what it is.
|
|
It is in no way secure unless you use config file encryption.
|
|
.PP
|
|
A long passphrase is recommended, or you can use a random one.
|
|
Note that if you reconfigure rclone with the same passwords/passphrases
|
|
elsewhere it will be compatible \- all the secrets used are derived from
|
|
those two passwords/passphrases.
|
|
.PP
|
|
Note that rclone does not encrypt * file length \- this can be calcuated
|
|
within 16 bytes * modification time \- used for syncing
|
|
.SS Specifying the remote
|
|
.PP
|
|
In normal use, make sure the remote has a \f[C]:\f[] in.
|
|
If you specify the remote without a \f[C]:\f[] then rclone will use a
|
|
local directory of that name.
|
|
So if you use a remote of \f[C]/path/to/secret/files\f[] then rclone
|
|
will encrypt stuff to that directory.
|
|
If you use a remote of \f[C]name\f[] then rclone will put files in a
|
|
directory called \f[C]name\f[] in the current directory.
|
|
.PP
|
|
If you specify the remote as \f[C]remote:path/to/dir\f[] then rclone
|
|
will store encrypted files in \f[C]path/to/dir\f[] on the remote.
|
|
If you are using file name encryption, then when you save files to
|
|
\f[C]secret:subdir/subfile\f[] this will store them in the unencrypted
|
|
path \f[C]path/to/dir\f[] but the \f[C]subdir/subpath\f[] bit will be
|
|
encrypted.
|
|
.PP
|
|
Note that unless you want encrypted bucket names (which are difficult to
|
|
manage because you won\[aq]t know what directory they represent in web
|
|
interfaces etc), you should probably specify a bucket, eg
|
|
\f[C]remote:secretbucket\f[] when using bucket based remotes such as S3,
|
|
Swift, Hubic, B2, GCS.
|
|
.SS Example
|
|
.PP
|
|
To test I made a little directory of files using "standard" file name
|
|
encryption.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
plaintext/
|
|
├──\ file0.txt
|
|
├──\ file1.txt
|
|
└──\ subdir
|
|
\ \ \ \ ├──\ file2.txt
|
|
\ \ \ \ ├──\ file3.txt
|
|
\ \ \ \ └──\ subsubdir
|
|
\ \ \ \ \ \ \ \ └──\ file4.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Copy these to the remote and list them back
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ copy\ plaintext\ secret:
|
|
$\ rclone\ \-q\ ls\ secret:
|
|
\ \ \ \ \ \ \ \ 7\ file1.txt
|
|
\ \ \ \ \ \ \ \ 6\ file0.txt
|
|
\ \ \ \ \ \ \ \ 8\ subdir/file2.txt
|
|
\ \ \ \ \ \ \ 10\ subdir/subsubdir/file4.txt
|
|
\ \ \ \ \ \ \ \ 9\ subdir/file3.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Now see what that looked like when encrypted
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ remote:path
|
|
\ \ \ \ \ \ \ 55\ hagjclgavj2mbiqm6u6cnjjqcg
|
|
\ \ \ \ \ \ \ 54\ v05749mltvv1tf4onltun46gls
|
|
\ \ \ \ \ \ \ 57\ 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
|
|
\ \ \ \ \ \ \ 58\ 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
|
|
\ \ \ \ \ \ \ 56\ 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that this retains the directory structure which means you can do
|
|
this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ secret:subdir
|
|
\ \ \ \ \ \ \ \ 8\ file2.txt
|
|
\ \ \ \ \ \ \ \ 9\ file3.txt
|
|
\ \ \ \ \ \ \ 10\ subsubdir/file4.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If don\[aq]t use file name encryption then the remote will look like
|
|
this \- note the \f[C]\&.bin\f[] extensions added to prevent the cloud
|
|
provider attempting to interpret the data.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ remote:path
|
|
\ \ \ \ \ \ \ 54\ file0.txt.bin
|
|
\ \ \ \ \ \ \ 57\ subdir/file3.txt.bin
|
|
\ \ \ \ \ \ \ 56\ subdir/file2.txt.bin
|
|
\ \ \ \ \ \ \ 58\ subdir/subsubdir/file4.txt.bin
|
|
\ \ \ \ \ \ \ 55\ file1.txt.bin
|
|
\f[]
|
|
.fi
|
|
.SS File name encryption modes
|
|
.PP
|
|
Here are some of the features of the file name encryption modes
|
|
.PP
|
|
Off * doesn\[aq]t hide file names or directory structure * allows for
|
|
longer file names (~246 characters) * can use sub paths and copy single
|
|
files
|
|
.PP
|
|
Standard * file names encrypted * file names can\[aq]t be as long (~156
|
|
characters) * can use sub paths and copy single files * directory
|
|
structure visibile * identical files names will have identical uploaded
|
|
names * can use shortcuts to shorten the directory recursion
|
|
.PP
|
|
Cloud storage systems have various limits on file name length and total
|
|
path length which you are more likely to hit using "Standard" file name
|
|
encryption.
|
|
If you keep your file names to below 156 characters in length then you
|
|
should be OK on all providers.
|
|
.PP
|
|
There may be an even more secure file name encryption mode in the future
|
|
which will address the long file name problem.
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Crypt stores modification times using the underlying remote so support
|
|
depends on that.
|
|
.PP
|
|
Hashes are not stored for crypt.
|
|
However the data integrity is protected by an extremely strong crypto
|
|
authenticator.
|
|
.SS File formats
|
|
.SS File encryption
|
|
.PP
|
|
Files are encrypted 1:1 source file to destination object.
|
|
The file has a header and is divided into chunks.
|
|
.SS Header
|
|
.IP \[bu] 2
|
|
8 bytes magic string \f[C]RCLONE\\x00\\x00\f[]
|
|
.IP \[bu] 2
|
|
24 bytes Nonce (IV)
|
|
.PP
|
|
The initial nonce is generated from the operating systems crypto strong
|
|
random number genrator.
|
|
The nonce is incremented for each chunk read making sure each nonce is
|
|
unique for each block written.
|
|
The chance of a nonce being re\-used is miniscule.
|
|
If you wrote an exabyte of data (10¹⁸ bytes) you would have a
|
|
probability of approximately 2×10⁻³² of re\-using a nonce.
|
|
.SS Chunk
|
|
.PP
|
|
Each chunk will contain 64kB of data, except for the last one which may
|
|
have less data.
|
|
The data chunk is in standard NACL secretbox format.
|
|
Secretbox uses XSalsa20 and Poly1305 to encrypt and authenticate
|
|
messages.
|
|
.PP
|
|
Each chunk contains:
|
|
.IP \[bu] 2
|
|
16 Bytes of Poly1305 authenticator
|
|
.IP \[bu] 2
|
|
1 \- 65536 bytes XSalsa20 encrypted data
|
|
.PP
|
|
64k chunk size was chosen as the best performing chunk size (the
|
|
authenticator takes too much time below this and the performance drops
|
|
off due to cache effects above this).
|
|
Note that these chunks are buffered in memory so they can\[aq]t be too
|
|
big.
|
|
.PP
|
|
This uses a 32 byte (256 bit key) key derived from the user password.
|
|
.SS Examples
|
|
.PP
|
|
1 byte file will encrypt to
|
|
.IP \[bu] 2
|
|
32 bytes header
|
|
.IP \[bu] 2
|
|
17 bytes data chunk
|
|
.PP
|
|
49 bytes total
|
|
.PP
|
|
1MB (1048576 bytes) file will encrypt to
|
|
.IP \[bu] 2
|
|
32 bytes header
|
|
.IP \[bu] 2
|
|
16 chunks of 65568 bytes
|
|
.PP
|
|
1049120 bytes total (a 0.05% overhead).
|
|
This is the overhead for big files.
|
|
.SS Name encryption
|
|
.PP
|
|
File names are encrypted segment by segment \- the path is broken up
|
|
into \f[C]/\f[] separated strings and these are encrypted individually.
|
|
.PP
|
|
File segments are padded using using PKCS#7 to a multiple of 16 bytes
|
|
before encryption.
|
|
.PP
|
|
They are then encrypted with EME using AES with 256 bit key.
|
|
EME (ECB\-Mix\-ECB) is a wide\-block encryption mode presented in the
|
|
2003 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
|
|
.PP
|
|
This makes for determinstic encryption which is what we want \- the same
|
|
filename must encrypt to the same thing otherwise we can\[aq]t find it
|
|
on the cloud storage system.
|
|
.PP
|
|
This means that
|
|
.IP \[bu] 2
|
|
filenames with the same name will encrypt the same
|
|
.IP \[bu] 2
|
|
filenames which start the same won\[aq]t have a common prefix
|
|
.PP
|
|
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
|
|
which are derived from the user password.
|
|
.PP
|
|
After encryption they are written out using a modified version of
|
|
standard \f[C]base32\f[] encoding as described in RFC4648.
|
|
The standard encoding is modified in two ways:
|
|
.IP \[bu] 2
|
|
it becomes lower case (no\-one likes upper case filenames!)
|
|
.IP \[bu] 2
|
|
we strip the padding character \f[C]=\f[]
|
|
.PP
|
|
\f[C]base32\f[] is used rather than the more efficient \f[C]base64\f[]
|
|
so rclone can be used on case insensitive remotes (eg Windows, Amazon
|
|
Drive).
|
|
.SS Key derivation
|
|
.PP
|
|
Rclone uses \f[C]scrypt\f[] with parameters \f[C]N=16384,\ r=8,\ p=1\f[]
|
|
with a an optional user supplied salt (password2) to derive the 32+32+16
|
|
= 80 bytes of key material required.
|
|
If the user doesn\[aq]t supply a salt then rclone uses an internal one.
|
|
.PP
|
|
\f[C]scrypt\f[] makes it impractical to mount a dictionary attack on
|
|
rclone encrypted data.
|
|
For full protection agains this you should always use a salt.
|
|
.SS Local Filesystem
|
|
.PP
|
|
Local paths are specified as normal filesystem paths, eg
|
|
\f[C]/path/to/wherever\f[], so
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/source\ /tmp/destination
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Will sync \f[C]/home/source\f[] to \f[C]/tmp/destination\f[]
|
|
.PP
|
|
These can be configured into the config file for consistencies sake, but
|
|
it is probably easier not to.
|
|
.SS Modified time
|
|
.PP
|
|
Rclone reads and writes the modified time using an accuracy determined
|
|
by the OS.
|
|
Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.
|
|
.SS Filenames
|
|
.PP
|
|
Filenames are expected to be encoded in UTF\-8 on disk.
|
|
This is the normal case for Windows and OS X.
|
|
.PP
|
|
There is a bit more uncertainty in the Linux world, but new
|
|
distributions will have UTF\-8 encoded files names.
|
|
If you are using an old Linux filesystem with non UTF\-8 file names (eg
|
|
latin1) then you can use the \f[C]convmv\f[] tool to convert the
|
|
filesystem to UTF\-8.
|
|
This tool is available in most distributions\[aq] package managers.
|
|
.PP
|
|
If an invalid (non\-UTF8) filename is read, the invalid caracters will
|
|
be replaced with the unicode replacement character, \[aq]�\[aq].
|
|
\f[C]rclone\f[] will emit a debug message in this case (use \f[C]\-v\f[]
|
|
to see), eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Local\ file\ system\ at\ .:\ Replacing\ invalid\ UTF\-8\ characters\ in\ "gro\\xdf"
|
|
\f[]
|
|
.fi
|
|
.SS Long paths on Windows
|
|
.PP
|
|
Rclone handles long paths automatically, by converting all paths to long
|
|
UNC
|
|
paths (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath)
|
|
which allows paths up to 32,767 characters.
|
|
.PP
|
|
This is why you will see that your paths, for instance
|
|
\f[C]c:\\files\f[] is converted to the UNC path
|
|
\f[C]\\\\?\\c:\\files\f[] in the output, and \f[C]\\\\server\\share\f[]
|
|
is converted to \f[C]\\\\?\\UNC\\server\\share\f[].
|
|
.PP
|
|
However, in rare cases this may cause problems with buggy file system
|
|
drivers like EncFS (https://github.com/ncw/rclone/issues/261).
|
|
To disable UNC conversion globally, add this to your
|
|
\f[C]\&.rclone.conf\f[] file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[local]
|
|
nounc\ =\ true
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you want to selectively disable UNC, you can add it to a separate
|
|
entry like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[nounc]
|
|
type\ =\ local
|
|
nounc\ =\ true
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And use rclone like this:
|
|
.PP
|
|
\f[C]rclone\ copy\ c:\\src\ nounc:z:\\dst\f[]
|
|
.PP
|
|
This will use UNC paths on \f[C]c:\\src\f[] but not on \f[C]z:\\dst\f[].
|
|
Of course this will cause problems if the absolute path length of a file
|
|
exceeds 258 characters on z, so only use this option if you have to.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to local storage
|
|
.SS \-\-one\-file\-system, \-x
|
|
.PP
|
|
This tells rclone to stay in the filesystem specified by the root and
|
|
not to recurse into different file systems.
|
|
.PP
|
|
For example if you have a directory heirachy like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
root
|
|
├──\ disk1\ \ \ \ \ \-\ disk1\ mounted\ on\ the\ root
|
|
│\ \ \ └──\ file3\ \-\ stored\ on\ disk1
|
|
├──\ disk2\ \ \ \ \ \-\ disk2\ mounted\ on\ the\ root
|
|
│\ \ \ └──\ file4\ \-\ stored\ on\ disk12
|
|
├──\ file1\ \ \ \ \ \-\ stored\ on\ the\ root\ disk
|
|
└──\ file2\ \ \ \ \ \-\ stored\ on\ the\ root\ disk
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Using \f[C]rclone\ \-\-one\-file\-system\ copy\ root\ remote:\f[] will
|
|
only copy \f[C]file1\f[] and \f[C]file2\f[].
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ \-\-one\-file\-system\ ls\ root
|
|
\ \ \ \ \ \ \ \ 0\ file1
|
|
\ \ \ \ \ \ \ \ 0\ file2
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ root
|
|
\ \ \ \ \ \ \ \ 0\ disk1/file3
|
|
\ \ \ \ \ \ \ \ 0\ disk2/file4
|
|
\ \ \ \ \ \ \ \ 0\ file1
|
|
\ \ \ \ \ \ \ \ 0\ file2
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
\f[B]NB\f[] Rclone (like most unix tools such as \f[C]du\f[],
|
|
\f[C]rsync\f[] and \f[C]tar\f[]) treats a bind mount to the same device
|
|
as being on the same filesystem.
|
|
.PP
|
|
\f[B]NB\f[] This flag is only available on Unix based systems.
|
|
On systems where it isn\[aq]t supported (eg Windows) it will not appear
|
|
as an valid flag.
|
|
.SS Changelog
|
|
.IP \[bu] 2
|
|
v1.35 \- 2017\-01\-02
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
moveto and copyto commands for choosing a destination name on copy/move
|
|
.IP \[bu] 2
|
|
rmdirs command to recursively delete empty directories
|
|
.IP \[bu] 2
|
|
Allow repeated \-\-include/\-\-exclude/\-\-filter options
|
|
.IP \[bu] 2
|
|
Only show transfer stats on commands which transfer stuff
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
show stats on any command using the \f[C]\-\-stats\f[] flag
|
|
.RE
|
|
.IP \[bu] 2
|
|
Allow overlapping directories in move when server side dir move is
|
|
supported
|
|
.IP \[bu] 2
|
|
Add \-\-stats\-unit option \- thanks Scott McGillivray
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix the config file being overwritten when two rclones are running
|
|
.IP \[bu] 2
|
|
Make rclone lsd obey the filters properly
|
|
.IP \[bu] 2
|
|
Fix compilation on mips
|
|
.IP \[bu] 2
|
|
Fix not transferring files that don\[aq]t differ in size
|
|
.IP \[bu] 2
|
|
Fix panic on nil retry/fatal error
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Retry reads on error \- should help with reliability a lot
|
|
.IP \[bu] 2
|
|
Report the modification times for directories from the remote
|
|
.IP \[bu] 2
|
|
Add bandwidth accounting and limiting (fixes \-\-bwlimit)
|
|
.IP \[bu] 2
|
|
If \-\-stats provided will show stats and which files are transferring
|
|
.IP \[bu] 2
|
|
Support R/W files if truncate is set.
|
|
.IP \[bu] 2
|
|
Implement statfs interface so df works
|
|
.IP \[bu] 2
|
|
Note that write is now supported on Amazon Drive
|
|
.IP \[bu] 2
|
|
Report number of blocks in a file \- thanks Stefan Breunig
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Prevent the user pointing crypt at itself
|
|
.IP \[bu] 2
|
|
Fix failed to authenticate decrypted block errors
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
these will now return the underlying unexpected EOF instead
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Add support for server side move and directory move \- thanks Stefan
|
|
Breunig
|
|
.IP \[bu] 2
|
|
Fix nil pointer deref on size attribute
|
|
.IP \[bu] 2
|
|
B2
|
|
.IP \[bu] 2
|
|
Use new prefix and delimiter parameters in directory listings
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This makes \-\-max\-depth 1 dir listings as used in mount much faster
|
|
.RE
|
|
.IP \[bu] 2
|
|
Reauth the account while doing uploads too \- should help with token
|
|
expiry
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Make DirMove more efficient and complain about moving the root
|
|
.IP \[bu] 2
|
|
Create destination directory on Move()
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.34 \- 2016\-11\-06
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Stop single file and \f[C]\-\-files\-from\f[] operations iterating
|
|
through the source bucket.
|
|
.IP \[bu] 2
|
|
Stop removing failed upload to cloud storage remotes
|
|
.IP \[bu] 2
|
|
Make ContentType be preserved for cloud to cloud copies
|
|
.IP \[bu] 2
|
|
Add support to toggle bandwidth limits via SIGUSR2 \- thanks Marco
|
|
Paganini
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ check\f[] shows count of hashes that couldn\[aq]t be
|
|
checked
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ listremotes\f[] command
|
|
.IP \[bu] 2
|
|
Support linux/arm64 build \- thanks Fredrik Fornwall
|
|
.IP \[bu] 2
|
|
Remove \f[C]Authorization:\f[] lines from \f[C]\-\-dump\-headers\f[]
|
|
output
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Ignore files with control characters in the names
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone\ move\f[] command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Delete src files which already existed in dst
|
|
.IP \[bu] 2
|
|
Fix deletion of src file when dst file older
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone\ check\f[] on crypted file systems
|
|
.IP \[bu] 2
|
|
Make failed uploads not count as "Transferred"
|
|
.IP \[bu] 2
|
|
Make sure high level retries show with \f[C]\-q\f[]
|
|
.IP \[bu] 2
|
|
Use a vendor directory with godep for repeatable builds
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ mount\f[] \- FUSE
|
|
.IP \[bu] 2
|
|
Implement FUSE mount options
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]\-\-no\-modtime\f[], \f[C]\-\-debug\-fuse\f[],
|
|
\f[C]\-\-read\-only\f[], \f[C]\-\-allow\-non\-empty\f[],
|
|
\f[C]\-\-allow\-root\f[], \f[C]\-\-allow\-other\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-default\-permissions\f[], \f[C]\-\-write\-back\-cache\f[],
|
|
\f[C]\-\-max\-read\-ahead\f[], \f[C]\-\-umask\f[], \f[C]\-\-uid\f[],
|
|
\f[C]\-\-gid\f[]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-dir\-cache\-time\f[] to control caching of directory
|
|
entries
|
|
.IP \[bu] 2
|
|
Implement seek for files opened for read (useful for video players)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
with \f[C]\-no\-seek\f[] flag to disable
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix crash on 32 bit ARM (alignment of 64 bit counter)
|
|
.IP \[bu] 2
|
|
\&...and many more internal fixes and improvements!
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Don\[aq]t show encrypted password in configurator to stop confusion
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
New wait for upload option \f[C]\-\-acd\-upload\-wait\-per\-gb\f[]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
upload timeouts scale by file size and can be disabled
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add 502 Bad Gateway to list of errors we retry
|
|
.IP \[bu] 2
|
|
Fix overwriting a file with a zero length file
|
|
.IP \[bu] 2
|
|
Fix ACD file size warning limit \- thanks Felix Bünemann
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Unix: implement \f[C]\-x\f[]/\f[C]\-\-one\-file\-system\f[] to stay on a
|
|
single file system
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
thanks Durval Menezes and Luiz Carlos Rumbelsperger Viana
|
|
.RE
|
|
.IP \[bu] 2
|
|
Windows: ignore the symlink bit on files
|
|
.IP \[bu] 2
|
|
Windows: Ignore directory based junction points
|
|
.IP \[bu] 2
|
|
B2
|
|
.IP \[bu] 2
|
|
Make sure each upload has at least one upload slot \- fixes strange
|
|
upload stats
|
|
.IP \[bu] 2
|
|
Fix uploads when using crypt
|
|
.IP \[bu] 2
|
|
Fix download of large files (sha1 mismatch)
|
|
.IP \[bu] 2
|
|
Return error when we try to create a bucket which someone else owns
|
|
.IP \[bu] 2
|
|
Update B2 docs with Data usage, and Crypt section \- thanks Tomasz Mazur
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Command line and config file support for
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Setting/overriding ACL \- thanks Radek Senfeld
|
|
.IP \[bu] 2
|
|
Setting storage class \- thanks Asko Tamm
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Make exponential backoff work exactly as per Google specification
|
|
.IP \[bu] 2
|
|
add \f[C]\&.epub\f[], \f[C]\&.odp\f[] and \f[C]\&.tsv\f[] as export
|
|
formats.
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Don\[aq]t read metadata for directory marker objects
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.33 \- 2016\-08\-24
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Implement encryption
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
data encrypted in NACL secretbox format
|
|
.IP \[bu] 2
|
|
with optional file name encryption
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone mount \- implements FUSE mounting of remotes (EXPERIMENTAL)
|
|
.IP \[bu] 2
|
|
works on Linux, FreeBSD and OS X (need testers for the last 2!)
|
|
.IP \[bu] 2
|
|
rclone cat \- outputs remote file or files to the terminal
|
|
.IP \[bu] 2
|
|
rclone genautocomplete \- command to make a bash completion script for
|
|
rclone
|
|
.RE
|
|
.IP \[bu] 2
|
|
Editing a remote using \f[C]rclone\ config\f[] now goes through the
|
|
wizard
|
|
.IP \[bu] 2
|
|
Compile with go 1.7 \- this fixes rclone on macOS Sierra and on 386
|
|
processors
|
|
.IP \[bu] 2
|
|
Use cobra for sub commands and docs generation
|
|
.IP \[bu] 2
|
|
drive
|
|
.IP \[bu] 2
|
|
Document how to make your own client_id
|
|
.IP \[bu] 2
|
|
s3
|
|
.IP \[bu] 2
|
|
User\-configurable Amazon S3 ACL (thanks Radek Šenfeld)
|
|
.IP \[bu] 2
|
|
b2
|
|
.IP \[bu] 2
|
|
Fix stats accounting for upload \- no more jumping to 100% done
|
|
.IP \[bu] 2
|
|
On cleanup delete hide marker if it is the current file
|
|
.IP \[bu] 2
|
|
New B2 API endpoint (thanks Per Cederberg)
|
|
.IP \[bu] 2
|
|
Set maximum backoff to 5 Minutes
|
|
.IP \[bu] 2
|
|
onedrive
|
|
.IP \[bu] 2
|
|
Fix URL escaping in file names \- eg uploading files with \f[C]+\f[] in
|
|
them.
|
|
.IP \[bu] 2
|
|
amazon cloud drive
|
|
.IP \[bu] 2
|
|
Fix token expiry during large uploads
|
|
.IP \[bu] 2
|
|
Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors
|
|
.IP \[bu] 2
|
|
local
|
|
.IP \[bu] 2
|
|
Fix filenames with invalid UTF\-8 not being uploaded
|
|
.IP \[bu] 2
|
|
Fix problem with some UTF\-8 characters on OS X
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.32 \- 2016\-07\-13
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Fix upload of files large files not in root
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.31 \- 2016\-07\-13
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Reduce memory on sync by about 50%
|
|
.IP \[bu] 2
|
|
Implement \-\-no\-traverse flag to stop copy traversing the destination
|
|
remote.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This can be used to reduce memory usage down to the smallest possible.
|
|
.IP \[bu] 2
|
|
Useful to copy a small number of files into a large destination folder.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement cleanup command for emptying trash / removing old versions of
|
|
files
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Currently B2 only
|
|
.RE
|
|
.IP \[bu] 2
|
|
Single file handling improved
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now copied with \-\-files\-from
|
|
.IP \[bu] 2
|
|
Automatically sets \-\-no\-traverse when copying a single file
|
|
.RE
|
|
.IP \[bu] 2
|
|
Info on using installing with ansible \- thanks Stefan Weichinger
|
|
.IP \[bu] 2
|
|
Implement \-\-no\-update\-modtime flag to stop rclone fixing the remote
|
|
modified times.
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix move command \- stop it running for overlapping Fses \- this was
|
|
causing data loss.
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Fix incomplete hashes \- this was causing problems for B2.
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Rename Amazon Cloud Drive to Amazon Drive \- no changes to config file
|
|
needed.
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Add support for non\-default project domain \- thanks Antonio Messina.
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Add instructions on how to use rclone with minio.
|
|
.IP \[bu] 2
|
|
Add ap\-northeast\-2 (Seoul) and ap\-south\-1 (Mumbai) regions.
|
|
.IP \[bu] 2
|
|
Skip setting the modified time for objects > 5GB as it isn\[aq]t
|
|
possible.
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Add \-\-b2\-versions flag so old versions can be listed and retreived.
|
|
.IP \[bu] 2
|
|
Treat 403 errors (eg cap exceeded) as fatal.
|
|
.IP \[bu] 2
|
|
Implement cleanup command for deleting old file versions.
|
|
.IP \[bu] 2
|
|
Make error handling compliant with B2 integrations notes.
|
|
.IP \[bu] 2
|
|
Fix handling of token expiry.
|
|
.IP \[bu] 2
|
|
Implement \-\-b2\-test\-mode to set \f[C]X\-Bz\-Test\-Mode\f[] header.
|
|
.IP \[bu] 2
|
|
Set cutoff for chunked upload to 200MB as per B2 guidelines.
|
|
.IP \[bu] 2
|
|
Make upload multi\-threaded.
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Don\[aq]t retry 461 errors.
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.30 \- 2016\-06\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Directory listing code reworked for more features and better error
|
|
reporting (thanks to Klaus Post for help).
|
|
This enables
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Directory include filtering for efficiency
|
|
.IP \[bu] 2
|
|
\-\-max\-depth parameter
|
|
.IP \[bu] 2
|
|
Better error reporting
|
|
.IP \[bu] 2
|
|
More to come
|
|
.RE
|
|
.IP \[bu] 2
|
|
Retry more errors
|
|
.IP \[bu] 2
|
|
Add \-\-ignore\-size flag \- for uploading images to onedrive
|
|
.IP \[bu] 2
|
|
Log \-v output to stdout by default
|
|
.IP \[bu] 2
|
|
Display the transfer stats in more human readable form
|
|
.IP \[bu] 2
|
|
Make 0 size files specifiable with \f[C]\-\-max\-size\ 0b\f[]
|
|
.IP \[bu] 2
|
|
Add \f[C]b\f[] suffix so we can specify bytes in \-\-bwlimit,
|
|
\-\-min\-size etc
|
|
.IP \[bu] 2
|
|
Use "password:" instead of "password>" prompt \- thanks Klaus Post and
|
|
Leigh Klotz
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix retry doing one too many retries
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Fix problems with OS X and UTF\-8 characters
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Check a file exists before uploading to help with 408 Conflict errors
|
|
.IP \[bu] 2
|
|
Reauth on 401 errors \- this has been causing a lot of problems
|
|
.IP \[bu] 2
|
|
Work around spurious 403 errors
|
|
.IP \[bu] 2
|
|
Restart directory listings on error
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Check a file exists before uploading to help with duplicates
|
|
.IP \[bu] 2
|
|
Fix retry of multipart uploads
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Implement large file uploading
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Add AES256 server\-side encryption for \- thanks Justin R.
|
|
Wilson
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Make sure we don\[aq]t use conflicting content types on upload
|
|
.IP \[bu] 2
|
|
Add service account support \- thanks Michal Witkowski
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Add auth version parameter
|
|
.IP \[bu] 2
|
|
Add domain option for openstack (v3 auth) \- thanks Fabian Ruff
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.29 \- 2016\-04\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-I,\ \-\-ignore\-times\f[] for unconditional upload
|
|
.IP \[bu] 2
|
|
Improve \f[C]dedupe\f[]command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now removes identical copies without asking
|
|
.IP \[bu] 2
|
|
Now obeys \f[C]\-\-dry\-run\f[]
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-\-dedupe\-mode\f[] for non interactive running
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ interactive\f[] \- interactive the default.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ skip\f[] \- removes identical files then skips
|
|
anything left.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ first\f[] \- removes identical files then keeps
|
|
the first one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ newest\f[] \- removes identical files then keeps
|
|
the newest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ oldest\f[] \- removes identical files then keeps
|
|
the oldest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ rename\f[] \- removes identical files then
|
|
renames the rest to be different.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug fixes
|
|
.IP \[bu] 2
|
|
Make rclone check obey the \f[C]\-\-size\-only\f[] flag.
|
|
.IP \[bu] 2
|
|
Use "application/octet\-stream" if discovered mime type is invalid.
|
|
.IP \[bu] 2
|
|
Fix missing "quit" option when there are no remotes.
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Increase default chunk size to 8 MB \- increases upload speed of big
|
|
files
|
|
.IP \[bu] 2
|
|
Speed up directory listings and make more reliable
|
|
.IP \[bu] 2
|
|
Add missing retries for Move and DirMove \- increases reliability
|
|
.IP \[bu] 2
|
|
Preserve mime type on file update
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Enable mod time syncing
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This means that B2 will now check modification times
|
|
.IP \[bu] 2
|
|
It will upload new files to update the modification times
|
|
.IP \[bu] 2
|
|
(there isn\[aq]t an API to just set the mod time.)
|
|
.IP \[bu] 2
|
|
If you want the old behaviour use \f[C]\-\-size\-only\f[].
|
|
.RE
|
|
.IP \[bu] 2
|
|
Update API to new version
|
|
.IP \[bu] 2
|
|
Fix parsing of mod time when not in metadata
|
|
.IP \[bu] 2
|
|
Swift/Hubic
|
|
.IP \[bu] 2
|
|
Don\[aq]t return an MD5SUM for static large objects
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Fix uploading files bigger than 50GB
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.28 \- 2016\-03\-01
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Configuration file encryption \- thanks Klaus Post
|
|
.IP \[bu] 2
|
|
Improve \f[C]rclone\ config\f[] adding more help and making it easier to
|
|
understand
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-u\f[]/\f[C]\-\-update\f[] so creation times can be used
|
|
on all remotes
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-\-low\-level\-retries\f[] flag
|
|
.IP \[bu] 2
|
|
Optionally disable gzip compression on downloads with
|
|
\f[C]\-\-no\-gzip\-encoding\f[]
|
|
.IP \[bu] 2
|
|
Bug fixes
|
|
.IP \[bu] 2
|
|
Don\[aq]t make directories if \f[C]\-\-dry\-run\f[] set
|
|
.IP \[bu] 2
|
|
Fix and document the \f[C]move\f[] command
|
|
.IP \[bu] 2
|
|
Fix redirecting stderr on unix\-like OSes when using
|
|
\f[C]\-\-log\-file\f[]
|
|
.IP \[bu] 2
|
|
Fix \f[C]delete\f[] command to wait until all finished \- fixes missing
|
|
deletes.
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Use one upload URL per go routine fixes
|
|
\f[C]more\ than\ one\ upload\ using\ auth\ token\f[]
|
|
.IP \[bu] 2
|
|
Add pacing, retries and reauthentication \- fixes token expiry problems
|
|
.IP \[bu] 2
|
|
Upload without using a temporary file from local (and remotes which
|
|
support SHA1)
|
|
.IP \[bu] 2
|
|
Fix reading metadata for all files when it shouldn\[aq]t have been
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Fix listing drive documents at root
|
|
.IP \[bu] 2
|
|
Disable copy and move for Google docs
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix uploading of chunked files with non ASCII characters
|
|
.IP \[bu] 2
|
|
Allow setting of \f[C]storage_url\f[] in the config \- thanks Xavier
|
|
Lucas
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Allow IAM role and credentials from environment variables \- thanks
|
|
Brian Stengaard
|
|
.IP \[bu] 2
|
|
Allow low privilege users to use S3 (check if directory exists during
|
|
Mkdir) \- thanks Jakub Gedeon
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Retry on more things to make directory listings more reliable
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.27 \- 2016\-01\-31
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Easier headless configuration with \f[C]rclone\ authorize\f[]
|
|
.IP \[bu] 2
|
|
Add support for multiple hash types \- we now check SHA1 as well as MD5
|
|
hashes.
|
|
.IP \[bu] 2
|
|
\f[C]delete\f[] command which does obey the filters (unlike
|
|
\f[C]purge\f[])
|
|
.IP \[bu] 2
|
|
\f[C]dedupe\f[] command to deduplicate a remote.
|
|
Useful with Google Drive.
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-ignore\-existing\f[] flag to skip all files that exist on
|
|
destination.
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-delete\-before\f[], \f[C]\-\-delete\-during\f[],
|
|
\f[C]\-\-delete\-after\f[] flags.
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-memprofile\f[] flag to debug memory use.
|
|
.IP \[bu] 2
|
|
Warn the user about files with same name but different case
|
|
.IP \[bu] 2
|
|
Make \f[C]\-\-include\f[] rules add their implict exclude * at the end
|
|
of the filter list
|
|
.IP \[bu] 2
|
|
Deprecate compiling with go1.3
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Fix download of files > 10 GB
|
|
.IP \[bu] 2
|
|
Fix directory traversal ("Next token is expired") for large directory
|
|
listings
|
|
.IP \[bu] 2
|
|
Remove 409 conflict from error codes we will retry \- stops very long
|
|
pauses
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
SHA1 hashes now checked by rclone core
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-auth\-owner\-only\f[] to only consider files owned
|
|
by the user \- thanks Björn Harrtell
|
|
.IP \[bu] 2
|
|
Export Google documents
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Make file exclusion error controllable with \-q
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix upload from unprivileged user.
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Fix updating of mod times of files with \f[C]+\f[] in.
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Add local file system option to disable UNC on Windows.
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.26 \- 2016\-01\-02
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Yandex storage backend \- thank you Dmitry Burdeev ("dibu")
|
|
.IP \[bu] 2
|
|
Implement Backblaze B2 storage backend
|
|
.IP \[bu] 2
|
|
Add \-\-min\-age and \-\-max\-age flags \- thank you Adriano Aurélio
|
|
Meirelles
|
|
.IP \[bu] 2
|
|
Make ls/lsl/md5sum/size/check obey includes and excludes
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix crash in http logging
|
|
.IP \[bu] 2
|
|
Upload releases to github too
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix sync for chunked files
|
|
.IP \[bu] 2
|
|
One Drive
|
|
.IP \[bu] 2
|
|
Re\-enable server side copy
|
|
.IP \[bu] 2
|
|
Don\[aq]t mask HTTP error codes with JSON decode error
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Fix corrupting Content\-Type on mod time update (thanks Joseph Spurrier)
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.25 \- 2015\-11\-14
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Implement Hubic storage system
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix deletion of some excluded files without \-\-delete\-excluded
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This could have deleted files unexpectedly on sync
|
|
.IP \[bu] 2
|
|
Always check first with \f[C]\-\-dry\-run\f[]!
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Stop SetModTime losing metadata (eg X\-Object\-Manifest)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This could have caused data loss for files > 5GB in size
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use ContentType from Object to avoid lookups in listings
|
|
.IP \[bu] 2
|
|
One Drive
|
|
.IP \[bu] 2
|
|
disable server side copy as it seems to be broken at Microsoft
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.24 \- 2015\-11\-07
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Add support for Microsoft One Drive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-no\-check\-certificate\f[] option to disable server
|
|
certificate verification
|
|
.IP \[bu] 2
|
|
Add async readahead buffer for faster transfer of big files
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Allow spaces in remotes and check remote names for validity at creation
|
|
time
|
|
.IP \[bu] 2
|
|
Allow \[aq]&\[aq] and disallow \[aq]:\[aq] in Windows filenames.
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Ignore directory marker objects where appropriate \- allows working with
|
|
Hubic
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the container if fs wasn\[aq]t at root
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the bucket if fs wasn\[aq]t at root
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the bucket if fs wasn\[aq]t at root
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.23 \- 2015\-10\-03
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone\ size\f[] for measuring remotes
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix headless config for drive and gcs
|
|
.IP \[bu] 2
|
|
Tell the user they should try again if the webserver method failed
|
|
.IP \[bu] 2
|
|
Improve output of \f[C]\-\-dump\-headers\f[]
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Allow anonymous access to public buckets
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Stop chunked operations logging "Failed to read info: Object Not Found"
|
|
.IP \[bu] 2
|
|
Use Content\-Length on uploads for extra reliability
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.22 \- 2015\-09\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement rsync like include and exclude flags
|
|
.IP \[bu] 2
|
|
swift
|
|
.IP \[bu] 2
|
|
Support files > 5GB \- thanks Sergey Tolmachev
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.21 \- 2015\-09\-22
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Display individual transfer progress
|
|
.IP \[bu] 2
|
|
Make lsl output times in localtime
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix allowing user to override credentials again in Drive, GCS and ACD
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Implement compliant pacing scheme
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Make directory reads concurrent for increased speed.
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.20 \- 2015\-09\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Amazon Drive support
|
|
.IP \[bu] 2
|
|
Oauth support redone \- fix many bugs and improve usability
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use "golang.org/x/oauth2" as oauth libary of choice
|
|
.IP \[bu] 2
|
|
Improve oauth usability for smoother initial signup
|
|
.IP \[bu] 2
|
|
drive, googlecloudstorage: optionally use auto config for the oauth
|
|
token
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement \-\-dump\-headers and \-\-dump\-bodies debug flags
|
|
.IP \[bu] 2
|
|
Show multiple matched commands if abbreviation too short
|
|
.IP \[bu] 2
|
|
Implement server side move where possible
|
|
.IP \[bu] 2
|
|
local
|
|
.IP \[bu] 2
|
|
Always use UNC paths internally on Windows \- fixes a lot of bugs
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.IP \[bu] 2
|
|
force use of our custom transport which makes timeouts work
|
|
.IP \[bu] 2
|
|
Thanks to Klaus Post for lots of help with this release
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.19 \- 2015\-08\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Server side copies for s3/swift/drive/dropbox/gcs
|
|
.IP \[bu] 2
|
|
Move command \- uses server side copies if it can
|
|
.IP \[bu] 2
|
|
Implement \-\-retries flag \- tries 3 times by default
|
|
.IP \[bu] 2
|
|
Build for plan9/amd64 and solaris/amd64 too
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Make a current version download with a fixed URL for scripting
|
|
.IP \[bu] 2
|
|
Ignore rmdir in limited fs rather than throwing error
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.IP \[bu] 2
|
|
Increase chunk size to improve upload speeds massively
|
|
.IP \[bu] 2
|
|
Issue an error message when trying to upload bad file name
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.18 \- 2015\-08\-17
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-use\-trash\f[] flag so rclone trashes instead of
|
|
deletes
|
|
.IP \[bu] 2
|
|
Add "Forbidden to download" message for files with no downloadURL
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.IP \[bu] 2
|
|
Remove datastore
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This was deprecated and it caused a lot of problems
|
|
.IP \[bu] 2
|
|
Modification times and MD5SUMs no longer stored
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix uploading files > 2GB
|
|
.IP \[bu] 2
|
|
s3
|
|
.IP \[bu] 2
|
|
use official AWS SDK from github.com/aws/aws\-sdk\-go
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[] will most likely require you to delete and recreate remote
|
|
.IP \[bu] 2
|
|
enable multipart upload which enables files > 5GB
|
|
.IP \[bu] 2
|
|
tested with Ceph / RadosGW / S3 emulation
|
|
.IP \[bu] 2
|
|
many thanks to Sam Liston and Brian Haymore at the Utah Center for High
|
|
Performance Computing (https://www.chpc.utah.edu/) for a Ceph test
|
|
account
|
|
.IP \[bu] 2
|
|
misc
|
|
.IP \[bu] 2
|
|
Show errors when reading the config file
|
|
.IP \[bu] 2
|
|
Do not print stats in quiet mode \- thanks Leonid Shalupov
|
|
.IP \[bu] 2
|
|
Add FAQ
|
|
.IP \[bu] 2
|
|
Fix created directories not obeying umask
|
|
.IP \[bu] 2
|
|
Linux installation instructions \- thanks Shimon Doodkin
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.17 \- 2015\-06\-14
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
dropbox: fix case insensitivity issues \- thanks Leonid Shalupov
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.16 \- 2015\-06\-09
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix uploading big files which was causing timeouts or panics
|
|
.IP \[bu] 2
|
|
Don\[aq]t check md5sum after download with \-\-size\-only
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.15 \- 2015\-06\-06
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \-\-checksum flag to only discard transfers by MD5SUM \- thanks Alex
|
|
Couper
|
|
.IP \[bu] 2
|
|
Implement \-\-size\-only flag to sync on size not checksum & modtime
|
|
.IP \[bu] 2
|
|
Expand docs and remove duplicated information
|
|
.IP \[bu] 2
|
|
Document rclone\[aq]s limitations with directories
|
|
.IP \[bu] 2
|
|
dropbox: update docs about case insensitivity
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.14 \- 2015\-05\-21
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
local: fix encoding of non utf\-8 file names \- fixes a duplicate file
|
|
problem
|
|
.IP \[bu] 2
|
|
drive: docs about rate limiting
|
|
.IP \[bu] 2
|
|
google cloud storage: Fix compile after API change in
|
|
"google.golang.org/api/storage/v1"
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.13 \- 2015\-05\-10
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Revise documentation (especially sync)
|
|
.IP \[bu] 2
|
|
Implement \-\-timeout and \-\-conntimeout
|
|
.IP \[bu] 2
|
|
s3: ignore etags from multipart uploads which aren\[aq]t md5sums
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.12 \- 2015\-03\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: Use chunked upload for files above a certain size
|
|
.IP \[bu] 2
|
|
drive: add \-\-drive\-chunk\-size and \-\-drive\-upload\-cutoff
|
|
parameters
|
|
.IP \[bu] 2
|
|
drive: switch to insert from update when a failed copy deletes the
|
|
upload
|
|
.IP \[bu] 2
|
|
core: Log duplicate files if they are detected
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.11 \- 2015\-03\-04
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
swift: add region parameter
|
|
.IP \[bu] 2
|
|
drive: fix crash on failed to update remote mtime
|
|
.IP \[bu] 2
|
|
In remote paths, change native directory separators to /
|
|
.IP \[bu] 2
|
|
Add synchronization to ls/lsl/lsd output to stop corruptions
|
|
.IP \[bu] 2
|
|
Ensure all stats/log messages to go stderr
|
|
.IP \[bu] 2
|
|
Add \-\-log\-file flag to log everything (including panics) to file
|
|
.IP \[bu] 2
|
|
Make it possible to disable stats printing with \-\-stats=0
|
|
.IP \[bu] 2
|
|
Implement \-\-bwlimit to limit data transfer bandwidth
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.10 \- 2015\-02\-12
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
s3: list an unlimited number of items
|
|
.IP \[bu] 2
|
|
Fix getting stuck in the configurator
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.09 \- 2015\-02\-07
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
windows: Stop drive letters (eg C:) getting mixed up with remotes (eg
|
|
drive:)
|
|
.IP \[bu] 2
|
|
local: Fix directory separators on Windows
|
|
.IP \[bu] 2
|
|
drive: fix rate limit exceeded errors
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.08 \- 2015\-02\-04
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: fix subdirectory listing to not list entire drive
|
|
.IP \[bu] 2
|
|
drive: Fix SetModTime
|
|
.IP \[bu] 2
|
|
dropbox: adapt code to recent library changes
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.07 \- 2014\-12\-23
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
google cloud storage: fix memory leak
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.06 \- 2014\-12\-12
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix "Couldn\[aq]t find home directory" on OSX
|
|
.IP \[bu] 2
|
|
swift: Add tenant parameter
|
|
.IP \[bu] 2
|
|
Use new location of Google API packages
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.05 \- 2014\-08\-09
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improved tests and consequently lots of minor fixes
|
|
.IP \[bu] 2
|
|
core: Fix race detected by go race detector
|
|
.IP \[bu] 2
|
|
core: Fixes after running errcheck
|
|
.IP \[bu] 2
|
|
drive: reset root directory on Rmdir and Purge
|
|
.IP \[bu] 2
|
|
fs: Document that Purger returns error on empty directory, test and fix
|
|
.IP \[bu] 2
|
|
google cloud storage: fix ListDir on subdirectory
|
|
.IP \[bu] 2
|
|
google cloud storage: re\-read metadata in SetModTime
|
|
.IP \[bu] 2
|
|
s3: make reading metadata more reliable to work around eventual
|
|
consistency problems
|
|
.IP \[bu] 2
|
|
s3: strip trailing / from ListDir()
|
|
.IP \[bu] 2
|
|
swift: return directories without / in ListDir
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.04 \- 2014\-07\-21
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
google cloud storage: Fix crash on Update
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.03 \- 2014\-07\-20
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
swift, s3, dropbox: fix updated files being marked as corrupted
|
|
.IP \[bu] 2
|
|
Make compile with go 1.1 again
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.02 \- 2014\-07\-19
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement Dropbox remote
|
|
.IP \[bu] 2
|
|
Implement Google Cloud Storage remote
|
|
.IP \[bu] 2
|
|
Verify Md5sums and Sizes after copies
|
|
.IP \[bu] 2
|
|
Remove times from "ls" command \- lists sizes only
|
|
.IP \[bu] 2
|
|
Add add "lsl" \- lists times and sizes
|
|
.IP \[bu] 2
|
|
Add "md5sum" command
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.01 \- 2014\-07\-04
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: fix transfer of big files using up lots of memory
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.00 \- 2014\-07\-03
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: fix whole second dates
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.99 \- 2014\-06\-26
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \-\-dry\-run not working
|
|
.IP \[bu] 2
|
|
Make compatible with go 1.1
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.98 \- 2014\-05\-30
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
s3: Treat missing Content\-Length as 0 for some ceph installations
|
|
.IP \[bu] 2
|
|
rclonetest: add file with a space in
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.97 \- 2014\-05\-05
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement copying of single files
|
|
.IP \[bu] 2
|
|
s3 & swift: support paths inside containers/buckets
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.96 \- 2014\-04\-24
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: Fix multiple files of same name being created
|
|
.IP \[bu] 2
|
|
drive: Use o.Update and fs.Put to optimise transfers
|
|
.IP \[bu] 2
|
|
Add version number, \-V and \-\-version
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.95 \- 2014\-03\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone.org: website, docs and graphics
|
|
.IP \[bu] 2
|
|
drive: fix path parsing
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.94 \- 2014\-03\-27
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Change remote format one last time
|
|
.IP \[bu] 2
|
|
GNU style flags
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.93 \- 2014\-03\-16
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: store token in config file
|
|
.IP \[bu] 2
|
|
cross compile other versions
|
|
.IP \[bu] 2
|
|
set strict permissions on config file
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.92 \- 2014\-03\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Config fixes and \-\-config option
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.91 \- 2014\-03\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make config file
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.90 \- 2013\-06\-27
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project named rclone
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.00 \- 2012\-11\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project started
|
|
.RE
|
|
.SS Bugs and Limitations
|
|
.SS Empty directories are left behind / not created
|
|
.PP
|
|
With remotes that have a concept of directory, eg Local and Drive, empty
|
|
directories may be left behind, or not created when one was expected.
|
|
.PP
|
|
This is because rclone doesn\[aq]t have a concept of a directory \- it
|
|
only works on objects.
|
|
Most of the object storage systems can\[aq]t actually store a directory
|
|
so there is nowhere for rclone to store anything about directories.
|
|
.PP
|
|
You can work round this to some extent with the\f[C]purge\f[] command
|
|
which will delete everything under the path, \f[B]inluding\f[] empty
|
|
directories.
|
|
.PP
|
|
This may be fixed at some point in Issue
|
|
#100 (https://github.com/ncw/rclone/issues/100)
|
|
.SS Directory timestamps aren\[aq]t preserved
|
|
.PP
|
|
For the same reason as the above, rclone doesn\[aq]t have a concept of a
|
|
directory \- it only works on objects, therefore it can\[aq]t preserve
|
|
the timestamps of directories.
|
|
.SS Frequently Asked Questions
|
|
.SS Do all cloud storage systems support all rclone commands
|
|
.PP
|
|
Yes they do.
|
|
All the rclone commands (eg \f[C]sync\f[], \f[C]copy\f[] etc) will work
|
|
on all the remote storage systems.
|
|
.SS Can I copy the config from one machine to another
|
|
.PP
|
|
Sure! Rclone stores all of its config in a single file.
|
|
If you want to find this file, the simplest way is to run
|
|
\f[C]rclone\ \-h\f[] and look at the help for the \f[C]\-\-config\f[]
|
|
flag which will tell you where it is.
|
|
.PP
|
|
See the remote setup docs (http://rclone.org/remote_setup/) for more
|
|
info.
|
|
.SS How do I configure rclone on a remote / headless box with no
|
|
browser?
|
|
.PP
|
|
This has now been documented in its own remote setup
|
|
page (http://rclone.org/remote_setup/).
|
|
.SS Can rclone sync directly from drive to s3
|
|
.PP
|
|
Rclone can sync between two remote cloud storage systems just fine.
|
|
.PP
|
|
Note that it effectively downloads the file and uploads it again, so the
|
|
node running rclone would need to have lots of bandwidth.
|
|
.PP
|
|
The syncs would be incremental (on a file by file basis).
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ drive:Folder\ s3:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Using rclone from multiple locations at the same time
|
|
.PP
|
|
You can use rclone from multiple places at the same time if you choose
|
|
different subdirectory for the output, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Server\ A>\ rclone\ sync\ /tmp/whatever\ remote:ServerA
|
|
Server\ B>\ rclone\ sync\ /tmp/whatever\ remote:ServerB
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you sync to the same directory then you should use rclone copy
|
|
otherwise the two rclones may delete each others files, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Server\ A>\ rclone\ copy\ /tmp/whatever\ remote:Backup
|
|
Server\ B>\ rclone\ copy\ /tmp/whatever\ remote:Backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The file names you upload from Server A and Server B should be different
|
|
in this case, otherwise some file systems (eg Drive) may make
|
|
duplicates.
|
|
.SS Why doesn\[aq]t rclone support partial transfers / binary diffs like
|
|
rsync?
|
|
.PP
|
|
Rclone stores each file you transfer as a native object on the remote
|
|
cloud storage system.
|
|
This means that you can see the files you upload as expected using
|
|
alternative access methods (eg using the Google Drive web interface).
|
|
There is a 1:1 mapping between files on your hard disk and objects
|
|
created in the cloud storage system.
|
|
.PP
|
|
Cloud storage systems (at least none I\[aq]ve come across yet) don\[aq]t
|
|
support partially uploading an object.
|
|
You can\[aq]t take an existing object, and change some bytes in the
|
|
middle of it.
|
|
.PP
|
|
It would be possible to make a sync system which stored binary diffs
|
|
instead of whole objects like rclone does, but that would break the 1:1
|
|
mapping of files on your hard disk to objects in the remote cloud
|
|
storage system.
|
|
.PP
|
|
All the cloud storage systems support partial downloads of content, so
|
|
it would be possible to make partial downloads work.
|
|
However to make this work efficiently this would require storing a
|
|
significant amount of metadata, which breaks the desired 1:1 mapping of
|
|
files to objects.
|
|
.SS Can rclone do bi\-directional sync?
|
|
.PP
|
|
No, not at present.
|
|
rclone only does uni\-directional sync from A \-> B.
|
|
It may do in the future though since it has all the primitives \- it
|
|
just requires writing the algorithm to do it.
|
|
.SS Can I use rclone with an HTTP proxy?
|
|
.PP
|
|
Yes.
|
|
rclone will use the environment variables \f[C]HTTP_PROXY\f[],
|
|
\f[C]HTTPS_PROXY\f[] and \f[C]NO_PROXY\f[], similar to cURL and other
|
|
programs.
|
|
.PP
|
|
\f[C]HTTPS_PROXY\f[] takes precedence over \f[C]HTTP_PROXY\f[] for https
|
|
requests.
|
|
.PP
|
|
The environment values may be either a complete URL or a "host[:port]",
|
|
in which case the "http" scheme is assumed.
|
|
.PP
|
|
The \f[C]NO_PROXY\f[] allows you to disable the proxy for specific
|
|
hosts.
|
|
Hosts must be comma separated, and can contain domains or parts.
|
|
For instance "foo.com" also matches "bar.foo.com".
|
|
.SS Rclone gives x509: failed to load system roots and no roots provided
|
|
error
|
|
.PP
|
|
This means that \f[C]rclone\f[] can\[aq]t file the SSL root
|
|
certificates.
|
|
Likely you are running \f[C]rclone\f[] on a NAS with a cut\-down Linux
|
|
OS, or possibly on Solaris.
|
|
.PP
|
|
Rclone (via the Go runtime) tries to load the root certificates from
|
|
these places on Linux.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
"/etc/ssl/certs/ca\-certificates.crt",\ //\ Debian/Ubuntu/Gentoo\ etc.
|
|
"/etc/pki/tls/certs/ca\-bundle.crt",\ \ \ //\ Fedora/RHEL
|
|
"/etc/ssl/ca\-bundle.pem",\ \ \ \ \ \ \ \ \ \ \ \ \ //\ OpenSUSE
|
|
"/etc/pki/tls/cacert.pem",\ \ \ \ \ \ \ \ \ \ \ \ //\ OpenELEC
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
So doing something like this should fix the problem.
|
|
It also sets the time which is important for SSL to work properly.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
mkdir\ \-p\ /etc/ssl/certs/
|
|
curl\ \-o\ /etc/ssl/certs/ca\-certificates.crt\ https://raw.githubusercontent.com/bagder/ca\-bundle/master/ca\-bundle.crt
|
|
ntpclient\ \-s\ \-h\ pool.ntp.org
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that you may need to add the \f[C]\-\-insecure\f[] option to the
|
|
\f[C]curl\f[] command line if it doesn\[aq]t work without.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-\-insecure\ \-o\ /etc/ssl/certs/ca\-certificates.crt\ https://raw.githubusercontent.com/bagder/ca\-bundle/master/ca\-bundle.crt
|
|
\f[]
|
|
.fi
|
|
.SS Rclone gives Failed to load config file: function not implemented
|
|
error
|
|
.PP
|
|
Likely this means that you are running rclone on Linux version not
|
|
supported by the go runtime, ie earlier than version 2.6.23.
|
|
.PP
|
|
See the system requirements section in the go install
|
|
docs (https://golang.org/doc/install) for full details.
|
|
.SS All my uploaded docx/xlsx/pptx files appear as archive/zip
|
|
.PP
|
|
This is caused by uploading these files from a Windows computer which
|
|
hasn\[aq]t got the Microsoft Office suite installed.
|
|
The easiest way to fix is to install the Word viewer and the Microsoft
|
|
Office Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
|
|
versions\[aq] file formats
|
|
.SS License
|
|
.PP
|
|
This is free software under the terms of MIT the license (check the
|
|
COPYING file included with the source code).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Copyright\ (C)\ 2012\ by\ Nick\ Craig\-Wood\ http://www.craig\-wood.com/nick/
|
|
|
|
Permission\ is\ hereby\ granted,\ free\ of\ charge,\ to\ any\ person\ obtaining\ a\ copy
|
|
of\ this\ software\ and\ associated\ documentation\ files\ (the\ "Software"),\ to\ deal
|
|
in\ the\ Software\ without\ restriction,\ including\ without\ limitation\ the\ rights
|
|
to\ use,\ copy,\ modify,\ merge,\ publish,\ distribute,\ sublicense,\ and/or\ sell
|
|
copies\ of\ the\ Software,\ and\ to\ permit\ persons\ to\ whom\ the\ Software\ is
|
|
furnished\ to\ do\ so,\ subject\ to\ the\ following\ conditions:
|
|
|
|
The\ above\ copyright\ notice\ and\ this\ permission\ notice\ shall\ be\ included\ in
|
|
all\ copies\ or\ substantial\ portions\ of\ the\ Software.
|
|
|
|
THE\ SOFTWARE\ IS\ PROVIDED\ "AS\ IS",\ WITHOUT\ WARRANTY\ OF\ ANY\ KIND,\ EXPRESS\ OR
|
|
IMPLIED,\ INCLUDING\ BUT\ NOT\ LIMITED\ TO\ THE\ WARRANTIES\ OF\ MERCHANTABILITY,
|
|
FITNESS\ FOR\ A\ PARTICULAR\ PURPOSE\ AND\ NONINFRINGEMENT.\ IN\ NO\ EVENT\ SHALL\ THE
|
|
AUTHORS\ OR\ COPYRIGHT\ HOLDERS\ BE\ LIABLE\ FOR\ ANY\ CLAIM,\ DAMAGES\ OR\ OTHER
|
|
LIABILITY,\ WHETHER\ IN\ AN\ ACTION\ OF\ CONTRACT,\ TORT\ OR\ OTHERWISE,\ ARISING\ FROM,
|
|
OUT\ OF\ OR\ IN\ CONNECTION\ WITH\ THE\ SOFTWARE\ OR\ THE\ USE\ OR\ OTHER\ DEALINGS\ IN
|
|
THE\ SOFTWARE.
|
|
\f[]
|
|
.fi
|
|
.SS Authors
|
|
.IP \[bu] 2
|
|
Nick Craig\-Wood <nick@craig-wood.com>
|
|
.SS Contributors
|
|
.IP \[bu] 2
|
|
Alex Couper <amcouper@gmail.com>
|
|
.IP \[bu] 2
|
|
Leonid Shalupov <leonid@shalupov.com>
|
|
.IP \[bu] 2
|
|
Shimon Doodkin <helpmepro1@gmail.com>
|
|
.IP \[bu] 2
|
|
Colin Nicholson <colin@colinn.com>
|
|
.IP \[bu] 2
|
|
Klaus Post <klauspost@gmail.com>
|
|
.IP \[bu] 2
|
|
Sergey Tolmachev <tolsi.ru@gmail.com>
|
|
.IP \[bu] 2
|
|
Adriano Aurélio Meirelles <adriano@atinge.com>
|
|
.IP \[bu] 2
|
|
C.
|
|
Bess <cbess@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dmitry Burdeev <dibu28@gmail.com>
|
|
.IP \[bu] 2
|
|
Joseph Spurrier <github@josephspurrier.com>
|
|
.IP \[bu] 2
|
|
Björn Harrtell <bjorn@wololo.org>
|
|
.IP \[bu] 2
|
|
Xavier Lucas <xavier.lucas@corp.ovh.com>
|
|
.IP \[bu] 2
|
|
Werner Beroux <werner@beroux.com>
|
|
.IP \[bu] 2
|
|
Brian Stengaard <brian@stengaard.eu>
|
|
.IP \[bu] 2
|
|
Jakub Gedeon <jgedeon@sofi.com>
|
|
.IP \[bu] 2
|
|
Jim Tittsler <jwt@onjapan.net>
|
|
.IP \[bu] 2
|
|
Michal Witkowski <michal@improbable.io>
|
|
.IP \[bu] 2
|
|
Fabian Ruff <fabian.ruff@sap.com>
|
|
.IP \[bu] 2
|
|
Leigh Klotz <klotz@quixey.com>
|
|
.IP \[bu] 2
|
|
Romain Lapray <lapray.romain@gmail.com>
|
|
.IP \[bu] 2
|
|
Justin R.
|
|
Wilson <jrw972@gmail.com>
|
|
.IP \[bu] 2
|
|
Antonio Messina <antonio.s.messina@gmail.com>
|
|
.IP \[bu] 2
|
|
Stefan G.
|
|
Weichinger <office@oops.co.at>
|
|
.IP \[bu] 2
|
|
Per Cederberg <cederberg@gmail.com>
|
|
.IP \[bu] 2
|
|
Radek Šenfeld <rush@logic.cz>
|
|
.IP \[bu] 2
|
|
Fredrik Fornwall <fredrik@fornwall.net>
|
|
.IP \[bu] 2
|
|
Asko Tamm <asko@deekit.net>
|
|
.IP \[bu] 2
|
|
xor\-zz <xor@gstocco.com>
|
|
.IP \[bu] 2
|
|
Tomasz Mazur <tmazur90@gmail.com>
|
|
.IP \[bu] 2
|
|
Marco Paganini <paganini@paganini.net>
|
|
.IP \[bu] 2
|
|
Felix Bünemann <buenemann@louis.info>
|
|
.IP \[bu] 2
|
|
Durval Menezes <jmrclone@durval.com>
|
|
.IP \[bu] 2
|
|
Luiz Carlos Rumbelsperger Viana <maxd13_luiz_carlos@hotmail.com>
|
|
.IP \[bu] 2
|
|
Stefan Breunig <stefan-github@yrden.de>
|
|
.IP \[bu] 2
|
|
Alishan Ladhani <ali-l@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
0xJAKE <0xJAKE@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Thibault Molleman <thibaultmol@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Scott McGillivray <scott.mcgillivray@gmail.com>
|
|
.SH Contact the rclone project
|
|
.SS Forum
|
|
.PP
|
|
Forum for general discussions and questions:
|
|
.IP \[bu] 2
|
|
https://forum.rclone.org
|
|
.SS Gitub project
|
|
.PP
|
|
The project website is at:
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone
|
|
.PP
|
|
There you can file bug reports, ask for help or contribute pull
|
|
requests.
|
|
.SS Google+
|
|
.PP
|
|
Rclone has a Google+ page which announcements are posted to
|
|
.IP \[bu] 2
|
|
Google+ page for general comments
|
|
.SS Twitter
|
|
.PP
|
|
You can also follow me on twitter for rclone announcments
|
|
.IP \[bu] 2
|
|
[\@njcw](https://twitter.com/njcw)
|
|
.SS Email
|
|
.PP
|
|
Or if all else fails or you want to ask something private or
|
|
confidential email Nick Craig\-Wood (mailto:nick@craig-wood.com)
|
|
.SH AUTHORS
|
|
Nick Craig\-Wood.
|