16739 lines
455 KiB
Groff
16739 lines
455 KiB
Groff
.\"t
|
|
.\" Automatically generated by Pandoc 1.19.2.1
|
|
.\"
|
|
.TH "rclone" "1" "Jun 16, 2018" "User Manual" ""
|
|
.hy
|
|
.SH Rclone
|
|
.PP
|
|
[IMAGE: Logo (https://rclone.org/img/rclone-120x120.png)] (https://rclone.org/)
|
|
.PP
|
|
Rclone is a command line program to sync files and directories to and
|
|
from:
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Amazon S3
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Box
|
|
.IP \[bu] 2
|
|
Ceph
|
|
.IP \[bu] 2
|
|
DigitalOcean Spaces
|
|
.IP \[bu] 2
|
|
Dreamhost
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
FTP
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.IP \[bu] 2
|
|
IBM COS S3
|
|
.IP \[bu] 2
|
|
Memset Memstore
|
|
.IP \[bu] 2
|
|
Mega
|
|
.IP \[bu] 2
|
|
Microsoft Azure Blob Storage
|
|
.IP \[bu] 2
|
|
Microsoft OneDrive
|
|
.IP \[bu] 2
|
|
Minio
|
|
.IP \[bu] 2
|
|
Nextcloud
|
|
.IP \[bu] 2
|
|
OVH
|
|
.IP \[bu] 2
|
|
OpenDrive
|
|
.IP \[bu] 2
|
|
Openstack Swift
|
|
.IP \[bu] 2
|
|
Oracle Cloud Storage
|
|
.IP \[bu] 2
|
|
ownCloud
|
|
.IP \[bu] 2
|
|
pCloud
|
|
.IP \[bu] 2
|
|
put.io
|
|
.IP \[bu] 2
|
|
QingStor
|
|
.IP \[bu] 2
|
|
Rackspace Cloud Files
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Wasabi
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.IP \[bu] 2
|
|
Yandex Disk
|
|
.IP \[bu] 2
|
|
The local filesystem
|
|
.PP
|
|
Features
|
|
.IP \[bu] 2
|
|
MD5/SHA1 hashes checked at all times for file integrity
|
|
.IP \[bu] 2
|
|
Timestamps preserved on files
|
|
.IP \[bu] 2
|
|
Partial syncs supported on a whole file basis
|
|
.IP \[bu] 2
|
|
Copy (https://rclone.org/commands/rclone_copy/) mode to just copy
|
|
new/changed files
|
|
.IP \[bu] 2
|
|
Sync (https://rclone.org/commands/rclone_sync/) (one way) mode to make a
|
|
directory identical
|
|
.IP \[bu] 2
|
|
Check (https://rclone.org/commands/rclone_check/) mode to check for file
|
|
hash equality
|
|
.IP \[bu] 2
|
|
Can sync to and from network, eg two different cloud accounts
|
|
.IP \[bu] 2
|
|
Optional encryption (Crypt (https://rclone.org/crypt/))
|
|
.IP \[bu] 2
|
|
Optional cache (Cache (https://rclone.org/cache/))
|
|
.IP \[bu] 2
|
|
Optional FUSE mount (rclone
|
|
mount (https://rclone.org/commands/rclone_mount/))
|
|
.PP
|
|
Links
|
|
.IP \[bu] 2
|
|
Home page (https://rclone.org/)
|
|
.IP \[bu] 2
|
|
Github project page for source and bug
|
|
tracker (https://github.com/ncw/rclone)
|
|
.IP \[bu] 2
|
|
Rclone Forum (https://forum.rclone.org)
|
|
.IP \[bu] 2
|
|
Google+ page
|
|
.IP \[bu] 2
|
|
Downloads (https://rclone.org/downloads/)
|
|
.SH Install
|
|
.PP
|
|
Rclone is a Go program and comes as a single binary file.
|
|
.SS Quickstart
|
|
.IP \[bu] 2
|
|
Download (https://rclone.org/downloads/) the relevant binary.
|
|
.IP \[bu] 2
|
|
Unpack and the \f[C]rclone\f[] binary.
|
|
.IP \[bu] 2
|
|
Run \f[C]rclone\ config\f[] to setup.
|
|
See rclone config docs (https://rclone.org/docs/) for more details.
|
|
.PP
|
|
See below for some expanded Linux / macOS instructions.
|
|
.PP
|
|
See the Usage section (https://rclone.org/docs/) of the docs for how to
|
|
use rclone, or run \f[C]rclone\ \-h\f[].
|
|
.SS Script installation
|
|
.PP
|
|
To install rclone on Linux/macOS/BSD systems, run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ https://rclone.org/install.sh\ |\ sudo\ bash
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
For beta installation, run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ https://rclone.org/install.sh\ |\ sudo\ bash\ \-s\ beta
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that this script checks the version of rclone installed first and
|
|
won\[aq]t re\-download if not needed.
|
|
.SS Linux installation from precompiled binary
|
|
.PP
|
|
Fetch and unpack
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-O\ https://downloads.rclone.org/rclone\-current\-linux\-amd64.zip
|
|
unzip\ rclone\-current\-linux\-amd64.zip
|
|
cd\ rclone\-*\-linux\-amd64
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Copy binary file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ cp\ rclone\ /usr/bin/
|
|
sudo\ chown\ root:root\ /usr/bin/rclone
|
|
sudo\ chmod\ 755\ /usr/bin/rclone
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Install manpage
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ mkdir\ \-p\ /usr/local/share/man/man1
|
|
sudo\ cp\ rclone.1\ /usr/local/share/man/man1/
|
|
sudo\ mandb\
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone\ config\f[] to setup.
|
|
See rclone config docs (https://rclone.org/docs/) for more details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.SS macOS installation from precompiled binary
|
|
.PP
|
|
Download the latest version of rclone.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cd\ &&\ curl\ \-O\ https://downloads.rclone.org/rclone\-current\-osx\-amd64.zip
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Unzip the download and cd to the extracted folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
unzip\ \-a\ rclone\-current\-osx\-amd64.zip\ &&\ cd\ rclone\-*\-osx\-amd64
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Move rclone to your $PATH.
|
|
You will be prompted for your password.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ mkdir\ \-p\ /usr/local/bin
|
|
sudo\ mv\ rclone\ /usr/local/bin/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
(the \f[C]mkdir\f[] command is safe to run, even if the directory
|
|
already exists).
|
|
.PP
|
|
Remove the leftover files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
cd\ ..\ &&\ rm\ \-rf\ rclone\-*\-osx\-amd64\ rclone\-current\-osx\-amd64.zip
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone\ config\f[] to setup.
|
|
See rclone config docs (https://rclone.org/docs/) for more details.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.SS Install from source
|
|
.PP
|
|
Make sure you have at least Go (https://golang.org/) 1.6 installed.
|
|
Make sure your \f[C]GOPATH\f[] is set, then:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go\ get\ \-u\ \-v\ github.com/ncw/rclone
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
and this will build the binary in \f[C]$GOPATH/bin\f[].
|
|
If you have built rclone before then you will want to update its
|
|
dependencies first with this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go\ get\ \-u\ \-v\ github.com/ncw/rclone/...
|
|
\f[]
|
|
.fi
|
|
.SS Installation with Ansible
|
|
.PP
|
|
This can be done with Stefan Weichinger\[aq]s ansible
|
|
role (https://github.com/stefangweichinger/ansible-rclone).
|
|
.PP
|
|
Instructions
|
|
.IP "1." 3
|
|
\f[C]git\ clone\ https://github.com/stefangweichinger/ansible\-rclone.git\f[]
|
|
into your local roles\-directory
|
|
.IP "2." 3
|
|
add the role to the hosts you want rclone installed to:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \-\ hosts:\ rclone\-hosts
|
|
\ \ \ \ \ \ roles:
|
|
\ \ \ \ \ \ \ \ \ \ \-\ rclone
|
|
\f[]
|
|
.fi
|
|
.SS Configure
|
|
.PP
|
|
First, you\[aq]ll need to configure rclone.
|
|
As the object storage systems have quite complicated authentication
|
|
these are kept in a config file.
|
|
(See the \f[C]\-\-config\f[] entry for how to find the config file and
|
|
choose its location.)
|
|
.PP
|
|
The easiest way to make the config is to run rclone with the config
|
|
option:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the following for detailed instructions for
|
|
.IP \[bu] 2
|
|
Alias (https://rclone.org/alias/)
|
|
.IP \[bu] 2
|
|
Amazon Drive (https://rclone.org/amazonclouddrive/)
|
|
.IP \[bu] 2
|
|
Amazon S3 (https://rclone.org/s3/)
|
|
.IP \[bu] 2
|
|
Backblaze B2 (https://rclone.org/b2/)
|
|
.IP \[bu] 2
|
|
Box (https://rclone.org/box/)
|
|
.IP \[bu] 2
|
|
Cache (https://rclone.org/cache/)
|
|
.IP \[bu] 2
|
|
Crypt (https://rclone.org/crypt/) \- to encrypt other remotes
|
|
.IP \[bu] 2
|
|
DigitalOcean Spaces (/s3/#digitalocean-spaces)
|
|
.IP \[bu] 2
|
|
Dropbox (https://rclone.org/dropbox/)
|
|
.IP \[bu] 2
|
|
FTP (https://rclone.org/ftp/)
|
|
.IP \[bu] 2
|
|
Google Cloud Storage (https://rclone.org/googlecloudstorage/)
|
|
.IP \[bu] 2
|
|
Google Drive (https://rclone.org/drive/)
|
|
.IP \[bu] 2
|
|
HTTP (https://rclone.org/http/)
|
|
.IP \[bu] 2
|
|
Hubic (https://rclone.org/hubic/)
|
|
.IP \[bu] 2
|
|
Mega (https://rclone.org/mega/)
|
|
.IP \[bu] 2
|
|
Microsoft Azure Blob Storage (https://rclone.org/azureblob/)
|
|
.IP \[bu] 2
|
|
Microsoft OneDrive (https://rclone.org/onedrive/)
|
|
.IP \[bu] 2
|
|
Openstack Swift / Rackspace Cloudfiles / Memset
|
|
Memstore (https://rclone.org/swift/)
|
|
.IP \[bu] 2
|
|
OpenDrive (https://rclone.org/opendrive/)
|
|
.IP \[bu] 2
|
|
Pcloud (https://rclone.org/pcloud/)
|
|
.IP \[bu] 2
|
|
QingStor (https://rclone.org/qingstor/)
|
|
.IP \[bu] 2
|
|
SFTP (https://rclone.org/sftp/)
|
|
.IP \[bu] 2
|
|
WebDAV (https://rclone.org/webdav/)
|
|
.IP \[bu] 2
|
|
Yandex Disk (https://rclone.org/yandex/)
|
|
.IP \[bu] 2
|
|
The local filesystem (https://rclone.org/local/)
|
|
.SS Usage
|
|
.PP
|
|
Rclone syncs a directory tree from one storage system to another.
|
|
.PP
|
|
Its syntax is like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Syntax:\ [options]\ subcommand\ <parameters>\ <parameters...>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Source and destination paths are specified by the name you gave the
|
|
storage system in the config file then the sub path, eg "drive:myfolder"
|
|
to look at "myfolder" in Google drive.
|
|
.PP
|
|
You can define as many storage paths as you like in the config file.
|
|
.SS Subcommands
|
|
.PP
|
|
rclone uses a system of subcommands.
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:path\ #\ lists\ a\ re
|
|
rclone\ copy\ /local/path\ remote:path\ #\ copies\ /local/path\ to\ the\ remote
|
|
rclone\ sync\ /local/path\ remote:path\ #\ syncs\ /local/path\ to\ the\ remote
|
|
\f[]
|
|
.fi
|
|
.SS rclone config
|
|
.PP
|
|
Enter an interactive configuration session.
|
|
.SS Synopsis
|
|
.PP
|
|
Enter an interactive configuration session where you can setup new
|
|
remotes and manage existing ones.
|
|
You may also set or remove a password to protect your configuration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ config
|
|
\f[]
|
|
.fi
|
|
.SS rclone copy
|
|
.PP
|
|
Copy files from source to dest, skipping already copied
|
|
.SS Synopsis
|
|
.PP
|
|
Copy the source to the destination.
|
|
Doesn\[aq]t transfer unchanged files, testing by size and modification
|
|
time or MD5SUM.
|
|
Doesn\[aq]t delete files from the destination.
|
|
.PP
|
|
Note that it is always the contents of the directory that is synced, not
|
|
the directory so when source:path is a directory, it\[aq]s the contents
|
|
of source:path that are copied, not the directory name and contents.
|
|
.PP
|
|
If dest:path doesn\[aq]t exist, it is created and the source:path
|
|
contents go there.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ source:sourcepath\ dest:destpath
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Let\[aq]s say there are two files in sourcepath
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sourcepath/one.txt
|
|
sourcepath/two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This copies them to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
destpath/one.txt
|
|
destpath/two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Not to
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
destpath/sourcepath/one.txt
|
|
destpath/sourcepath/two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you are familiar with \f[C]rsync\f[], rclone always works as if you
|
|
had written a trailing / \- meaning "copy the contents of this
|
|
directory".
|
|
This applies to all commands and whether you are talking about the
|
|
source or destination.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ source:path\ dest:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ copy
|
|
\f[]
|
|
.fi
|
|
.SS rclone sync
|
|
.PP
|
|
Make source and dest identical, modifying destination only.
|
|
.SS Synopsis
|
|
.PP
|
|
Sync the source to the destination, changing the destination only.
|
|
Doesn\[aq]t transfer unchanged files, testing by size and modification
|
|
time or MD5SUM.
|
|
Destination is updated to match source, including deleting files if
|
|
necessary.
|
|
.PP
|
|
\f[B]Important\f[]: Since this can cause data loss, test first with the
|
|
\f[C]\-\-dry\-run\f[] flag to see exactly what would be copied and
|
|
deleted.
|
|
.PP
|
|
Note that files in the destination won\[aq]t be deleted if there were
|
|
any errors at any point.
|
|
.PP
|
|
It is always the contents of the directory that is synced, not the
|
|
directory so when source:path is a directory, it\[aq]s the contents of
|
|
source:path that are copied, not the directory name and contents.
|
|
See extended explanation in the \f[C]copy\f[] command above if unsure.
|
|
.PP
|
|
If dest:path doesn\[aq]t exist, it is created and the source:path
|
|
contents go there.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ source:path\ dest:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ sync
|
|
\f[]
|
|
.fi
|
|
.SS rclone move
|
|
.PP
|
|
Move files from source to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
Moves the contents of the source directory to the destination directory.
|
|
Rclone will error if the source and destination overlap and the remote
|
|
does not support a server side directory move operation.
|
|
.PP
|
|
If no filters are in use and if possible this will server side move
|
|
\f[C]source:path\f[] into \f[C]dest:path\f[].
|
|
After this \f[C]source:path\f[] will no longer longer exist.
|
|
.PP
|
|
Otherwise for each file in \f[C]source:path\f[] selected by the filters
|
|
(if any) this will move it into \f[C]dest:path\f[].
|
|
If possible a server side move will be used, otherwise it will copy it
|
|
(server side if possible) into \f[C]dest:path\f[] then delete the
|
|
original (if no errors on copy) in \f[C]source:path\f[].
|
|
.PP
|
|
If you want to delete empty source directories after move, use the
|
|
\-\-delete\-empty\-src\-dirs flag.
|
|
.PP
|
|
\f[B]Important\f[]: Since this can cause data loss, test first with the
|
|
\-\-dry\-run flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ move\ source:path\ dest:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-delete\-empty\-src\-dirs\ \ \ Delete\ empty\ source\ dirs\ after\ move
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ move
|
|
\f[]
|
|
.fi
|
|
.SS rclone delete
|
|
.PP
|
|
Remove the contents of path.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the contents of path.
|
|
Unlike \f[C]purge\f[] it obeys include/exclude filters so can be used to
|
|
selectively delete files.
|
|
.PP
|
|
Eg delete all files bigger than 100MBytes
|
|
.PP
|
|
Check what would be deleted first (use either)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-min\-size\ 100M\ lsl\ remote:path
|
|
rclone\ \-\-dry\-run\ \-\-min\-size\ 100M\ delete\ remote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then delete
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-min\-size\ 100M\ delete\ remote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
That reads "delete everything with a minimum size of 100 MB", hence
|
|
delete all files bigger than 100MBytes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ delete\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ delete
|
|
\f[]
|
|
.fi
|
|
.SS rclone purge
|
|
.PP
|
|
Remove the path and all of its contents.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the path and all of its contents.
|
|
Note that this does not obey include/exclude filters \- everything will
|
|
be removed.
|
|
Use \f[C]delete\f[] if you want to selectively delete files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ purge\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ purge
|
|
\f[]
|
|
.fi
|
|
.SS rclone mkdir
|
|
.PP
|
|
Make the path if it doesn\[aq]t already exist.
|
|
.SS Synopsis
|
|
.PP
|
|
Make the path if it doesn\[aq]t already exist.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ mkdir
|
|
\f[]
|
|
.fi
|
|
.SS rclone rmdir
|
|
.PP
|
|
Remove the path if empty.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove the path.
|
|
Note that you can\[aq]t remove a path with objects in it, use purge for
|
|
that.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rmdir\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ rmdir
|
|
\f[]
|
|
.fi
|
|
.SS rclone check
|
|
.PP
|
|
Checks the files in the source and destination match.
|
|
.SS Synopsis
|
|
.PP
|
|
Checks the files in the source and destination match.
|
|
It compares sizes and hashes (MD5 or SHA1) and logs a report of files
|
|
which don\[aq]t match.
|
|
It doesn\[aq]t alter the source or destination.
|
|
.PP
|
|
If you supply the \-\-size\-only flag, it will only compare the sizes
|
|
not the hashes as well.
|
|
Use this for a quick check.
|
|
.PP
|
|
If you supply the \-\-download flag, it will download the data from both
|
|
remotes and check them against each other on the fly.
|
|
This can be useful for remotes that don\[aq]t support hashes or if you
|
|
really want to check all the data.
|
|
.PP
|
|
If you supply the \-\-one\-way flag, it will only check that files in
|
|
source match the files in destination, not the other way around.
|
|
Meaning extra files in destination that are not in the source will not
|
|
trigger an error.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ check\ source:path\ dest:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-download\ \ \ Check\ by\ downloading\ rather\ than\ with\ hash.
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ help\ for\ check
|
|
\ \ \ \ \ \ \-\-one\-way\ \ \ \ Check\ one\ way\ only,\ source\ files\ must\ exist\ on\ remote
|
|
\f[]
|
|
.fi
|
|
.SS rclone ls
|
|
.PP
|
|
List the objects in the path with size and path.
|
|
.SS Synopsis
|
|
.PP
|
|
Lists the objects in the source path to standard output in a human
|
|
readable format with size and path.
|
|
Recurses by default.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ ls\ swift:bucket
|
|
\ \ \ \ 60295\ bevajer5jef
|
|
\ \ \ \ 90613\ canole
|
|
\ \ \ \ 94467\ diwogej7
|
|
\ \ \ \ 37600\ fubuwic
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Any of the filtering options can be applied to this commmand.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[],\f[C]lsl\f[],\f[C]lsd\f[] are designed to be human readable.
|
|
\f[C]lsf\f[] is designed to be human and machine readable.
|
|
\f[C]lsjson\f[] is designed to be machine readable.
|
|
.PP
|
|
Note that \f[C]ls\f[] and \f[C]lsl\f[] recurse by default \- use
|
|
"\-\-max\-depth 1" to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[],\f[C]lsf\f[],\f[C]lsjson\f[] do not
|
|
recurse by default \- use "\-R" to make them recurse.
|
|
.PP
|
|
Listing a non existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (eg s3, swift, gcs, etc
|
|
\- the bucket based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ ls
|
|
\f[]
|
|
.fi
|
|
.SS rclone lsd
|
|
.PP
|
|
List all directories/containers/buckets in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Lists the directories in the source path to standard output.
|
|
Does not recurse by default.
|
|
Use the \-R flag to recurse.
|
|
.PP
|
|
This command lists the total size of the directory (if known, \-1 if
|
|
not), the modification time (if known, the current time if not), the
|
|
number of objects in the directory (if known, \-1 if not) and the name
|
|
of the directory, Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsd\ swift:
|
|
\ \ \ \ \ \ 494000\ 2018\-04\-26\ 08:43:20\ \ \ \ \ 10000\ 10000files
|
|
\ \ \ \ \ \ \ \ \ \ 65\ 2018\-04\-26\ 08:43:20\ \ \ \ \ \ \ \ \ 1\ 1File
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsd\ drive:test
|
|
\ \ \ \ \ \ \ \ \ \ \-1\ 2016\-10\-17\ 17:41:53\ \ \ \ \ \ \ \ \-1\ 1000files
|
|
\ \ \ \ \ \ \ \ \ \ \-1\ 2017\-01\-03\ 14:40:54\ \ \ \ \ \ \ \ \-1\ 2500files
|
|
\ \ \ \ \ \ \ \ \ \ \-1\ 2017\-07\-08\ 14:39:28\ \ \ \ \ \ \ \ \-1\ 4000files
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you just want the directory names use "rclone lsf \-\-dirs\-only".
|
|
.PP
|
|
Any of the filtering options can be applied to this commmand.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[],\f[C]lsl\f[],\f[C]lsd\f[] are designed to be human readable.
|
|
\f[C]lsf\f[] is designed to be human and machine readable.
|
|
\f[C]lsjson\f[] is designed to be machine readable.
|
|
.PP
|
|
Note that \f[C]ls\f[] and \f[C]lsl\f[] recurse by default \- use
|
|
"\-\-max\-depth 1" to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[],\f[C]lsf\f[],\f[C]lsjson\f[] do not
|
|
recurse by default \- use "\-R" to make them recurse.
|
|
.PP
|
|
Listing a non existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (eg s3, swift, gcs, etc
|
|
\- the bucket based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ help\ for\ lsd
|
|
\ \ \-R,\ \-\-recursive\ \ \ Recurse\ into\ the\ listing.
|
|
\f[]
|
|
.fi
|
|
.SS rclone lsl
|
|
.PP
|
|
List the objects in path with modification time, size and path.
|
|
.SS Synopsis
|
|
.PP
|
|
Lists the objects in the source path to standard output in a human
|
|
readable format with modification time, size and path.
|
|
Recurses by default.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsl\ swift:bucket
|
|
\ \ \ \ 60295\ 2016\-06\-25\ 18:55:41.062626927\ bevajer5jef
|
|
\ \ \ \ 90613\ 2016\-06\-25\ 18:55:43.302607074\ canole
|
|
\ \ \ \ 94467\ 2016\-06\-25\ 18:55:43.046609333\ diwogej7
|
|
\ \ \ \ 37600\ 2016\-06\-25\ 18:55:40.814629136\ fubuwic
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Any of the filtering options can be applied to this commmand.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[],\f[C]lsl\f[],\f[C]lsd\f[] are designed to be human readable.
|
|
\f[C]lsf\f[] is designed to be human and machine readable.
|
|
\f[C]lsjson\f[] is designed to be machine readable.
|
|
.PP
|
|
Note that \f[C]ls\f[] and \f[C]lsl\f[] recurse by default \- use
|
|
"\-\-max\-depth 1" to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[],\f[C]lsf\f[],\f[C]lsjson\f[] do not
|
|
recurse by default \- use "\-R" to make them recurse.
|
|
.PP
|
|
Listing a non existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (eg s3, swift, gcs, etc
|
|
\- the bucket based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsl\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ lsl
|
|
\f[]
|
|
.fi
|
|
.SS rclone md5sum
|
|
.PP
|
|
Produces an md5sum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces an md5sum file for all the objects in the path.
|
|
This is in the same format as the standard md5sum tool produces.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ md5sum\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ md5sum
|
|
\f[]
|
|
.fi
|
|
.SS rclone sha1sum
|
|
.PP
|
|
Produces an sha1sum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces an sha1sum file for all the objects in the path.
|
|
This is in the same format as the standard sha1sum tool produces.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sha1sum\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ sha1sum
|
|
\f[]
|
|
.fi
|
|
.SS rclone size
|
|
.PP
|
|
Prints the total size and number of objects in remote:path.
|
|
.SS Synopsis
|
|
.PP
|
|
Prints the total size and number of objects in remote:path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ size\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ size
|
|
\ \ \ \ \ \ \-\-json\ \ \ format\ output\ as\ JSON
|
|
\f[]
|
|
.fi
|
|
.SS rclone version
|
|
.PP
|
|
Show the version number.
|
|
.SS Synopsis
|
|
.PP
|
|
Show the version number.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ version\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ version
|
|
\f[]
|
|
.fi
|
|
.SS rclone cleanup
|
|
.PP
|
|
Clean up the remote if possible
|
|
.SS Synopsis
|
|
.PP
|
|
Clean up the remote if possible.
|
|
Empty the trash or delete old file versions.
|
|
Not supported by all remotes.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cleanup\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ cleanup
|
|
\f[]
|
|
.fi
|
|
.SS rclone dedupe
|
|
.PP
|
|
Interactively find duplicate files and delete/rename them.
|
|
.SS Synopsis
|
|
.PP
|
|
By default \f[C]dedupe\f[] interactively finds duplicate files and
|
|
offers to delete all but one or rename them to be different.
|
|
Only useful with Google Drive which can have duplicate file names.
|
|
.PP
|
|
In the first pass it will merge directories with the same name.
|
|
It will do this iteratively until all the identical directories have
|
|
been merged.
|
|
.PP
|
|
The \f[C]dedupe\f[] command will delete all but one of any identical
|
|
(same md5sum) files it finds without confirmation.
|
|
This means that for most duplicated files the \f[C]dedupe\f[] command
|
|
will not be interactive.
|
|
You can use \f[C]\-\-dry\-run\f[] to see what would happen without doing
|
|
anything.
|
|
.PP
|
|
Here is an example run.
|
|
.PP
|
|
Before \- with duplicates
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsl\ drive:dupes
|
|
\ \ 6048320\ 2016\-03\-05\ 16:23:16.798000000\ one.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:23:11.775000000\ one.txt
|
|
\ \ \ 564374\ 2016\-03\-05\ 16:23:06.731000000\ one.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:18:26.092000000\ one.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:22:46.185000000\ two.txt
|
|
\ \ 1744073\ 2016\-03\-05\ 16:22:38.104000000\ two.txt
|
|
\ \ \ 564374\ 2016\-03\-05\ 16:22:52.118000000\ two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Now the \f[C]dedupe\f[] session
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ dedupe\ drive:dupes
|
|
2016/03/05\ 16:24:37\ Google\ drive\ root\ \[aq]dupes\[aq]:\ Looking\ for\ duplicates\ using\ interactive\ mode.
|
|
one.txt:\ Found\ 4\ duplicates\ \-\ deleting\ identical\ copies
|
|
one.txt:\ Deleting\ 2/3\ identical\ duplicates\ (md5sum\ "1eedaa9fe86fd4b8632e2ac549403b36")
|
|
one.txt:\ 2\ duplicates\ remain
|
|
\ \ 1:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-03\-05\ 16:23:16.798000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
|
|
\ \ 2:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-03\-05\ 16:23:06.731000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
|
|
s)\ Skip\ and\ do\ nothing
|
|
k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
|
|
r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
|
|
s/k/r>\ k
|
|
Enter\ the\ number\ of\ the\ file\ to\ keep>\ 1
|
|
one.txt:\ Deleted\ 1\ extra\ copies
|
|
two.txt:\ Found\ 3\ duplicates\ \-\ deleting\ identical\ copies
|
|
two.txt:\ 3\ duplicates\ remain
|
|
\ \ 1:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-03\-05\ 16:22:52.118000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
|
|
\ \ 2:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-03\-05\ 16:22:46.185000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
|
|
\ \ 3:\ \ \ \ \ \ 1744073\ bytes,\ 2016\-03\-05\ 16:22:38.104000000,\ md5sum\ 851957f7fb6f0bc4ce76be966d336802
|
|
s)\ Skip\ and\ do\ nothing
|
|
k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
|
|
r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
|
|
s/k/r>\ r
|
|
two\-1.txt:\ renamed\ from:\ two.txt
|
|
two\-2.txt:\ renamed\ from:\ two.txt
|
|
two\-3.txt:\ renamed\ from:\ two.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The result being
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsl\ drive:dupes
|
|
\ \ 6048320\ 2016\-03\-05\ 16:23:16.798000000\ one.txt
|
|
\ \ \ 564374\ 2016\-03\-05\ 16:22:52.118000000\ two\-1.txt
|
|
\ \ 6048320\ 2016\-03\-05\ 16:22:46.185000000\ two\-2.txt
|
|
\ \ 1744073\ 2016\-03\-05\ 16:22:38.104000000\ two\-3.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Dedupe can be run non interactively using the \f[C]\-\-dedupe\-mode\f[]
|
|
flag or by using an extra parameter with the same value
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ interactive\f[] \- interactive as above.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ skip\f[] \- removes identical files then skips
|
|
anything left.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ first\f[] \- removes identical files then keeps
|
|
the first one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ newest\f[] \- removes identical files then keeps
|
|
the newest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ oldest\f[] \- removes identical files then keeps
|
|
the oldest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ largest\f[] \- removes identical files then keeps
|
|
the largest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ rename\f[] \- removes identical files then
|
|
renames the rest to be different.
|
|
.PP
|
|
For example to rename all the identically named photos in your Google
|
|
Photos directory, do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dedupe\ \-\-dedupe\-mode\ rename\ "drive:Google\ Photos"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dedupe\ rename\ "drive:Google\ Photos"
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dedupe\ [mode]\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-dedupe\-mode\ string\ \ \ Dedupe\ mode\ interactive|skip|first|newest|oldest|rename.\ (default\ "interactive")
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ dedupe
|
|
\f[]
|
|
.fi
|
|
.SS rclone about
|
|
.PP
|
|
Get quota information from the remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Get quota information from the remote, like bytes used/free/quota and
|
|
bytes used in the trash.
|
|
Not supported by all remotes.
|
|
.PP
|
|
This will print to stdout something like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Total:\ \ \ 17G
|
|
Used:\ \ \ \ 7.444G
|
|
Free:\ \ \ \ 1.315G
|
|
Trashed:\ 100.000M
|
|
Other:\ \ \ 8.241G
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Where the fields are:
|
|
.IP \[bu] 2
|
|
Total: total size available.
|
|
.IP \[bu] 2
|
|
Used: total size used
|
|
.IP \[bu] 2
|
|
Free: total amount this user could upload.
|
|
.IP \[bu] 2
|
|
Trashed: total amount in the trash
|
|
.IP \[bu] 2
|
|
Other: total amount in other storage (eg Gmail, Google Photos)
|
|
.IP \[bu] 2
|
|
Objects: total number of objects in the storage
|
|
.PP
|
|
Note that not all the backends provide all the fields \- they will be
|
|
missing if they are not known for that backend.
|
|
Where it is known that the value is unlimited the value will also be
|
|
omitted.
|
|
.PP
|
|
Use the \-\-full flag to see the numbers written out in full, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Total:\ \ \ 18253611008
|
|
Used:\ \ \ \ 7993453766
|
|
Free:\ \ \ \ 1411001220
|
|
Trashed:\ 104857602
|
|
Other:\ \ \ 8849156022
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Use the \-\-json flag for a computer readable output, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "total":\ 18253611008,
|
|
\ \ \ \ "used":\ 7993453766,
|
|
\ \ \ \ "trashed":\ 104857602,
|
|
\ \ \ \ "other":\ 8849156022,
|
|
\ \ \ \ "free":\ 1411001220
|
|
}
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ about\ remote:\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-full\ \ \ Full\ numbers\ instead\ of\ SI\ units
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ about
|
|
\ \ \ \ \ \ \-\-json\ \ \ Format\ output\ as\ JSON
|
|
\f[]
|
|
.fi
|
|
.SS rclone authorize
|
|
.PP
|
|
Remote authorization.
|
|
.SS Synopsis
|
|
.PP
|
|
Remote authorization.
|
|
Used to authorize a remote or headless rclone from a machine with a
|
|
browser \- use as instructed by rclone config.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ authorize\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ authorize
|
|
\f[]
|
|
.fi
|
|
.SS rclone cachestats
|
|
.PP
|
|
Print cache stats for a remote
|
|
.SS Synopsis
|
|
.PP
|
|
Print cache stats for a remote in JSON format
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cachestats\ source:\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ cachestats
|
|
\f[]
|
|
.fi
|
|
.SS rclone cat
|
|
.PP
|
|
Concatenates any files and sends them to stdout.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cat sends any files to standard output.
|
|
.PP
|
|
You can use it like this to output a single file
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cat\ remote:path/to/file
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or like this to output any file in dir or subdirectories.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cat\ remote:path/to/dir
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or like this to output any .txt files in dir or subdirectories.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-include\ "*.txt"\ cat\ remote:path/to/dir
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Use the \-\-head flag to print characters only at the start, \-\-tail
|
|
for the end and \-\-offset and \-\-count to print a section in the
|
|
middle.
|
|
Note that if offset is negative it will count from the end, so
|
|
\-\-offset \-1 \-\-count 1 is equivalent to \-\-tail 1.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cat\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-count\ int\ \ \ \ Only\ print\ N\ characters.\ (default\ \-1)
|
|
\ \ \ \ \ \ \-\-discard\ \ \ \ \ \ Discard\ the\ output\ instead\ of\ printing.
|
|
\ \ \ \ \ \ \-\-head\ int\ \ \ \ \ Only\ print\ the\ first\ N\ characters.
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ help\ for\ cat
|
|
\ \ \ \ \ \ \-\-offset\ int\ \ \ Start\ printing\ at\ offset\ N\ (or\ from\ end\ if\ \-ve).
|
|
\ \ \ \ \ \ \-\-tail\ int\ \ \ \ \ Only\ print\ the\ last\ N\ characters.
|
|
\f[]
|
|
.fi
|
|
.SS rclone config create
|
|
.PP
|
|
Create a new remote with name, type and options.
|
|
.SS Synopsis
|
|
.PP
|
|
Create a new remote of with and options.
|
|
The options should be passed in in pairs of .
|
|
.PP
|
|
For example to make a swift remote of name myremote using auto config
|
|
you would do:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ create\ myremote\ swift\ env_auth\ true
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ create\ <name>\ <type>\ [<key>\ <value>]*\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ create
|
|
\f[]
|
|
.fi
|
|
.SS rclone config delete
|
|
.PP
|
|
Delete an existing remote .
|
|
.SS Synopsis
|
|
.PP
|
|
Delete an existing remote .
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ delete\ <name>\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ delete
|
|
\f[]
|
|
.fi
|
|
.SS rclone config dump
|
|
.PP
|
|
Dump the config file as JSON.
|
|
.SS Synopsis
|
|
.PP
|
|
Dump the config file as JSON.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ dump\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ dump
|
|
\f[]
|
|
.fi
|
|
.SS rclone config edit
|
|
.PP
|
|
Enter an interactive configuration session.
|
|
.SS Synopsis
|
|
.PP
|
|
Enter an interactive configuration session where you can setup new
|
|
remotes and manage existing ones.
|
|
You may also set or remove a password to protect your configuration.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ edit\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ edit
|
|
\f[]
|
|
.fi
|
|
.SS rclone config file
|
|
.PP
|
|
Show path of configuration file in use.
|
|
.SS Synopsis
|
|
.PP
|
|
Show path of configuration file in use.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ file\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ file
|
|
\f[]
|
|
.fi
|
|
.SS rclone config password
|
|
.PP
|
|
Update password in an existing remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Update an existing remote\[aq]s password.
|
|
The password should be passed in in pairs of .
|
|
.PP
|
|
For example to set password of a remote of name myremote you would do:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ password\ myremote\ fieldname\ mypassword
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ password\ <name>\ [<key>\ <value>]+\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ password
|
|
\f[]
|
|
.fi
|
|
.SS rclone config providers
|
|
.PP
|
|
List in JSON format all the providers and options.
|
|
.SS Synopsis
|
|
.PP
|
|
List in JSON format all the providers and options.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ providers\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ providers
|
|
\f[]
|
|
.fi
|
|
.SS rclone config show
|
|
.PP
|
|
Print (decrypted) config file, or the config for a single remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Print (decrypted) config file, or the config for a single remote.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ show\ [<remote>]\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ show
|
|
\f[]
|
|
.fi
|
|
.SS rclone config update
|
|
.PP
|
|
Update options in an existing remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Update an existing remote\[aq]s options.
|
|
The options should be passed in in pairs of .
|
|
.PP
|
|
For example to update the env_auth field of a remote of name myremote
|
|
you would do:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ update\ myremote\ swift\ env_auth\ true
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config\ update\ <name>\ [<key>\ <value>]+\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ update
|
|
\f[]
|
|
.fi
|
|
.SS rclone copyto
|
|
.PP
|
|
Copy files from source to dest, skipping already copied
|
|
.SS Synopsis
|
|
.PP
|
|
If source:path is a file or directory then it copies it to a file or
|
|
directory named dest:path.
|
|
.PP
|
|
This can be used to upload single files to other than their current
|
|
name.
|
|
If the source is a directory then it acts exactly like the copy command.
|
|
.PP
|
|
So
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copyto\ src\ dst
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
where src and dst are rclone paths, either remote:path or /path/to/local
|
|
or C:.
|
|
.PP
|
|
This will:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
if\ src\ is\ file
|
|
\ \ \ \ copy\ it\ to\ dst,\ overwriting\ an\ existing\ file\ if\ it\ exists
|
|
if\ src\ is\ directory
|
|
\ \ \ \ copy\ it\ to\ dst,\ overwriting\ existing\ files\ if\ they\ exist
|
|
\ \ \ \ see\ copy\ command\ for\ full\ details
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This doesn\[aq]t transfer unchanged files, testing by size and
|
|
modification time or MD5SUM.
|
|
It doesn\[aq]t delete files from the destination.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copyto\ source:path\ dest:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ copyto
|
|
\f[]
|
|
.fi
|
|
.SS rclone cryptcheck
|
|
.PP
|
|
Cryptcheck checks the integrity of a crypted remote.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cryptcheck checks a remote against a crypted remote.
|
|
This is the equivalent of running rclone check, but able to check the
|
|
checksums of the crypted remote.
|
|
.PP
|
|
For it to work the underlying remote of the cryptedremote must support
|
|
some kind of checksum.
|
|
.PP
|
|
It works by reading the nonce from each file on the cryptedremote: and
|
|
using that to encrypt each file on the remote:.
|
|
It then checks the checksum of the underlying file on the cryptedremote:
|
|
against the checksum of the file it has just encrypted.
|
|
.PP
|
|
Use it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cryptcheck\ /path/to/files\ encryptedremote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can use it like this also, but that will involve downloading all the
|
|
files in remote:path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cryptcheck\ remote:path\ encryptedremote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
After it has run it will log the status of the encryptedremote:.
|
|
.PP
|
|
If you supply the \-\-one\-way flag, it will only check that files in
|
|
source match the files in destination, not the other way around.
|
|
Meaning extra files in destination that are not in the source will not
|
|
trigger an error.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cryptcheck\ remote:path\ cryptedremote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ help\ for\ cryptcheck
|
|
\ \ \ \ \ \ \-\-one\-way\ \ \ Check\ one\ way\ only,\ source\ files\ must\ exist\ on\ destination
|
|
\f[]
|
|
.fi
|
|
.SS rclone cryptdecode
|
|
.PP
|
|
Cryptdecode returns unencrypted file names.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone cryptdecode returns unencrypted file names when provided with a
|
|
list of encrypted file names.
|
|
List limit is 10 items.
|
|
.PP
|
|
If you supply the \-\-reverse flag, it will return encrypted file names.
|
|
.PP
|
|
use it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cryptdecode\ encryptedremote:\ encryptedfilename1\ encryptedfilename2
|
|
|
|
rclone\ cryptdecode\ \-\-reverse\ encryptedremote:\ filename1\ filename2
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ cryptdecode\ encryptedremote:\ encryptedfilename\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ help\ for\ cryptdecode
|
|
\ \ \ \ \ \ \-\-reverse\ \ \ Reverse\ cryptdecode,\ encrypts\ filenames
|
|
\f[]
|
|
.fi
|
|
.SS rclone dbhashsum
|
|
.PP
|
|
Produces a Dropbox hash file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces a Dropbox hash file for all the objects in the path.
|
|
The hashes are calculated according to Dropbox content hash
|
|
rules (https://www.dropbox.com/developers/reference/content-hash).
|
|
The output is in the same format as md5sum and sha1sum.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ dbhashsum\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ dbhashsum
|
|
\f[]
|
|
.fi
|
|
.SS rclone deletefile
|
|
.PP
|
|
Remove a single file path from remote.
|
|
.SS Synopsis
|
|
.PP
|
|
Remove a single file path from remote.
|
|
Unlike \f[C]delete\f[] it cannot be used to remove a directory and it
|
|
doesn\[aq]t obey include/exclude filters \- if the specified file
|
|
exists, it will always be removed.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ deletefile\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ deletefile
|
|
\f[]
|
|
.fi
|
|
.SS rclone genautocomplete
|
|
.PP
|
|
Output completion script for a given shell.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a shell completion script for rclone.
|
|
Run with \-\-help to list the supported shells.
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ genautocomplete
|
|
\f[]
|
|
.fi
|
|
.SS rclone genautocomplete bash
|
|
.PP
|
|
Output bash completion script for rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a bash shell autocompletion script for rclone.
|
|
.PP
|
|
This writes to /etc/bash_completion.d/rclone by default so will probably
|
|
need to be run with sudo or as root, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ rclone\ genautocomplete\ bash
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Logout and login again to use the autocompletion scripts, or source them
|
|
directly
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&.\ /etc/bash_completion
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you supply a command line argument the script will be written there.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ genautocomplete\ bash\ [output_file]\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ bash
|
|
\f[]
|
|
.fi
|
|
.SS rclone genautocomplete zsh
|
|
.PP
|
|
Output zsh completion script for rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
Generates a zsh autocompletion script for rclone.
|
|
.PP
|
|
This writes to /usr/share/zsh/vendor\-completions/_rclone by default so
|
|
will probably need to be run with sudo or as root, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
sudo\ rclone\ genautocomplete\ zsh
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Logout and login again to use the autocompletion scripts, or source them
|
|
directly
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
autoload\ \-U\ compinit\ &&\ compinit
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you supply a command line argument the script will be written there.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ genautocomplete\ zsh\ [output_file]\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ zsh
|
|
\f[]
|
|
.fi
|
|
.SS rclone gendocs
|
|
.PP
|
|
Output markdown docs for rclone to the directory supplied.
|
|
.SS Synopsis
|
|
.PP
|
|
This produces markdown docs for the rclone commands to the directory
|
|
supplied.
|
|
These are in a format suitable for hugo to render into the rclone.org
|
|
website.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ gendocs\ output_directory\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ gendocs
|
|
\f[]
|
|
.fi
|
|
.SS rclone hashsum
|
|
.PP
|
|
Produces an hashsum file for all the objects in the path.
|
|
.SS Synopsis
|
|
.PP
|
|
Produces a hash file for all the objects in the path using the hash
|
|
named.
|
|
The output is in the same format as the standard md5sum/sha1sum tool.
|
|
.PP
|
|
Run without a hash to see the list of supported hashes, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ hashsum
|
|
Supported\ hashes\ are:
|
|
\ \ *\ MD5
|
|
\ \ *\ SHA\-1
|
|
\ \ *\ DropboxHash
|
|
\ \ *\ QuickXorHash
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ hashsum\ MD5\ remote:path
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ hashsum\ <hash>\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ hashsum
|
|
\f[]
|
|
.fi
|
|
.SS rclone link
|
|
.PP
|
|
Generate public link to file/folder.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone link will create or retrieve a public link to the given file or
|
|
folder.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ link\ remote:path/to/file
|
|
rclone\ link\ remote:path/to/folder/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If successful, the last line of the output will contain the link.
|
|
Exact capabilities depend on the remote, but the link will always be
|
|
created with the least constraints \[en] e.g.
|
|
no expiry, no password protection, accessible without account.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ link\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ link
|
|
\f[]
|
|
.fi
|
|
.SS rclone listremotes
|
|
.PP
|
|
List all the remotes in the config file.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone listremotes lists all the available remotes from the config file.
|
|
.PP
|
|
When uses with the \-l flag it lists the types too.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ listremotes\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ listremotes
|
|
\ \ \-l,\ \-\-long\ \ \ Show\ the\ type\ as\ well\ as\ names.
|
|
\f[]
|
|
.fi
|
|
.SS rclone lsf
|
|
.PP
|
|
List directories and objects in remote:path formatted for parsing
|
|
.SS Synopsis
|
|
.PP
|
|
List the contents of the source path (directories and objects) to
|
|
standard output in a form which is easy to parse by scripts.
|
|
By default this will just be the names of the objects and directories,
|
|
one per line.
|
|
The directories will have a / suffix.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsf\ swift:bucket
|
|
bevajer5jef
|
|
canole
|
|
diwogej7
|
|
ferejej3gux/
|
|
fubuwic
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Use the \-\-format option to control what gets listed.
|
|
By default this is just the path, but you can use these parameters to
|
|
control the output:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
p\ \-\ path
|
|
s\ \-\ size
|
|
t\ \-\ modification\ time
|
|
h\ \-\ hash
|
|
i\ \-\ ID\ of\ object\ if\ known
|
|
m\ \-\ MimeType\ of\ object\ if\ known
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
So if you wanted the path, size and modification time, you would use
|
|
\-\-format "pst", or maybe \-\-format "tsp" to put the path last.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsf\ \ \-\-format\ "tsp"\ swift:bucket
|
|
2016\-06\-25\ 18:55:41;60295;bevajer5jef
|
|
2016\-06\-25\ 18:55:43;90613;canole
|
|
2016\-06\-25\ 18:55:43;94467;diwogej7
|
|
2018\-04\-26\ 08:50:45;0;ferejej3gux/
|
|
2016\-06\-25\ 18:55:40;37600;fubuwic
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you specify "h" in the format you will get the MD5 hash by default,
|
|
use the "\-\-hash" flag to change which hash you want.
|
|
Note that this can be returned as an empty string if it isn\[aq]t
|
|
available on the object (and for directories), "ERROR" if there was an
|
|
error reading it from the object and "UNSUPPORTED" if that object does
|
|
not support that hash type.
|
|
.PP
|
|
For example to emulate the md5sum command you can use
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsf\ \-R\ \-\-hash\ MD5\ \-\-format\ hp\ \-\-separator\ "\ \ "\ \-\-files\-only\ .
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsf\ \-R\ \-\-hash\ MD5\ \-\-format\ hp\ \-\-separator\ "\ \ "\ \-\-files\-only\ swift:bucket\
|
|
7908e352297f0f530b84a756f188baa3\ \ bevajer5jef
|
|
cd65ac234e6fea5925974a51cdd865cc\ \ canole
|
|
03b5341b4f234b9d984d03ad076bae91\ \ diwogej7
|
|
8fd37c3810dd660778137ac3a66cc06d\ \ fubuwic
|
|
99713e14a4c4ff553acaf1930fad985b\ \ gixacuh7ku
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
(Though "rclone md5sum ." is an easier way of typing this.)
|
|
.PP
|
|
By default the separator is ";" this can be changed with the
|
|
\-\-separator flag.
|
|
Note that separators aren\[aq]t escaped in the path so putting it last
|
|
is a good strategy.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsf\ \ \-\-separator\ ","\ \-\-format\ "tshp"\ swift:bucket
|
|
2016\-06\-25\ 18:55:41,60295,7908e352297f0f530b84a756f188baa3,bevajer5jef
|
|
2016\-06\-25\ 18:55:43,90613,cd65ac234e6fea5925974a51cdd865cc,canole
|
|
2016\-06\-25\ 18:55:43,94467,03b5341b4f234b9d984d03ad076bae91,diwogej7
|
|
2018\-04\-26\ 08:52:53,0,,ferejej3gux/
|
|
2016\-06\-25\ 18:55:40,37600,8fd37c3810dd660778137ac3a66cc06d,fubuwic
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can output in CSV standard format.
|
|
This will escape things in " if they contain ,
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ lsf\ \-\-csv\ \-\-files\-only\ \-\-format\ ps\ remote:path
|
|
test.log,22355
|
|
test.sh,449
|
|
"this\ file\ contains\ a\ comma,\ in\ the\ file\ name.txt",6
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that the \-\-absolute parameter is useful for making lists of files
|
|
to pass to an rclone copy with the \-\-files\-from flag.
|
|
.PP
|
|
For example to find all the files modified within one day and copy those
|
|
only (without traversing the whole directory structure):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsf\ \-\-absolute\ \-\-files\-only\ \-\-max\-age\ 1d\ /path/to/local\ >\ new_files
|
|
rclone\ copy\ \-\-files\-from\ new_files\ /path/to/local\ remote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Any of the filtering options can be applied to this commmand.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[],\f[C]lsl\f[],\f[C]lsd\f[] are designed to be human readable.
|
|
\f[C]lsf\f[] is designed to be human and machine readable.
|
|
\f[C]lsjson\f[] is designed to be machine readable.
|
|
.PP
|
|
Note that \f[C]ls\f[] and \f[C]lsl\f[] recurse by default \- use
|
|
"\-\-max\-depth 1" to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[],\f[C]lsf\f[],\f[C]lsjson\f[] do not
|
|
recurse by default \- use "\-R" to make them recurse.
|
|
.PP
|
|
Listing a non existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (eg s3, swift, gcs, etc
|
|
\- the bucket based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsf\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-absolute\ \ \ \ \ \ \ \ \ \ \ Put\ a\ leading\ /\ in\ front\ of\ path\ names.
|
|
\ \ \ \ \ \ \-\-csv\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Output\ in\ CSV\ format.
|
|
\ \ \-d,\ \-\-dir\-slash\ \ \ \ \ \ \ \ \ \ Append\ a\ slash\ to\ directory\ names.\ (default\ true)
|
|
\ \ \ \ \ \ \-\-dirs\-only\ \ \ \ \ \ \ \ \ \ Only\ list\ directories.
|
|
\ \ \ \ \ \ \-\-files\-only\ \ \ \ \ \ \ \ \ Only\ list\ files.
|
|
\ \ \-F,\ \-\-format\ string\ \ \ \ \ \ Output\ format\ \-\ see\ \ help\ for\ details\ (default\ "p")
|
|
\ \ \ \ \ \ \-\-hash\ h\ \ \ \ \ \ \ \ \ \ \ \ \ Use\ this\ hash\ when\ h\ is\ used\ in\ the\ format\ MD5|SHA\-1|DropboxHash\ (default\ "MD5")
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ lsf
|
|
\ \ \-R,\ \-\-recursive\ \ \ \ \ \ \ \ \ \ Recurse\ into\ the\ listing.
|
|
\ \ \-s,\ \-\-separator\ string\ \ \ Separator\ for\ the\ items\ in\ the\ format.\ (default\ ";")
|
|
\f[]
|
|
.fi
|
|
.SS rclone lsjson
|
|
.PP
|
|
List directories and objects in the path in JSON format.
|
|
.SS Synopsis
|
|
.PP
|
|
List directories and objects in the path in JSON format.
|
|
.PP
|
|
The output is an array of Items, where each Item looks like this
|
|
.PP
|
|
{ "Hashes" : { "SHA\-1" : "f572d396fae9206628714fb2ce00f72e94f2258f",
|
|
"MD5" : "b1946ac92492d2347c6235b4d2611184", "DropboxHash" :
|
|
"ecb65bb98f9d905b70458986c39fcbad7715e5f2fcc3b1f07767d7c83e2438cc" },
|
|
"ID": "y2djkhiujf83u33", "IsDir" : false, "MimeType" :
|
|
"application/octet\-stream", "ModTime" :
|
|
"2017\-05\-31T16:15:57.034468261+01:00", "Name" : "file.txt",
|
|
"Encrypted" : "v0qpsdq8anpci8n929v3uu9338", "Path" :
|
|
"full/path/goes/here/file.txt", "Size" : 6 }
|
|
.PP
|
|
If \-\-hash is not specified the Hashes property won\[aq]t be emitted.
|
|
.PP
|
|
If \-\-no\-modtime is specified then ModTime will be blank.
|
|
.PP
|
|
If \-\-encrypted is not specified the Encrypted won\[aq]t be emitted.
|
|
.PP
|
|
The Path field will only show folders below the remote path being
|
|
listed.
|
|
If "remote:path" contains the file "subfolder/file.txt", the Path for
|
|
"file.txt" will be "subfolder/file.txt", not
|
|
"remote:path/subfolder/file.txt".
|
|
When used without \-\-recursive the Path will always be the same as
|
|
Name.
|
|
.PP
|
|
The time is in RFC3339 format with nanosecond precision.
|
|
.PP
|
|
The whole output can be processed as a JSON blob, or alternatively it
|
|
can be processed line by line as each item is written one to a line.
|
|
.PP
|
|
Any of the filtering options can be applied to this commmand.
|
|
.PP
|
|
There are several related list commands
|
|
.IP \[bu] 2
|
|
\f[C]ls\f[] to list size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsl\f[] to list modification time, size and path of objects only
|
|
.IP \[bu] 2
|
|
\f[C]lsd\f[] to list directories only
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[] to list objects and directories in easy to parse format
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[] to list objects and directories in JSON format
|
|
.PP
|
|
\f[C]ls\f[],\f[C]lsl\f[],\f[C]lsd\f[] are designed to be human readable.
|
|
\f[C]lsf\f[] is designed to be human and machine readable.
|
|
\f[C]lsjson\f[] is designed to be machine readable.
|
|
.PP
|
|
Note that \f[C]ls\f[] and \f[C]lsl\f[] recurse by default \- use
|
|
"\-\-max\-depth 1" to stop the recursion.
|
|
.PP
|
|
The other list commands \f[C]lsd\f[],\f[C]lsf\f[],\f[C]lsjson\f[] do not
|
|
recurse by default \- use "\-R" to make them recurse.
|
|
.PP
|
|
Listing a non existent directory will produce an error except for
|
|
remotes which can\[aq]t have empty directories (eg s3, swift, gcs, etc
|
|
\- the bucket based remotes).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsjson\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-M,\ \-\-encrypted\ \ \ \ Show\ the\ encrypted\ names.
|
|
\ \ \ \ \ \ \-\-hash\ \ \ \ \ \ \ \ \ Include\ hashes\ in\ the\ output\ (may\ take\ longer).
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ help\ for\ lsjson
|
|
\ \ \ \ \ \ \-\-no\-modtime\ \ \ Don\[aq]t\ read\ the\ modification\ time\ (can\ speed\ things\ up).
|
|
\ \ \-R,\ \-\-recursive\ \ \ \ Recurse\ into\ the\ listing.
|
|
\f[]
|
|
.fi
|
|
.SS rclone mount
|
|
.PP
|
|
Mount the remote as a mountpoint.
|
|
\f[B]EXPERIMENTAL\f[]
|
|
.SS Synopsis
|
|
.PP
|
|
rclone mount allows Linux, FreeBSD, macOS and Windows to mount any of
|
|
Rclone\[aq]s cloud storage systems as a file system with FUSE.
|
|
.PP
|
|
This is \f[B]EXPERIMENTAL\f[] \- use with care.
|
|
.PP
|
|
First set up your remote using \f[C]rclone\ config\f[].
|
|
Check it works with \f[C]rclone\ ls\f[] etc.
|
|
.PP
|
|
Start the mount like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mount\ remote:path/to/files\ /path/to/local/mount
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or on Windows like this where X: is an unused drive letter
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mount\ remote:path/to/files\ X:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
When the program ends, either via Ctrl+C or receiving a SIGINT or
|
|
SIGTERM signal, the mount is automatically stopped.
|
|
.PP
|
|
The umount operation can fail, for example when the mountpoint is busy.
|
|
When that happens, it is the user\[aq]s responsibility to stop the mount
|
|
manually with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ Linux
|
|
fusermount\ \-u\ /path/to/local/mount
|
|
#\ OS\ X
|
|
umount\ /path/to/local/mount
|
|
\f[]
|
|
.fi
|
|
.SS Installing on Windows
|
|
.PP
|
|
To run rclone mount on Windows, you will need to download and install
|
|
WinFsp (http://www.secfs.net/winfsp/).
|
|
.PP
|
|
WinFsp is an open source (https://github.com/billziss-gh/winfsp) Windows
|
|
File System Proxy which makes it easy to write user space file systems
|
|
for Windows.
|
|
It provides a FUSE emulation layer which rclone uses combination with
|
|
cgofuse (https://github.com/billziss-gh/cgofuse).
|
|
Both of these packages are by Bill Zissimopoulos who was very helpful
|
|
during the implementation of rclone mount for Windows.
|
|
.SS Windows caveats
|
|
.PP
|
|
Note that drives created as Administrator are not visible by other
|
|
accounts (including the account that was elevated as Administrator).
|
|
So if you start a Windows drive from an Administrative Command Prompt
|
|
and then try to access the same drive from Explorer (which does not run
|
|
as Administrator), you will not be able to see the new drive.
|
|
.PP
|
|
The easiest way around this is to start the drive from a normal command
|
|
prompt.
|
|
It is also possible to start a drive from the SYSTEM account (using the
|
|
WinFsp.Launcher
|
|
infrastructure (https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture))
|
|
which creates drives accessible for everyone on the system or
|
|
alternatively using the nssm service manager (https://nssm.cc/usage).
|
|
.SS Limitations
|
|
.PP
|
|
Without the use of "\-\-vfs\-cache\-mode" this can only write files
|
|
sequentially, it can only seek when reading.
|
|
This means that many applications won\[aq]t work with their files on an
|
|
rclone mount without "\-\-vfs\-cache\-mode writes" or
|
|
"\-\-vfs\-cache\-mode full".
|
|
See the File Caching (#file-caching) section for more info.
|
|
.PP
|
|
The bucket based remotes (eg Swift, S3, Google Compute Storage, B2,
|
|
Hubic) won\[aq]t work from the root \- you will need to specify a
|
|
bucket, or a path within the bucket.
|
|
So \f[C]swift:\f[] won\[aq]t work whereas \f[C]swift:bucket\f[] will as
|
|
will \f[C]swift:bucket/path\f[].
|
|
None of these support the concept of directories, so empty directories
|
|
will have a tendency to disappear once they fall out of the directory
|
|
cache.
|
|
.PP
|
|
Only supported on Linux, FreeBSD, OS X and Windows at the moment.
|
|
.SS rclone mount vs rclone sync/copy
|
|
.PP
|
|
File systems expect things to be 100% reliable, whereas cloud storage
|
|
systems are a long way from 100% reliable.
|
|
The rclone sync/copy commands cope with this with lots of retries.
|
|
However rclone mount can\[aq]t use retries in the same way without
|
|
making local copies of the uploads.
|
|
Look at the \f[B]EXPERIMENTAL\f[] file caching (#file-caching) for
|
|
solutions to make mount mount more reliable.
|
|
.SS Attribute caching
|
|
.PP
|
|
You can use the flag \-\-attr\-timeout to set the time the kernel caches
|
|
the attributes (size, modification time etc) for directory entries.
|
|
.PP
|
|
The default is "1s" which caches files just long enough to avoid too
|
|
many callbacks to rclone from the kernel.
|
|
.PP
|
|
In theory 0s should be the correct value for filesystems which can
|
|
change outside the control of the kernel.
|
|
However this causes quite a few problems such as rclone using too much
|
|
memory (https://github.com/ncw/rclone/issues/2157), rclone not serving
|
|
files to
|
|
samba (https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112)
|
|
and excessive time listing
|
|
directories (https://github.com/ncw/rclone/issues/2095#issuecomment-371141147).
|
|
.PP
|
|
The kernel can cache the info about a file for the time given by
|
|
"\-\-attr\-timeout".
|
|
You may see corruption if the remote file changes length during this
|
|
window.
|
|
It will show up as either a truncated file or a file with garbage on the
|
|
end.
|
|
With "\-\-attr\-timeout 1s" this is very unlikely but not impossible.
|
|
The higher you set "\-\-attr\-timeout" the more likely it is.
|
|
The default setting of "1s" is the lowest setting which mitigates the
|
|
problems above.
|
|
.PP
|
|
If you set it higher (\[aq]10s\[aq] or \[aq]1m\[aq] say) then the kernel
|
|
will call back to rclone less often making it more efficient, however
|
|
there is more chance of the corruption issue above.
|
|
.PP
|
|
If files don\[aq]t change on the remote outside of the control of rclone
|
|
then there is no chance of corruption.
|
|
.PP
|
|
This is the same as setting the attr_timeout option in mount.fuse.
|
|
.SS Filters
|
|
.PP
|
|
Note that all the rclone filters can be used to select a subset of the
|
|
files to be visible in the mount.
|
|
.SS systemd
|
|
.PP
|
|
When running rclone mount as a systemd service, it is possible to use
|
|
Type=notify.
|
|
In this case the service will enter the started state after the
|
|
mountpoint has been successfully set up.
|
|
Units having the rclone mount service specified as a requirement will
|
|
see all files and folders immediately in this mode.
|
|
.SS chunked reading
|
|
.PP
|
|
\-\-vfs\-read\-chunk\-size will enable reading the source objects in
|
|
parts.
|
|
This can reduce the used download quota for some remotes by requesting
|
|
only chunks from the remote that are actually read at the cost of an
|
|
increased number of requests.
|
|
.PP
|
|
When \-\-vfs\-read\-chunk\-size\-limit is also specified and greater
|
|
than \-\-vfs\-read\-chunk\-size, the chunk size for each open file will
|
|
get doubled for each chunk read, until the specified value is reached.
|
|
A value of \-1 will disable the limit and the chunk size will grow
|
|
indefinitely.
|
|
.PP
|
|
With \-\-vfs\-read\-chunk\-size 100M and
|
|
\-\-vfs\-read\-chunk\-size\-limit 0 the following parts will be
|
|
downloaded: 0\-100M, 100M\-200M, 200M\-300M, 300M\-400M and so on.
|
|
When \-\-vfs\-read\-chunk\-size\-limit 500M is specified, the result
|
|
would be 0\-100M, 100M\-300M, 300M\-700M, 700M\-1200M, 1200M\-1700M and
|
|
so on.
|
|
.PP
|
|
Chunked reading will only work with \-\-vfs\-cache\-mode < full, as the
|
|
file will always be copied to the vfs cache before opening with
|
|
\-\-vfs\-cache\-mode full.
|
|
.SS Directory Cache
|
|
.PP
|
|
Using the \f[C]\-\-dir\-cache\-time\f[] flag, you can set how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made locally in the mount may appear immediately or invalidate
|
|
the cache.
|
|
However, changes done on the remote will only be picked up once the
|
|
cache expires.
|
|
.PP
|
|
Alternatively, you can send a \f[C]SIGHUP\f[] signal to rclone for it to
|
|
flush all directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill\ \-SIGHUP\ $(pidof\ rclone)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control (/rc) then you can use
|
|
rclone rc to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget\ file=path/to/file\ dir=path/to/dir
|
|
\f[]
|
|
.fi
|
|
.SS File Caching
|
|
.PP
|
|
\f[B]NB\f[] File caching is \f[B]EXPERIMENTAL\f[] \- use with care!
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
The VFS layer is used by rclone mount to make a cloud storage system
|
|
work more like a normal file system.
|
|
.PP
|
|
You\[aq]ll need to enable VFS caching if you want, for example, to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache works in addition to the cache backend and you
|
|
may find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\-cache\-dir\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Directory\ rclone\ will\ use\ for\ caching.
|
|
\-\-vfs\-cache\-max\-age\ duration\ \ \ \ \ \ \ \ \ Max\ age\ of\ objects\ in\ the\ cache.\ (default\ 1h0m0s)
|
|
\-\-vfs\-cache\-mode\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ Cache\ mode\ off|minimal|writes|full\ (default\ "off")
|
|
\-\-vfs\-cache\-poll\-interval\ duration\ \ \ Interval\ to\ poll\ the\ cache\ for\ stale\ objects.\ (default\ 1m0s)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]\-vv\f[] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]\-\-cache\-dir\f[] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by
|
|
\f[C]\-\-vfs\-cache\-mode\f[].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
so if rclone is quit or dies with open files then these won\[aq]t get
|
|
written back to the remote.
|
|
However they will still be in the on disk cache.
|
|
.SS \-\-vfs\-cache\-mode off
|
|
.PP
|
|
In this mode the cache will read directly from the remote and write
|
|
directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS \-\-vfs\-cache\-mode minimal
|
|
.PP
|
|
This is very similar to "off" except that files opened for read AND
|
|
write will be buffered to disks.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS \-\-vfs\-cache\-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried up to \-\-low\-level\-retries
|
|
times.
|
|
.SS \-\-vfs\-cache\-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When a file is opened for read it will be downloaded in its entirety
|
|
first.
|
|
.PP
|
|
This may be appropriate for your needs, or you may prefer to look at the
|
|
cache backend which does a much more sophisticated job of caching,
|
|
including caching directory hierarchies and chunks of files.
|
|
.PP
|
|
In this mode, unlike the others, when a file is written to the disk, it
|
|
will be kept on the disk after it is written to the remote.
|
|
It will be purged on a schedule according to
|
|
\f[C]\-\-vfs\-cache\-max\-age\f[].
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload or download fails it will be retried up to
|
|
\-\-low\-level\-retries times.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mount\ remote:path\ /path/to/mountpoint\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-allow\-non\-empty\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Allow\ mounting\ over\ a\ non\-empty\ directory.
|
|
\ \ \ \ \ \ \-\-allow\-other\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Allow\ access\ to\ other\ users.
|
|
\ \ \ \ \ \ \-\-allow\-root\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Allow\ access\ to\ root\ user.
|
|
\ \ \ \ \ \ \-\-attr\-timeout\ duration\ \ \ \ \ \ \ \ \ \ \ \ \ \ Time\ for\ which\ file/directory\ attributes\ are\ cached.\ (default\ 1s)
|
|
\ \ \ \ \ \ \-\-daemon\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Run\ mount\ as\ a\ daemon\ (background\ mode).
|
|
\ \ \ \ \ \ \-\-debug\-fuse\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Debug\ the\ FUSE\ internals\ \-\ needs\ \-v.
|
|
\ \ \ \ \ \ \-\-default\-permissions\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Makes\ kernel\ enforce\ access\ control\ based\ on\ the\ file\ mode.
|
|
\ \ \ \ \ \ \-\-dir\-cache\-time\ duration\ \ \ \ \ \ \ \ \ \ \ \ Time\ to\ cache\ directory\ entries\ for.\ (default\ 5m0s)
|
|
\ \ \ \ \ \ \-\-fuse\-flag\ stringArray\ \ \ \ \ \ \ \ \ \ \ \ \ \ Flags\ or\ arguments\ to\ be\ passed\ direct\ to\ libfuse/WinFsp.\ Repeat\ if\ required.
|
|
\ \ \ \ \ \ \-\-gid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ gid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ mount
|
|
\ \ \ \ \ \ \-\-max\-read\-ahead\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ The\ number\ of\ bytes\ that\ can\ be\ prefetched\ for\ sequential\ reads.\ (default\ 128k)
|
|
\ \ \ \ \ \ \-\-no\-checksum\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ compare\ checksums\ on\ up/download.
|
|
\ \ \ \ \ \ \-\-no\-modtime\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ read/write\ the\ modification\ time\ (can\ speed\ things\ up).
|
|
\ \ \ \ \ \ \-\-no\-seek\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ allow\ seeking\ in\ files.
|
|
\ \ \-o,\ \-\-option\ stringArray\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Option\ for\ libfuse/WinFsp.\ Repeat\ if\ required.
|
|
\ \ \ \ \ \ \-\-poll\-interval\ duration\ \ \ \ \ \ \ \ \ \ \ \ \ Time\ to\ wait\ between\ polling\ for\ changes.\ Must\ be\ smaller\ than\ dir\-cache\-time.\ Only\ on\ supported\ remotes.\ Set\ to\ 0\ to\ disable.\ (default\ 1m0s)
|
|
\ \ \ \ \ \ \-\-read\-only\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Mount\ read\-only.
|
|
\ \ \ \ \ \ \-\-uid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ uid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \ \ \ \ \-\-umask\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ permission\ bits\ set\ by\ the\ filesystem.
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-max\-age\ duration\ \ \ \ \ \ \ \ \ Max\ age\ of\ objects\ in\ the\ cache.\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-mode\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ Cache\ mode\ off|minimal|writes|full\ (default\ "off")
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-poll\-interval\ duration\ \ \ Interval\ to\ poll\ the\ cache\ for\ stale\ objects.\ (default\ 1m0s)
|
|
\ \ \ \ \ \ \-\-vfs\-read\-chunk\-size\ int\ \ \ \ \ \ \ \ \ \ \ \ Read\ the\ source\ objects\ in\ chunks.
|
|
\ \ \ \ \ \ \-\-vfs\-read\-chunk\-size\-limit\ int\ \ \ \ \ \ If\ greater\ than\ \-\-vfs\-read\-chunk\-size,\ double\ the\ chunk\ size\ after\ each\ chunk\ read,\ until\ the\ limit\ is\ reached.\ \-1\ is\ unlimited.
|
|
\ \ \ \ \ \ \-\-volname\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Set\ the\ volume\ name\ (not\ supported\ by\ all\ OSes).
|
|
\ \ \ \ \ \ \-\-write\-back\-cache\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Makes\ kernel\ buffer\ writes\ before\ sending\ them\ to\ rclone.\ Without\ this,\ writethrough\ caching\ is\ used.
|
|
\f[]
|
|
.fi
|
|
.SS rclone moveto
|
|
.PP
|
|
Move file or directory from source to dest.
|
|
.SS Synopsis
|
|
.PP
|
|
If source:path is a file or directory then it moves it to a file or
|
|
directory named dest:path.
|
|
.PP
|
|
This can be used to rename files or upload single files to other than
|
|
their existing name.
|
|
If the source is a directory then it acts exacty like the move command.
|
|
.PP
|
|
So
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ moveto\ src\ dst
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
where src and dst are rclone paths, either remote:path or /path/to/local
|
|
or C:.
|
|
.PP
|
|
This will:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
if\ src\ is\ file
|
|
\ \ \ \ move\ it\ to\ dst,\ overwriting\ an\ existing\ file\ if\ it\ exists
|
|
if\ src\ is\ directory
|
|
\ \ \ \ move\ it\ to\ dst,\ overwriting\ existing\ files\ if\ they\ exist
|
|
\ \ \ \ see\ move\ command\ for\ full\ details
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This doesn\[aq]t transfer unchanged files, testing by size and
|
|
modification time or MD5SUM.
|
|
src will be deleted on successful transfer.
|
|
.PP
|
|
\f[B]Important\f[]: Since this can cause data loss, test first with the
|
|
\-\-dry\-run flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ moveto\ source:path\ dest:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ moveto
|
|
\f[]
|
|
.fi
|
|
.SS rclone ncdu
|
|
.PP
|
|
Explore a remote with a text based user interface.
|
|
.SS Synopsis
|
|
.PP
|
|
This displays a text based user interface allowing the navigation of a
|
|
remote.
|
|
It is most useful for answering the question \- "What is using all my
|
|
disk space?".
|
|
.PP
|
|
To make the user interface it first scans the entire remote given and
|
|
builds an in memory representation.
|
|
rclone ncdu can be used during this scanning phase and you will see it
|
|
building up the directory structure as it goes along.
|
|
.PP
|
|
Here are the keys \- press \[aq]?\[aq] to toggle the help on and off
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ ↑,↓\ or\ k,j\ to\ Move
|
|
\ →,l\ to\ enter
|
|
\ ←,h\ to\ return
|
|
\ c\ toggle\ counts
|
|
\ g\ toggle\ graph
|
|
\ n,s,C\ sort\ by\ name,size,count
|
|
\ ^L\ refresh\ screen
|
|
\ ?\ to\ toggle\ help\ on\ and\ off
|
|
\ q/ESC/c\-C\ to\ quit
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This an homage to the ncdu tool (https://dev.yorhel.nl/ncdu) but for
|
|
rclone remotes.
|
|
It is missing lots of features at the moment, most importantly deleting
|
|
files, but is useful as it stands.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ncdu\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ ncdu
|
|
\f[]
|
|
.fi
|
|
.SS rclone obscure
|
|
.PP
|
|
Obscure password for use in the rclone.conf
|
|
.SS Synopsis
|
|
.PP
|
|
Obscure password for use in the rclone.conf
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ obscure\ password\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ obscure
|
|
\f[]
|
|
.fi
|
|
.SS rclone rc
|
|
.PP
|
|
Run a command against a running rclone.
|
|
.SS Synopsis
|
|
.PP
|
|
This runs a command against a running rclone.
|
|
By default it will use that specified in the \-\-rc\-addr command.
|
|
.PP
|
|
Arguments should be passed in as parameter=value.
|
|
.PP
|
|
The result will be returned as a JSON object by default.
|
|
.PP
|
|
Use "rclone rc" to see a list of all possible commands.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ commands\ parameter\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ help\ for\ rc
|
|
\ \ \ \ \ \ \-\-no\-output\ \ \ \ If\ set\ don\[aq]t\ output\ the\ JSON\ result.
|
|
\ \ \ \ \ \ \-\-url\ string\ \ \ URL\ to\ connect\ to\ rclone\ remote\ control.\ (default\ "http://localhost:5572/")
|
|
\f[]
|
|
.fi
|
|
.SS rclone rcat
|
|
.PP
|
|
Copies standard input to file on remote.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone rcat reads from standard input (stdin) and copies it to a single
|
|
remote file.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
echo\ "hello\ world"\ |\ rclone\ rcat\ remote:path/to/file
|
|
ffmpeg\ \-\ |\ rclone\ rcat\ remote:path/to/file
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If the remote file already exists, it will be overwritten.
|
|
.PP
|
|
rcat will try to upload small files in a single request, which is
|
|
usually more efficient than the streaming/chunked upload endpoints,
|
|
which use multiple requests.
|
|
Exact behaviour depends on the remote.
|
|
What is considered a small file may be set through
|
|
\f[C]\-\-streaming\-upload\-cutoff\f[].
|
|
Uploading only starts after the cutoff is reached or if the file ends
|
|
before that.
|
|
The data must fit into RAM.
|
|
The cutoff needs to be small enough to adhere the limits of your remote,
|
|
please see there.
|
|
Generally speaking, setting this cutoff too high will decrease your
|
|
performance.
|
|
.PP
|
|
Note that the upload can also not be retried because the data is not
|
|
kept around until the upload succeeds.
|
|
If you need to transfer a lot of data, you\[aq]re better off caching
|
|
locally and then \f[C]rclone\ move\f[] it to the destination.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rcat\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ rcat
|
|
\f[]
|
|
.fi
|
|
.SS rclone rmdirs
|
|
.PP
|
|
Remove empty directories under the path.
|
|
.SS Synopsis
|
|
.PP
|
|
This removes any empty directories (or directories that only contain
|
|
empty directories) under the path that it finds, including the path if
|
|
it has nothing in.
|
|
.PP
|
|
If you supply the \-\-leave\-root flag, it will not remove the root
|
|
directory.
|
|
.PP
|
|
This is useful for tidying up remotes that rclone has left a lot of
|
|
empty directories in.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rmdirs\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ help\ for\ rmdirs
|
|
\ \ \ \ \ \ \-\-leave\-root\ \ \ Do\ not\ remove\ root\ directory\ if\ empty
|
|
\f[]
|
|
.fi
|
|
.SS rclone serve
|
|
.PP
|
|
Serve a remote over a protocol.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve is used to serve a remote over a given protocol.
|
|
This command requires the use of a subcommand to specify the protocol,
|
|
eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ serve\ http\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Each subcommand has its own options which you can see in their help.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ serve\ <protocol>\ [opts]\ <remote>\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ help\ for\ serve
|
|
\f[]
|
|
.fi
|
|
.SS rclone serve http
|
|
.PP
|
|
Serve the remote over HTTP.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve http implements a basic web server to serve the remote over
|
|
HTTP.
|
|
This can be viewed in a web browser or you can make a remote of type
|
|
http read from it.
|
|
.PP
|
|
You can use the filter flags (eg \-\-include, \-\-exclude) to control
|
|
what is served.
|
|
.PP
|
|
The server will log errors.
|
|
Use \-v to see access logs.
|
|
.PP
|
|
\-\-bwlimit will be respected for file transfers.
|
|
Use \-\-stats to control the stats printing.
|
|
.SS Server options
|
|
.PP
|
|
Use \-\-addr to specify which IP address and port the server should
|
|
listen on, eg \-\-addr 1.2.3.4:8000 or \-\-addr :8080 to listen to all
|
|
IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set \-\-addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised \- see the next section for info.
|
|
.PP
|
|
\-\-server\-read\-timeout and \-\-server\-write\-timeout can be used to
|
|
control the timeouts on the server.
|
|
Note that this is the total time for a transfer.
|
|
.PP
|
|
\-\-max\-header\-bytes controls the maximum number of bytes the server
|
|
will accept in the HTTP header.
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can either use an htpasswd file which can take lots of users, or set
|
|
a single username and password with the \-\-user and \-\-pass flags.
|
|
.PP
|
|
Use \-\-htpasswd /path/to/htpasswd to provide an htpasswd file.
|
|
This is in standard apache format and supports MD5, SHA1 and BCrypt for
|
|
basic authentication.
|
|
Bcrypt is recommended.
|
|
.PP
|
|
To create an htpasswd file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
touch\ htpasswd
|
|
htpasswd\ \-B\ htpasswd\ user
|
|
htpasswd\ \-B\ htpasswd\ anotherUser
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The password file can be updated while rclone is running.
|
|
.PP
|
|
Use \-\-realm to set the authentication realm.
|
|
.SS SSL/TLS
|
|
.PP
|
|
By default this will serve over http.
|
|
If you want you can serve over https.
|
|
You will need to supply the \-\-cert and \-\-key flags.
|
|
If you wish to do client side certificate validation then you will need
|
|
to supply \-\-client\-ca also.
|
|
.PP
|
|
\-\-cert should be a either a PEM encoded certificate or a concatenation
|
|
of that with the CA certificate.
|
|
\-\-key should be the PEM encoded private key and \-\-client\-ca should
|
|
be the PEM encoded client certificate authority certificate.
|
|
.SS Directory Cache
|
|
.PP
|
|
Using the \f[C]\-\-dir\-cache\-time\f[] flag, you can set how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made locally in the mount may appear immediately or invalidate
|
|
the cache.
|
|
However, changes done on the remote will only be picked up once the
|
|
cache expires.
|
|
.PP
|
|
Alternatively, you can send a \f[C]SIGHUP\f[] signal to rclone for it to
|
|
flush all directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill\ \-SIGHUP\ $(pidof\ rclone)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control (/rc) then you can use
|
|
rclone rc to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget\ file=path/to/file\ dir=path/to/dir
|
|
\f[]
|
|
.fi
|
|
.SS File Caching
|
|
.PP
|
|
\f[B]NB\f[] File caching is \f[B]EXPERIMENTAL\f[] \- use with care!
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
The VFS layer is used by rclone mount to make a cloud storage system
|
|
work more like a normal file system.
|
|
.PP
|
|
You\[aq]ll need to enable VFS caching if you want, for example, to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache works in addition to the cache backend and you
|
|
may find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\-cache\-dir\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Directory\ rclone\ will\ use\ for\ caching.
|
|
\-\-vfs\-cache\-max\-age\ duration\ \ \ \ \ \ \ \ \ Max\ age\ of\ objects\ in\ the\ cache.\ (default\ 1h0m0s)
|
|
\-\-vfs\-cache\-mode\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ Cache\ mode\ off|minimal|writes|full\ (default\ "off")
|
|
\-\-vfs\-cache\-poll\-interval\ duration\ \ \ Interval\ to\ poll\ the\ cache\ for\ stale\ objects.\ (default\ 1m0s)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]\-vv\f[] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]\-\-cache\-dir\f[] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by
|
|
\f[C]\-\-vfs\-cache\-mode\f[].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
so if rclone is quit or dies with open files then these won\[aq]t get
|
|
written back to the remote.
|
|
However they will still be in the on disk cache.
|
|
.SS \-\-vfs\-cache\-mode off
|
|
.PP
|
|
In this mode the cache will read directly from the remote and write
|
|
directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS \-\-vfs\-cache\-mode minimal
|
|
.PP
|
|
This is very similar to "off" except that files opened for read AND
|
|
write will be buffered to disks.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS \-\-vfs\-cache\-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried up to \-\-low\-level\-retries
|
|
times.
|
|
.SS \-\-vfs\-cache\-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When a file is opened for read it will be downloaded in its entirety
|
|
first.
|
|
.PP
|
|
This may be appropriate for your needs, or you may prefer to look at the
|
|
cache backend which does a much more sophisticated job of caching,
|
|
including caching directory hierarchies and chunks of files.
|
|
.PP
|
|
In this mode, unlike the others, when a file is written to the disk, it
|
|
will be kept on the disk after it is written to the remote.
|
|
It will be purged on a schedule according to
|
|
\f[C]\-\-vfs\-cache\-max\-age\f[].
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload or download fails it will be retried up to
|
|
\-\-low\-level\-retries times.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ serve\ http\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-addr\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ IPaddress:Port\ or\ :Port\ to\ bind\ server\ to.\ (default\ "localhost:8080")
|
|
\ \ \ \ \ \ \-\-cert\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SSL\ PEM\ key\ (concatenation\ of\ certificate\ and\ CA\ certificate)
|
|
\ \ \ \ \ \ \-\-client\-ca\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Client\ certificate\ authority\ to\ verify\ clients\ with
|
|
\ \ \ \ \ \ \-\-dir\-cache\-time\ duration\ \ \ \ \ \ \ \ \ \ \ \ Time\ to\ cache\ directory\ entries\ for.\ (default\ 5m0s)
|
|
\ \ \ \ \ \ \-\-gid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ gid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ http
|
|
\ \ \ \ \ \ \-\-htpasswd\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ htpasswd\ file\ \-\ if\ not\ provided\ no\ authentication\ is\ done
|
|
\ \ \ \ \ \ \-\-key\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SSL\ PEM\ Private\ key
|
|
\ \ \ \ \ \ \-\-max\-header\-bytes\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Maximum\ size\ of\ request\ header\ (default\ 4096)
|
|
\ \ \ \ \ \ \-\-no\-checksum\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ compare\ checksums\ on\ up/download.
|
|
\ \ \ \ \ \ \-\-no\-modtime\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ read/write\ the\ modification\ time\ (can\ speed\ things\ up).
|
|
\ \ \ \ \ \ \-\-no\-seek\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ allow\ seeking\ in\ files.
|
|
\ \ \ \ \ \ \-\-pass\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Password\ for\ authentication.
|
|
\ \ \ \ \ \ \-\-poll\-interval\ duration\ \ \ \ \ \ \ \ \ \ \ \ \ Time\ to\ wait\ between\ polling\ for\ changes.\ Must\ be\ smaller\ than\ dir\-cache\-time.\ Only\ on\ supported\ remotes.\ Set\ to\ 0\ to\ disable.\ (default\ 1m0s)
|
|
\ \ \ \ \ \ \-\-read\-only\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Mount\ read\-only.
|
|
\ \ \ \ \ \ \-\-realm\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ realm\ for\ authentication\ (default\ "rclone")
|
|
\ \ \ \ \ \ \-\-server\-read\-timeout\ duration\ \ \ \ \ \ \ Timeout\ for\ server\ reading\ data\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-server\-write\-timeout\ duration\ \ \ \ \ \ Timeout\ for\ server\ writing\ data\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-uid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ uid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \ \ \ \ \-\-umask\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ permission\ bits\ set\ by\ the\ filesystem.\ (default\ 2)
|
|
\ \ \ \ \ \ \-\-user\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ User\ name\ for\ authentication.
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-max\-age\ duration\ \ \ \ \ \ \ \ \ Max\ age\ of\ objects\ in\ the\ cache.\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-mode\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ Cache\ mode\ off|minimal|writes|full\ (default\ "off")
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-poll\-interval\ duration\ \ \ Interval\ to\ poll\ the\ cache\ for\ stale\ objects.\ (default\ 1m0s)
|
|
\ \ \ \ \ \ \-\-vfs\-read\-chunk\-size\ int\ \ \ \ \ \ \ \ \ \ \ \ Read\ the\ source\ objects\ in\ chunks.
|
|
\ \ \ \ \ \ \-\-vfs\-read\-chunk\-size\-limit\ int\ \ \ \ \ \ If\ greater\ than\ \-\-vfs\-read\-chunk\-size,\ double\ the\ chunk\ size\ after\ each\ chunk\ read,\ until\ the\ limit\ is\ reached.\ \-1\ is\ unlimited.
|
|
\f[]
|
|
.fi
|
|
.SS rclone serve restic
|
|
.PP
|
|
Serve the remote for restic\[aq]s REST API.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve restic implements restic\[aq]s REST backend API over HTTP.
|
|
This allows restic to use rclone as a data storage mechanism for cloud
|
|
providers that restic does not support directly.
|
|
.PP
|
|
Restic (https://restic.net/) is a command line program for doing
|
|
backups.
|
|
.PP
|
|
The server will log errors.
|
|
Use \-v to see access logs.
|
|
.PP
|
|
\-\-bwlimit will be respected for file transfers.
|
|
Use \-\-stats to control the stats printing.
|
|
.SS Setting up rclone for use by restic
|
|
.PP
|
|
First set up a remote for your chosen cloud provider (/docs/#configure).
|
|
.PP
|
|
Once you have set up the remote, check it is working with, for example
|
|
"rclone lsd remote:".
|
|
You may have called the remote something other than "remote:" \- just
|
|
substitute whatever you called it in the following instructions.
|
|
.PP
|
|
Now start the rclone restic server
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ serve\ restic\ \-v\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Where you can replace "backup" in the above by whatever path in the
|
|
remote you wish to use.
|
|
.PP
|
|
By default this will serve on "localhost:8080" you can change this with
|
|
use of the "\-\-addr" flag.
|
|
.PP
|
|
You might wish to start this server on boot.
|
|
.SS Setting up restic to use rclone
|
|
.PP
|
|
Now you can follow the restic
|
|
instructions (http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server)
|
|
on setting up restic.
|
|
.PP
|
|
Note that you will need restic 0.8.2 or later to interoperate with
|
|
rclone.
|
|
.PP
|
|
For the example above you will want to use "http://localhost:8080/" as
|
|
the URL for the REST server.
|
|
.PP
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ export\ RESTIC_REPOSITORY=rest:http://localhost:8080/
|
|
$\ export\ RESTIC_PASSWORD=yourpassword
|
|
$\ restic\ init
|
|
created\ restic\ backend\ 8b1a4b56ae\ at\ rest:http://localhost:8080/
|
|
|
|
Please\ note\ that\ knowledge\ of\ your\ password\ is\ required\ to\ access
|
|
the\ repository.\ Losing\ your\ password\ means\ that\ your\ data\ is
|
|
irrecoverably\ lost.
|
|
$\ restic\ backup\ /path/to/files/to/backup
|
|
scan\ [/path/to/files/to/backup]
|
|
scanned\ 189\ directories,\ 312\ files\ in\ 0:00
|
|
[0:00]\ 100.00%\ \ 38.128\ MiB\ /\ 38.128\ MiB\ \ 501\ /\ 501\ items\ \ 0\ errors\ \ ETA\ 0:00\
|
|
duration:\ 0:00
|
|
snapshot\ 45c8fdd8\ saved
|
|
\f[]
|
|
.fi
|
|
.SS Multiple repositories
|
|
.PP
|
|
Note that you can use the endpoint to host multiple repositories.
|
|
Do this by adding a directory name or path after the URL.
|
|
Note that these \f[B]must\f[] end with /.
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ export\ RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
|
|
#\ backup\ user1\ stuff
|
|
$\ export\ RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
|
|
#\ backup\ user2\ stuff
|
|
\f[]
|
|
.fi
|
|
.SS Server options
|
|
.PP
|
|
Use \-\-addr to specify which IP address and port the server should
|
|
listen on, eg \-\-addr 1.2.3.4:8000 or \-\-addr :8080 to listen to all
|
|
IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set \-\-addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised \- see the next section for info.
|
|
.PP
|
|
\-\-server\-read\-timeout and \-\-server\-write\-timeout can be used to
|
|
control the timeouts on the server.
|
|
Note that this is the total time for a transfer.
|
|
.PP
|
|
\-\-max\-header\-bytes controls the maximum number of bytes the server
|
|
will accept in the HTTP header.
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can either use an htpasswd file which can take lots of users, or set
|
|
a single username and password with the \-\-user and \-\-pass flags.
|
|
.PP
|
|
Use \-\-htpasswd /path/to/htpasswd to provide an htpasswd file.
|
|
This is in standard apache format and supports MD5, SHA1 and BCrypt for
|
|
basic authentication.
|
|
Bcrypt is recommended.
|
|
.PP
|
|
To create an htpasswd file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
touch\ htpasswd
|
|
htpasswd\ \-B\ htpasswd\ user
|
|
htpasswd\ \-B\ htpasswd\ anotherUser
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The password file can be updated while rclone is running.
|
|
.PP
|
|
Use \-\-realm to set the authentication realm.
|
|
.SS SSL/TLS
|
|
.PP
|
|
By default this will serve over http.
|
|
If you want you can serve over https.
|
|
You will need to supply the \-\-cert and \-\-key flags.
|
|
If you wish to do client side certificate validation then you will need
|
|
to supply \-\-client\-ca also.
|
|
.PP
|
|
\-\-cert should be a either a PEM encoded certificate or a concatenation
|
|
of that with the CA certificate.
|
|
\-\-key should be the PEM encoded private key and \-\-client\-ca should
|
|
be the PEM encoded client certificate authority certificate.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ serve\ restic\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-addr\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ IPaddress:Port\ or\ :Port\ to\ bind\ server\ to.\ (default\ "localhost:8080")
|
|
\ \ \ \ \ \ \-\-append\-only\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ disallow\ deletion\ of\ repository\ data
|
|
\ \ \ \ \ \ \-\-cert\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SSL\ PEM\ key\ (concatenation\ of\ certificate\ and\ CA\ certificate)
|
|
\ \ \ \ \ \ \-\-client\-ca\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Client\ certificate\ authority\ to\ verify\ clients\ with
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ restic
|
|
\ \ \ \ \ \ \-\-htpasswd\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ htpasswd\ file\ \-\ if\ not\ provided\ no\ authentication\ is\ done
|
|
\ \ \ \ \ \ \-\-key\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SSL\ PEM\ Private\ key
|
|
\ \ \ \ \ \ \-\-max\-header\-bytes\ int\ \ \ \ \ \ \ \ \ \ \ \ Maximum\ size\ of\ request\ header\ (default\ 4096)
|
|
\ \ \ \ \ \ \-\-pass\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Password\ for\ authentication.
|
|
\ \ \ \ \ \ \-\-realm\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ realm\ for\ authentication\ (default\ "rclone")
|
|
\ \ \ \ \ \ \-\-server\-read\-timeout\ duration\ \ \ \ Timeout\ for\ server\ reading\ data\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-server\-write\-timeout\ duration\ \ \ Timeout\ for\ server\ writing\ data\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-stdio\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ run\ an\ HTTP2\ server\ on\ stdin/stdout
|
|
\ \ \ \ \ \ \-\-user\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ User\ name\ for\ authentication.
|
|
\f[]
|
|
.fi
|
|
.SS rclone serve webdav
|
|
.PP
|
|
Serve remote:path over webdav.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone serve webdav implements a basic webdav server to serve the remote
|
|
over HTTP via the webdav protocol.
|
|
This can be viewed with a webdav client or you can make a remote of type
|
|
webdav to read and write it.
|
|
.PP
|
|
NB at the moment each directory listing reads the start of each file
|
|
which is undesirable: see https://github.com/golang/go/issues/22577
|
|
.SS Server options
|
|
.PP
|
|
Use \-\-addr to specify which IP address and port the server should
|
|
listen on, eg \-\-addr 1.2.3.4:8000 or \-\-addr :8080 to listen to all
|
|
IPs.
|
|
By default it only listens on localhost.
|
|
You can use port :0 to let the OS choose an available port.
|
|
.PP
|
|
If you set \-\-addr to listen on a public or LAN accessible IP address
|
|
then using Authentication is advised \- see the next section for info.
|
|
.PP
|
|
\-\-server\-read\-timeout and \-\-server\-write\-timeout can be used to
|
|
control the timeouts on the server.
|
|
Note that this is the total time for a transfer.
|
|
.PP
|
|
\-\-max\-header\-bytes controls the maximum number of bytes the server
|
|
will accept in the HTTP header.
|
|
.SS Authentication
|
|
.PP
|
|
By default this will serve files without needing a login.
|
|
.PP
|
|
You can either use an htpasswd file which can take lots of users, or set
|
|
a single username and password with the \-\-user and \-\-pass flags.
|
|
.PP
|
|
Use \-\-htpasswd /path/to/htpasswd to provide an htpasswd file.
|
|
This is in standard apache format and supports MD5, SHA1 and BCrypt for
|
|
basic authentication.
|
|
Bcrypt is recommended.
|
|
.PP
|
|
To create an htpasswd file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
touch\ htpasswd
|
|
htpasswd\ \-B\ htpasswd\ user
|
|
htpasswd\ \-B\ htpasswd\ anotherUser
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The password file can be updated while rclone is running.
|
|
.PP
|
|
Use \-\-realm to set the authentication realm.
|
|
.SS SSL/TLS
|
|
.PP
|
|
By default this will serve over http.
|
|
If you want you can serve over https.
|
|
You will need to supply the \-\-cert and \-\-key flags.
|
|
If you wish to do client side certificate validation then you will need
|
|
to supply \-\-client\-ca also.
|
|
.PP
|
|
\-\-cert should be a either a PEM encoded certificate or a concatenation
|
|
of that with the CA certificate.
|
|
\-\-key should be the PEM encoded private key and \-\-client\-ca should
|
|
be the PEM encoded client certificate authority certificate.
|
|
.SS Directory Cache
|
|
.PP
|
|
Using the \f[C]\-\-dir\-cache\-time\f[] flag, you can set how long a
|
|
directory should be considered up to date and not refreshed from the
|
|
backend.
|
|
Changes made locally in the mount may appear immediately or invalidate
|
|
the cache.
|
|
However, changes done on the remote will only be picked up once the
|
|
cache expires.
|
|
.PP
|
|
Alternatively, you can send a \f[C]SIGHUP\f[] signal to rclone for it to
|
|
flush all directory caches, regardless of how old they are.
|
|
Assuming only one rclone instance is running, you can reset the cache
|
|
like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill\ \-SIGHUP\ $(pidof\ rclone)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control (/rc) then you can use
|
|
rclone rc to flush the whole directory cache:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or individual files or directories:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget\ file=path/to/file\ dir=path/to/dir
|
|
\f[]
|
|
.fi
|
|
.SS File Caching
|
|
.PP
|
|
\f[B]NB\f[] File caching is \f[B]EXPERIMENTAL\f[] \- use with care!
|
|
.PP
|
|
These flags control the VFS file caching options.
|
|
The VFS layer is used by rclone mount to make a cloud storage system
|
|
work more like a normal file system.
|
|
.PP
|
|
You\[aq]ll need to enable VFS caching if you want, for example, to read
|
|
and write simultaneously to a file.
|
|
See below for more details.
|
|
.PP
|
|
Note that the VFS cache works in addition to the cache backend and you
|
|
may find that you need one or the other or both.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\-cache\-dir\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Directory\ rclone\ will\ use\ for\ caching.
|
|
\-\-vfs\-cache\-max\-age\ duration\ \ \ \ \ \ \ \ \ Max\ age\ of\ objects\ in\ the\ cache.\ (default\ 1h0m0s)
|
|
\-\-vfs\-cache\-mode\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ Cache\ mode\ off|minimal|writes|full\ (default\ "off")
|
|
\-\-vfs\-cache\-poll\-interval\ duration\ \ \ Interval\ to\ poll\ the\ cache\ for\ stale\ objects.\ (default\ 1m0s)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If run with \f[C]\-vv\f[] rclone will print the location of the file
|
|
cache.
|
|
The files are stored in the user cache file area which is OS dependent
|
|
but can be controlled with \f[C]\-\-cache\-dir\f[] or setting the
|
|
appropriate environment variable.
|
|
.PP
|
|
The cache has 4 different modes selected by
|
|
\f[C]\-\-vfs\-cache\-mode\f[].
|
|
The higher the cache mode the more compatible rclone becomes at the cost
|
|
of using disk space.
|
|
.PP
|
|
Note that files are written back to the remote only when they are closed
|
|
so if rclone is quit or dies with open files then these won\[aq]t get
|
|
written back to the remote.
|
|
However they will still be in the on disk cache.
|
|
.SS \-\-vfs\-cache\-mode off
|
|
.PP
|
|
In this mode the cache will read directly from the remote and write
|
|
directly to the remote without caching anything on disk.
|
|
.PP
|
|
This will mean some operations are not possible
|
|
.IP \[bu] 2
|
|
Files can\[aq]t be opened for both read AND write
|
|
.IP \[bu] 2
|
|
Files opened for write can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files open for read with O_TRUNC will be opened write only
|
|
.IP \[bu] 2
|
|
Files open for write only will behave as if O_TRUNC was supplied
|
|
.IP \[bu] 2
|
|
Open modes O_APPEND, O_TRUNC are ignored
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS \-\-vfs\-cache\-mode minimal
|
|
.PP
|
|
This is very similar to "off" except that files opened for read AND
|
|
write will be buffered to disks.
|
|
This means that files opened for write will be a lot more compatible,
|
|
but uses the minimal disk space.
|
|
.PP
|
|
These operations are not possible
|
|
.IP \[bu] 2
|
|
Files opened for write only can\[aq]t be seeked
|
|
.IP \[bu] 2
|
|
Existing files opened for write must have O_TRUNC set
|
|
.IP \[bu] 2
|
|
Files opened for write only will ignore O_APPEND, O_TRUNC
|
|
.IP \[bu] 2
|
|
If an upload fails it can\[aq]t be retried
|
|
.SS \-\-vfs\-cache\-mode writes
|
|
.PP
|
|
In this mode files opened for read only are still read directly from the
|
|
remote, write only and read/write files are buffered to disk first.
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload fails it will be retried up to \-\-low\-level\-retries
|
|
times.
|
|
.SS \-\-vfs\-cache\-mode full
|
|
.PP
|
|
In this mode all reads and writes are buffered to and from disk.
|
|
When a file is opened for read it will be downloaded in its entirety
|
|
first.
|
|
.PP
|
|
This may be appropriate for your needs, or you may prefer to look at the
|
|
cache backend which does a much more sophisticated job of caching,
|
|
including caching directory hierarchies and chunks of files.
|
|
.PP
|
|
In this mode, unlike the others, when a file is written to the disk, it
|
|
will be kept on the disk after it is written to the remote.
|
|
It will be purged on a schedule according to
|
|
\f[C]\-\-vfs\-cache\-max\-age\f[].
|
|
.PP
|
|
This mode should support all normal file system operations.
|
|
.PP
|
|
If an upload or download fails it will be retried up to
|
|
\-\-low\-level\-retries times.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ serve\ webdav\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \ \ \ \ \-\-addr\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ IPaddress:Port\ or\ :Port\ to\ bind\ server\ to.\ (default\ "localhost:8080")
|
|
\ \ \ \ \ \ \-\-cert\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SSL\ PEM\ key\ (concatenation\ of\ certificate\ and\ CA\ certificate)
|
|
\ \ \ \ \ \ \-\-client\-ca\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Client\ certificate\ authority\ to\ verify\ clients\ with
|
|
\ \ \ \ \ \ \-\-dir\-cache\-time\ duration\ \ \ \ \ \ \ \ \ \ \ \ Time\ to\ cache\ directory\ entries\ for.\ (default\ 5m0s)
|
|
\ \ \ \ \ \ \-\-gid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ gid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ webdav
|
|
\ \ \ \ \ \ \-\-htpasswd\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ htpasswd\ file\ \-\ if\ not\ provided\ no\ authentication\ is\ done
|
|
\ \ \ \ \ \ \-\-key\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ SSL\ PEM\ Private\ key
|
|
\ \ \ \ \ \ \-\-max\-header\-bytes\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Maximum\ size\ of\ request\ header\ (default\ 4096)
|
|
\ \ \ \ \ \ \-\-no\-checksum\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ compare\ checksums\ on\ up/download.
|
|
\ \ \ \ \ \ \-\-no\-modtime\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ read/write\ the\ modification\ time\ (can\ speed\ things\ up).
|
|
\ \ \ \ \ \ \-\-no\-seek\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Don\[aq]t\ allow\ seeking\ in\ files.
|
|
\ \ \ \ \ \ \-\-pass\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Password\ for\ authentication.
|
|
\ \ \ \ \ \ \-\-poll\-interval\ duration\ \ \ \ \ \ \ \ \ \ \ \ \ Time\ to\ wait\ between\ polling\ for\ changes.\ Must\ be\ smaller\ than\ dir\-cache\-time.\ Only\ on\ supported\ remotes.\ Set\ to\ 0\ to\ disable.\ (default\ 1m0s)
|
|
\ \ \ \ \ \ \-\-read\-only\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Mount\ read\-only.
|
|
\ \ \ \ \ \ \-\-realm\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ realm\ for\ authentication\ (default\ "rclone")
|
|
\ \ \ \ \ \ \-\-server\-read\-timeout\ duration\ \ \ \ \ \ \ Timeout\ for\ server\ reading\ data\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-server\-write\-timeout\ duration\ \ \ \ \ \ Timeout\ for\ server\ writing\ data\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-uid\ uint32\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ uid\ field\ set\ by\ the\ filesystem.\ (default\ 502)
|
|
\ \ \ \ \ \ \-\-umask\ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Override\ the\ permission\ bits\ set\ by\ the\ filesystem.\ (default\ 2)
|
|
\ \ \ \ \ \ \-\-user\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ User\ name\ for\ authentication.
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-max\-age\ duration\ \ \ \ \ \ \ \ \ Max\ age\ of\ objects\ in\ the\ cache.\ (default\ 1h0m0s)
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-mode\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ Cache\ mode\ off|minimal|writes|full\ (default\ "off")
|
|
\ \ \ \ \ \ \-\-vfs\-cache\-poll\-interval\ duration\ \ \ Interval\ to\ poll\ the\ cache\ for\ stale\ objects.\ (default\ 1m0s)
|
|
\ \ \ \ \ \ \-\-vfs\-read\-chunk\-size\ int\ \ \ \ \ \ \ \ \ \ \ \ Read\ the\ source\ objects\ in\ chunks.
|
|
\ \ \ \ \ \ \-\-vfs\-read\-chunk\-size\-limit\ int\ \ \ \ \ \ If\ greater\ than\ \-\-vfs\-read\-chunk\-size,\ double\ the\ chunk\ size\ after\ each\ chunk\ read,\ until\ the\ limit\ is\ reached.\ \-1\ is\ unlimited.
|
|
\f[]
|
|
.fi
|
|
.SS rclone touch
|
|
.PP
|
|
Create new file or change file modification time.
|
|
.SS Synopsis
|
|
.PP
|
|
Create new file or change file modification time.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ touch\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ help\ for\ touch
|
|
\ \ \-C,\ \-\-no\-create\ \ \ \ \ \ \ \ \ \ Do\ not\ create\ the\ file\ if\ it\ does\ not\ exist.
|
|
\ \ \-t,\ \-\-timestamp\ string\ \ \ Change\ the\ modification\ times\ to\ the\ specified\ time\ instead\ of\ the\ current\ time\ of\ day.\ The\ argument\ is\ of\ the\ form\ \[aq]YYMMDD\[aq]\ (ex.\ 17.10.30)\ or\ \[aq]YYYY\-MM\-DDTHH:MM:SS\[aq]\ (ex.\ 2006\-01\-02T15:04:05)
|
|
\f[]
|
|
.fi
|
|
.SS rclone tree
|
|
.PP
|
|
List the contents of the remote in a tree like fashion.
|
|
.SS Synopsis
|
|
.PP
|
|
rclone tree lists the contents of a remote in a similar way to the unix
|
|
tree command.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ tree\ remote:path
|
|
/
|
|
├──\ file1
|
|
├──\ file2
|
|
├──\ file3
|
|
└──\ subdir
|
|
\ \ \ \ ├──\ file4
|
|
\ \ \ \ └──\ file5
|
|
|
|
1\ directories,\ 5\ files
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can use any of the filtering options with the tree command (eg
|
|
\-\-include and \-\-exclude).
|
|
You can also use \-\-fast\-list.
|
|
.PP
|
|
The tree command has many options for controlling the listing which are
|
|
compatible with the tree command.
|
|
Note that not all of them have short options as they conflict with
|
|
rclone\[aq]s short options.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ tree\ remote:path\ [flags]
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ \ \-a,\ \-\-all\ \ \ \ \ \ \ \ \ \ \ \ \ All\ files\ are\ listed\ (list\ .\ files\ too).
|
|
\ \ \-C,\ \-\-color\ \ \ \ \ \ \ \ \ \ \ Turn\ colorization\ on\ always.
|
|
\ \ \-d,\ \-\-dirs\-only\ \ \ \ \ \ \ List\ directories\ only.
|
|
\ \ \ \ \ \ \-\-dirsfirst\ \ \ \ \ \ \ List\ directories\ before\ files\ (\-U\ disables).
|
|
\ \ \ \ \ \ \-\-full\-path\ \ \ \ \ \ \ Print\ the\ full\ path\ prefix\ for\ each\ file.
|
|
\ \ \-h,\ \-\-help\ \ \ \ \ \ \ \ \ \ \ \ help\ for\ tree
|
|
\ \ \ \ \ \ \-\-human\ \ \ \ \ \ \ \ \ \ \ Print\ the\ size\ in\ a\ more\ human\ readable\ way.
|
|
\ \ \ \ \ \ \-\-level\ int\ \ \ \ \ \ \ Descend\ only\ level\ directories\ deep.
|
|
\ \ \-D,\ \-\-modtime\ \ \ \ \ \ \ \ \ Print\ the\ date\ of\ last\ modification.
|
|
\ \ \-i,\ \-\-noindent\ \ \ \ \ \ \ \ Don\[aq]t\ print\ indentation\ lines.
|
|
\ \ \ \ \ \ \-\-noreport\ \ \ \ \ \ \ \ Turn\ off\ file/directory\ count\ at\ end\ of\ tree\ listing.
|
|
\ \ \-o,\ \-\-output\ string\ \ \ Output\ to\ file\ instead\ of\ stdout.
|
|
\ \ \-p,\ \-\-protections\ \ \ \ \ Print\ the\ protections\ for\ each\ file.
|
|
\ \ \-Q,\ \-\-quote\ \ \ \ \ \ \ \ \ \ \ Quote\ filenames\ with\ double\ quotes.
|
|
\ \ \-s,\ \-\-size\ \ \ \ \ \ \ \ \ \ \ \ Print\ the\ size\ in\ bytes\ of\ each\ file.
|
|
\ \ \ \ \ \ \-\-sort\ string\ \ \ \ \ Select\ sort:\ name,version,size,mtime,ctime.
|
|
\ \ \ \ \ \ \-\-sort\-ctime\ \ \ \ \ \ Sort\ files\ by\ last\ status\ change\ time.
|
|
\ \ \-t,\ \-\-sort\-modtime\ \ \ \ Sort\ files\ by\ last\ modification\ time.
|
|
\ \ \-r,\ \-\-sort\-reverse\ \ \ \ Reverse\ the\ order\ of\ the\ sort.
|
|
\ \ \-U,\ \-\-unsorted\ \ \ \ \ \ \ \ Leave\ files\ unsorted.
|
|
\ \ \ \ \ \ \-\-version\ \ \ \ \ \ \ \ \ Sort\ files\ alphanumerically\ by\ version.
|
|
\f[]
|
|
.fi
|
|
.SS Copying single files
|
|
.PP
|
|
rclone normally syncs or copies directories.
|
|
However, if the source remote points to a file, rclone will just copy
|
|
that file.
|
|
The destination remote must point to a directory \- rclone will give the
|
|
error
|
|
\f[C]Failed\ to\ create\ file\ system\ for\ "remote:file":\ is\ a\ file\ not\ a\ directory\f[]
|
|
if it isn\[aq]t.
|
|
.PP
|
|
For example, suppose you have a remote with a file in called
|
|
\f[C]test.jpg\f[], then you could copy just that file like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ remote:test.jpg\ /tmp/download
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The file \f[C]test.jpg\f[] will be placed inside \f[C]/tmp/download\f[].
|
|
.PP
|
|
This is equivalent to specifying
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-files\-from\ /tmp/files\ remote:\ /tmp/download
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Where \f[C]/tmp/files\f[] contains the single line
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
test.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
It is recommended to use \f[C]copy\f[] when copying individual files,
|
|
not \f[C]sync\f[].
|
|
They have pretty much the same effect but \f[C]copy\f[] will use a lot
|
|
less memory.
|
|
.SS Quoting and the shell
|
|
.PP
|
|
When you are typing commands to your computer you are using something
|
|
called the command line shell.
|
|
This interprets various characters in an OS specific way.
|
|
.PP
|
|
Here are some gotchas which may help users unfamiliar with the shell
|
|
rules
|
|
.SS Linux / OSX
|
|
.PP
|
|
If your names have spaces or shell metacharacters (eg \f[C]*\f[],
|
|
\f[C]?\f[], \f[C]$\f[], \f[C]\[aq]\f[], \f[C]"\f[] etc) then you must
|
|
quote them.
|
|
Use single quotes \f[C]\[aq]\f[] by default.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \[aq]Important\ files?\[aq]\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you want to send a \f[C]\[aq]\f[] you will need to use \f[C]"\f[], eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ "O\[aq]Reilly\ Reviews"\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The rules for quoting metacharacters are complicated and if you want the
|
|
full details you\[aq]ll have to consult the manual page for your shell.
|
|
.SS Windows
|
|
.PP
|
|
If your names have spaces in you need to put them in \f[C]"\f[], eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ "E:\\folder\ name\\folder\ name\\folder\ name"\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you are using the root directory on its own then don\[aq]t quote it
|
|
(see #464 (https://github.com/ncw/rclone/issues/464) for why), eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ E:\\\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Copying files or directories with \f[C]:\f[] in the names
|
|
.PP
|
|
rclone uses \f[C]:\f[] to mark a remote name.
|
|
This is, however, a valid filename component in non\-Windows OSes.
|
|
The remote name parser will only search for a \f[C]:\f[] up to the first
|
|
\f[C]/\f[] so if you need to act on a file or directory like this then
|
|
use the full path starting with a \f[C]/\f[], or use \f[C]\&./\f[] as a
|
|
current directory prefix.
|
|
.PP
|
|
So to sync a directory called \f[C]sync:me\f[] to a remote called
|
|
\f[C]remote:\f[] use
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ ./sync:me\ remote:path
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
or
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /full/path/to/sync:me\ remote:path
|
|
\f[]
|
|
.fi
|
|
.SS Server Side Copy
|
|
.PP
|
|
Most remotes (but not all \- see the
|
|
overview (/overview/#optional-features)) support server side copy.
|
|
.PP
|
|
This means if you want to copy one folder to another then rclone
|
|
won\[aq]t download all the files and re\-upload them; it will instruct
|
|
the server to copy them in place.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ s3:oldbucket\ s3:newbucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Will copy the contents of \f[C]oldbucket\f[] to \f[C]newbucket\f[]
|
|
without downloading and re\-uploading.
|
|
.PP
|
|
Remotes which don\[aq]t support server side copy \f[B]will\f[] download
|
|
and re\-upload in this case.
|
|
.PP
|
|
Server side copies are used with \f[C]sync\f[] and \f[C]copy\f[] and
|
|
will be identified in the log when using the \f[C]\-v\f[] flag.
|
|
The \f[C]move\f[] command may also use them if remote doesn\[aq]t
|
|
support server side move directly.
|
|
This is done by issuing a server side copy then a delete which is much
|
|
quicker than a download and re\-upload.
|
|
.PP
|
|
Server side copies will only be attempted if the remote names are the
|
|
same.
|
|
.PP
|
|
This can be used when scripting to make aged backups efficiently, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ remote:current\-backup\ remote:previous\-backup
|
|
rclone\ sync\ /path/to/files\ remote:current\-backup
|
|
\f[]
|
|
.fi
|
|
.SS Options
|
|
.PP
|
|
Rclone has a number of options to control its behaviour.
|
|
.PP
|
|
Options which use TIME use the go time parser.
|
|
A duration string is a possibly signed sequence of decimal numbers, each
|
|
with optional fraction and a unit suffix, such as "300ms", "\-1.5h" or
|
|
"2h45m".
|
|
Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
|
|
.PP
|
|
Options which use SIZE use kByte by default.
|
|
However, a suffix of \f[C]b\f[] for bytes, \f[C]k\f[] for kBytes,
|
|
\f[C]M\f[] for MBytes, \f[C]G\f[] for GBytes, \f[C]T\f[] for TBytes and
|
|
\f[C]P\f[] for PBytes may be used.
|
|
These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively.
|
|
.SS \-\-backup\-dir=DIR
|
|
.PP
|
|
When using \f[C]sync\f[], \f[C]copy\f[] or \f[C]move\f[] any files which
|
|
would have been overwritten or deleted are moved in their original
|
|
hierarchy into this directory.
|
|
.PP
|
|
If \f[C]\-\-suffix\f[] is set, then the moved files will have the suffix
|
|
added to them.
|
|
If there is a file with the same path (after the suffix has been added)
|
|
in DIR, then it will be overwritten.
|
|
.PP
|
|
The remote in use must support server side move or copy and you must use
|
|
the same remote as the destination of the sync.
|
|
The backup directory must not overlap the destination directory.
|
|
.PP
|
|
For example
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /path/to/local\ remote:current\ \-\-backup\-dir\ remote:old
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
will sync \f[C]/path/to/local\f[] to \f[C]remote:current\f[], but for
|
|
any files which would have been updated or deleted will be stored in
|
|
\f[C]remote:old\f[].
|
|
.PP
|
|
If running rclone from a script you might want to use today\[aq]s date
|
|
as the directory name passed to \f[C]\-\-backup\-dir\f[] to store the
|
|
old files, or you might want to pass \f[C]\-\-suffix\f[] with
|
|
today\[aq]s date.
|
|
.SS \-\-bind string
|
|
.PP
|
|
Local address to bind to for outgoing connections.
|
|
This can be an IPv4 address (1.2.3.4), an IPv6 address (1234::789A) or
|
|
host name.
|
|
If the host name doesn\[aq]t resolve or resolves to more than one IP
|
|
address it will give an error.
|
|
.SS \-\-bwlimit=BANDWIDTH_SPEC
|
|
.PP
|
|
This option controls the bandwidth limit.
|
|
Limits can be specified in two ways: As a single limit, or as a
|
|
timetable.
|
|
.PP
|
|
Single limits last for the duration of the session.
|
|
To use a single limit, specify the desired bandwidth in kBytes/s, or use
|
|
a suffix b|k|M|G.
|
|
The default is \f[C]0\f[] which means to not limit bandwidth.
|
|
.PP
|
|
For example, to limit bandwidth usage to 10 MBytes/s use
|
|
\f[C]\-\-bwlimit\ 10M\f[]
|
|
.PP
|
|
It is also possible to specify a "timetable" of limits, which will cause
|
|
certain limits to be applied at certain times.
|
|
To specify a timetable, format your entries as "HH:MM,BANDWIDTH
|
|
HH:MM,BANDWIDTH...".
|
|
.PP
|
|
An example of a typical timetable to avoid link saturation during
|
|
daytime working hours could be:
|
|
.PP
|
|
\f[C]\-\-bwlimit\ "08:00,512\ 12:00,10M\ 13:00,512\ 18:00,30M\ 23:00,off"\f[]
|
|
.PP
|
|
In this example, the transfer bandwidth will be set to 512kBytes/sec at
|
|
8am.
|
|
At noon, it will raise to 10Mbytes/s, and drop back to 512kBytes/sec at
|
|
1pm.
|
|
At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it
|
|
will be completely disabled (full speed).
|
|
Anything between 11pm and 8am will remain unlimited.
|
|
.PP
|
|
Bandwidth limits only apply to the data transfer.
|
|
They don\[aq]t apply to the bandwidth of the directory listings etc.
|
|
.PP
|
|
Note that the units are Bytes/s, not Bits/s.
|
|
Typically connections are measured in Bits/s \- to convert divide by 8.
|
|
For example, let\[aq]s say you have a 10 Mbit/s connection and you wish
|
|
rclone to use half of it \- 5 Mbit/s.
|
|
This is 5/8 = 0.625MByte/s so you would use a
|
|
\f[C]\-\-bwlimit\ 0.625M\f[] parameter for rclone.
|
|
.PP
|
|
On Unix systems (Linux, MacOS, \&...) the bandwidth limiter can be
|
|
toggled by sending a \f[C]SIGUSR2\f[] signal to rclone.
|
|
This allows to remove the limitations of a long running rclone transfer
|
|
and to restore it back to the value specified with \f[C]\-\-bwlimit\f[]
|
|
quickly when needed.
|
|
Assuming there is only one rclone instance running, you can toggle the
|
|
limiter like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
kill\ \-SIGUSR2\ $(pidof\ rclone)
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you configure rclone with a remote control (/rc) then you can use
|
|
change the bwlimit dynamically:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ core/bwlimit\ rate=1M
|
|
\f[]
|
|
.fi
|
|
.SS \-\-buffer\-size=SIZE
|
|
.PP
|
|
Use this sized buffer to speed up file transfers.
|
|
Each \f[C]\-\-transfer\f[] will use this much memory for buffering.
|
|
.PP
|
|
Set to 0 to disable the buffering for the minimum memory usage.
|
|
.SS \-\-checkers=N
|
|
.PP
|
|
The number of checkers to run in parallel.
|
|
Checkers do the equality checking of files during a sync.
|
|
For some storage systems (eg S3, Swift, Dropbox) this can take a
|
|
significant amount of time so they are run in parallel.
|
|
.PP
|
|
The default is to run 8 checkers in parallel.
|
|
.SS \-c, \-\-checksum
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check the file hash and size to
|
|
determine if files are equal.
|
|
.PP
|
|
This is useful when the remote doesn\[aq]t support setting modified time
|
|
and a more accurate sync is desired than just checking the file size.
|
|
.PP
|
|
This is very useful when transferring between remotes which store the
|
|
same hash type on the object, eg Drive and Swift.
|
|
For details of which remotes support which hash type see the table in
|
|
the overview section (https://rclone.org/overview/).
|
|
.PP
|
|
Eg \f[C]rclone\ \-\-checksum\ sync\ s3:/bucket\ swift:/bucket\f[] would
|
|
run much quicker than without the \f[C]\-\-checksum\f[] flag.
|
|
.PP
|
|
When using this flag, rclone won\[aq]t update mtimes of remote files if
|
|
they are incorrect as it would normally.
|
|
.SS \-\-config=CONFIG_FILE
|
|
.PP
|
|
Specify the location of the rclone config file.
|
|
.PP
|
|
Normally the config file is in your home directory as a file called
|
|
\f[C]\&.config/rclone/rclone.conf\f[] (or \f[C]\&.rclone.conf\f[] if
|
|
created with an older version).
|
|
If \f[C]$XDG_CONFIG_HOME\f[] is set it will be at
|
|
\f[C]$XDG_CONFIG_HOME/rclone/rclone.conf\f[]
|
|
.PP
|
|
If you run \f[C]rclone\ \-h\f[] and look at the help for the
|
|
\f[C]\-\-config\f[] option you will see where the default location is
|
|
for you.
|
|
.PP
|
|
Use this flag to override the config location, eg
|
|
\f[C]rclone\ \-\-config=".myconfig"\ .config\f[].
|
|
.SS \-\-contimeout=TIME
|
|
.PP
|
|
Set the connection timeout.
|
|
This should be in go time format which looks like \f[C]5s\f[] for 5
|
|
seconds, \f[C]10m\f[] for 10 minutes, or \f[C]3h30m\f[].
|
|
.PP
|
|
The connection timeout is the amount of time rclone will wait for a
|
|
connection to go through to a remote object storage system.
|
|
It is \f[C]1m\f[] by default.
|
|
.SS \-\-dedupe\-mode MODE
|
|
.PP
|
|
Mode to run dedupe command in.
|
|
One of \f[C]interactive\f[], \f[C]skip\f[], \f[C]first\f[],
|
|
\f[C]newest\f[], \f[C]oldest\f[], \f[C]rename\f[].
|
|
The default is \f[C]interactive\f[].
|
|
See the dedupe command for more information as to what these options
|
|
mean.
|
|
.SS \-\-disable FEATURE,FEATURE,...
|
|
.PP
|
|
This disables a comma separated list of optional features.
|
|
For example to disable server side move and server side copy use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\-disable\ move,copy
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The features can be put in in any case.
|
|
.PP
|
|
To see a list of which features can be disabled use:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\-disable\ help
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the overview features (/overview/#features) and optional
|
|
features (/overview/#optional-features) to get an idea of which feature
|
|
does what.
|
|
.PP
|
|
This flag can be useful for debugging and in exceptional circumstances
|
|
(eg Google Drive limiting the total volume of Server Side Copies to
|
|
100GB/day).
|
|
.SS \-n, \-\-dry\-run
|
|
.PP
|
|
Do a trial run with no permanent changes.
|
|
Use this to see what rclone would do without actually doing it.
|
|
Useful when setting up the \f[C]sync\f[] command which deletes files in
|
|
the destination.
|
|
.SS \-\-ignore\-checksum
|
|
.PP
|
|
Normally rclone will check that the checksums of transferred files
|
|
match, and give an error "corrupted on transfer" if they don\[aq]t.
|
|
.PP
|
|
You can use this option to skip that check.
|
|
You should only use it if you have had the "corrupted on transfer" error
|
|
message and you are sure you might want to transfer potentially
|
|
corrupted data.
|
|
.SS \-\-ignore\-existing
|
|
.PP
|
|
Using this option will make rclone unconditionally skip all files that
|
|
exist on the destination, no matter the content of these files.
|
|
.PP
|
|
While this isn\[aq]t a generally recommended option, it can be useful in
|
|
cases where your files change due to encryption.
|
|
However, it cannot correct partial transfers in case a transfer was
|
|
interrupted.
|
|
.SS \-\-ignore\-size
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check only the modification time.
|
|
If \f[C]\-\-checksum\f[] is set then it only checks the checksum.
|
|
.PP
|
|
It will also cause rclone to skip verifying the sizes are the same after
|
|
transfer.
|
|
.PP
|
|
This can be useful for transferring files to and from OneDrive which
|
|
occasionally misreports the size of image files (see
|
|
#399 (https://github.com/ncw/rclone/issues/399) for more info).
|
|
.SS \-I, \-\-ignore\-times
|
|
.PP
|
|
Using this option will cause rclone to unconditionally upload all files
|
|
regardless of the state of files on the destination.
|
|
.PP
|
|
Normally rclone would skip any files that have the same modification
|
|
time and are the same size (or have the same checksum if using
|
|
\f[C]\-\-checksum\f[]).
|
|
.SS \-\-immutable
|
|
.PP
|
|
Treat source and destination files as immutable and disallow
|
|
modification.
|
|
.PP
|
|
With this option set, files will be created and deleted as requested,
|
|
but existing files will never be updated.
|
|
If an existing file does not match between the source and destination,
|
|
rclone will give the error
|
|
\f[C]Source\ and\ destination\ exist\ but\ do\ not\ match:\ immutable\ file\ modified\f[].
|
|
.PP
|
|
Note that only commands which transfer files (e.g.
|
|
\f[C]sync\f[], \f[C]copy\f[], \f[C]move\f[]) are affected by this
|
|
behavior, and only modification is disallowed.
|
|
Files may still be deleted explicitly (e.g.
|
|
\f[C]delete\f[], \f[C]purge\f[]) or implicitly (e.g.
|
|
\f[C]sync\f[], \f[C]move\f[]).
|
|
Use \f[C]copy\ \-\-immutable\f[] if it is desired to avoid deletion as
|
|
well as modification.
|
|
.PP
|
|
This can be useful as an additional layer of protection for immutable or
|
|
append\-only data sets (notably backup archives), where modification
|
|
implies corruption and should not be propagated.
|
|
.SS \-\-leave\-root
|
|
.PP
|
|
During rmdirs it will not remove root directory, even if it\[aq]s empty.
|
|
.SS \-\-log\-file=FILE
|
|
.PP
|
|
Log all of rclone\[aq]s output to FILE.
|
|
This is not active by default.
|
|
This can be useful for tracking down problems with syncs in combination
|
|
with the \f[C]\-v\f[] flag.
|
|
See the Logging section (#logging) for more info.
|
|
.PP
|
|
Note that if you are using the \f[C]logrotate\f[] program to manage
|
|
rclone\[aq]s logs, then you should use the \f[C]copytruncate\f[] option
|
|
as rclone doesn\[aq]t have a signal to rotate logs.
|
|
.SS \-\-log\-level LEVEL
|
|
.PP
|
|
This sets the log level for rclone.
|
|
The default log level is \f[C]NOTICE\f[].
|
|
.PP
|
|
\f[C]DEBUG\f[] is equivalent to \f[C]\-vv\f[].
|
|
It outputs lots of debug info \- useful for bug reports and really
|
|
finding out what rclone is doing.
|
|
.PP
|
|
\f[C]INFO\f[] is equivalent to \f[C]\-v\f[].
|
|
It outputs information about each transfer and prints stats once a
|
|
minute by default.
|
|
.PP
|
|
\f[C]NOTICE\f[] is the default log level if no logging flags are
|
|
supplied.
|
|
It outputs very little when things are working normally.
|
|
It outputs warnings and significant events.
|
|
.PP
|
|
\f[C]ERROR\f[] is equivalent to \f[C]\-q\f[].
|
|
It only outputs error messages.
|
|
.SS \-\-low\-level\-retries NUMBER
|
|
.PP
|
|
This controls the number of low level retries rclone does.
|
|
.PP
|
|
A low level retry is used to retry a failing operation \- typically one
|
|
HTTP request.
|
|
This might be uploading a chunk of a big file for example.
|
|
You will see low level retries in the log with the \f[C]\-v\f[] flag.
|
|
.PP
|
|
This shouldn\[aq]t need to be changed from the default in normal
|
|
operations.
|
|
However, if you get a lot of low level retries you may wish to reduce
|
|
the value so rclone moves on to a high level retry (see the
|
|
\f[C]\-\-retries\f[] flag) quicker.
|
|
.PP
|
|
Disable low level retries with \f[C]\-\-low\-level\-retries\ 1\f[].
|
|
.SS \-\-max\-delete=N
|
|
.PP
|
|
This tells rclone not to delete more than N files.
|
|
If that limit is exceeded then a fatal error will be generated and
|
|
rclone will stop the operation in progress.
|
|
.SS \-\-max\-depth=N
|
|
.PP
|
|
This modifies the recursion depth for all the commands except purge.
|
|
.PP
|
|
So if you do \f[C]rclone\ \-\-max\-depth\ 1\ ls\ remote:path\f[] you
|
|
will see only the files in the top level directory.
|
|
Using \f[C]\-\-max\-depth\ 2\f[] means you will see all the files in
|
|
first two directory levels and so on.
|
|
.PP
|
|
For historical reasons the \f[C]lsd\f[] command defaults to using a
|
|
\f[C]\-\-max\-depth\f[] of 1 \- you can override this with the command
|
|
line flag.
|
|
.PP
|
|
You can use this command to disable recursion (with
|
|
\f[C]\-\-max\-depth\ 1\f[]).
|
|
.PP
|
|
Note that if you use this with \f[C]sync\f[] and
|
|
\f[C]\-\-delete\-excluded\f[] the files not recursed through are
|
|
considered excluded and will be deleted on the destination.
|
|
Test first with \f[C]\-\-dry\-run\f[] if you are not sure what will
|
|
happen.
|
|
.SS \-\-max\-transfer=SIZE
|
|
.PP
|
|
Rclone will stop transferring when it has reached the size specified.
|
|
Defaults to off.
|
|
.PP
|
|
When the limit is reached all transfers will stop immediately.
|
|
.PP
|
|
Rclone will exit with exit code 8 if the transfer limit is reached.
|
|
.SS \-\-modify\-window=TIME
|
|
.PP
|
|
When checking whether a file has been modified, this is the maximum
|
|
allowed time difference that a file can have and still be considered
|
|
equivalent.
|
|
.PP
|
|
The default is \f[C]1ns\f[] unless this is overridden by a remote.
|
|
For example OS X only stores modification times to the nearest second so
|
|
if you are reading and writing to an OS X filing system this will be
|
|
\f[C]1s\f[] by default.
|
|
.PP
|
|
This command line flag allows you to override that computed default.
|
|
.SS \-\-no\-gzip\-encoding
|
|
.PP
|
|
Don\[aq]t set \f[C]Accept\-Encoding:\ gzip\f[].
|
|
This means that rclone won\[aq]t ask the server for compressed files
|
|
automatically.
|
|
Useful if you\[aq]ve set the server to return files with
|
|
\f[C]Content\-Encoding:\ gzip\f[] but you uploaded compressed files.
|
|
.PP
|
|
There is no need to set this in normal operation, and doing so will
|
|
decrease the network transfer efficiency of rclone.
|
|
.SS \-\-no\-update\-modtime
|
|
.PP
|
|
When using this flag, rclone won\[aq]t update modification times of
|
|
remote files if they are incorrect as it would normally.
|
|
.PP
|
|
This can be used if the remote is being synced with another tool also
|
|
(eg the Google Drive client).
|
|
.SS \-q, \-\-quiet
|
|
.PP
|
|
Normally rclone outputs stats and a completion message.
|
|
If you set this flag it will make as little output as possible.
|
|
.SS \-\-retries int
|
|
.PP
|
|
Retry the entire sync if it fails this many times it fails (default 3).
|
|
.PP
|
|
Some remotes can be unreliable and a few retries help pick up the files
|
|
which didn\[aq]t get transferred because of errors.
|
|
.PP
|
|
Disable retries with \f[C]\-\-retries\ 1\f[].
|
|
.SS \-\-retries\-sleep=TIME
|
|
.PP
|
|
This sets the interval between each retry specified by
|
|
\f[C]\-\-retries\f[]
|
|
.PP
|
|
The default is 0.
|
|
Use 0 to disable.
|
|
.SS \-\-size\-only
|
|
.PP
|
|
Normally rclone will look at modification time and size of files to see
|
|
if they are equal.
|
|
If you set this flag then rclone will check only the size.
|
|
.PP
|
|
This can be useful transferring files from Dropbox which have been
|
|
modified by the desktop sync client which doesn\[aq]t set checksums of
|
|
modification times in the same way as rclone.
|
|
.SS \-\-stats=TIME
|
|
.PP
|
|
Commands which transfer data (\f[C]sync\f[], \f[C]copy\f[],
|
|
\f[C]copyto\f[], \f[C]move\f[], \f[C]moveto\f[]) will print data
|
|
transfer stats at regular intervals to show their progress.
|
|
.PP
|
|
This sets the interval.
|
|
.PP
|
|
The default is \f[C]1m\f[].
|
|
Use 0 to disable.
|
|
.PP
|
|
If you set the stats interval then all commands can show stats.
|
|
This can be useful when running other commands, \f[C]check\f[] or
|
|
\f[C]mount\f[] for example.
|
|
.PP
|
|
Stats are logged at \f[C]INFO\f[] level by default which means they
|
|
won\[aq]t show at default log level \f[C]NOTICE\f[].
|
|
Use \f[C]\-\-stats\-log\-level\ NOTICE\f[] or \f[C]\-v\f[] to make them
|
|
show.
|
|
See the Logging section (#logging) for more info on log levels.
|
|
.PP
|
|
Note that on macOS you can send a SIGINFO (which is normally ctrl\-T in
|
|
the terminal) to make the stats print immediately.
|
|
.SS \-\-stats\-file\-name\-length integer
|
|
.PP
|
|
By default, the \f[C]\-\-stats\f[] output will truncate file names and
|
|
paths longer than 40 characters.
|
|
This is equivalent to providing
|
|
\f[C]\-\-stats\-file\-name\-length\ 40\f[].
|
|
Use \f[C]\-\-stats\-file\-name\-length\ 0\f[] to disable any truncation
|
|
of file names printed by stats.
|
|
.SS \-\-stats\-log\-level string
|
|
.PP
|
|
Log level to show \f[C]\-\-stats\f[] output at.
|
|
This can be \f[C]DEBUG\f[], \f[C]INFO\f[], \f[C]NOTICE\f[], or
|
|
\f[C]ERROR\f[].
|
|
The default is \f[C]INFO\f[].
|
|
This means at the default level of logging which is \f[C]NOTICE\f[] the
|
|
stats won\[aq]t show \- if you want them to then use
|
|
\f[C]\-\-stats\-log\-level\ NOTICE\f[].
|
|
See the Logging section (#logging) for more info on log levels.
|
|
.SS \-\-stats\-unit=bits|bytes
|
|
.PP
|
|
By default, data transfer rates will be printed in bytes/second.
|
|
.PP
|
|
This option allows the data rate to be printed in bits/second.
|
|
.PP
|
|
Data transfer volume will still be reported in bytes.
|
|
.PP
|
|
The rate is reported as a binary unit, not SI unit.
|
|
So 1 Mbit/s equals 1,048,576 bits/s and not 1,000,000 bits/s.
|
|
.PP
|
|
The default is \f[C]bytes\f[].
|
|
.SS \-\-suffix=SUFFIX
|
|
.PP
|
|
This is for use with \f[C]\-\-backup\-dir\f[] only.
|
|
If this isn\[aq]t set then \f[C]\-\-backup\-dir\f[] will move files with
|
|
their original name.
|
|
If it is set then the files will have SUFFIX added on to them.
|
|
.PP
|
|
See \f[C]\-\-backup\-dir\f[] for more info.
|
|
.SS \-\-syslog
|
|
.PP
|
|
On capable OSes (not Windows or Plan9) send all log output to syslog.
|
|
.PP
|
|
This can be useful for running rclone in a script or
|
|
\f[C]rclone\ mount\f[].
|
|
.SS \-\-syslog\-facility string
|
|
.PP
|
|
If using \f[C]\-\-syslog\f[] this sets the syslog facility (eg
|
|
\f[C]KERN\f[], \f[C]USER\f[]).
|
|
See \f[C]man\ syslog\f[] for a list of possible facilities.
|
|
The default facility is \f[C]DAEMON\f[].
|
|
.SS \-\-tpslimit float
|
|
.PP
|
|
Limit HTTP transactions per second to this.
|
|
Default is 0 which is used to mean unlimited transactions per second.
|
|
.PP
|
|
For example to limit rclone to 10 HTTP transactions per second use
|
|
\f[C]\-\-tpslimit\ 10\f[], or to 1 transaction every 2 seconds use
|
|
\f[C]\-\-tpslimit\ 0.5\f[].
|
|
.PP
|
|
Use this when the number of transactions per second from rclone is
|
|
causing a problem with the cloud storage provider (eg getting you banned
|
|
or rate limited).
|
|
.PP
|
|
This can be very useful for \f[C]rclone\ mount\f[] to control the
|
|
behaviour of applications using it.
|
|
.PP
|
|
See also \f[C]\-\-tpslimit\-burst\f[].
|
|
.SS \-\-tpslimit\-burst int
|
|
.PP
|
|
Max burst of transactions for \f[C]\-\-tpslimit\f[].
|
|
(default 1)
|
|
.PP
|
|
Normally \f[C]\-\-tpslimit\f[] will do exactly the number of transaction
|
|
per second specified.
|
|
However if you supply \f[C]\-\-tps\-burst\f[] then rclone can save up
|
|
some transactions from when it was idle giving a burst of up to the
|
|
parameter supplied.
|
|
.PP
|
|
For example if you provide \f[C]\-\-tpslimit\-burst\ 10\f[] then if
|
|
rclone has been idle for more than 10*\f[C]\-\-tpslimit\f[] then it can
|
|
do 10 transactions very quickly before they are limited again.
|
|
.PP
|
|
This may be used to increase performance of \f[C]\-\-tpslimit\f[]
|
|
without changing the long term average number of transactions per
|
|
second.
|
|
.SS \-\-track\-renames
|
|
.PP
|
|
By default, rclone doesn\[aq]t keep track of renamed files, so if you
|
|
rename a file locally then sync it to a remote, rclone will delete the
|
|
old file on the remote and upload a new copy.
|
|
.PP
|
|
If you use this flag, and the remote supports server side copy or server
|
|
side move, and the source and destination have a compatible hash, then
|
|
this will track renames during \f[C]sync\f[], \f[C]copy\f[], and
|
|
\f[C]move\f[] operations and perform renaming server\-side.
|
|
.PP
|
|
Files will be matched by size and hash \- if both match then a rename
|
|
will be considered.
|
|
.PP
|
|
If the destination does not support server\-side copy or move, rclone
|
|
will fall back to the default behaviour and log an error level message
|
|
to the console.
|
|
.PP
|
|
Note that \f[C]\-\-track\-renames\f[] uses extra memory to keep track of
|
|
all the rename candidates.
|
|
.PP
|
|
Note also that \f[C]\-\-track\-renames\f[] is incompatible with
|
|
\f[C]\-\-delete\-before\f[] and will select \f[C]\-\-delete\-after\f[]
|
|
instead of \f[C]\-\-delete\-during\f[].
|
|
.SS \-\-delete\-(before,during,after)
|
|
.PP
|
|
This option allows you to specify when files on your destination are
|
|
deleted when you sync folders.
|
|
.PP
|
|
Specifying the value \f[C]\-\-delete\-before\f[] will delete all files
|
|
present on the destination, but not on the source \f[I]before\f[]
|
|
starting the transfer of any new or updated files.
|
|
This uses two passes through the file systems, one for the deletions and
|
|
one for the copies.
|
|
.PP
|
|
Specifying \f[C]\-\-delete\-during\f[] will delete files while checking
|
|
and uploading files.
|
|
This is the fastest option and uses the least memory.
|
|
.PP
|
|
Specifying \f[C]\-\-delete\-after\f[] (the default value) will delay
|
|
deletion of files until all new/updated files have been successfully
|
|
transferred.
|
|
The files to be deleted are collected in the copy pass then deleted
|
|
after the copy pass has completed successfully.
|
|
The files to be deleted are held in memory so this mode may use more
|
|
memory.
|
|
This is the safest mode as it will only delete files if there have been
|
|
no errors subsequent to that.
|
|
If there have been errors before the deletions start then you will get
|
|
the message \f[C]not\ deleting\ files\ as\ there\ were\ IO\ errors\f[].
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
When doing anything which involves a directory listing (eg
|
|
\f[C]sync\f[], \f[C]copy\f[], \f[C]ls\f[] \- in fact nearly every
|
|
command), rclone normally lists a directory and processes it before
|
|
using more directory lists to process any subdirectories.
|
|
This can be parallelised and works very quickly using the least amount
|
|
of memory.
|
|
.PP
|
|
However, some remotes have a way of listing all files beneath a
|
|
directory in one (or a small number) of transactions.
|
|
These tend to be the bucket based remotes (eg S3, B2, GCS, Swift,
|
|
Hubic).
|
|
.PP
|
|
If you use the \f[C]\-\-fast\-list\f[] flag then rclone will use this
|
|
method for listing directories.
|
|
This will have the following consequences for the listing:
|
|
.IP \[bu] 2
|
|
It \f[B]will\f[] use fewer transactions (important if you pay for them)
|
|
.IP \[bu] 2
|
|
It \f[B]will\f[] use more memory.
|
|
Rclone has to load the whole listing into memory.
|
|
.IP \[bu] 2
|
|
It \f[I]may\f[] be faster because it uses fewer transactions
|
|
.IP \[bu] 2
|
|
It \f[I]may\f[] be slower because it can\[aq]t be parallelized
|
|
.PP
|
|
rclone should always give identical results with and without
|
|
\f[C]\-\-fast\-list\f[].
|
|
.PP
|
|
If you pay for transactions and can fit your entire sync listing into
|
|
memory then \f[C]\-\-fast\-list\f[] is recommended.
|
|
If you have a very big sync to do then don\[aq]t use
|
|
\f[C]\-\-fast\-list\f[] otherwise you will run out of memory.
|
|
.PP
|
|
If you use \f[C]\-\-fast\-list\f[] on a remote which doesn\[aq]t support
|
|
it, then rclone will just ignore it.
|
|
.SS \-\-timeout=TIME
|
|
.PP
|
|
This sets the IO idle timeout.
|
|
If a transfer has started but then becomes idle for this long it is
|
|
considered broken and disconnected.
|
|
.PP
|
|
The default is \f[C]5m\f[].
|
|
Set to 0 to disable.
|
|
.SS \-\-transfers=N
|
|
.PP
|
|
The number of file transfers to run in parallel.
|
|
It can sometimes be useful to set this to a smaller number if the remote
|
|
is giving a lot of timeouts or bigger if you have lots of bandwidth and
|
|
a fast remote.
|
|
.PP
|
|
The default is to run 4 file transfers in parallel.
|
|
.SS \-u, \-\-update
|
|
.PP
|
|
This forces rclone to skip any files which exist on the destination and
|
|
have a modified time that is newer than the source file.
|
|
.PP
|
|
If an existing destination file has a modification time equal (within
|
|
the computed modify window precision) to the source file\[aq]s, it will
|
|
be updated if the sizes are different.
|
|
.PP
|
|
On remotes which don\[aq]t support mod time directly the time checked
|
|
will be the uploaded time.
|
|
This means that if uploading to one of these remotes, rclone will skip
|
|
any files which exist on the destination and have an uploaded time that
|
|
is newer than the modification time of the source file.
|
|
.PP
|
|
This can be useful when transferring to a remote which doesn\[aq]t
|
|
support mod times directly as it is more accurate than a
|
|
\f[C]\-\-size\-only\f[] check and faster than using
|
|
\f[C]\-\-checksum\f[].
|
|
.SS \-\-use\-server\-modtime
|
|
.PP
|
|
Some object\-store backends (e.g, Swift, S3) do not preserve file
|
|
modification times (modtime).
|
|
On these backends, rclone stores the original modtime as additional
|
|
metadata on the object.
|
|
By default it will make an API call to retrieve the metadata when the
|
|
modtime is needed by an operation.
|
|
.PP
|
|
Use this flag to disable the extra API call and rely instead on the
|
|
server\[aq]s modified time.
|
|
In cases such as a local to remote sync, knowing the local file is newer
|
|
than the time it was last uploaded to the remote is sufficient.
|
|
In those cases, this flag can speed up the process and reduce the number
|
|
of API calls necessary.
|
|
.SS \-v, \-vv, \-\-verbose
|
|
.PP
|
|
With \f[C]\-v\f[] rclone will tell you about each file that is
|
|
transferred and a small number of significant events.
|
|
.PP
|
|
With \f[C]\-vv\f[] rclone will become very verbose telling you about
|
|
every file it considers and transfers.
|
|
Please send bug reports with a log with this setting.
|
|
.SS \-V, \-\-version
|
|
.PP
|
|
Prints the version number
|
|
.SS Configuration Encryption
|
|
.PP
|
|
Your configuration file contains information for logging in to your
|
|
cloud services.
|
|
This means that you should keep your \f[C]\&.rclone.conf\f[] file in a
|
|
secure location.
|
|
.PP
|
|
If you are in an environment where that isn\[aq]t possible, you can add
|
|
a password to your configuration.
|
|
This means that you will have to enter the password every time you start
|
|
rclone.
|
|
.PP
|
|
To add a password to your rclone configuration, execute
|
|
\f[C]rclone\ config\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
>rclone\ config
|
|
Current\ remotes:
|
|
|
|
e)\ Edit\ existing\ remote
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
e/n/d/s/q>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Go into \f[C]s\f[], Set configuration password:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
e/n/d/s/q>\ s
|
|
Your\ configuration\ is\ not\ encrypted.
|
|
If\ you\ add\ a\ password,\ you\ will\ protect\ your\ login\ information\ to\ cloud\ services.
|
|
a)\ Add\ Password
|
|
q)\ Quit\ to\ main\ menu
|
|
a/q>\ a
|
|
Enter\ NEW\ configuration\ password:
|
|
password:
|
|
Confirm\ NEW\ password:
|
|
password:
|
|
Password\ set
|
|
Your\ configuration\ is\ encrypted.
|
|
c)\ Change\ Password
|
|
u)\ Unencrypt\ configuration
|
|
q)\ Quit\ to\ main\ menu
|
|
c/u/q>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Your configuration is now encrypted, and every time you start rclone you
|
|
will now be asked for the password.
|
|
In the same menu, you can change the password or completely remove
|
|
encryption from your configuration.
|
|
.PP
|
|
There is no way to recover the configuration if you lose your password.
|
|
.PP
|
|
rclone uses nacl
|
|
secretbox (https://godoc.org/golang.org/x/crypto/nacl/secretbox) which
|
|
in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your
|
|
configuration with secret\-key cryptography.
|
|
The password is SHA\-256 hashed, which produces the key for secretbox.
|
|
The hashed password is not stored.
|
|
.PP
|
|
While this provides very good security, we do not recommend storing your
|
|
encrypted rclone configuration in public if it contains sensitive
|
|
information, maybe except if you use a very strong password.
|
|
.PP
|
|
If it is safe in your environment, you can set the
|
|
\f[C]RCLONE_CONFIG_PASS\f[] environment variable to contain your
|
|
password, in which case it will be used for decrypting the
|
|
configuration.
|
|
.PP
|
|
You can set this for a session from a script.
|
|
For unix like systems save this to a file called
|
|
\f[C]set\-rclone\-password\f[]:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#!/bin/echo\ Source\ this\ file\ don\[aq]t\ run\ it
|
|
|
|
read\ \-s\ RCLONE_CONFIG_PASS
|
|
export\ RCLONE_CONFIG_PASS
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then source the file when you want to use it.
|
|
From the shell you would do \f[C]source\ set\-rclone\-password\f[].
|
|
It will then ask you for the password and set it in the environment
|
|
variable.
|
|
.PP
|
|
If you are running rclone inside a script, you might want to disable
|
|
password prompts.
|
|
To do that, pass the parameter \f[C]\-\-ask\-password=false\f[] to
|
|
rclone.
|
|
This will make rclone fail instead of asking for a password if
|
|
\f[C]RCLONE_CONFIG_PASS\f[] doesn\[aq]t contain a valid password.
|
|
.SS Developer options
|
|
.PP
|
|
These options are useful when developing or debugging rclone.
|
|
There are also some more remote specific options which aren\[aq]t
|
|
documented here which are used for testing.
|
|
These start with remote name eg \f[C]\-\-drive\-test\-option\f[] \- see
|
|
the docs for the remote in question.
|
|
.SS \-\-cpuprofile=FILE
|
|
.PP
|
|
Write CPU profile to file.
|
|
This can be analysed with \f[C]go\ tool\ pprof\f[].
|
|
.SS \-\-dump flag,flag,flag
|
|
.PP
|
|
The \f[C]\-\-dump\f[] flag takes a comma separated list of flags to dump
|
|
info about.
|
|
These are:
|
|
.SS \-\-dump headers
|
|
.PP
|
|
Dump HTTP headers with \f[C]Authorization:\f[] lines removed.
|
|
May still contain sensitive info.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.PP
|
|
Use \f[C]\-\-dump\ auth\f[] if you do want the \f[C]Authorization:\f[]
|
|
headers.
|
|
.SS \-\-dump bodies
|
|
.PP
|
|
Dump HTTP headers and bodies \- may contain sensitive info.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.PP
|
|
Note that the bodies are buffered in memory so don\[aq]t use this for
|
|
enormous files.
|
|
.SS \-\-dump requests
|
|
.PP
|
|
Like \f[C]\-\-dump\ bodies\f[] but dumps the request bodies and the
|
|
response headers.
|
|
Useful for debugging download problems.
|
|
.SS \-\-dump responses
|
|
.PP
|
|
Like \f[C]\-\-dump\ bodies\f[] but dumps the response bodies and the
|
|
request headers.
|
|
Useful for debugging upload problems.
|
|
.SS \-\-dump auth
|
|
.PP
|
|
Dump HTTP headers \- will contain sensitive info such as
|
|
\f[C]Authorization:\f[] headers \- use \f[C]\-\-dump\ headers\f[] to
|
|
dump without \f[C]Authorization:\f[] headers.
|
|
Can be very verbose.
|
|
Useful for debugging only.
|
|
.SS \-\-dump filters
|
|
.PP
|
|
Dump the filters to the output.
|
|
Useful to see exactly what include and exclude options are filtering on.
|
|
.SS \-\-dump goroutines
|
|
.PP
|
|
This dumps a list of the running go\-routines at the end of the command
|
|
to standard output.
|
|
.SS \-\-dump openfiles
|
|
.PP
|
|
This dumps a list of the open files at the end of the command.
|
|
It uses the \f[C]lsof\f[] command to do that so you\[aq]ll need that
|
|
installed to use it.
|
|
.SS \-\-memprofile=FILE
|
|
.PP
|
|
Write memory profile to file.
|
|
This can be analysed with \f[C]go\ tool\ pprof\f[].
|
|
.SS \-\-no\-check\-certificate=true/false
|
|
.PP
|
|
\f[C]\-\-no\-check\-certificate\f[] controls whether a client verifies
|
|
the server\[aq]s certificate chain and host name.
|
|
If \f[C]\-\-no\-check\-certificate\f[] is true, TLS accepts any
|
|
certificate presented by the server and any host name in that
|
|
certificate.
|
|
In this mode, TLS is susceptible to man\-in\-the\-middle attacks.
|
|
.PP
|
|
This option defaults to \f[C]false\f[].
|
|
.PP
|
|
\f[B]This should be used only for testing.\f[]
|
|
.SS Filtering
|
|
.PP
|
|
For the filtering options
|
|
.IP \[bu] 2
|
|
\f[C]\-\-delete\-excluded\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-files\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-min\-size\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-max\-size\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-min\-age\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-max\-age\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dump\ filters\f[]
|
|
.PP
|
|
See the filtering section (https://rclone.org/filtering/).
|
|
.SS Remote control
|
|
.PP
|
|
For the remote control options and for instructions on how to remote
|
|
control rclone
|
|
.IP \[bu] 2
|
|
\f[C]\-\-rc\f[]
|
|
.IP \[bu] 2
|
|
and anything starting with \f[C]\-\-rc\-\f[]
|
|
.PP
|
|
See the remote control section (https://rclone.org/rc/).
|
|
.SS Logging
|
|
.PP
|
|
rclone has 4 levels of logging, \f[C]ERROR\f[], \f[C]NOTICE\f[],
|
|
\f[C]INFO\f[] and \f[C]DEBUG\f[].
|
|
.PP
|
|
By default, rclone logs to standard error.
|
|
This means you can redirect standard error and still see the normal
|
|
output of rclone commands (eg \f[C]rclone\ ls\f[]).
|
|
.PP
|
|
By default, rclone will produce \f[C]Error\f[] and \f[C]Notice\f[] level
|
|
messages.
|
|
.PP
|
|
If you use the \f[C]\-q\f[] flag, rclone will only produce
|
|
\f[C]Error\f[] messages.
|
|
.PP
|
|
If you use the \f[C]\-v\f[] flag, rclone will produce \f[C]Error\f[],
|
|
\f[C]Notice\f[] and \f[C]Info\f[] messages.
|
|
.PP
|
|
If you use the \f[C]\-vv\f[] flag, rclone will produce \f[C]Error\f[],
|
|
\f[C]Notice\f[], \f[C]Info\f[] and \f[C]Debug\f[] messages.
|
|
.PP
|
|
You can also control the log levels with the \f[C]\-\-log\-level\f[]
|
|
flag.
|
|
.PP
|
|
If you use the \f[C]\-\-log\-file=FILE\f[] option, rclone will redirect
|
|
\f[C]Error\f[], \f[C]Info\f[] and \f[C]Debug\f[] messages along with
|
|
standard error to FILE.
|
|
.PP
|
|
If you use the \f[C]\-\-syslog\f[] flag then rclone will log to syslog
|
|
and the \f[C]\-\-syslog\-facility\f[] control which facility it uses.
|
|
.PP
|
|
Rclone prefixes all log messages with their level in capitals, eg INFO
|
|
which makes it easy to grep the log file for different kinds of
|
|
information.
|
|
.SS Exit Code
|
|
.PP
|
|
If any errors occur during the command execution, rclone will exit with
|
|
a non\-zero exit code.
|
|
This allows scripts to detect when rclone operations have failed.
|
|
.PP
|
|
During the startup phase, rclone will exit immediately if an error is
|
|
detected in the configuration.
|
|
There will always be a log message immediately before exiting.
|
|
.PP
|
|
When rclone is running it will accumulate errors as it goes along, and
|
|
only exit with a non\-zero exit code if (after retries) there were still
|
|
failed transfers.
|
|
For every error counted there will be a high priority log message
|
|
(visible with \f[C]\-q\f[]) showing the message and which file caused
|
|
the problem.
|
|
A high priority message is also shown when starting a retry so the user
|
|
can see that any previous error messages may not be valid after the
|
|
retry.
|
|
If rclone has done a retry it will log a high priority message if the
|
|
retry was successful.
|
|
.SS List of exit codes
|
|
.IP \[bu] 2
|
|
\f[C]0\f[] \- success
|
|
.IP \[bu] 2
|
|
\f[C]1\f[] \- Syntax or usage error
|
|
.IP \[bu] 2
|
|
\f[C]2\f[] \- Error not otherwise categorised
|
|
.IP \[bu] 2
|
|
\f[C]3\f[] \- Directory not found
|
|
.IP \[bu] 2
|
|
\f[C]4\f[] \- File not found
|
|
.IP \[bu] 2
|
|
\f[C]5\f[] \- Temporary error (one that more retries might fix) (Retry
|
|
errors)
|
|
.IP \[bu] 2
|
|
\f[C]6\f[] \- Less serious errors (like 461 errors from dropbox)
|
|
(NoRetry errors)
|
|
.IP \[bu] 2
|
|
\f[C]7\f[] \- Fatal error (one that more retries won\[aq]t fix, like
|
|
account suspended) (Fatal errors)
|
|
.IP \[bu] 2
|
|
\f[C]8\f[] \- Transfer exceeded \- limit set by \-\-max\-transfer
|
|
reached
|
|
.SS Environment Variables
|
|
.PP
|
|
Rclone can be configured entirely using environment variables.
|
|
These can be used to set defaults for options or config file entries.
|
|
.SS Options
|
|
.PP
|
|
Every option in rclone can have its default set by environment variable.
|
|
.PP
|
|
To find the name of the environment variable, first, take the long
|
|
option name, strip the leading \f[C]\-\-\f[], change \f[C]\-\f[] to
|
|
\f[C]_\f[], make upper case and prepend \f[C]RCLONE_\f[].
|
|
.PP
|
|
For example, to always set \f[C]\-\-stats\ 5s\f[], set the environment
|
|
variable \f[C]RCLONE_STATS=5s\f[].
|
|
If you set stats on the command line this will override the environment
|
|
variable setting.
|
|
.PP
|
|
Or to always use the trash in drive \f[C]\-\-drive\-use\-trash\f[], set
|
|
\f[C]RCLONE_DRIVE_USE_TRASH=true\f[].
|
|
.PP
|
|
The same parser is used for the options and the environment variables so
|
|
they take exactly the same form.
|
|
.SS Config file
|
|
.PP
|
|
You can set defaults for values in the config file on an individual
|
|
remote basis.
|
|
If you want to use this feature, you will need to discover the name of
|
|
the config items that you want.
|
|
The easiest way is to run through \f[C]rclone\ config\f[] by hand, then
|
|
look in the config file to see what the values are (the config file can
|
|
be found by looking at the help for \f[C]\-\-config\f[] in
|
|
\f[C]rclone\ help\f[]).
|
|
.PP
|
|
To find the name of the environment variable, you need to set, take
|
|
\f[C]RCLONE_CONFIG_\f[] + name of remote + \f[C]_\f[] + name of config
|
|
file option and make it all uppercase.
|
|
.PP
|
|
For example, to configure an S3 remote named \f[C]mys3:\f[] without a
|
|
config file (using unix ways of setting environment variables):
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ export\ RCLONE_CONFIG_MYS3_TYPE=s3
|
|
$\ export\ RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
|
|
$\ export\ RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
|
|
$\ rclone\ lsd\ MYS3:
|
|
\ \ \ \ \ \ \ \ \ \ \-1\ 2016\-09\-21\ 12:54:21\ \ \ \ \ \ \ \ \-1\ my\-bucket
|
|
$\ rclone\ listremotes\ |\ grep\ mys3
|
|
mys3:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that if you want to create a remote using environment variables you
|
|
must create the \f[C]\&..._TYPE\f[] variable as above.
|
|
.SS Other environment variables
|
|
.IP \[bu] 2
|
|
RCLONE_CONFIG_PASS` set to contain your config file password (see
|
|
Configuration Encryption (#configuration-encryption) section)
|
|
.IP \[bu] 2
|
|
HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions
|
|
thereof).
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
HTTPS_PROXY takes precedence over HTTP_PROXY for https requests.
|
|
.IP \[bu] 2
|
|
The environment values may be either a complete URL or a "host[:port]"
|
|
for, in which case the "http" scheme is assumed.
|
|
.RE
|
|
.SH Configuring rclone on a remote / headless machine
|
|
.PP
|
|
Some of the configurations (those involving oauth2) require an Internet
|
|
connected web browser.
|
|
.PP
|
|
If you are trying to set rclone up on a remote or headless box with no
|
|
browser available on it (eg a NAS or a server in a datacenter) then you
|
|
will need to use an alternative means of configuration.
|
|
There are two ways of doing it, described below.
|
|
.SS Configuring using rclone authorize
|
|
.PP
|
|
On the headless box
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\&...
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ n
|
|
For\ this\ to\ work,\ you\ will\ need\ rclone\ available\ on\ a\ machine\ that\ has\ a\ web\ browser\ available.
|
|
Execute\ the\ following\ on\ your\ machine:
|
|
\ \ \ \ rclone\ authorize\ "amazon\ cloud\ drive"
|
|
Then\ paste\ the\ result\ below:
|
|
result>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then on your main desktop machine
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ authorize\ "amazon\ cloud\ drive"
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
Paste\ the\ following\ into\ your\ remote\ machine\ \-\-\->
|
|
SECRET_TOKEN
|
|
<\-\-\-End\ paste
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then back to the headless box, paste in the code
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
result>\ SECRET_TOKEN
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[acd12]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ SECRET_TOKEN
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>
|
|
\f[]
|
|
.fi
|
|
.SS Configuring by copying the config file
|
|
.PP
|
|
Rclone stores all of its config in a single configuration file.
|
|
This can easily be copied to configure a remote rclone.
|
|
.PP
|
|
So first configure rclone on your desktop machine
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
to set up the config file.
|
|
.PP
|
|
Find the config file by running \f[C]rclone\ \-h\f[] and looking for the
|
|
help for the \f[C]\-\-config\f[] option
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-h
|
|
[snip]
|
|
\ \ \ \ \ \ \-\-config="/home/user/.rclone.conf":\ Config\ file.
|
|
[snip]
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Now transfer it to the remote box (scp, cut paste, ftp, sftp etc) and
|
|
place it in the correct place (use \f[C]rclone\ \-h\f[] on the remote
|
|
box to find out where).
|
|
.SH Filtering, includes and excludes
|
|
.PP
|
|
Rclone has a sophisticated set of include and exclude rules.
|
|
Some of these are based on patterns and some on other things like file
|
|
size.
|
|
.PP
|
|
The filters are applied for the \f[C]copy\f[], \f[C]sync\f[],
|
|
\f[C]move\f[], \f[C]ls\f[], \f[C]lsl\f[], \f[C]md5sum\f[],
|
|
\f[C]sha1sum\f[], \f[C]size\f[], \f[C]delete\f[] and \f[C]check\f[]
|
|
operations.
|
|
Note that \f[C]purge\f[] does not obey the filters.
|
|
.PP
|
|
Each path as it passes through rclone is matched against the include and
|
|
exclude rules like \f[C]\-\-include\f[], \f[C]\-\-exclude\f[],
|
|
\f[C]\-\-include\-from\f[], \f[C]\-\-exclude\-from\f[],
|
|
\f[C]\-\-filter\f[], or \f[C]\-\-filter\-from\f[].
|
|
The simplest way to try them out is using the \f[C]ls\f[] command, or
|
|
\f[C]\-\-dry\-run\f[] together with \f[C]\-v\f[].
|
|
.SS Patterns
|
|
.PP
|
|
The patterns used to match files for inclusion or exclusion are based on
|
|
"file globs" as used by the unix shell.
|
|
.PP
|
|
If the pattern starts with a \f[C]/\f[] then it only matches at the top
|
|
level of the directory tree, \f[B]relative to the root of the remote\f[]
|
|
(not necessarily the root of the local drive).
|
|
If it doesn\[aq]t start with \f[C]/\f[] then it is matched starting at
|
|
the \f[B]end of the path\f[], but it will only match a complete path
|
|
element:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
file.jpg\ \ \-\ matches\ "file.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ matches\ "directory/file.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "afile.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "directory/afile.jpg"
|
|
/file.jpg\ \-\ matches\ "file.jpg"\ in\ the\ root\ directory\ of\ the\ remote
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "afile.jpg"
|
|
\ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "directory/file.jpg"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
\f[B]Important\f[] Note that you must use \f[C]/\f[] in patterns and not
|
|
\f[C]\\\f[] even if running on Windows.
|
|
.PP
|
|
A \f[C]*\f[] matches anything but not a \f[C]/\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
*.jpg\ \ \-\ matches\ "file.jpg"
|
|
\ \ \ \ \ \ \ \-\ matches\ "directory/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "file.jpg/something"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Use \f[C]**\f[] to match anything, including slashes (\f[C]/\f[]).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
dir/**\ \-\ matches\ "dir/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ matches\ "dir/dir1/dir2/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "directory/file.jpg"
|
|
\ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "adir/file.jpg"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
A \f[C]?\f[] matches any character except a slash \f[C]/\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
l?ss\ \ \-\ matches\ "less"
|
|
\ \ \ \ \ \ \-\ matches\ "lass"
|
|
\ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "floss"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
A \f[C][\f[] and \f[C]]\f[] together make a a character class, such as
|
|
\f[C][a\-z]\f[] or \f[C][aeiou]\f[] or \f[C][[:alpha:]]\f[].
|
|
See the go regexp docs (https://golang.org/pkg/regexp/syntax/) for more
|
|
info on these.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
h[ae]llo\ \-\ matches\ "hello"
|
|
\ \ \ \ \ \ \ \ \ \-\ matches\ "hallo"
|
|
\ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "hullo"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
A \f[C]{\f[] and \f[C]}\f[] define a choice between elements.
|
|
It should contain a comma separated list of patterns, any of which might
|
|
match.
|
|
These patterns can contain wildcards.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{one,two}_potato\ \-\ matches\ "one_potato"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \-\ matches\ "two_potato"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "three_potato"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \-\ doesn\[aq]t\ match\ "_potato"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Special characters can be escaped with a \f[C]\\\f[] before them.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\\*.jpg\ \ \ \ \ \ \ \-\ matches\ "*.jpg"
|
|
\\\\.jpg\ \ \ \ \ \ \ \-\ matches\ "\\.jpg"
|
|
\\[one\\].jpg\ \ \-\ matches\ "[one].jpg"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note also that rclone filter globs can only be used in one of the filter
|
|
command line flags, not in the specification of the remote, so
|
|
\f[C]rclone\ copy\ "remote:dir*.jpg"\ /path/to/dir\f[] won\[aq]t work \-
|
|
what is required is
|
|
\f[C]rclone\ \-\-include\ "*.jpg"\ copy\ remote:dir\ /path/to/dir\f[]
|
|
.SS Directories
|
|
.PP
|
|
Rclone keeps track of directories that could match any file patterns.
|
|
.PP
|
|
Eg if you add the include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/a/*.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Rclone will synthesize the directory include rule
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/a/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you put any rules which end in \f[C]/\f[] then it will only match
|
|
directories.
|
|
.PP
|
|
Directory matches are \f[B]only\f[] used to optimise directory access
|
|
patterns \- you must still match the files that you want to match.
|
|
Directory matches won\[aq]t optimise anything on bucket based remotes
|
|
(eg s3, swift, google compute storage, b2) which don\[aq]t have a
|
|
concept of directory.
|
|
.SS Differences between rsync and rclone patterns
|
|
.PP
|
|
Rclone implements bash style \f[C]{a,b,c}\f[] glob matching which rsync
|
|
doesn\[aq]t.
|
|
.PP
|
|
Rclone always does a wildcard match so \f[C]\\\f[] must always escape a
|
|
\f[C]\\\f[].
|
|
.SS How the rules are used
|
|
.PP
|
|
Rclone maintains a combined list of include rules and exclude rules.
|
|
.PP
|
|
Each file is matched in order, starting from the top, against the rule
|
|
in the list until it finds a match.
|
|
The file is then included or excluded according to the rule type.
|
|
.PP
|
|
If the matcher fails to find a match after testing against all the
|
|
entries in the list then the path is included.
|
|
.PP
|
|
For example given the following rules, \f[C]+\f[] being include,
|
|
\f[C]\-\f[] being exclude,
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\-\ secret*.jpg
|
|
+\ *.jpg
|
|
+\ *.png
|
|
+\ file2.avi
|
|
\-\ *
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This would include
|
|
.IP \[bu] 2
|
|
\f[C]file1.jpg\f[]
|
|
.IP \[bu] 2
|
|
\f[C]file3.png\f[]
|
|
.IP \[bu] 2
|
|
\f[C]file2.avi\f[]
|
|
.PP
|
|
This would exclude
|
|
.IP \[bu] 2
|
|
\f[C]secret17.jpg\f[]
|
|
.IP \[bu] 2
|
|
non \f[C]*.jpg\f[] and \f[C]*.png\f[]
|
|
.PP
|
|
A similar process is done on directory entries before recursing into
|
|
them.
|
|
This only works on remotes which have a concept of directory (Eg local,
|
|
google drive, onedrive, amazon drive) and not on bucket based remotes
|
|
(eg s3, swift, google compute storage, b2).
|
|
.SS Adding filtering rules
|
|
.PP
|
|
Filtering rules are added with the following command line flags.
|
|
.SS Repeating options
|
|
.PP
|
|
You can repeat the following options to add more than one rule of that
|
|
type.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-exclude\-from\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-filter\-from\f[]
|
|
.PP
|
|
\f[B]Important\f[] You should not use \f[C]\-\-include*\f[] together
|
|
with \f[C]\-\-exclude*\f[].
|
|
It may produce different results than you expected.
|
|
In that case try to use: \f[C]\-\-filter*\f[].
|
|
.PP
|
|
Note that all the options of the same type are processed together in the
|
|
order above, regardless of what order they were placed on the command
|
|
line.
|
|
.PP
|
|
So all \f[C]\-\-include\f[] options are processed first in the order
|
|
they appeared on the command line, then all \f[C]\-\-include\-from\f[]
|
|
options etc.
|
|
.PP
|
|
To mix up the order includes and excludes, the \f[C]\-\-filter\f[] flag
|
|
can be used.
|
|
.SS \f[C]\-\-exclude\f[] \- Exclude files matching pattern
|
|
.PP
|
|
Add a single exclude rule with \f[C]\-\-exclude\f[].
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Eg \f[C]\-\-exclude\ *.bak\f[] to exclude all bak files from the sync.
|
|
.SS \f[C]\-\-exclude\-from\f[] \- Read exclude patterns from file
|
|
.PP
|
|
Add exclude rules from a file.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Prepare a file like this \f[C]exclude\-file.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ a\ sample\ exclude\ rule\ file
|
|
*.bak
|
|
file2.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-exclude\-from\ exclude\-file.txt\f[].
|
|
This will sync all files except those ending in \f[C]bak\f[] and
|
|
\f[C]file2.jpg\f[].
|
|
.PP
|
|
This is useful if you have a lot of rules.
|
|
.SS \f[C]\-\-include\f[] \- Include files matching pattern
|
|
.PP
|
|
Add a single include rule with \f[C]\-\-include\f[].
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Eg \f[C]\-\-include\ *.{png,jpg}\f[] to include all \f[C]png\f[] and
|
|
\f[C]jpg\f[] files in the backup and no others.
|
|
.PP
|
|
This adds an implicit \f[C]\-\-exclude\ *\f[] at the very end of the
|
|
filter list.
|
|
This means you can mix \f[C]\-\-include\f[] and
|
|
\f[C]\-\-include\-from\f[] with the other filters (eg
|
|
\f[C]\-\-exclude\f[]) but you must include all the files you want in the
|
|
include statement.
|
|
If this doesn\[aq]t provide enough flexibility then you must use
|
|
\f[C]\-\-filter\-from\f[].
|
|
.SS \f[C]\-\-include\-from\f[] \- Read include patterns from file
|
|
.PP
|
|
Add include rules from a file.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Prepare a file like this \f[C]include\-file.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ a\ sample\ include\ rule\ file
|
|
*.jpg
|
|
*.png
|
|
file2.avi
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-include\-from\ include\-file.txt\f[].
|
|
This will sync all \f[C]jpg\f[], \f[C]png\f[] files and
|
|
\f[C]file2.avi\f[].
|
|
.PP
|
|
This is useful if you have a lot of rules.
|
|
.PP
|
|
This adds an implicit \f[C]\-\-exclude\ *\f[] at the very end of the
|
|
filter list.
|
|
This means you can mix \f[C]\-\-include\f[] and
|
|
\f[C]\-\-include\-from\f[] with the other filters (eg
|
|
\f[C]\-\-exclude\f[]) but you must include all the files you want in the
|
|
include statement.
|
|
If this doesn\[aq]t provide enough flexibility then you must use
|
|
\f[C]\-\-filter\-from\f[].
|
|
.SS \f[C]\-\-filter\f[] \- Add a file\-filtering rule
|
|
.PP
|
|
This can be used to add a single include or exclude rule.
|
|
Include rules start with \f[C]+\f[] and exclude rules start with
|
|
\f[C]\-\f[].
|
|
A special rule called \f[C]!\f[] can be used to clear the existing
|
|
rules.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Eg \f[C]\-\-filter\ "\-\ *.bak"\f[] to exclude all bak files from the
|
|
sync.
|
|
.SS \f[C]\-\-filter\-from\f[] \- Read filtering patterns from a file
|
|
.PP
|
|
Add include/exclude rules from a file.
|
|
.PP
|
|
This flag can be repeated.
|
|
See above for the order the flags are processed in.
|
|
.PP
|
|
Prepare a file like this \f[C]filter\-file.txt\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ a\ sample\ filter\ rule\ file
|
|
\-\ secret*.jpg
|
|
+\ *.jpg
|
|
+\ *.png
|
|
+\ file2.avi
|
|
\-\ /dir/Trash/**
|
|
+\ /dir/**
|
|
#\ exclude\ everything\ else
|
|
\-\ *
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use as \f[C]\-\-filter\-from\ filter\-file.txt\f[].
|
|
The rules are processed in the order that they are defined.
|
|
.PP
|
|
This example will include all \f[C]jpg\f[] and \f[C]png\f[] files,
|
|
exclude any files matching \f[C]secret*.jpg\f[] and include
|
|
\f[C]file2.avi\f[].
|
|
It will also include everything in the directory \f[C]dir\f[] at the
|
|
root of the sync, except \f[C]dir/Trash\f[] which it will exclude.
|
|
Everything else will be excluded from the sync.
|
|
.SS \f[C]\-\-files\-from\f[] \- Read list of source\-file names
|
|
.PP
|
|
This reads a list of file names from the file passed in and
|
|
\f[B]only\f[] these files are transferred.
|
|
The \f[B]filtering rules are ignored\f[] completely if you use this
|
|
option.
|
|
.PP
|
|
This option can be repeated to read from more than one file.
|
|
These are read in the order that they are placed on the command line.
|
|
.PP
|
|
Paths within the \f[C]\-\-files\-from\f[] file will be interpreted as
|
|
starting with the root specified in the command.
|
|
Leading \f[C]/\f[] characters are ignored.
|
|
.PP
|
|
For example, suppose you had \f[C]files\-from.txt\f[] with this content:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ comment
|
|
file1.jpg
|
|
subdir/file2.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You could then use it like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /home/me/pics\ remote:pics
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will transfer these files only (if they exist)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/me/pics/file1.jpg\ \ \ \ \ \ \ \ →\ remote:pics/file1.jpg
|
|
/home/me/pics/subdir/file2.jpg\ →\ remote:pics/subdirfile1.jpg
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To take a more complicated example, let\[aq]s say you had a few files
|
|
you want to back up regularly with these absolute paths:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/important
|
|
/home/user1/dir/file
|
|
/home/user2/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy these you\[aq]d find a common subdirectory \- in this case
|
|
\f[C]/home\f[] and put the remaining files in \f[C]files\-from.txt\f[]
|
|
with or without leading \f[C]/\f[], eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
user1/important
|
|
user1/dir/file
|
|
user2/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You could then copy these to a remote like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /home\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The 3 files will arrive in \f[C]remote:backup\f[] with the paths as in
|
|
the \f[C]files\-from.txt\f[] like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/important\ →\ remote:backup/user1/important
|
|
/home/user1/dir/file\ \ →\ remote:backup/user1/dir/file
|
|
/home/user2/stuff\ \ \ \ \ →\ remote:backup/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You could of course choose \f[C]/\f[] as the root too in which case your
|
|
\f[C]files\-from.txt\f[] might look like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/important
|
|
/home/user1/dir/file
|
|
/home/user2/stuff
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And you would transfer it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ \-\-files\-from\ files\-from.txt\ /\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
In this case there will be an extra \f[C]home\f[] directory on the
|
|
remote:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/home/user1/important\ →\ remote:home/backup/user1/important
|
|
/home/user1/dir/file\ \ →\ remote:home/backup/user1/dir/file
|
|
/home/user2/stuff\ \ \ \ \ →\ remote:home/backup/stuff
|
|
\f[]
|
|
.fi
|
|
.SS \f[C]\-\-min\-size\f[] \- Don\[aq]t transfer any file smaller than
|
|
this
|
|
.PP
|
|
This option controls the minimum size file which will be transferred.
|
|
This defaults to \f[C]kBytes\f[] but a suffix of \f[C]k\f[], \f[C]M\f[],
|
|
or \f[C]G\f[] can be used.
|
|
.PP
|
|
For example \f[C]\-\-min\-size\ 50k\f[] means no files smaller than
|
|
50kByte will be transferred.
|
|
.SS \f[C]\-\-max\-size\f[] \- Don\[aq]t transfer any file larger than
|
|
this
|
|
.PP
|
|
This option controls the maximum size file which will be transferred.
|
|
This defaults to \f[C]kBytes\f[] but a suffix of \f[C]k\f[], \f[C]M\f[],
|
|
or \f[C]G\f[] can be used.
|
|
.PP
|
|
For example \f[C]\-\-max\-size\ 1G\f[] means no files larger than 1GByte
|
|
will be transferred.
|
|
.SS \f[C]\-\-max\-age\f[] \- Don\[aq]t transfer any file older than this
|
|
.PP
|
|
This option controls the maximum age of files to transfer.
|
|
Give in seconds or with a suffix of:
|
|
.IP \[bu] 2
|
|
\f[C]ms\f[] \- Milliseconds
|
|
.IP \[bu] 2
|
|
\f[C]s\f[] \- Seconds
|
|
.IP \[bu] 2
|
|
\f[C]m\f[] \- Minutes
|
|
.IP \[bu] 2
|
|
\f[C]h\f[] \- Hours
|
|
.IP \[bu] 2
|
|
\f[C]d\f[] \- Days
|
|
.IP \[bu] 2
|
|
\f[C]w\f[] \- Weeks
|
|
.IP \[bu] 2
|
|
\f[C]M\f[] \- Months
|
|
.IP \[bu] 2
|
|
\f[C]y\f[] \- Years
|
|
.PP
|
|
For example \f[C]\-\-max\-age\ 2d\f[] means no files older than 2 days
|
|
will be transferred.
|
|
.SS \f[C]\-\-min\-age\f[] \- Don\[aq]t transfer any file younger than
|
|
this
|
|
.PP
|
|
This option controls the minimum age of files to transfer.
|
|
Give in seconds or with a suffix (see \f[C]\-\-max\-age\f[] for list of
|
|
suffixes)
|
|
.PP
|
|
For example \f[C]\-\-min\-age\ 2d\f[] means no files younger than 2 days
|
|
will be transferred.
|
|
.SS \f[C]\-\-delete\-excluded\f[] \- Delete files on dest excluded from
|
|
sync
|
|
.PP
|
|
\f[B]Important\f[] this flag is dangerous \- use with
|
|
\f[C]\-\-dry\-run\f[] and \f[C]\-v\f[] first.
|
|
.PP
|
|
When doing \f[C]rclone\ sync\f[] this will delete any files which are
|
|
excluded from the sync on the destination.
|
|
.PP
|
|
If for example you did a sync from \f[C]A\f[] to \f[C]B\f[] without the
|
|
\f[C]\-\-min\-size\ 50k\f[] flag
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ A:\ B:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then you repeated it like this with the \f[C]\-\-delete\-excluded\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ \-\-min\-size\ 50k\ \-\-delete\-excluded\ sync\ A:\ B:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This would delete all files on \f[C]B\f[] which are less than 50 kBytes
|
|
as these are now excluded from the sync.
|
|
.PP
|
|
Always test first with \f[C]\-\-dry\-run\f[] and \f[C]\-v\f[] before
|
|
using this flag.
|
|
.SS \f[C]\-\-dump\ filters\f[] \- dump the filters to the output
|
|
.PP
|
|
This dumps the defined filters to the output as regular expressions.
|
|
.PP
|
|
Useful for debugging.
|
|
.SS Quoting shell metacharacters
|
|
.PP
|
|
The examples above may not work verbatim in your shell as they have
|
|
shell metacharacters in them (eg \f[C]*\f[]), and may require quoting.
|
|
.PP
|
|
Eg linux, OSX
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\ \\*.jpg\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\ \[aq]*.jpg\[aq]\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include=\[aq]*.jpg\[aq]\f[]
|
|
.PP
|
|
In Windows the expansion is done by the command not the shell so this
|
|
should work fine
|
|
.IP \[bu] 2
|
|
\f[C]\-\-include\ *.jpg\f[]
|
|
.SS Exclude directory based on a file
|
|
.PP
|
|
It is possible to exclude a directory based on a file, which is present
|
|
in this directory.
|
|
Filename should be specified using the \f[C]\-\-exclude\-if\-present\f[]
|
|
flag.
|
|
This flag has a priority over the other filtering flags.
|
|
.PP
|
|
Imagine, you have the following directory structure:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
dir1/file1
|
|
dir1/dir2/file2
|
|
dir1/dir2/dir3/file3
|
|
dir1/dir2/dir3/.ignore
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can exclude \f[C]dir3\f[] from sync by running the following
|
|
command:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ \-\-exclude\-if\-present\ .ignore\ dir1\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Currently only one filename is supported, i.e.
|
|
\f[C]\-\-exclude\-if\-present\f[] should not be used multiple times.
|
|
.SH Remote controlling rclone
|
|
.PP
|
|
If rclone is run with the \f[C]\-\-rc\f[] flag then it starts an http
|
|
server which can be used to remote control rclone.
|
|
.PP
|
|
\f[B]NB\f[] this is experimental and everything here is subject to
|
|
change!
|
|
.SS Supported parameters
|
|
.SS \-\-rc
|
|
.PP
|
|
Flag to start the http server listen on remote requests
|
|
.SS \-\-rc\-addr=IP
|
|
.PP
|
|
IPaddress:Port or :Port to bind server to.
|
|
(default "localhost:5572")
|
|
.SS \-\-rc\-cert=KEY
|
|
.PP
|
|
SSL PEM key (concatenation of certificate and CA certificate)
|
|
.SS \-\-rc\-client\-ca=PATH
|
|
.PP
|
|
Client certificate authority to verify clients with
|
|
.SS \-\-rc\-htpasswd=PATH
|
|
.PP
|
|
htpasswd file \- if not provided no authentication is done
|
|
.SS \-\-rc\-key=PATH
|
|
.PP
|
|
SSL PEM Private key
|
|
.SS \-\-rc\-max\-header\-bytes=VALUE
|
|
.PP
|
|
Maximum size of request header (default 4096)
|
|
.SS \-\-rc\-user=VALUE
|
|
.PP
|
|
User name for authentication.
|
|
.SS \-\-rc\-pass=VALUE
|
|
.PP
|
|
Password for authentication.
|
|
.SS \-\-rc\-realm=VALUE
|
|
.PP
|
|
Realm for authentication (default "rclone")
|
|
.SS \-\-rc\-server\-read\-timeout=DURATION
|
|
.PP
|
|
Timeout for server reading data (default 1h0m0s)
|
|
.SS \-\-rc\-server\-write\-timeout=DURATION
|
|
.PP
|
|
Timeout for server writing data (default 1h0m0s)
|
|
.SS Accessing the remote control via the rclone rc command
|
|
.PP
|
|
Rclone itself implements the remote control protocol in its
|
|
\f[C]rclone\ rc\f[] command.
|
|
.PP
|
|
You can use it like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ rc\ rc/noop\ param1=one\ param2=two
|
|
{
|
|
\ \ \ \ "param1":\ "one",
|
|
\ \ \ \ "param2":\ "two"
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Run \f[C]rclone\ rc\f[] on its own to see the help for the installed
|
|
remote control commands.
|
|
.SS Supported commands
|
|
.SS cache/expire: Purge a remote from cache
|
|
.PP
|
|
Purge a remote from the cache backend.
|
|
Supports either a directory or a file.
|
|
Params: \- remote = path to remote (required) \- withData = true/false
|
|
to delete cached data (chunks) as well (optional)
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ cache/expire\ remote=path/to/sub/folder/
|
|
rclone\ rc\ cache/expire\ remote=/\ withData=true
|
|
\f[]
|
|
.fi
|
|
.SS cache/stats: Get cache stats
|
|
.PP
|
|
Show statistics for the cache remote.
|
|
.SS core/bwlimit: Set the bandwidth limit.
|
|
.PP
|
|
This sets the bandwidth limit to that passed in.
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ core/bwlimit\ rate=1M
|
|
rclone\ rc\ core/bwlimit\ rate=off
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The format of the parameter is exactly the same as passed to \-\-bwlimit
|
|
except only one bandwidth may be specified.
|
|
.SS core/gc: Runs a garbage collection.
|
|
.PP
|
|
This tells the go runtime to do a garbage collection run.
|
|
It isn\[aq]t necessary to call this normally, but it can be useful for
|
|
debugging memory problems.
|
|
.SS core/memstats: Returns the memory statistics
|
|
.PP
|
|
This returns the memory statistics of the running program.
|
|
What the values mean are explained in the go docs:
|
|
https://golang.org/pkg/runtime/#MemStats
|
|
.PP
|
|
The most interesting values for most people are:
|
|
.IP \[bu] 2
|
|
HeapAlloc: This is the amount of memory rclone is actually using
|
|
.IP \[bu] 2
|
|
HeapSys: This is the amount of memory rclone has obtained from the OS
|
|
.IP \[bu] 2
|
|
Sys: this is the total amount of memory requested from the OS
|
|
.IP \[bu] 2
|
|
It is virtual memory so may include unused memory
|
|
.SS core/pid: Return PID of current process
|
|
.PP
|
|
This returns PID of current process.
|
|
Useful for stopping rclone process.
|
|
.SS rc/error: This returns an error
|
|
.PP
|
|
This returns an error with the input as part of its error string.
|
|
Useful for testing error handling.
|
|
.SS rc/list: List all the registered remote control commands
|
|
.PP
|
|
This lists all the registered remote control commands as a JSON map in
|
|
the commands response.
|
|
.SS rc/noop: Echo the input to the output parameters
|
|
.PP
|
|
This echoes the input parameters to the output parameters for testing
|
|
purposes.
|
|
It can be used to check that rclone is still alive and to check that
|
|
parameter passing is working properly.
|
|
.SS vfs/forget: Forget files or directories in the directory cache.
|
|
.PP
|
|
This forgets the paths in the directory cache causing them to be
|
|
re\-read from the remote when needed.
|
|
.PP
|
|
If no paths are passed in then it will forget all the paths in the
|
|
directory cache.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Otherwise pass files or dirs in as file=path or dir=path.
|
|
Any parameter key starting with file will forget that file and any
|
|
starting with dir will forget that dir, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ rc\ vfs/forget\ file=hello\ file2=goodbye\ dir=home/junk
|
|
\f[]
|
|
.fi
|
|
.SS Accessing the remote control via HTTP
|
|
.PP
|
|
Rclone implements a simple HTTP based protocol.
|
|
.PP
|
|
Each endpoint takes an JSON object and returns a JSON object or an
|
|
error.
|
|
The JSON objects are essentially a map of string names to values.
|
|
.PP
|
|
All calls must made using POST.
|
|
.PP
|
|
The input objects can be supplied using URL parameters, POST parameters
|
|
or by supplying "Content\-Type: application/json" and a JSON blob in the
|
|
body.
|
|
There are examples of these below using \f[C]curl\f[].
|
|
.PP
|
|
The response will be a JSON blob in the body of the response.
|
|
This is formatted to be reasonably human readable.
|
|
.PP
|
|
If an error occurs then there will be an HTTP error status (usually 400)
|
|
and the body of the response will contain a JSON encoded error object.
|
|
.SS Using POST with URL parameters only
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-X\ POST\ \[aq]http://localhost:5572/rc/noop/?potato=1&sausage=2\[aq]
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "potato":\ "1",
|
|
\ \ \ \ "sausage":\ "2"
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Here is what an error response looks like:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-X\ POST\ \[aq]http://localhost:5572/rc/error/?potato=1&sausage=2\[aq]
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "error":\ "arbitrary\ error\ on\ input\ map[potato:1\ sausage:2]",
|
|
\ \ \ \ "input":\ {
|
|
\ \ \ \ \ \ \ \ "potato":\ "1",
|
|
\ \ \ \ \ \ \ \ "sausage":\ "2"
|
|
\ \ \ \ }
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that curl doesn\[aq]t return errors to the shell unless you use the
|
|
\f[C]\-f\f[] option
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ curl\ \-f\ \-X\ POST\ \[aq]http://localhost:5572/rc/error/?potato=1&sausage=2\[aq]
|
|
curl:\ (22)\ The\ requested\ URL\ returned\ error:\ 400\ Bad\ Request
|
|
$\ echo\ $?
|
|
22
|
|
\f[]
|
|
.fi
|
|
.SS Using POST with a form
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-\-data\ "potato=1"\ \-\-data\ "sausage=2"\ http://localhost:5572/rc/noop/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "potato":\ "1",
|
|
\ \ \ \ "sausage":\ "2"
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that you can combine these with URL parameters too with the POST
|
|
parameters taking precedence.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-\-data\ "potato=1"\ \-\-data\ "sausage=2"\ "http://localhost:5572/rc/noop/?rutabaga=3&sausage=4"
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "potato":\ "1",
|
|
\ \ \ \ "rutabaga":\ "3",
|
|
\ \ \ \ "sausage":\ "4"
|
|
}
|
|
\f[]
|
|
.fi
|
|
.SS Using POST with a JSON blob
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-H\ "Content\-Type:\ application/json"\ \-X\ POST\ \-d\ \[aq]{"potato":2,"sausage":1}\[aq]\ http://localhost:5572/rc/noop/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
response
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "password":\ "xyz",
|
|
\ \ \ \ "username":\ "xyz"
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This can be combined with URL parameters too if required.
|
|
The JSON blob takes precedence.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-H\ "Content\-Type:\ application/json"\ \-X\ POST\ \-d\ \[aq]{"potato":2,"sausage":1}\[aq]\ \[aq]http://localhost:5572/rc/noop/?rutabaga=3&potato=4\[aq]
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "potato":\ 2,
|
|
\ \ \ \ "rutabaga":\ "3",
|
|
\ \ \ \ "sausage":\ 1
|
|
}
|
|
\f[]
|
|
.fi
|
|
.SS Debugging rclone with pprof
|
|
.PP
|
|
If you use the \f[C]\-\-rc\f[] flag this will also enable the use of the
|
|
go profiling tools on the same port.
|
|
.PP
|
|
To use these, first install go (https://golang.org/doc/install).
|
|
.PP
|
|
Then (for example) to profile rclone\[aq]s memory use you can run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
go\ tool\ pprof\ \-web\ http://localhost:5572/debug/pprof/heap
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This should open a page in your browser showing what is using what
|
|
memory.
|
|
.PP
|
|
You can also use the \f[C]\-text\f[] flag to produce a textual summary
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ go\ tool\ pprof\ \-text\ http://localhost:5572/debug/pprof/heap
|
|
Showing\ nodes\ accounting\ for\ 1537.03kB,\ 100%\ of\ 1537.03kB\ total
|
|
\ \ \ \ \ \ flat\ \ flat%\ \ \ sum%\ \ \ \ \ \ \ \ cum\ \ \ cum%
|
|
\ 1024.03kB\ 66.62%\ 66.62%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
|
|
\ \ \ \ \ 513kB\ 33.38%\ \ \ 100%\ \ \ \ \ \ 513kB\ 33.38%\ \ net/http.newBufioWriterSize
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/cmd/all.init
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/cmd/serve.init
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/cmd/serve/restic.init
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/vendor/golang.org/x/net/http2.init
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init.0
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ main.init
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ \ \ \ \ 513kB\ 33.38%\ \ net/http.(*conn).readRequest
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ \ \ \ \ 513kB\ 33.38%\ \ net/http.(*conn).serve
|
|
\ \ \ \ \ \ \ \ \ 0\ \ \ \ \ 0%\ \ \ 100%\ \ 1024.03kB\ 66.62%\ \ runtime.main
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Possible profiles to look at:
|
|
.IP \[bu] 2
|
|
Memory: \f[C]go\ tool\ pprof\ http://localhost:5572/debug/pprof/heap\f[]
|
|
.IP \[bu] 2
|
|
30\-second CPU profile:
|
|
\f[C]go\ tool\ pprof\ http://localhost:5572/debug/pprof/profile\f[]
|
|
.IP \[bu] 2
|
|
5\-second execution trace:
|
|
\f[C]wget\ http://localhost:5572/debug/pprof/trace?seconds=5\f[]
|
|
.PP
|
|
See the net/http/pprof docs (https://golang.org/pkg/net/http/pprof/) for
|
|
more info on how to use the profiling and for a general overview see the
|
|
Go team\[aq]s blog post on profiling go
|
|
programs (https://blog.golang.org/profiling-go-programs).
|
|
.PP
|
|
The profiling hook is zero overhead unless it is
|
|
used (https://stackoverflow.com/q/26545159/164234).
|
|
.SH Overview of cloud storage systems
|
|
.PP
|
|
Each cloud storage system is slightly different.
|
|
Rclone attempts to provide a unified interface to them, but some
|
|
underlying differences show through.
|
|
.SS Features
|
|
.PP
|
|
Here is an overview of the major features of each cloud storage system.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c c c c.
|
|
T{
|
|
Name
|
|
T}@T{
|
|
Hash
|
|
T}@T{
|
|
ModTime
|
|
T}@T{
|
|
Case Insensitive
|
|
T}@T{
|
|
Duplicate Files
|
|
T}@T{
|
|
MIME Type
|
|
T}
|
|
_
|
|
T{
|
|
Amazon Drive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Amazon S3
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Backblaze B2
|
|
T}@T{
|
|
SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Box
|
|
T}@T{
|
|
SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
Dropbox
|
|
T}@T{
|
|
DBHASH †
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
FTP
|
|
T}@T{
|
|
\-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
Google Cloud Storage
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Google Drive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
HTTP
|
|
T}@T{
|
|
\-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
Hubic
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Mega
|
|
T}@T{
|
|
\-
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
Microsoft Azure Blob Storage
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
Microsoft OneDrive
|
|
T}@T{
|
|
SHA1 ‡‡
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R
|
|
T}
|
|
T{
|
|
OpenDrive
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
Openstack Swift
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
pCloud
|
|
T}@T{
|
|
MD5, SHA1
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
W
|
|
T}
|
|
T{
|
|
QingStor
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
SFTP
|
|
T}@T{
|
|
MD5, SHA1 ‡
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
WebDAV
|
|
T}@T{
|
|
\-
|
|
T}@T{
|
|
Yes ††
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
T{
|
|
Yandex Disk
|
|
T}@T{
|
|
MD5
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
R/W
|
|
T}
|
|
T{
|
|
The local filesystem
|
|
T}@T{
|
|
All
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Depends
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
\-
|
|
T}
|
|
.TE
|
|
.SS Hash
|
|
.PP
|
|
The cloud storage system supports various hash types of the objects.
|
|
The hashes are used when transferring data as an integrity check and can
|
|
be specifically used with the \f[C]\-\-checksum\f[] flag in syncs and in
|
|
the \f[C]check\f[] command.
|
|
.PP
|
|
To use the verify checksums when transferring between cloud storage
|
|
systems they must support a common hash type.
|
|
.PP
|
|
† Note that Dropbox supports its own custom
|
|
hash (https://www.dropbox.com/developers/reference/content-hash).
|
|
This is an SHA256 sum of all the 4MB block SHA256s.
|
|
.PP
|
|
‡ SFTP supports checksums if the same login has shell access and
|
|
\f[C]md5sum\f[] or \f[C]sha1sum\f[] as well as \f[C]echo\f[] are in the
|
|
remote\[aq]s PATH.
|
|
.PP
|
|
†† WebDAV supports modtimes when used with Owncloud and Nextcloud only.
|
|
.PP
|
|
‡‡ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive
|
|
for business and SharePoint server support Microsoft\[aq]s own
|
|
QuickXorHash (https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash).
|
|
.SS ModTime
|
|
.PP
|
|
The cloud storage system supports setting modification times on objects.
|
|
If it does then this enables a using the modification times as part of
|
|
the sync.
|
|
If not then only the size will be checked by default, though the MD5SUM
|
|
can be checked with the \f[C]\-\-checksum\f[] flag.
|
|
.PP
|
|
All cloud storage systems support some kind of date on the object and
|
|
these will be set when transferring from the cloud storage system.
|
|
.SS Case Insensitive
|
|
.PP
|
|
If a cloud storage systems is case sensitive then it is possible to have
|
|
two files which differ only in case, eg \f[C]file.txt\f[] and
|
|
\f[C]FILE.txt\f[].
|
|
If a cloud storage system is case insensitive then that isn\[aq]t
|
|
possible.
|
|
.PP
|
|
This can cause problems when syncing between a case insensitive system
|
|
and a case sensitive system.
|
|
The symptom of this is that no matter how many times you run the sync it
|
|
never completes fully.
|
|
.PP
|
|
The local filesystem and SFTP may or may not be case sensitive depending
|
|
on OS.
|
|
.IP \[bu] 2
|
|
Windows \- usually case insensitive, though case is preserved
|
|
.IP \[bu] 2
|
|
OSX \- usually case insensitive, though it is possible to format case
|
|
sensitive
|
|
.IP \[bu] 2
|
|
Linux \- usually case sensitive, but there are case insensitive file
|
|
systems (eg FAT formatted USB keys)
|
|
.PP
|
|
Most of the time this doesn\[aq]t cause any problems as people tend to
|
|
avoid files whose name differs only by case even on case sensitive
|
|
systems.
|
|
.SS Duplicate files
|
|
.PP
|
|
If a cloud storage system allows duplicate files then it can have two
|
|
objects with the same name.
|
|
.PP
|
|
This confuses rclone greatly when syncing \- use the
|
|
\f[C]rclone\ dedupe\f[] command to rename or remove duplicates.
|
|
.SS MIME Type
|
|
.PP
|
|
MIME types (also known as media types) classify types of documents using
|
|
a simple text classification, eg \f[C]text/html\f[] or
|
|
\f[C]application/pdf\f[].
|
|
.PP
|
|
Some cloud storage systems support reading (\f[C]R\f[]) the MIME type of
|
|
objects and some support writing (\f[C]W\f[]) the MIME type of objects.
|
|
.PP
|
|
The MIME type can be important if you are serving files directly to HTTP
|
|
from the storage system.
|
|
.PP
|
|
If you are copying from a remote which supports reading (\f[C]R\f[]) to
|
|
a remote which supports writing (\f[C]W\f[]) then rclone will preserve
|
|
the MIME types.
|
|
Otherwise they will be guessed from the extension, or the remote itself
|
|
may assign the MIME type.
|
|
.SS Optional Features
|
|
.PP
|
|
All the remotes support a basic set of features, but there are some
|
|
optional features supported by some remotes used to make some operations
|
|
more efficient.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
l c c c c c c c c c.
|
|
T{
|
|
Name
|
|
T}@T{
|
|
Purge
|
|
T}@T{
|
|
Copy
|
|
T}@T{
|
|
Move
|
|
T}@T{
|
|
DirMove
|
|
T}@T{
|
|
CleanUp
|
|
T}@T{
|
|
ListR
|
|
T}@T{
|
|
StreamUpload
|
|
T}@T{
|
|
LinkSharing
|
|
T}@T{
|
|
About
|
|
T}
|
|
_
|
|
T{
|
|
Amazon Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Amazon S3
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Backblaze B2
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Box
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Dropbox
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
FTP
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Google Cloud Storage
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Google Drive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
HTTP
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Hubic
|
|
T}@T{
|
|
Yes †
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Mega
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
Microsoft Azure Blob Storage
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Microsoft OneDrive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #197 (https://github.com/ncw/rclone/issues/197)
|
|
T}@T{
|
|
No #575 (https://github.com/ncw/rclone/issues/575)
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
OpenDrive
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Openstack Swift
|
|
T}@T{
|
|
Yes †
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
pCloud
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
T{
|
|
QingStor
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
SFTP
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
WebDAV
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes ‡
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
Yandex Disk
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No #2178 (https://github.com/ncw/rclone/issues/2178)
|
|
T}@T{
|
|
No
|
|
T}
|
|
T{
|
|
The local filesystem
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}@T{
|
|
No
|
|
T}@T{
|
|
Yes
|
|
T}
|
|
.TE
|
|
.SS Purge
|
|
.PP
|
|
This deletes a directory quicker than just deleting all the files in the
|
|
directory.
|
|
.PP
|
|
† Note Swift and Hubic implement this in order to delete directory
|
|
markers but they don\[aq]t actually have a quicker way of deleting files
|
|
other than deleting them individually.
|
|
.PP
|
|
‡ StreamUpload is not supported with Nextcloud
|
|
.SS Copy
|
|
.PP
|
|
Used when copying an object to and from the same remote.
|
|
This known as a server side copy so you can copy a file without
|
|
downloading it and uploading it again.
|
|
It is used if you use \f[C]rclone\ copy\f[] or \f[C]rclone\ move\f[] if
|
|
the remote doesn\[aq]t support \f[C]Move\f[] directly.
|
|
.PP
|
|
If the server doesn\[aq]t support \f[C]Copy\f[] directly then for copy
|
|
operations the file is downloaded then re\-uploaded.
|
|
.SS Move
|
|
.PP
|
|
Used when moving/renaming an object on the same remote.
|
|
This is known as a server side move of a file.
|
|
This is used in \f[C]rclone\ move\f[] if the server doesn\[aq]t support
|
|
\f[C]DirMove\f[].
|
|
.PP
|
|
If the server isn\[aq]t capable of \f[C]Move\f[] then rclone simulates
|
|
it with \f[C]Copy\f[] then delete.
|
|
If the server doesn\[aq]t support \f[C]Copy\f[] then rclone will
|
|
download the file and re\-upload it.
|
|
.SS DirMove
|
|
.PP
|
|
This is used to implement \f[C]rclone\ move\f[] to move a directory if
|
|
possible.
|
|
If it isn\[aq]t then it will use \f[C]Move\f[] on each file (which falls
|
|
back to \f[C]Copy\f[] then download and upload \- see \f[C]Move\f[]
|
|
section).
|
|
.SS CleanUp
|
|
.PP
|
|
This is used for emptying the trash for a remote by
|
|
\f[C]rclone\ cleanup\f[].
|
|
.PP
|
|
If the server can\[aq]t do \f[C]CleanUp\f[] then
|
|
\f[C]rclone\ cleanup\f[] will return an error.
|
|
.SS ListR
|
|
.PP
|
|
The remote supports a recursive list to list all the contents beneath a
|
|
directory quickly.
|
|
This enables the \f[C]\-\-fast\-list\f[] flag to work.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS StreamUpload
|
|
.PP
|
|
Some remotes allow files to be uploaded without knowing the file size in
|
|
advance.
|
|
This allows certain operations to work without spooling the file to
|
|
local disk first, e.g.
|
|
\f[C]rclone\ rcat\f[].
|
|
.SS LinkSharing
|
|
.PP
|
|
Sets the necessary permissions on a file or folder and prints a link
|
|
that allows others to access them, even if they don\[aq]t have an
|
|
account on the particular cloud provider.
|
|
.SS About
|
|
.PP
|
|
This is used to fetch quota information from the remote, like bytes
|
|
used/free/quota and bytes used in the trash.
|
|
.PP
|
|
If the server can\[aq]t do \f[C]About\f[] then \f[C]rclone\ about\f[]
|
|
will return an error.
|
|
.SS Alias
|
|
.PP
|
|
The \f[C]alias\f[] remote provides a new name for another remote.
|
|
.PP
|
|
Paths may be as deep as required or a local path, eg
|
|
\f[C]remote:directory/subdirectory\f[] or
|
|
\f[C]/directory/subdirectory\f[].
|
|
.PP
|
|
During the initial setup with \f[C]rclone\ config\f[] you will specify
|
|
the target remote.
|
|
The target remote can either be a local path or another remote.
|
|
.PP
|
|
Subfolders can be used in target remote.
|
|
Asume a alias remote named \f[C]backup\f[] with the target
|
|
\f[C]mydrive:private/backup\f[].
|
|
Invoking \f[C]rclone\ mkdir\ backup:desktop\f[] is exactly the same as
|
|
invoking \f[C]rclone\ mkdir\ mydrive:private/backup/desktop\f[].
|
|
.PP
|
|
There will be no special handling of paths containing \f[C]\&..\f[]
|
|
segments.
|
|
Invoking \f[C]rclone\ mkdir\ backup:../desktop\f[] is exactly the same
|
|
as invoking \f[C]rclone\ mkdir\ mydrive:private/backup/../desktop\f[].
|
|
The empty path is not allowed as a remote.
|
|
To alias the current directory use \f[C]\&.\f[] instead.
|
|
.PP
|
|
Here is an example of how to make a alias called \f[C]remote\f[] for
|
|
local folder.
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Alias\ for\ a\ existing\ remote
|
|
\ \ \ \\\ "alias"
|
|
\ 2\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 3\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 4\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 5\ /\ Box
|
|
\ \ \ \\\ "box"
|
|
\ 6\ /\ Cache\ a\ remote
|
|
\ \ \ \\\ "cache"
|
|
\ 7\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 8\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 9\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
10\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
11\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
12\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
13\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
14\ /\ Microsoft\ Azure\ Blob\ Storage
|
|
\ \ \ \\\ "azureblob"
|
|
15\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
16\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
17\ /\ Pcloud
|
|
\ \ \ \\\ "pcloud"
|
|
18\ /\ QingCloud\ Object\ Storage
|
|
\ \ \ \\\ "qingstor"
|
|
19\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
20\ /\ Webdav
|
|
\ \ \ \\\ "webdav"
|
|
21\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
22\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ 1
|
|
Remote\ or\ path\ to\ alias.
|
|
Can\ be\ "myremote:path/to/dir",\ "myremote:bucket",\ "myremote:"\ or\ "/local/path".
|
|
remote>\ /mnt/storage/backup
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
remote\ =\ /mnt/storage/backup
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
Current\ remotes:
|
|
|
|
Name\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Type
|
|
====\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ====
|
|
remote\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ alias
|
|
|
|
e)\ Edit\ existing\ remote
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
e/n/d/r/c/s/q>\ q
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level in \f[C]/mnt/storage/backup\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in \f[C]/mnt/storage/backup\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Copy another local directory to the alias directory called source
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:source
|
|
\f[]
|
|
.fi
|
|
.SS Amazon Drive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for Amazon Drive involves getting a token from Amazon
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
The configuration process for Amazon Drive may involve using an oauth
|
|
proxy (https://github.com/ncw/oauthproxy).
|
|
This is used to keep the Amazon credentials out of the source code.
|
|
The proxy runs in Google\[aq]s very secure App Engine environment and
|
|
doesn\[aq]t store any credentials which pass through it.
|
|
.PP
|
|
\f[B]NB\f[] rclone doesn\[aq]t not currently have its own Amazon Drive
|
|
credentials (see the
|
|
forum (https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/)
|
|
for why) so you will either need to have your own \f[C]client_id\f[] and
|
|
\f[C]client_secret\f[] with Amazon Drive, or use a a third party ouath
|
|
proxy in which case you will need to enter \f[C]client_id\f[],
|
|
\f[C]client_secret\f[], \f[C]auth_url\f[] and \f[C]token_url\f[].
|
|
.PP
|
|
Note also if you are not using Amazon\[aq]s \f[C]auth_url\f[] and
|
|
\f[C]token_url\f[], (ie you filled in something for those) then if
|
|
setting up on a remote machine you can only use the copying the config
|
|
method of
|
|
configuration (https://rclone.org/remote_setup/#configuring-by-copying-the-config-file)
|
|
\- \f[C]rclone\ authorize\f[] will not work.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/r/c/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 7\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 8\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 9\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
10\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
11\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
12\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
13\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
14\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 1
|
|
Amazon\ Application\ Client\ Id\ \-\ required.
|
|
client_id>\ your\ client\ ID\ goes\ here
|
|
Amazon\ Application\ Client\ Secret\ \-\ required.
|
|
client_secret>\ your\ client\ secret\ goes\ here
|
|
Auth\ server\ URL\ \-\ leave\ blank\ to\ use\ Amazon\[aq]s.
|
|
auth_url>\ Optional\ auth\ URL
|
|
Token\ server\ url\ \-\ leave\ blank\ to\ use\ Amazon\[aq]s.
|
|
token_url>\ Optional\ token\ URL
|
|
Remote\ config
|
|
Make\ sure\ your\ Redirect\ URL\ is\ set\ to\ "http://127.0.0.1:53682/"\ in\ your\ custom\ config.
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\ your\ client\ ID\ goes\ here
|
|
client_secret\ =\ your\ client\ secret\ goes\ here
|
|
auth_url\ =\ Optional\ auth\ URL
|
|
token_url\ =\ Optional\ token\ URL
|
|
token\ =\ {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015\-09\-06T16:07:39.658438471+01:00"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Amazon.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your Amazon Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your Amazon Drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Amazon Drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and MD5SUMs
|
|
.PP
|
|
Amazon Drive doesn\[aq]t allow modification times to be changed via the
|
|
API so these won\[aq]t be accurate or used for syncing.
|
|
.PP
|
|
It does store MD5SUMs so for a more accurate sync, you can use the
|
|
\f[C]\-\-checksum\f[] flag.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Amazon don\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Amazon\[aq]s
|
|
apps or via the Amazon Drive website.
|
|
As of November 17, 2016, files are automatically deleted by Amazon from
|
|
the trash after 30 days.
|
|
.SS Using with non \f[C]\&.com\f[] Amazon accounts
|
|
.PP
|
|
Let\[aq]s say you usually use \f[C]amazon.co.uk\f[].
|
|
When you authenticate with rclone it will take you to an
|
|
\f[C]amazon.com\f[] page to log in.
|
|
Your \f[C]amazon.co.uk\f[] email and password should work here just
|
|
fine.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-acd\-templink\-threshold=SIZE
|
|
.PP
|
|
Files this size or more will be downloaded via their \f[C]tempLink\f[].
|
|
This is to work around a problem with Amazon Drive which blocks
|
|
downloads of files bigger than about 10GB.
|
|
The default for this is 9GB which shouldn\[aq]t need to be changed.
|
|
.PP
|
|
To download files above this threshold, rclone requests a
|
|
\f[C]tempLink\f[] which downloads the file through a temporary URL
|
|
directly from the underlying S3 storage.
|
|
.SS \-\-acd\-upload\-wait\-per\-gb=TIME
|
|
.PP
|
|
Sometimes Amazon Drive gives an error when a file has been fully
|
|
uploaded but the file appears anyway after a little while.
|
|
This happens sometimes for files over 1GB in size and nearly every time
|
|
for files bigger than 10GB.
|
|
This parameter controls the time rclone waits for the file to appear.
|
|
.PP
|
|
The default value for this parameter is 3 minutes per GB, so by default
|
|
it will wait 3 minutes for every GB uploaded to see if the file appears.
|
|
.PP
|
|
You can disable this feature by setting it to 0.
|
|
This may cause conflict errors as rclone retries the failed upload but
|
|
the file will most likely appear correctly eventually.
|
|
.PP
|
|
These values were determined empirically by observing lots of uploads of
|
|
big files for a range of file sizes.
|
|
.PP
|
|
Upload with the \f[C]\-v\f[] flag to see more info about what rclone is
|
|
doing in this situation.
|
|
.SS Limitations
|
|
.PP
|
|
Note that Amazon Drive is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
Amazon Drive has rate limiting so you may notice errors in the sync (429
|
|
errors).
|
|
rclone will automatically retry the sync up to 3 times by default (see
|
|
\f[C]\-\-retries\f[] flag) which should hopefully work around this
|
|
problem.
|
|
.PP
|
|
Amazon Drive has an internal limit of file sizes that can be uploaded to
|
|
the service.
|
|
This limit is not officially published, but all files larger than this
|
|
will fail.
|
|
.PP
|
|
At the time of writing (Jan 2016) is in the area of 50GB per file.
|
|
This means that larger files are likely to fail.
|
|
.PP
|
|
Unfortunately there is no way for rclone to see that this failure is
|
|
because of file size, so it will retry the operation, as any other
|
|
failure.
|
|
To avoid this problem, use \f[C]\-\-max\-size\ 50000M\f[] option to
|
|
limit the maximum size of uploaded files.
|
|
Note that \f[C]\-\-max\-size\f[] does not split files into segments, it
|
|
only ignores files over this size.
|
|
.SS Amazon S3 Storage Providers
|
|
.PP
|
|
The S3 backend can be used with a number of different providers:
|
|
.IP \[bu] 2
|
|
AWS S3
|
|
.IP \[bu] 2
|
|
Ceph
|
|
.IP \[bu] 2
|
|
DigitalOcean Spaces
|
|
.IP \[bu] 2
|
|
Dreamhost
|
|
.IP \[bu] 2
|
|
IBM COS S3
|
|
.IP \[bu] 2
|
|
Minio
|
|
.IP \[bu] 2
|
|
Wasabi
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
Once you have made a remote (see the provider specific section above)
|
|
you can use it like this:
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS AWS S3
|
|
.PP
|
|
Here is an example of making an s3 configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Alias\ for\ a\ existing\ remote
|
|
\ \ \ \\\ "alias"
|
|
\ 2\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 3\ /\ Amazon\ S3\ Compliant\ Storage\ Providers\ (AWS,\ Ceph,\ Dreamhost,\ IBM\ COS,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 4\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
[snip]
|
|
23\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ s3
|
|
Choose\ your\ S3\ provider.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Web\ Services\ (AWS)\ S3
|
|
\ \ \ \\\ "AWS"
|
|
\ 2\ /\ Ceph\ Object\ Storage
|
|
\ \ \ \\\ "Ceph"
|
|
\ 3\ /\ Digital\ Ocean\ Spaces
|
|
\ \ \ \\\ "DigitalOcean"
|
|
\ 4\ /\ Dreamhost\ DreamObjects
|
|
\ \ \ \\\ "Dreamhost"
|
|
\ 5\ /\ IBM\ COS\ S3
|
|
\ \ \ \\\ "IBMCOS"
|
|
\ 6\ /\ Minio\ Object\ Storage
|
|
\ \ \ \\\ "Minio"
|
|
\ 7\ /\ Wasabi\ Object\ Storage
|
|
\ \ \ \\\ "Wasabi"
|
|
\ 8\ /\ Any\ other\ S3\ compatible\ provider
|
|
\ \ \ \\\ "Other"
|
|
provider>\ 1
|
|
Get\ AWS\ credentials\ from\ runtime\ (environment\ variables\ or\ EC2/ECS\ meta\ data\ if\ no\ env\ vars).\ Only\ applies\ if\ access_key_id\ and\ secret_access_key\ is\ blank.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Enter\ AWS\ credentials\ in\ the\ next\ step
|
|
\ \ \ \\\ "false"
|
|
\ 2\ /\ Get\ AWS\ credentials\ from\ the\ environment\ (env\ vars\ or\ IAM)
|
|
\ \ \ \\\ "true"
|
|
env_auth>\ 1
|
|
AWS\ Access\ Key\ ID\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
access_key_id>\ XXX
|
|
AWS\ Secret\ Access\ Key\ (password)\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
secret_access_key>\ YYY
|
|
Region\ to\ connect\ to.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ \ \ /\ The\ default\ endpoint\ \-\ a\ good\ choice\ if\ you\ are\ unsure.
|
|
\ 1\ |\ US\ Region,\ Northern\ Virginia\ or\ Pacific\ Northwest.
|
|
\ \ \ |\ Leave\ location\ constraint\ empty.
|
|
\ \ \ \\\ "us\-east\-1"
|
|
\ \ \ /\ US\ East\ (Ohio)\ Region
|
|
\ 2\ |\ Needs\ location\ constraint\ us\-east\-2.
|
|
\ \ \ \\\ "us\-east\-2"
|
|
\ \ \ /\ US\ West\ (Oregon)\ Region
|
|
\ 3\ |\ Needs\ location\ constraint\ us\-west\-2.
|
|
\ \ \ \\\ "us\-west\-2"
|
|
\ \ \ /\ US\ West\ (Northern\ California)\ Region
|
|
\ 4\ |\ Needs\ location\ constraint\ us\-west\-1.
|
|
\ \ \ \\\ "us\-west\-1"
|
|
\ \ \ /\ Canada\ (Central)\ Region
|
|
\ 5\ |\ Needs\ location\ constraint\ ca\-central\-1.
|
|
\ \ \ \\\ "ca\-central\-1"
|
|
\ \ \ /\ EU\ (Ireland)\ Region
|
|
\ 6\ |\ Needs\ location\ constraint\ EU\ or\ eu\-west\-1.
|
|
\ \ \ \\\ "eu\-west\-1"
|
|
\ \ \ /\ EU\ (London)\ Region
|
|
\ 7\ |\ Needs\ location\ constraint\ eu\-west\-2.
|
|
\ \ \ \\\ "eu\-west\-2"
|
|
\ \ \ /\ EU\ (Frankfurt)\ Region
|
|
\ 8\ |\ Needs\ location\ constraint\ eu\-central\-1.
|
|
\ \ \ \\\ "eu\-central\-1"
|
|
\ \ \ /\ Asia\ Pacific\ (Singapore)\ Region
|
|
\ 9\ |\ Needs\ location\ constraint\ ap\-southeast\-1.
|
|
\ \ \ \\\ "ap\-southeast\-1"
|
|
\ \ \ /\ Asia\ Pacific\ (Sydney)\ Region
|
|
10\ |\ Needs\ location\ constraint\ ap\-southeast\-2.
|
|
\ \ \ \\\ "ap\-southeast\-2"
|
|
\ \ \ /\ Asia\ Pacific\ (Tokyo)\ Region
|
|
11\ |\ Needs\ location\ constraint\ ap\-northeast\-1.
|
|
\ \ \ \\\ "ap\-northeast\-1"
|
|
\ \ \ /\ Asia\ Pacific\ (Seoul)
|
|
12\ |\ Needs\ location\ constraint\ ap\-northeast\-2.
|
|
\ \ \ \\\ "ap\-northeast\-2"
|
|
\ \ \ /\ Asia\ Pacific\ (Mumbai)
|
|
13\ |\ Needs\ location\ constraint\ ap\-south\-1.
|
|
\ \ \ \\\ "ap\-south\-1"
|
|
\ \ \ /\ South\ America\ (Sao\ Paulo)\ Region
|
|
14\ |\ Needs\ location\ constraint\ sa\-east\-1.
|
|
\ \ \ \\\ "sa\-east\-1"
|
|
region>\ 1
|
|
Endpoint\ for\ S3\ API.
|
|
Leave\ blank\ if\ using\ AWS\ to\ use\ the\ default\ endpoint\ for\ the\ region.
|
|
endpoint>\
|
|
Location\ constraint\ \-\ must\ be\ set\ to\ match\ the\ Region.\ Used\ when\ creating\ buckets\ only.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Empty\ for\ US\ Region,\ Northern\ Virginia\ or\ Pacific\ Northwest.
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ US\ East\ (Ohio)\ Region.
|
|
\ \ \ \\\ "us\-east\-2"
|
|
\ 3\ /\ US\ West\ (Oregon)\ Region.
|
|
\ \ \ \\\ "us\-west\-2"
|
|
\ 4\ /\ US\ West\ (Northern\ California)\ Region.
|
|
\ \ \ \\\ "us\-west\-1"
|
|
\ 5\ /\ Canada\ (Central)\ Region.
|
|
\ \ \ \\\ "ca\-central\-1"
|
|
\ 6\ /\ EU\ (Ireland)\ Region.
|
|
\ \ \ \\\ "eu\-west\-1"
|
|
\ 7\ /\ EU\ (London)\ Region.
|
|
\ \ \ \\\ "eu\-west\-2"
|
|
\ 8\ /\ EU\ Region.
|
|
\ \ \ \\\ "EU"
|
|
\ 9\ /\ Asia\ Pacific\ (Singapore)\ Region.
|
|
\ \ \ \\\ "ap\-southeast\-1"
|
|
10\ /\ Asia\ Pacific\ (Sydney)\ Region.
|
|
\ \ \ \\\ "ap\-southeast\-2"
|
|
11\ /\ Asia\ Pacific\ (Tokyo)\ Region.
|
|
\ \ \ \\\ "ap\-northeast\-1"
|
|
12\ /\ Asia\ Pacific\ (Seoul)
|
|
\ \ \ \\\ "ap\-northeast\-2"
|
|
13\ /\ Asia\ Pacific\ (Mumbai)
|
|
\ \ \ \\\ "ap\-south\-1"
|
|
14\ /\ South\ America\ (Sao\ Paulo)\ Region.
|
|
\ \ \ \\\ "sa\-east\-1"
|
|
location_constraint>\ 1
|
|
Canned\ ACL\ used\ when\ creating\ buckets\ and/or\ storing\ objects\ in\ S3.
|
|
For\ more\ info\ visit\ https://docs.aws.amazon.com/AmazonS3/latest/dev/acl\-overview.html#canned\-acl
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Owner\ gets\ FULL_CONTROL.\ No\ one\ else\ has\ access\ rights\ (default).
|
|
\ \ \ \\\ "private"
|
|
\ 2\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ access.
|
|
\ \ \ \\\ "public\-read"
|
|
\ \ \ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ and\ WRITE\ access.
|
|
\ 3\ |\ Granting\ this\ on\ a\ bucket\ is\ generally\ not\ recommended.
|
|
\ \ \ \\\ "public\-read\-write"
|
|
\ 4\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AuthenticatedUsers\ group\ gets\ READ\ access.
|
|
\ \ \ \\\ "authenticated\-read"
|
|
\ \ \ /\ Object\ owner\ gets\ FULL_CONTROL.\ Bucket\ owner\ gets\ READ\ access.
|
|
\ 5\ |\ If\ you\ specify\ this\ canned\ ACL\ when\ creating\ a\ bucket,\ Amazon\ S3\ ignores\ it.
|
|
\ \ \ \\\ "bucket\-owner\-read"
|
|
\ \ \ /\ Both\ the\ object\ owner\ and\ the\ bucket\ owner\ get\ FULL_CONTROL\ over\ the\ object.
|
|
\ 6\ |\ If\ you\ specify\ this\ canned\ ACL\ when\ creating\ a\ bucket,\ Amazon\ S3\ ignores\ it.
|
|
\ \ \ \\\ "bucket\-owner\-full\-control"
|
|
acl>\ 1
|
|
The\ server\-side\ encryption\ algorithm\ used\ when\ storing\ this\ object\ in\ S3.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ None
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ AES256
|
|
\ \ \ \\\ "AES256"
|
|
server_side_encryption>\ 1
|
|
The\ storage\ class\ to\ use\ when\ storing\ objects\ in\ S3.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Default
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ Standard\ storage\ class
|
|
\ \ \ \\\ "STANDARD"
|
|
\ 3\ /\ Reduced\ redundancy\ storage\ class
|
|
\ \ \ \\\ "REDUCED_REDUNDANCY"
|
|
\ 4\ /\ Standard\ Infrequent\ Access\ storage\ class
|
|
\ \ \ \\\ "STANDARD_IA"
|
|
\ 5\ /\ One\ Zone\ Infrequent\ Access\ storage\ class
|
|
\ \ \ \\\ "ONEZONE_IA"
|
|
storage_class>\ 1
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
type\ =\ s3
|
|
provider\ =\ AWS
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ XXX
|
|
secret_access_key\ =\ YYY
|
|
region\ =\ us\-east\-1
|
|
endpoint\ =\
|
|
location_constraint\ =\
|
|
acl\ =\ private
|
|
server_side_encryption\ =\
|
|
storage_class\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS \-\-update and \-\-use\-server\-modtime
|
|
.PP
|
|
As noted below, the modified time is stored on metadata on the object.
|
|
It is used by default for all operations that require checking the time
|
|
a file was last updated.
|
|
It allows rclone to treat the remote more like a true filesystem, but it
|
|
is inefficient because it requires an extra API call to retrieve the
|
|
metadata.
|
|
.PP
|
|
For many operations, the time the object was last uploaded to the remote
|
|
is sufficient to determine if it is "dirty".
|
|
By using \f[C]\-\-update\f[] along with
|
|
\f[C]\-\-use\-server\-modtime\f[], you can avoid the extra API call and
|
|
simply upload files whose local modtime is newer than the time it was
|
|
last uploaded.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Amz\-Meta\-Mtime\f[] as floating point since the epoch accurate
|
|
to 1 ns.
|
|
.SS Multipart uploads
|
|
.PP
|
|
rclone supports multipart uploads with S3 which means that it can upload
|
|
files bigger than 5GB.
|
|
Note that files uploaded \f[I]both\f[] with multipart upload
|
|
\f[I]and\f[] through crypt remotes do not have MD5 sums.
|
|
.SS Buckets and Regions
|
|
.PP
|
|
With Amazon S3 you can list buckets (\f[C]rclone\ lsd\f[]) using any
|
|
region, but you can only access the content of a bucket from the region
|
|
it was created in.
|
|
If you attempt to access a bucket from the wrong region, you will get an
|
|
error,
|
|
\f[C]incorrect\ region,\ the\ bucket\ is\ not\ in\ \[aq]XXX\[aq]\ region\f[].
|
|
.SS Authentication
|
|
.PP
|
|
There are a number of ways to supply \f[C]rclone\f[] with a set of AWS
|
|
credentials, with and without using the environment.
|
|
.PP
|
|
The different authentication methods are tried in this order:
|
|
.IP \[bu] 2
|
|
Directly in the rclone configuration file (\f[C]env_auth\ =\ false\f[]
|
|
in the config file):
|
|
.IP \[bu] 2
|
|
\f[C]access_key_id\f[] and \f[C]secret_access_key\f[] are required.
|
|
.IP \[bu] 2
|
|
\f[C]session_token\f[] can be optionally set when using AWS STS.
|
|
.IP \[bu] 2
|
|
Runtime configuration (\f[C]env_auth\ =\ true\f[] in the config file):
|
|
.IP \[bu] 2
|
|
Export the following environment variables before running
|
|
\f[C]rclone\f[]:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access Key ID: \f[C]AWS_ACCESS_KEY_ID\f[] or \f[C]AWS_ACCESS_KEY\f[]
|
|
.IP \[bu] 2
|
|
Secret Access Key: \f[C]AWS_SECRET_ACCESS_KEY\f[] or
|
|
\f[C]AWS_SECRET_KEY\f[]
|
|
.IP \[bu] 2
|
|
Session Token: \f[C]AWS_SESSION_TOKEN\f[] (optional)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Or, use a named
|
|
profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html):
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Profile files are standard files used by AWS CLI tools
|
|
.IP \[bu] 2
|
|
By default it will use the profile in your home directory (eg
|
|
\f[C]~/.aws/credentials\f[] on unix based systems) file and the
|
|
"default" profile, to change set these environment variables:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]AWS_SHARED_CREDENTIALS_FILE\f[] to control which file.
|
|
.IP \[bu] 2
|
|
\f[C]AWS_PROFILE\f[] to control which profile to use.
|
|
.RE
|
|
.RE
|
|
.IP \[bu] 2
|
|
Or, run \f[C]rclone\f[] in an ECS task with an IAM role (AWS only).
|
|
.IP \[bu] 2
|
|
Or, run \f[C]rclone\f[] on an EC2 instance with an IAM role (AWS only).
|
|
.PP
|
|
If none of these option actually end up providing \f[C]rclone\f[] with
|
|
AWS credentials then S3 interaction will be non\-authenticated (see
|
|
below).
|
|
.SS S3 Permissions
|
|
.PP
|
|
When using the \f[C]sync\f[] subcommand of \f[C]rclone\f[] the following
|
|
minimum permissions are required to be available on the bucket being
|
|
written to:
|
|
.IP \[bu] 2
|
|
\f[C]ListBucket\f[]
|
|
.IP \[bu] 2
|
|
\f[C]DeleteObject\f[]
|
|
.IP \[bu] 2
|
|
\f[C]GetObject\f[]
|
|
.IP \[bu] 2
|
|
\f[C]PutObject\f[]
|
|
.IP \[bu] 2
|
|
\f[C]PutObjectACL\f[]
|
|
.PP
|
|
Example policy:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "Version":\ "2012\-10\-17",
|
|
\ \ \ \ "Statement":\ [
|
|
\ \ \ \ \ \ \ \ {
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "Effect":\ "Allow",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "Principal":\ {
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ "AWS":\ "arn:aws:iam::USER_SID:user/USER_NAME"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ },
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "Action":\ [
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ "s3:ListBucket",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ "s3:DeleteObject",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ "s3:GetObject",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ "s3:PutObject",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ "s3:PutObjectAcl"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ ],
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "Resource":\ [
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ "arn:aws:s3:::BUCKET_NAME/*",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ \ \ "arn:aws:s3:::BUCKET_NAME"
|
|
\ \ \ \ \ \ \ \ \ \ \ \ ]
|
|
\ \ \ \ \ \ \ \ }
|
|
\ \ \ \ ]
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Notes on above:
|
|
.IP "1." 3
|
|
This is a policy that can be used when creating bucket.
|
|
It assumes that \f[C]USER_NAME\f[] has been created.
|
|
.IP "2." 3
|
|
The Resource entry must include both resource ARNs, as one implies the
|
|
bucket and the other implies the bucket\[aq]s objects.
|
|
.PP
|
|
For reference, here\[aq]s an Ansible
|
|
script (https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b)
|
|
that will generate one or more buckets that will work with
|
|
\f[C]rclone\ sync\f[].
|
|
.SS Key Management System (KMS)
|
|
.PP
|
|
If you are using server side encryption with KMS then you will find you
|
|
can\[aq]t transfer small objects.
|
|
As a work\-around you can use the \f[C]\-\-ignore\-checksum\f[] flag.
|
|
.PP
|
|
A proper fix is being worked on in issue
|
|
#1824 (https://github.com/ncw/rclone/issues/1824).
|
|
.SS Glacier
|
|
.PP
|
|
You can transition objects to glacier storage using a lifecycle
|
|
policy (http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html).
|
|
The bucket can still be synced or copied into normally, but if rclone
|
|
tries to access the data you will see an error like below.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2017/09/11\ 19:07:43\ Failed\ to\ sync:\ failed\ to\ open\ source\ object:\ Object\ in\ GLACIER,\ restore\ first:\ path/to/file
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
In this case you need to
|
|
restore (http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html)
|
|
the object(s) in question before using rclone.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-s3\-acl=STRING
|
|
.PP
|
|
Canned ACL used when creating buckets and/or storing objects in S3.
|
|
.PP
|
|
For more info visit the canned ACL
|
|
docs (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl).
|
|
.SS \-\-s3\-storage\-class=STRING
|
|
.PP
|
|
Storage class to upload new objects with.
|
|
.PP
|
|
Available options include:
|
|
.IP \[bu] 2
|
|
STANDARD \- default storage class
|
|
.IP \[bu] 2
|
|
STANDARD_IA \- for less frequently accessed data (e.g backups)
|
|
.IP \[bu] 2
|
|
ONEZONE_IA \- for storing data in only one Availability Zone
|
|
.IP \[bu] 2
|
|
REDUCED_REDUNDANCY (only for noncritical, reproducible data, has lower
|
|
redundancy)
|
|
.SS \-\-s3\-chunk\-size=SIZE
|
|
.PP
|
|
Any files larger than this will be uploaded in chunks of this size.
|
|
The default is 5MB.
|
|
The minimum is 5MB.
|
|
.PP
|
|
Note that 2 chunks of this size are buffered in memory per transfer.
|
|
.PP
|
|
If you are transferring large files over high speed links and you have
|
|
enough memory, then increasing this will speed up the transfers.
|
|
.SS \-\-s3\-upload\-concurrency
|
|
.PP
|
|
Number of chunks of the same file that are uploaded concurrently.
|
|
Default is 2.
|
|
.PP
|
|
If you are uploading small amount of large file over high speed link and
|
|
these uploads do not fully utilize your bandwidth, then increasing this
|
|
may help to speed up the transfers.
|
|
.SS Anonymous access to public buckets
|
|
.PP
|
|
If you want to use rclone to access a public bucket, configure with a
|
|
blank \f[C]access_key_id\f[] and \f[C]secret_access_key\f[].
|
|
Your config should end up looking like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[anons3]
|
|
type\ =\ s3
|
|
provider\ =\ AWS
|
|
env_auth\ =\ false
|
|
access_key_id\ =\
|
|
secret_access_key\ =\
|
|
region\ =\ us\-east\-1
|
|
endpoint\ =\
|
|
location_constraint\ =\
|
|
acl\ =\ private
|
|
server_side_encryption\ =\
|
|
storage_class\ =\
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then use it as normal with the name of the public bucket, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ anons3:1000genomes
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You will be able to list and copy data but not upload it.
|
|
.SS Ceph
|
|
.PP
|
|
Ceph (https://ceph.com/) is an open source unified, distributed storage
|
|
system designed for excellent performance, reliability and scalability.
|
|
It has an S3 compatible object storage interface.
|
|
.PP
|
|
To use rclone with Ceph, configure as above but leave the region blank
|
|
and set the endpoint.
|
|
You should end up with something like this in your config:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[ceph]
|
|
type\ =\ s3
|
|
provider\ =\ Ceph
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ XXX
|
|
secret_access_key\ =\ YYY
|
|
region\ =
|
|
endpoint\ =\ https://ceph.endpoint.example.com
|
|
location_constraint\ =
|
|
acl\ =
|
|
server_side_encryption\ =
|
|
storage_class\ =
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note also that Ceph sometimes puts \f[C]/\f[] in the passwords it gives
|
|
users.
|
|
If you read the secret access key using the command line tools you will
|
|
get a JSON blob with the \f[C]/\f[] escaped as \f[C]\\/\f[].
|
|
Make sure you only write \f[C]/\f[] in the secret access key.
|
|
.PP
|
|
Eg the dump from Ceph looks something like this (irrelevant keys
|
|
removed).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
{
|
|
\ \ \ \ "user_id":\ "xxx",
|
|
\ \ \ \ "display_name":\ "xxxx",
|
|
\ \ \ \ "keys":\ [
|
|
\ \ \ \ \ \ \ \ {
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "user":\ "xxx",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "access_key":\ "xxxxxx",
|
|
\ \ \ \ \ \ \ \ \ \ \ \ "secret_key":\ "xxxxxx\\/xxxx"
|
|
\ \ \ \ \ \ \ \ }
|
|
\ \ \ \ ],
|
|
}
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Because this is a json dump, it is encoding the \f[C]/\f[] as
|
|
\f[C]\\/\f[], so if you use the secret key as \f[C]xxxxxx/xxxx\f[] it
|
|
will work fine.
|
|
.SS Dreamhost
|
|
.PP
|
|
Dreamhost DreamObjects (https://www.dreamhost.com/cloud/storage/) is an
|
|
object storage system based on CEPH.
|
|
.PP
|
|
To use rclone with Dreamhost, configure as above but leave the region
|
|
blank and set the endpoint.
|
|
You should end up with something like this in your config:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[dreamobjects]
|
|
type\ =\ s3
|
|
provider\ =\ DreamHost
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ your_access_key
|
|
secret_access_key\ =\ your_secret_key
|
|
region\ =
|
|
endpoint\ =\ objects\-us\-west\-1.dream.io
|
|
location_constraint\ =
|
|
acl\ =\ private
|
|
server_side_encryption\ =
|
|
storage_class\ =
|
|
\f[]
|
|
.fi
|
|
.SS DigitalOcean Spaces
|
|
.PP
|
|
Spaces (https://www.digitalocean.com/products/object-storage/) is an
|
|
S3\-interoperable (https://developers.digitalocean.com/documentation/spaces/)
|
|
object storage service from cloud provider DigitalOcean.
|
|
.PP
|
|
To connect to DigitalOcean Spaces you will need an access key and secret
|
|
key.
|
|
These can be retrieved on the "Applications &
|
|
API (https://cloud.digitalocean.com/settings/api/tokens)" page of the
|
|
DigitalOcean control panel.
|
|
They will be needed when promted by \f[C]rclone\ config\f[] for your
|
|
\f[C]access_key_id\f[] and \f[C]secret_access_key\f[].
|
|
.PP
|
|
When prompted for a \f[C]region\f[] or \f[C]location_constraint\f[],
|
|
press enter to use the default value.
|
|
The region must be included in the \f[C]endpoint\f[] setting (e.g.
|
|
\f[C]nyc3.digitaloceanspaces.com\f[]).
|
|
The defualt values can be used for other settings.
|
|
.PP
|
|
Going through the whole process of creating a new remote by running
|
|
\f[C]rclone\ config\f[], each prompt should be answered as shown below:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Storage>\ s3
|
|
env_auth>\ 1
|
|
access_key_id>\ YOUR_ACCESS_KEY
|
|
secret_access_key>\ YOUR_SECRET_KEY
|
|
region>
|
|
endpoint>\ nyc3.digitaloceanspaces.com
|
|
location_constraint>
|
|
acl>
|
|
storage_class>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The resulting configuration file should look like:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[spaces]
|
|
type\ =\ s3
|
|
provider\ =\ DigitalOcean
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ YOUR_ACCESS_KEY
|
|
secret_access_key\ =\ YOUR_SECRET_KEY
|
|
region\ =
|
|
endpoint\ =\ nyc3.digitaloceanspaces.com
|
|
location_constraint\ =
|
|
acl\ =
|
|
server_side_encryption\ =
|
|
storage_class\ =
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Once configured, you can create a new Space and begin copying files.
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ spaces:my\-new\-space
|
|
rclone\ copy\ /path/to/files\ spaces:my\-new\-space
|
|
\f[]
|
|
.fi
|
|
.SS IBM COS (S3)
|
|
.PP
|
|
Information stored with IBM Cloud Object Storage is encrypted and
|
|
dispersed across multiple geographic locations, and accessed through an
|
|
implementation of the S3 API.
|
|
This service makes use of the distributed storage technologies provided
|
|
by IBM's Cloud Object Storage System (formerly Cleversafe).
|
|
For more information visit: (http://www.ibm.com/cloud/object\-storage)
|
|
.PP
|
|
To configure access to IBM COS S3, follow the steps below:
|
|
.IP " 1." 4
|
|
Run rclone config and select n for a new remote.
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
2018/02/14\ 14:13:11\ NOTICE:\ Config\ file\ "C:\\\\Users\\\\a\\\\.config\\\\rclone\\\\rclone.conf"\ not\ found\ \-\ using\ defaults
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 2." 4
|
|
Enter the name for the configuration
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
name>\ <YOUR\ NAME>
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 3." 4
|
|
Select "s3" storage.
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
1\ /\ Alias\ for\ a\ existing\ remote
|
|
\\\ "alias"
|
|
2\ /\ Amazon\ Drive
|
|
\\\ "amazon\ cloud\ drive"
|
|
3\ /\ Amazon\ S3\ Complaint\ Storage\ Providers\ (Dreamhost,\ Ceph,\ Minio,\ IBM\ COS)
|
|
\\\ "s3"
|
|
4\ /\ Backblaze\ B2
|
|
\\\ "b2"
|
|
[snip]
|
|
23\ /\ http\ Connection
|
|
\\\ "http"
|
|
Storage>\ 3
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 4." 4
|
|
Select IBM COS as the S3 Storage Provider.
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose\ the\ S3\ provider.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Choose\ this\ option\ to\ configure\ Storage\ to\ AWS\ S3
|
|
\ \ \ \\\ "AWS"
|
|
\ 2\ /\ Choose\ this\ option\ to\ configure\ Storage\ to\ Ceph\ Systems
|
|
\ \\\ "Ceph"
|
|
\ 3\ /\ \ Choose\ this\ option\ to\ configure\ Storage\ to\ Dreamhost
|
|
\ \\\ "Dreamhost"
|
|
\ \ \ 4\ /\ Choose\ this\ option\ to\ the\ configure\ Storage\ to\ IBM\ COS\ S3
|
|
\ \\\ "IBMCOS"
|
|
\ 5\ /\ Choose\ this\ option\ to\ the\ configure\ Storage\ to\ Minio
|
|
\ \\\ "Minio"
|
|
\ Provider>4
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 5." 4
|
|
Enter the Access Key and Secret.
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
AWS\ Access\ Key\ ID\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
access_key_id>\ <>
|
|
AWS\ Secret\ Access\ Key\ (password)\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
secret_access_key>\ <>
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 6." 4
|
|
Specify the endpoint for IBM COS.
|
|
For Public IBM COS, choose from the option below.
|
|
For On Premise IBM COS, enter an enpoint address.
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Endpoint\ for\ IBM\ COS\ S3\ API.
|
|
Specify\ if\ using\ an\ IBM\ COS\ On\ Premise.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ US\ Cross\ Region\ Endpoint
|
|
\ \ \ \\\ "s3\-api.us\-geo.objectstorage.softlayer.net"
|
|
\ 2\ /\ US\ Cross\ Region\ Dallas\ Endpoint
|
|
\ \ \ \\\ "s3\-api.dal.us\-geo.objectstorage.softlayer.net"
|
|
\ 3\ /\ US\ Cross\ Region\ Washington\ DC\ Endpoint
|
|
\ \ \ \\\ "s3\-api.wdc\-us\-geo.objectstorage.softlayer.net"
|
|
\ 4\ /\ US\ Cross\ Region\ San\ Jose\ Endpoint
|
|
\ \ \ \\\ "s3\-api.sjc\-us\-geo.objectstorage.softlayer.net"
|
|
\ 5\ /\ US\ Cross\ Region\ Private\ Endpoint
|
|
\ \ \ \\\ "s3\-api.us\-geo.objectstorage.service.networklayer.com"
|
|
\ 6\ /\ US\ Cross\ Region\ Dallas\ Private\ Endpoint
|
|
\ \ \ \\\ "s3\-api.dal\-us\-geo.objectstorage.service.networklayer.com"
|
|
\ 7\ /\ US\ Cross\ Region\ Washington\ DC\ Private\ Endpoint
|
|
\ \ \ \\\ "s3\-api.wdc\-us\-geo.objectstorage.service.networklayer.com"
|
|
\ 8\ /\ US\ Cross\ Region\ San\ Jose\ Private\ Endpoint
|
|
\ \ \ \\\ "s3\-api.sjc\-us\-geo.objectstorage.service.networklayer.com"
|
|
\ 9\ /\ US\ Region\ East\ Endpoint
|
|
\ \ \ \\\ "s3.us\-east.objectstorage.softlayer.net"
|
|
10\ /\ US\ Region\ East\ Private\ Endpoint
|
|
\ \ \ \\\ "s3.us\-east.objectstorage.service.networklayer.com"
|
|
11\ /\ US\ Region\ South\ Endpoint
|
|
[snip]
|
|
34\ /\ Toronto\ Single\ Site\ Private\ Endpoint
|
|
\ \ \ \\\ "s3.tor01.objectstorage.service.networklayer.com"
|
|
endpoint>1
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 7." 4
|
|
Specify a IBM COS Location Constraint.
|
|
The location constraint must match endpoint when using IBM Cloud Public.
|
|
For on\-prem COS, do not make a selection from this list, hit enter
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ 1\ /\ US\ Cross\ Region\ Standard
|
|
\ \ \ \\\ "us\-standard"
|
|
\ 2\ /\ US\ Cross\ Region\ Vault
|
|
\ \ \ \\\ "us\-vault"
|
|
\ 3\ /\ US\ Cross\ Region\ Cold
|
|
\ \ \ \\\ "us\-cold"
|
|
\ 4\ /\ US\ Cross\ Region\ Flex
|
|
\ \ \ \\\ "us\-flex"
|
|
\ 5\ /\ US\ East\ Region\ Standard
|
|
\ \ \ \\\ "us\-east\-standard"
|
|
\ 6\ /\ US\ East\ Region\ Vault
|
|
\ \ \ \\\ "us\-east\-vault"
|
|
\ 7\ /\ US\ East\ Region\ Cold
|
|
\ \ \ \\\ "us\-east\-cold"
|
|
\ 8\ /\ US\ East\ Region\ Flex
|
|
\ \ \ \\\ "us\-east\-flex"
|
|
\ 9\ /\ US\ South\ Region\ Standard
|
|
\ \ \ \\\ "us\-south\-standard"
|
|
10\ /\ US\ South\ Region\ Vault
|
|
\ \ \ \\\ "us\-south\-vault"
|
|
[snip]
|
|
32\ /\ Toronto\ Flex
|
|
\ \ \ \\\ "tor01\-flex"
|
|
location_constraint>1
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 8." 4
|
|
Specify a canned ACL.
|
|
IBM Cloud (Strorage) supports "public\-read" and "private".
|
|
IBM Cloud(Infra) supports all the canned ACLs.
|
|
On\-Premise COS supports all the canned ACLs.
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Canned\ ACL\ used\ when\ creating\ buckets\ and/or\ storing\ objects\ in\ S3.
|
|
For\ more\ info\ visit\ https://docs.aws.amazon.com/AmazonS3/latest/dev/acl\-overview.html#canned\-acl
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ \ 1\ /\ Owner\ gets\ FULL_CONTROL.\ No\ one\ else\ has\ access\ rights\ (default).\ This\ acl\ is\ available\ on\ IBM\ Cloud\ (Infra),\ IBM\ Cloud\ (Storage),\ On\-Premise\ COS
|
|
\ \ \\\ "private"
|
|
\ \ 2\ \ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ access.\ This\ acl\ is\ available\ on\ IBM\ Cloud\ (Infra),\ IBM\ Cloud\ (Storage),\ On\-Premise\ IBM\ COS
|
|
\ \ \\\ "public\-read"
|
|
\ \ 3\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ and\ WRITE\ access.\ This\ acl\ is\ available\ on\ IBM\ Cloud\ (Infra),\ On\-Premise\ IBM\ COS
|
|
\ \ \\\ "public\-read\-write"
|
|
\ \ 4\ \ /\ Owner\ gets\ FULL_CONTROL.\ The\ AuthenticatedUsers\ group\ gets\ READ\ access.\ Not\ supported\ on\ Buckets.\ This\ acl\ is\ available\ on\ IBM\ Cloud\ (Infra)\ and\ On\-Premise\ IBM\ COS
|
|
\ \ \\\ "authenticated\-read"
|
|
acl>\ 1
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP " 9." 4
|
|
Review the displayed configuration and accept to save the "remote" then
|
|
quit.
|
|
The config file should look like this
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[xxx]
|
|
type\ =\ s3
|
|
Provider\ =\ IBMCOS
|
|
access_key_id\ =\ xxx
|
|
secret_access_key\ =\ yyy
|
|
endpoint\ =\ s3\-api.us\-geo.objectstorage.softlayer.net
|
|
location_constraint\ =\ us\-standard
|
|
acl\ =\ private
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.IP "10." 4
|
|
Execute rclone commands
|
|
.RS 4
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
1)\ \ Create\ a\ bucket.
|
|
\ \ \ \ rclone\ mkdir\ IBM\-COS\-XREGION:newbucket
|
|
2)\ \ List\ available\ buckets.
|
|
\ \ \ \ rclone\ lsd\ IBM\-COS\-XREGION:
|
|
\ \ \ \ \-1\ 2017\-11\-08\ 21:16:22\ \ \ \ \ \ \ \ \-1\ test
|
|
\ \ \ \ \-1\ 2018\-02\-14\ 20:16:39\ \ \ \ \ \ \ \ \-1\ newbucket
|
|
3)\ \ List\ contents\ of\ a\ bucket.
|
|
\ \ \ \ rclone\ ls\ IBM\-COS\-XREGION:newbucket
|
|
\ \ \ \ 18685952\ test.exe
|
|
4)\ \ Copy\ a\ file\ from\ local\ to\ remote.
|
|
\ \ \ \ rclone\ copy\ /Users/file.txt\ IBM\-COS\-XREGION:newbucket
|
|
5)\ \ Copy\ a\ file\ from\ remote\ to\ local.
|
|
\ \ \ \ rclone\ copy\ IBM\-COS\-XREGION:newbucket/file.txt\ .
|
|
6)\ \ Delete\ a\ file\ on\ remote.
|
|
\ \ \ \ rclone\ delete\ IBM\-COS\-XREGION:newbucket/file.txt
|
|
\f[]
|
|
.fi
|
|
.RE
|
|
.SS Minio
|
|
.PP
|
|
Minio (https://minio.io/) is an object storage server built for cloud
|
|
application developers and devops.
|
|
.PP
|
|
It is very easy to install and provides an S3 compatible server which
|
|
can be used by rclone.
|
|
.PP
|
|
To use it, install Minio following the instructions
|
|
here (https://docs.minio.io/docs/minio-quickstart-guide).
|
|
.PP
|
|
When it configures itself Minio will print something like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Endpoint:\ \ http://192.168.1.106:9000\ \ http://172.23.0.1:9000
|
|
AccessKey:\ USWUXHGYZQYFYFFIT3RE
|
|
SecretKey:\ MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
Region:\ \ \ \ us\-east\-1
|
|
SQS\ ARNs:\ \ arn:minio:sqs:us\-east\-1:1:redis\ arn:minio:sqs:us\-east\-1:2:redis
|
|
|
|
Browser\ Access:
|
|
\ \ \ http://192.168.1.106:9000\ \ http://172.23.0.1:9000
|
|
|
|
Command\-line\ Access:\ https://docs.minio.io/docs/minio\-client\-quickstart\-guide
|
|
\ \ \ $\ mc\ config\ host\ add\ myminio\ http://192.168.1.106:9000\ USWUXHGYZQYFYFFIT3RE\ MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
|
|
Object\ API\ (Amazon\ S3\ compatible):
|
|
\ \ \ Go:\ \ \ \ \ \ \ \ \ https://docs.minio.io/docs/golang\-client\-quickstart\-guide
|
|
\ \ \ Java:\ \ \ \ \ \ \ https://docs.minio.io/docs/java\-client\-quickstart\-guide
|
|
\ \ \ Python:\ \ \ \ \ https://docs.minio.io/docs/python\-client\-quickstart\-guide
|
|
\ \ \ JavaScript:\ https://docs.minio.io/docs/javascript\-client\-quickstart\-guide
|
|
\ \ \ .NET:\ \ \ \ \ \ \ https://docs.minio.io/docs/dotnet\-client\-quickstart\-guide
|
|
|
|
Drive\ Capacity:\ 26\ GiB\ Free,\ 165\ GiB\ Total
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
These details need to go into \f[C]rclone\ config\f[] like this.
|
|
Note that it is important to put the region in as stated above.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
env_auth>\ 1
|
|
access_key_id>\ USWUXHGYZQYFYFFIT3RE
|
|
secret_access_key>\ MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
region>\ us\-east\-1
|
|
endpoint>\ http://192.168.1.106:9000
|
|
location_constraint>
|
|
server_side_encryption>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Which makes the config file look like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[minio]
|
|
type\ =\ s3
|
|
provider\ =\ Minio
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ USWUXHGYZQYFYFFIT3RE
|
|
secret_access_key\ =\ MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
|
|
region\ =\ us\-east\-1
|
|
endpoint\ =\ http://192.168.1.106:9000
|
|
location_constraint\ =
|
|
server_side_encryption\ =
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
So once set up, for example to copy files into a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /path/to/files\ minio:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Wasabi
|
|
.PP
|
|
Wasabi (https://wasabi.com) is a cloud\-based object storage service for
|
|
a broad range of applications and use cases.
|
|
Wasabi is designed for individuals and organizations that require a
|
|
high\-performance, reliable, and secure data storage infrastructure at
|
|
minimal cost.
|
|
.PP
|
|
Wasabi provides an S3 interface which can be configured for use with
|
|
rclone like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ wasabi
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
[snip]
|
|
Storage>\ s3
|
|
Get\ AWS\ credentials\ from\ runtime\ (environment\ variables\ or\ EC2/ECS\ meta\ data\ if\ no\ env\ vars).\ Only\ applies\ if\ access_key_id\ and\ secret_access_key\ is\ blank.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Enter\ AWS\ credentials\ in\ the\ next\ step
|
|
\ \ \ \\\ "false"
|
|
\ 2\ /\ Get\ AWS\ credentials\ from\ the\ environment\ (env\ vars\ or\ IAM)
|
|
\ \ \ \\\ "true"
|
|
env_auth>\ 1
|
|
AWS\ Access\ Key\ ID\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
access_key_id>\ YOURACCESSKEY
|
|
AWS\ Secret\ Access\ Key\ (password)\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
secret_access_key>\ YOURSECRETACCESSKEY
|
|
Region\ to\ connect\ to.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ \ \ /\ The\ default\ endpoint\ \-\ a\ good\ choice\ if\ you\ are\ unsure.
|
|
\ 1\ |\ US\ Region,\ Northern\ Virginia\ or\ Pacific\ Northwest.
|
|
\ \ \ |\ Leave\ location\ constraint\ empty.
|
|
\ \ \ \\\ "us\-east\-1"
|
|
[snip]
|
|
region>\ us\-east\-1
|
|
Endpoint\ for\ S3\ API.
|
|
Leave\ blank\ if\ using\ AWS\ to\ use\ the\ default\ endpoint\ for\ the\ region.
|
|
Specify\ if\ using\ an\ S3\ clone\ such\ as\ Ceph.
|
|
endpoint>\ s3.wasabisys.com
|
|
Location\ constraint\ \-\ must\ be\ set\ to\ match\ the\ Region.\ Used\ when\ creating\ buckets\ only.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Empty\ for\ US\ Region,\ Northern\ Virginia\ or\ Pacific\ Northwest.
|
|
\ \ \ \\\ ""
|
|
[snip]
|
|
location_constraint>
|
|
Canned\ ACL\ used\ when\ creating\ buckets\ and/or\ storing\ objects\ in\ S3.
|
|
For\ more\ info\ visit\ https://docs.aws.amazon.com/AmazonS3/latest/dev/acl\-overview.html#canned\-acl
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Owner\ gets\ FULL_CONTROL.\ No\ one\ else\ has\ access\ rights\ (default).
|
|
\ \ \ \\\ "private"
|
|
[snip]
|
|
acl>
|
|
The\ server\-side\ encryption\ algorithm\ used\ when\ storing\ this\ object\ in\ S3.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ None
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ AES256
|
|
\ \ \ \\\ "AES256"
|
|
server_side_encryption>
|
|
The\ storage\ class\ to\ use\ when\ storing\ objects\ in\ S3.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Default
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ Standard\ storage\ class
|
|
\ \ \ \\\ "STANDARD"
|
|
\ 3\ /\ Reduced\ redundancy\ storage\ class
|
|
\ \ \ \\\ "REDUCED_REDUNDANCY"
|
|
\ 4\ /\ Standard\ Infrequent\ Access\ storage\ class
|
|
\ \ \ \\\ "STANDARD_IA"
|
|
storage_class>
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[wasabi]
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ YOURACCESSKEY
|
|
secret_access_key\ =\ YOURSECRETACCESSKEY
|
|
region\ =\ us\-east\-1
|
|
endpoint\ =\ s3.wasabisys.com
|
|
location_constraint\ =
|
|
acl\ =
|
|
server_side_encryption\ =
|
|
storage_class\ =
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will leave the config file looking like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[wasabi]
|
|
type\ =\ s3
|
|
provider\ =\ Wasabi
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ YOURACCESSKEY
|
|
secret_access_key\ =\ YOURSECRETACCESSKEY
|
|
region\ =
|
|
endpoint\ =\ s3.wasabisys.com
|
|
location_constraint\ =
|
|
acl\ =
|
|
server_side_encryption\ =
|
|
storage_class\ =
|
|
\f[]
|
|
.fi
|
|
.SS Backblaze B2
|
|
.PP
|
|
B2 is Backblaze\[aq]s cloud storage
|
|
system (https://www.backblaze.com/b2/).
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making a b2 configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
You will need your account number (a short hex number) and key (a long
|
|
hex number) which you can get from the b2 control panel.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
q)\ Quit\ config
|
|
n/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 3
|
|
Account\ ID
|
|
account>\ 123456789abc
|
|
Application\ Key
|
|
key>\ 0123456789abcdef0123456789abcdef0123456789
|
|
Endpoint\ for\ the\ service\ \-\ leave\ blank\ normally.
|
|
endpoint>
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
account\ =\ 123456789abc
|
|
key\ =\ 0123456789abcdef0123456789abcdef0123456789
|
|
endpoint\ =
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Bz\-Info\-src_last_modified_millis\f[] as milliseconds since
|
|
1970\-01\-01 in the Backblaze standard.
|
|
Other tools should be able to use this as a modified time.
|
|
.PP
|
|
Modified times are used in syncing and are fully supported except in the
|
|
case of updating a modification time on an existing object.
|
|
In this case the object will be uploaded again as B2 doesn\[aq]t have an
|
|
API method to set the modification time independent of doing an upload.
|
|
.SS SHA1 checksums
|
|
.PP
|
|
The SHA1 checksums of the files are checked on upload and download and
|
|
will be used in the syncing process.
|
|
.PP
|
|
Large files (bigger than the limit in \f[C]\-\-b2\-upload\-cutoff\f[])
|
|
which are uploaded in chunks will store their SHA1 on the object as
|
|
\f[C]X\-Bz\-Info\-large_file_sha1\f[] as recommended by Backblaze.
|
|
.PP
|
|
For a large file to be uploaded with an SHA1 checksum, the source needs
|
|
to support SHA1 checksums.
|
|
The local disk supports SHA1 checksums so large file transfers from
|
|
local disk will have an SHA1.
|
|
See the overview (/overview/#features) for exactly which remotes support
|
|
SHA1.
|
|
.PP
|
|
Sources which don\[aq]t support SHA1, in particular \f[C]crypt\f[] will
|
|
upload large files without SHA1 checksums.
|
|
This may be fixed in the future (see
|
|
#1767 (https://github.com/ncw/rclone/issues/1767)).
|
|
.PP
|
|
Files sizes below \f[C]\-\-b2\-upload\-cutoff\f[] will always have an
|
|
SHA1 regardless of the source.
|
|
.SS Transfers
|
|
.PP
|
|
Backblaze recommends that you do lots of transfers simultaneously for
|
|
maximum speed.
|
|
In tests from my SSD equipped laptop the optimum setting is about
|
|
\f[C]\-\-transfers\ 32\f[] though higher numbers may be used for a
|
|
slight speed improvement.
|
|
The optimum number for you may vary depending on your hardware, how big
|
|
the files are, how much you want to load your computer, etc.
|
|
The default of \f[C]\-\-transfers\ 4\f[] is definitely too low for
|
|
Backblaze B2 though.
|
|
.PP
|
|
Note that uploading big files (bigger than 200 MB by default) will use a
|
|
96 MB RAM buffer by default.
|
|
There can be at most \f[C]\-\-transfers\f[] of these in use at any
|
|
moment, so this sets the upper limit on the memory used.
|
|
.SS Versions
|
|
.PP
|
|
When rclone uploads a new version of a file it creates a new version of
|
|
it (https://www.backblaze.com/b2/docs/file_versions.html).
|
|
Likewise when you delete a file, the old version will be marked hidden
|
|
and still be available.
|
|
Conversely, you may opt in to a "hard delete" of files with the
|
|
\f[C]\-\-b2\-hard\-delete\f[] flag which would permanently remove the
|
|
file instead of hiding it.
|
|
.PP
|
|
Old versions of files, where available, are visible using the
|
|
\f[C]\-\-b2\-versions\f[] flag.
|
|
.PP
|
|
If you wish to remove all the old versions then you can use the
|
|
\f[C]rclone\ cleanup\ remote:bucket\f[] command which will delete all
|
|
the old versions of files, leaving the current ones intact.
|
|
You can also supply a path and only old versions under that path will be
|
|
deleted, eg \f[C]rclone\ cleanup\ remote:bucket/path/to/stuff\f[].
|
|
.PP
|
|
When you \f[C]purge\f[] a bucket, the current and the old versions will
|
|
be deleted then the bucket will be deleted.
|
|
.PP
|
|
However \f[C]delete\f[] will cause the current versions of the files to
|
|
become hidden old versions.
|
|
.PP
|
|
Here is a session showing the listing and retrieval of an old version
|
|
followed by a \f[C]cleanup\f[] of the old versions.
|
|
.PP
|
|
Show current version and all the versions with \f[C]\-\-b2\-versions\f[]
|
|
flag.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\ \ \ \ \ \ \ \ 8\ one\-v2016\-07\-04\-141032\-000.txt
|
|
\ \ \ \ \ \ \ 16\ one\-v2016\-07\-04\-141003\-000.txt
|
|
\ \ \ \ \ \ \ 15\ one\-v2016\-07\-02\-155621\-000.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Retrieve an old version
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ copy\ b2:cleanup\-test/one\-v2016\-07\-04\-141003\-000.txt\ /tmp
|
|
|
|
$\ ls\ \-l\ /tmp/one\-v2016\-07\-04\-141003\-000.txt
|
|
\-rw\-rw\-r\-\-\ 1\ ncw\ ncw\ 16\ Jul\ \ 2\ 17:46\ /tmp/one\-v2016\-07\-04\-141003\-000.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Clean up all the old versions and show that they\[aq]ve gone.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ cleanup\ b2:cleanup\-test
|
|
|
|
$\ rclone\ \-q\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\f[]
|
|
.fi
|
|
.SS Data usage
|
|
.PP
|
|
It is useful to know how many requests are sent to the server in
|
|
different scenarios.
|
|
.PP
|
|
All copy commands send the following 4 requests:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_authorize_account
|
|
/b2api/v1/b2_create_bucket
|
|
/b2api/v1/b2_list_buckets
|
|
/b2api/v1/b2_list_file_names
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The \f[C]b2_list_file_names\f[] request will be sent once for every 1k
|
|
files in the remote path, providing the checksum and modification time
|
|
of the listed files.
|
|
As of version 1.33 issue #818 (https://github.com/ncw/rclone/issues/818)
|
|
causes extra requests to be sent when using B2 with Crypt.
|
|
When a copy operation does not require any files to be uploaded, no more
|
|
requests will be sent.
|
|
.PP
|
|
Uploading files that do not require chunking, will send 2 requests per
|
|
file upload:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_get_upload_url
|
|
/b2api/v1/b2_upload_file/
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Uploading files requiring chunking, will send 2 requests (one each to
|
|
start and finish the upload) and another 2 requests for each chunk:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
/b2api/v1/b2_start_large_file
|
|
/b2api/v1/b2_get_upload_part_url
|
|
/b2api/v1/b2_upload_part/
|
|
/b2api/v1/b2_finish_large_file
|
|
\f[]
|
|
.fi
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-b2\-chunk\-size valuee=SIZE
|
|
.PP
|
|
When uploading large files chunk the file into this size.
|
|
Note that these chunks are buffered in memory and there might a maximum
|
|
of \f[C]\-\-transfers\f[] chunks in progress at once.
|
|
5,000,000 Bytes is the minimim size (default 96M).
|
|
.SS \-\-b2\-upload\-cutoff=SIZE
|
|
.PP
|
|
Cutoff for switching to chunked upload (default 190.735 MiB == 200 MB).
|
|
Files above this size will be uploaded in chunks of
|
|
\f[C]\-\-b2\-chunk\-size\f[].
|
|
.PP
|
|
This value should be set no larger than 4.657GiB (== 5GB) as this is the
|
|
largest file size that can be uploaded.
|
|
.SS \-\-b2\-test\-mode=FLAG
|
|
.PP
|
|
This is for debugging purposes only.
|
|
.PP
|
|
Setting FLAG to one of the strings below will cause b2 to return
|
|
specific errors for debugging purposes.
|
|
.IP \[bu] 2
|
|
\f[C]fail_some_uploads\f[]
|
|
.IP \[bu] 2
|
|
\f[C]expire_some_account_authorization_tokens\f[]
|
|
.IP \[bu] 2
|
|
\f[C]force_cap_exceeded\f[]
|
|
.PP
|
|
These will be set in the \f[C]X\-Bz\-Test\-Mode\f[] header which is
|
|
documented in the b2 integrations
|
|
checklist (https://www.backblaze.com/b2/docs/integration_checklist.html).
|
|
.SS \-\-b2\-versions
|
|
.PP
|
|
When set rclone will show and act on older versions of files.
|
|
For example
|
|
.PP
|
|
Listing without \f[C]\-\-b2\-versions\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And with
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ \-\-b2\-versions\ ls\ b2:cleanup\-test
|
|
\ \ \ \ \ \ \ \ 9\ one.txt
|
|
\ \ \ \ \ \ \ \ 8\ one\-v2016\-07\-04\-141032\-000.txt
|
|
\ \ \ \ \ \ \ 16\ one\-v2016\-07\-04\-141003\-000.txt
|
|
\ \ \ \ \ \ \ 15\ one\-v2016\-07\-02\-155621\-000.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Showing that the current version is unchanged but older versions can be
|
|
seen.
|
|
These have the UTC date that they were uploaded to the server to the
|
|
nearest millisecond appended to them.
|
|
.PP
|
|
Note that when using \f[C]\-\-b2\-versions\f[] no file write operations
|
|
are permitted, so you can\[aq]t upload files or delete them.
|
|
.SS Box
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for Box involves getting a token from Box which you
|
|
need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Box
|
|
\ \ \ \\\ "box"
|
|
\ 5\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 6\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 7\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 8\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 9\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
10\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
11\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
12\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
13\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
14\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
15\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
16\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ box
|
|
Box\ App\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Box\ App\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Box.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your Box
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your Box
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Box directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Invalid refresh token
|
|
.PP
|
|
According to the box
|
|
docs (https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens):
|
|
.RS
|
|
.PP
|
|
Each refresh_token is valid for one use in 60 days.
|
|
.RE
|
|
.PP
|
|
This means that if you
|
|
.IP \[bu] 2
|
|
Don\[aq]t use the box remote for 60 days
|
|
.IP \[bu] 2
|
|
Copy the config file with a box refresh token in and use it in two
|
|
places
|
|
.IP \[bu] 2
|
|
Get an error on a token refresh
|
|
.PP
|
|
then rclone will return an error which includes the text
|
|
\f[C]Invalid\ refresh\ token\f[].
|
|
.PP
|
|
To fix this you will need to use oauth2 again to update the refresh
|
|
token.
|
|
You can use the methods in the remote setup
|
|
docs (https://rclone.org/remote_setup/), bearing in mind that if you use
|
|
the copy the config file method, you should not use that remote on the
|
|
computer you did the authentication on.
|
|
.PP
|
|
Here is how to do it.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ config
|
|
Current\ remotes:
|
|
|
|
Name\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Type
|
|
====\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ====
|
|
remote\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ box
|
|
|
|
e)\ Edit\ existing\ remote
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
e/n/d/r/c/s/q>\ e
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ an\ existing\ value
|
|
\ 1\ >\ remote
|
|
remote>\ remote
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
type\ =\ box
|
|
token\ =\ {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017\-07\-08T23:40:08.059167677+01:00"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
Edit\ remote
|
|
Value\ "client_id"\ =\ ""
|
|
Edit?\ (y/n)>
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ n
|
|
Value\ "client_secret"\ =\ ""
|
|
Edit?\ (y/n)>
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ n
|
|
Remote\ config
|
|
Already\ have\ a\ token\ \-\ refresh?
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
type\ =\ box
|
|
token\ =\ {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017\-07\-23T12:22:29.259137901+01:00"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Box allows modification times to be set on objects accurate to 1 second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
One drive supports SHA1 type hashes, so you can use the
|
|
\f[C]\-\-checksum\f[] flag.
|
|
.SS Transfers
|
|
.PP
|
|
For files above 50MB rclone will use a chunked transfer.
|
|
Rclone will upload up to \f[C]\-\-transfers\f[] chunks at the same time
|
|
(shared among all the multipart uploads).
|
|
Chunks are buffered in memory and are normally 8MB so increasing
|
|
\f[C]\-\-transfers\f[] will increase memory use.
|
|
.SS Deleting files
|
|
.PP
|
|
Depending on the enterprise settings for your user, the item will either
|
|
be actually deleted from Box or moved to the trash.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-box\-upload\-cutoff=SIZE
|
|
.PP
|
|
Cutoff for switching to chunked upload \- must be >= 50MB.
|
|
The default is 50MB.
|
|
.SS Limitations
|
|
.PP
|
|
Note that Box is case insensitive so you can\[aq]t have a file called
|
|
"Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
Box file names can\[aq]t have the \f[C]\\\f[] character in.
|
|
rclone maps this to and from an identical looking unicode equivalent
|
|
\f[C]＼\f[].
|
|
.PP
|
|
Box only supports filenames up to 255 characters in length.
|
|
.SS Cache (BETA)
|
|
.PP
|
|
The \f[C]cache\f[] remote wraps another existing remote and stores file
|
|
structure and its data for long running tasks like
|
|
\f[C]rclone\ mount\f[].
|
|
.PP
|
|
To get started you just need to have an existing remote which can be
|
|
configured with \f[C]cache\f[].
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]test\-cache\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/r/c/s/q>\ n
|
|
name>\ test\-cache
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\&...
|
|
\ 5\ /\ Cache\ a\ remote
|
|
\ \ \ \\\ "cache"
|
|
\&...
|
|
Storage>\ 5
|
|
Remote\ to\ cache.
|
|
Normally\ should\ contain\ a\ \[aq]:\[aq]\ and\ a\ path,\ eg\ "myremote:path/to/dir",
|
|
"myremote:bucket"\ or\ maybe\ "myremote:"\ (not\ recommended).
|
|
remote>\ local:/test
|
|
Optional:\ The\ URL\ of\ the\ Plex\ server
|
|
plex_url>\ http://127.0.0.1:32400
|
|
Optional:\ The\ username\ of\ the\ Plex\ user
|
|
plex_username>\ dummyusername
|
|
Optional:\ The\ password\ of\ the\ Plex\ user
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
n)\ No\ leave\ this\ optional\ password\ blank
|
|
y/g/n>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
The\ size\ of\ a\ chunk.\ Lower\ value\ good\ for\ slow\ connections\ but\ can\ affect\ seamless\ reading.
|
|
Default:\ 5M
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ 1MB
|
|
\ \ \ \\\ "1m"
|
|
\ 2\ /\ 5\ MB
|
|
\ \ \ \\\ "5M"
|
|
\ 3\ /\ 10\ MB
|
|
\ \ \ \\\ "10M"
|
|
chunk_size>\ 2
|
|
How\ much\ time\ should\ object\ info\ (file\ size,\ file\ hashes\ etc)\ be\ stored\ in\ cache.\ Use\ a\ very\ high\ value\ if\ you\ don\[aq]t\ plan\ on\ changing\ the\ source\ FS\ from\ outside\ the\ cache.
|
|
Accepted\ units\ are:\ "s",\ "m",\ "h".
|
|
Default:\ 5m
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ 1\ hour
|
|
\ \ \ \\\ "1h"
|
|
\ 2\ /\ 24\ hours
|
|
\ \ \ \\\ "24h"
|
|
\ 3\ /\ 24\ hours
|
|
\ \ \ \\\ "48h"
|
|
info_age>\ 2
|
|
The\ maximum\ size\ of\ stored\ chunks.\ When\ the\ storage\ grows\ beyond\ this\ size,\ the\ oldest\ chunks\ will\ be\ deleted.
|
|
Default:\ 10G
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ 500\ MB
|
|
\ \ \ \\\ "500M"
|
|
\ 2\ /\ 1\ GB
|
|
\ \ \ \\\ "1G"
|
|
\ 3\ /\ 10\ GB
|
|
\ \ \ \\\ "10G"
|
|
chunk_total_size>\ 3
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[test\-cache]
|
|
remote\ =\ local:/test
|
|
plex_url\ =\ http://127.0.0.1:32400
|
|
plex_username\ =\ dummyusername
|
|
plex_password\ =\ ***\ ENCRYPTED\ ***
|
|
chunk_size\ =\ 5M
|
|
info_age\ =\ 48h
|
|
chunk_total_size\ =\ 10G
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ test\-cache:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ test\-cache:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To start a cached mount
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mount\ \-\-allow\-other\ test\-cache:\ /var/tmp/test\-cache
|
|
\f[]
|
|
.fi
|
|
.SS Write Features
|
|
.SS Offline uploading
|
|
.PP
|
|
In an effort to make writing through cache more reliable, the backend
|
|
now supports this feature which can be activated by specifying a
|
|
\f[C]cache\-tmp\-upload\-path\f[].
|
|
.PP
|
|
A files goes through these states when using this feature:
|
|
.IP "1." 3
|
|
An upload is started (usually by copying a file on the cache remote)
|
|
.IP "2." 3
|
|
When the copy to the temporary location is complete the file is part of
|
|
the cached remote and looks and behaves like any other file (reading
|
|
included)
|
|
.IP "3." 3
|
|
After \f[C]cache\-tmp\-wait\-time\f[] passes and the file is next in
|
|
line, \f[C]rclone\ move\f[] is used to move the file to the cloud
|
|
provider
|
|
.IP "4." 3
|
|
Reading the file still works during the upload but most modifications on
|
|
it will be prohibited
|
|
.IP "5." 3
|
|
Once the move is complete the file is unlocked for modifications as it
|
|
becomes as any other regular file
|
|
.IP "6." 3
|
|
If the file is being read through \f[C]cache\f[] when it\[aq]s actually
|
|
deleted from the temporary path then \f[C]cache\f[] will simply swap the
|
|
source to the cloud provider without interrupting the reading (small
|
|
blip can happen though)
|
|
.PP
|
|
Files are uploaded in sequence and only one file is uploaded at a time.
|
|
Uploads will be stored in a queue and be processed based on the order
|
|
they were added.
|
|
The queue and the temporary storage is persistent across restarts and
|
|
even purges of the cache.
|
|
.SS Write Support
|
|
.PP
|
|
Writes are supported through \f[C]cache\f[].
|
|
One caveat is that a mounted cache remote does not add any retry or
|
|
fallback mechanism to the upload operation.
|
|
This will depend on the implementation of the wrapped remote.
|
|
Consider using \f[C]Offline\ uploading\f[] for reliable writes.
|
|
.PP
|
|
One special case is covered with \f[C]cache\-writes\f[] which will cache
|
|
the file data at the same time as the upload when it is enabled making
|
|
it available from the cache store immediately once the upload is
|
|
finished.
|
|
.SS Read Features
|
|
.SS Multiple connections
|
|
.PP
|
|
To counter the high latency between a local PC where rclone is running
|
|
and cloud providers, the cache remote can split multiple requests to the
|
|
cloud provider for smaller file chunks and combines them together
|
|
locally where they can be available almost immediately before the reader
|
|
usually needs them.
|
|
.PP
|
|
This is similar to buffering when media files are played online.
|
|
Rclone will stay around the current marker but always try its best to
|
|
stay ahead and prepare the data before.
|
|
.SS Plex Integration
|
|
.PP
|
|
There is a direct integration with Plex which allows cache to detect
|
|
during reading if the file is in playback or not.
|
|
This helps cache to adapt how it queries the cloud provider depending on
|
|
what is needed for.
|
|
.PP
|
|
Scans will have a minimum amount of workers (1) while in a confirmed
|
|
playback cache will deploy the configured number of workers.
|
|
.PP
|
|
This integration opens the doorway to additional performance
|
|
improvements which will be explored in the near future.
|
|
.PP
|
|
\f[B]Note:\f[] If Plex options are not configured, \f[C]cache\f[] will
|
|
function with its configured options without adapting any of its
|
|
settings.
|
|
.PP
|
|
How to enable?
|
|
Run \f[C]rclone\ config\f[] and add all the Plex options (endpoint,
|
|
username and password) in your remote and it will be automatically
|
|
enabled.
|
|
.PP
|
|
Affected settings: \- \f[C]cache\-workers\f[]: \f[I]Configured value\f[]
|
|
during confirmed playback or \f[I]1\f[] all the other times
|
|
.SS Known issues
|
|
.SS Mount and \-\-dir\-cache\-time
|
|
.PP
|
|
\-\-dir\-cache\-time controls the first layer of directory caching which
|
|
works at the mount layer.
|
|
Being an independent caching mechanism from the \f[C]cache\f[] backend,
|
|
it will manage its own entries based on the configured time.
|
|
.PP
|
|
To avoid getting in a scenario where dir cache has obsolete data and
|
|
cache would have the correct one, try to set
|
|
\f[C]\-\-dir\-cache\-time\f[] to a lower time than
|
|
\f[C]\-\-cache\-info\-age\f[].
|
|
Default values are already configured in this way.
|
|
.SS Windows support \- Experimental
|
|
.PP
|
|
There are a couple of issues with Windows \f[C]mount\f[] functionality
|
|
that still require some investigations.
|
|
It should be considered as experimental thus far as fixes come in for
|
|
this OS.
|
|
.PP
|
|
Most of the issues seem to be related to the difference between
|
|
filesystems on Linux flavors and Windows as cache is heavily dependant
|
|
on them.
|
|
.PP
|
|
Any reports or feedback on how cache behaves on this OS is greatly
|
|
appreciated.
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone/issues/1935
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone/issues/1907
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone/issues/1834
|
|
.SS Risk of throttling
|
|
.PP
|
|
Future iterations of the cache backend will make use of the pooling
|
|
functionality of the cloud provider to synchronize and at the same time
|
|
make writing through it more tolerant to failures.
|
|
.PP
|
|
There are a couple of enhancements in track to add these but in the
|
|
meantime there is a valid concern that the expiring cache listings can
|
|
lead to cloud provider throttles or bans due to repeated queries on it
|
|
for very large mounts.
|
|
.PP
|
|
Some recommendations: \- don\[aq]t use a very small interval for entry
|
|
informations (\f[C]\-\-cache\-info\-age\f[]) \- while writes aren\[aq]t
|
|
yet optimised, you can still write through \f[C]cache\f[] which gives
|
|
you the advantage of adding the file in the cache at the same time if
|
|
configured to do so.
|
|
.PP
|
|
Future enhancements:
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone/issues/1937
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone/issues/1936
|
|
.SS cache and crypt
|
|
.PP
|
|
One common scenario is to keep your data encrypted in the cloud provider
|
|
using the \f[C]crypt\f[] remote.
|
|
\f[C]crypt\f[] uses a similar technique to wrap around an existing
|
|
remote and handles this translation in a seamless way.
|
|
.PP
|
|
There is an issue with wrapping the remotes in this order: \f[B]cloud
|
|
remote\f[] \-> \f[B]crypt\f[] \-> \f[B]cache\f[]
|
|
.PP
|
|
During testing, I experienced a lot of bans with the remotes in this
|
|
order.
|
|
I suspect it might be related to how crypt opens files on the cloud
|
|
provider which makes it think we\[aq]re downloading the full file
|
|
instead of small chunks.
|
|
Organizing the remotes in this order yelds better results: \f[B]cloud
|
|
remote\f[] \-> \f[B]cache\f[] \-> \f[B]crypt\f[]
|
|
.SS Cache and Remote Control (\-\-rc)
|
|
.PP
|
|
Cache supports the new \f[C]\-\-rc\f[] mode in rclone and can be remote
|
|
controlled through the following end points: By default, the listener is
|
|
disabled if you do not add the flag.
|
|
.SS rc cache/expire
|
|
.PP
|
|
Purge a remote from the cache backend.
|
|
Supports either a directory or a file.
|
|
It supports both encrypted and unencrypted file names if cache is
|
|
wrapped by crypt.
|
|
.PP
|
|
Params: \- \f[B]remote\f[] = path to remote \f[B](required)\f[] \-
|
|
\f[B]withData\f[] = true/false to delete cached data (chunks) as well
|
|
\f[I](optional, false by default)\f[]
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-cache\-chunk\-path=PATH
|
|
.PP
|
|
Path to where partial file data (chunks) is stored locally.
|
|
The remote name is appended to the final path.
|
|
.PP
|
|
This config follows the \f[C]\-\-cache\-db\-path\f[].
|
|
If you specify a custom location for \f[C]\-\-cache\-db\-path\f[] and
|
|
don\[aq]t specify one for \f[C]\-\-cache\-chunk\-path\f[] then
|
|
\f[C]\-\-cache\-chunk\-path\f[] will use the same path as
|
|
\f[C]\-\-cache\-db\-path\f[].
|
|
.PP
|
|
\f[B]Default\f[]: /cache\-backend/ \f[B]Example\f[]:
|
|
/.cache/cache\-backend/test\-cache
|
|
.SS \-\-cache\-db\-path=PATH
|
|
.PP
|
|
Path to where the file structure metadata (DB) is stored locally.
|
|
The remote name is used as the DB file name.
|
|
.PP
|
|
\f[B]Default\f[]: /cache\-backend/ \f[B]Example\f[]:
|
|
/.cache/cache\-backend/test\-cache
|
|
.SS \-\-cache\-db\-purge
|
|
.PP
|
|
Flag to clear all the cached data for this remote before.
|
|
.PP
|
|
\f[B]Default\f[]: not set
|
|
.SS \-\-cache\-chunk\-size=SIZE
|
|
.PP
|
|
The size of a chunk (partial file data).
|
|
Use lower numbers for slower connections.
|
|
If the chunk size is changed, any downloaded chunks will be invalid and
|
|
cache\-chunk\-path will need to be cleared or unexpected EOF errors will
|
|
occur.
|
|
.PP
|
|
\f[B]Default\f[]: 5M
|
|
.SS \-\-cache\-total\-chunk\-size=SIZE
|
|
.PP
|
|
The total size that the chunks can take up on the local disk.
|
|
If \f[C]cache\f[] exceeds this value then it will start to the delete
|
|
the oldest chunks until it goes under this value.
|
|
.PP
|
|
\f[B]Default\f[]: 10G
|
|
.SS \-\-cache\-chunk\-clean\-interval=DURATION
|
|
.PP
|
|
How often should \f[C]cache\f[] perform cleanups of the chunk storage.
|
|
The default value should be ok for most people.
|
|
If you find that \f[C]cache\f[] goes over
|
|
\f[C]cache\-total\-chunk\-size\f[] too often then try to lower this
|
|
value to force it to perform cleanups more often.
|
|
.PP
|
|
\f[B]Default\f[]: 1m
|
|
.SS \-\-cache\-info\-age=DURATION
|
|
.PP
|
|
How long to keep file structure information (directory listings, file
|
|
size, mod times etc) locally.
|
|
.PP
|
|
If all write operations are done through \f[C]cache\f[] then you can
|
|
safely make this value very large as the cache store will also be
|
|
updated in real time.
|
|
.PP
|
|
\f[B]Default\f[]: 6h
|
|
.SS \-\-cache\-read\-retries=RETRIES
|
|
.PP
|
|
How many times to retry a read from a cache storage.
|
|
.PP
|
|
Since reading from a \f[C]cache\f[] stream is independent from
|
|
downloading file data, readers can get to a point where there\[aq]s no
|
|
more data in the cache.
|
|
Most of the times this can indicate a connectivity issue if
|
|
\f[C]cache\f[] isn\[aq]t able to provide file data anymore.
|
|
.PP
|
|
For really slow connections, increase this to a point where the stream
|
|
is able to provide data but your experience will be very stuttering.
|
|
.PP
|
|
\f[B]Default\f[]: 10
|
|
.SS \-\-cache\-workers=WORKERS
|
|
.PP
|
|
How many workers should run in parallel to download chunks.
|
|
.PP
|
|
Higher values will mean more parallel processing (better CPU needed) and
|
|
more concurrent requests on the cloud provider.
|
|
This impacts several aspects like the cloud provider API limits, more
|
|
stress on the hardware that rclone runs on but it also means that
|
|
streams will be more fluid and data will be available much more faster
|
|
to readers.
|
|
.PP
|
|
\f[B]Note\f[]: If the optional Plex integration is enabled then this
|
|
setting will adapt to the type of reading performed and the value
|
|
specified here will be used as a maximum number of workers to use.
|
|
\f[B]Default\f[]: 4
|
|
.SS \-\-cache\-chunk\-no\-memory
|
|
.PP
|
|
By default, \f[C]cache\f[] will keep file data during streaming in RAM
|
|
as well to provide it to readers as fast as possible.
|
|
.PP
|
|
This transient data is evicted as soon as it is read and the number of
|
|
chunks stored doesn\[aq]t exceed the number of workers.
|
|
However, depending on other settings like \f[C]cache\-chunk\-size\f[]
|
|
and \f[C]cache\-workers\f[] this footprint can increase if there are
|
|
parallel streams too (multiple files being read at the same time).
|
|
.PP
|
|
If the hardware permits it, use this feature to provide an overall
|
|
better performance during streaming but it can also be disabled if RAM
|
|
is not available on the local machine.
|
|
.PP
|
|
\f[B]Default\f[]: not set
|
|
.SS \-\-cache\-rps=NUMBER
|
|
.PP
|
|
This setting places a hard limit on the number of requests per second
|
|
that \f[C]cache\f[] will be doing to the cloud provider remote and try
|
|
to respect that value by setting waits between reads.
|
|
.PP
|
|
If you find that you\[aq]re getting banned or limited on the cloud
|
|
provider through cache and know that a smaller number of requests per
|
|
second will allow you to work with it then you can use this setting for
|
|
that.
|
|
.PP
|
|
A good balance of all the other settings should make this setting
|
|
useless but it is available to set for more special cases.
|
|
.PP
|
|
\f[B]NOTE\f[]: This will limit the number of requests during streams but
|
|
other API calls to the cloud provider like directory listings will still
|
|
pass.
|
|
.PP
|
|
\f[B]Default\f[]: disabled
|
|
.SS \-\-cache\-writes
|
|
.PP
|
|
If you need to read files immediately after you upload them through
|
|
\f[C]cache\f[] you can enable this flag to have their data stored in the
|
|
cache store at the same time during upload.
|
|
.PP
|
|
\f[B]Default\f[]: not set
|
|
.SS \-\-cache\-tmp\-upload\-path=PATH
|
|
.PP
|
|
This is the path where \f[C]cache\f[] will use as a temporary storage
|
|
for new files that need to be uploaded to the cloud provider.
|
|
.PP
|
|
Specifying a value will enable this feature.
|
|
Without it, it is completely disabled and files will be uploaded
|
|
directly to the cloud provider
|
|
.PP
|
|
\f[B]Default\f[]: empty
|
|
.SS \-\-cache\-tmp\-wait\-time=DURATION
|
|
.PP
|
|
This is the duration that a file must wait in the temporary location
|
|
\f[I]cache\-tmp\-upload\-path\f[] before it is selected for upload.
|
|
.PP
|
|
Note that only one file is uploaded at a time and it can take longer to
|
|
start the upload if a queue formed for this purpose.
|
|
.PP
|
|
\f[B]Default\f[]: 15m
|
|
.SS \-\-cache\-db\-wait\-time=DURATION
|
|
.PP
|
|
Only one process can have the DB open at any one time, so rclone waits
|
|
for this duration for the DB to become available before it gives an
|
|
error.
|
|
.PP
|
|
If you set it to 0 then it will wait forever.
|
|
.PP
|
|
\f[B]Default\f[]: 1s
|
|
.SS Crypt
|
|
.PP
|
|
The \f[C]crypt\f[] remote encrypts and decrypts another remote.
|
|
.PP
|
|
To use it first set up the underlying remote following the config
|
|
instructions for that remote.
|
|
You can also use a local pathname instead of a remote which will encrypt
|
|
and decrypt from that directory which might be useful for encrypting
|
|
onto a USB stick for example.
|
|
.PP
|
|
First check your chosen remote is working \- we\[aq]ll call it
|
|
\f[C]remote:path\f[] in these docs.
|
|
Note that anything inside \f[C]remote:path\f[] will be encrypted and
|
|
anything outside won\[aq]t.
|
|
This means that if you are using a bucket based remote (eg S3, B2,
|
|
swift) then you should probably put the bucket in the remote
|
|
\f[C]s3:bucket\f[].
|
|
If you just use \f[C]s3:\f[] then rclone will make encrypted bucket
|
|
names too (if using file name encryption) which may or may not be what
|
|
you want.
|
|
.PP
|
|
Now configure \f[C]crypt\f[] using \f[C]rclone\ config\f[].
|
|
We will call this one \f[C]secret\f[] to differentiate it from the
|
|
\f[C]remote\f[].
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n\ \ \
|
|
name>\ secret
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 5
|
|
Remote\ to\ encrypt/decrypt.
|
|
Normally\ should\ contain\ a\ \[aq]:\[aq]\ and\ a\ path,\ eg\ "myremote:path/to/dir",
|
|
"myremote:bucket"\ or\ maybe\ "myremote:"\ (not\ recommended).
|
|
remote>\ remote:path
|
|
How\ to\ encrypt\ the\ filenames.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Don\[aq]t\ encrypt\ the\ file\ names.\ \ Adds\ a\ ".bin"\ extension\ only.
|
|
\ \ \ \\\ "off"
|
|
\ 2\ /\ Encrypt\ the\ filenames\ see\ the\ docs\ for\ the\ details.
|
|
\ \ \ \\\ "standard"
|
|
\ 3\ /\ Very\ simple\ filename\ obfuscation.
|
|
\ \ \ \\\ "obfuscate"
|
|
filename_encryption>\ 2
|
|
Option\ to\ either\ encrypt\ directory\ names\ or\ leave\ them\ intact.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Encrypt\ directory\ names.
|
|
\ \ \ \\\ "true"
|
|
\ 2\ /\ Don\[aq]t\ encrypt\ directory\ names,\ leave\ them\ intact.
|
|
\ \ \ \\\ "false"
|
|
filename_encryption>\ 1
|
|
Password\ or\ pass\ phrase\ for\ encryption.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
y/g>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
Password\ or\ pass\ phrase\ for\ salt.\ Optional\ but\ recommended.
|
|
Should\ be\ different\ to\ the\ previous\ password.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
n)\ No\ leave\ this\ optional\ password\ blank
|
|
y/g/n>\ g
|
|
Password\ strength\ in\ bits.
|
|
64\ is\ just\ about\ memorable
|
|
128\ is\ secure
|
|
1024\ is\ the\ maximum
|
|
Bits>\ 128
|
|
Your\ password\ is:\ JAsJvRcgR\-_veXNfy_sGmQ
|
|
Use\ this\ password?
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[secret]
|
|
remote\ =\ remote:path
|
|
filename_encryption\ =\ standard
|
|
password\ =\ ***\ ENCRYPTED\ ***
|
|
password2\ =\ ***\ ENCRYPTED\ ***
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
\f[B]Important\f[] The password is stored in the config file is lightly
|
|
obscured so it isn\[aq]t immediately obvious what it is.
|
|
It is in no way secure unless you use config file encryption.
|
|
.PP
|
|
A long passphrase is recommended, or you can use a random one.
|
|
Note that if you reconfigure rclone with the same passwords/passphrases
|
|
elsewhere it will be compatible \- all the secrets used are derived from
|
|
those two passwords/passphrases.
|
|
.PP
|
|
Note that rclone does not encrypt
|
|
.IP \[bu] 2
|
|
file length \- this can be calcuated within 16 bytes
|
|
.IP \[bu] 2
|
|
modification time \- used for syncing
|
|
.SS Specifying the remote
|
|
.PP
|
|
In normal use, make sure the remote has a \f[C]:\f[] in.
|
|
If you specify the remote without a \f[C]:\f[] then rclone will use a
|
|
local directory of that name.
|
|
So if you use a remote of \f[C]/path/to/secret/files\f[] then rclone
|
|
will encrypt stuff to that directory.
|
|
If you use a remote of \f[C]name\f[] then rclone will put files in a
|
|
directory called \f[C]name\f[] in the current directory.
|
|
.PP
|
|
If you specify the remote as \f[C]remote:path/to/dir\f[] then rclone
|
|
will store encrypted files in \f[C]path/to/dir\f[] on the remote.
|
|
If you are using file name encryption, then when you save files to
|
|
\f[C]secret:subdir/subfile\f[] this will store them in the unencrypted
|
|
path \f[C]path/to/dir\f[] but the \f[C]subdir/subpath\f[] bit will be
|
|
encrypted.
|
|
.PP
|
|
Note that unless you want encrypted bucket names (which are difficult to
|
|
manage because you won\[aq]t know what directory they represent in web
|
|
interfaces etc), you should probably specify a bucket, eg
|
|
\f[C]remote:secretbucket\f[] when using bucket based remotes such as S3,
|
|
Swift, Hubic, B2, GCS.
|
|
.SS Example
|
|
.PP
|
|
To test I made a little directory of files using "standard" file name
|
|
encryption.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
plaintext/
|
|
├──\ file0.txt
|
|
├──\ file1.txt
|
|
└──\ subdir
|
|
\ \ \ \ ├──\ file2.txt
|
|
\ \ \ \ ├──\ file3.txt
|
|
\ \ \ \ └──\ subsubdir
|
|
\ \ \ \ \ \ \ \ └──\ file4.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Copy these to the remote and list them back
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ copy\ plaintext\ secret:
|
|
$\ rclone\ \-q\ ls\ secret:
|
|
\ \ \ \ \ \ \ \ 7\ file1.txt
|
|
\ \ \ \ \ \ \ \ 6\ file0.txt
|
|
\ \ \ \ \ \ \ \ 8\ subdir/file2.txt
|
|
\ \ \ \ \ \ \ 10\ subdir/subsubdir/file4.txt
|
|
\ \ \ \ \ \ \ \ 9\ subdir/file3.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Now see what that looked like when encrypted
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ remote:path
|
|
\ \ \ \ \ \ \ 55\ hagjclgavj2mbiqm6u6cnjjqcg
|
|
\ \ \ \ \ \ \ 54\ v05749mltvv1tf4onltun46gls
|
|
\ \ \ \ \ \ \ 57\ 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
|
|
\ \ \ \ \ \ \ 58\ 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
|
|
\ \ \ \ \ \ \ 56\ 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that this retains the directory structure which means you can do
|
|
this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ secret:subdir
|
|
\ \ \ \ \ \ \ \ 8\ file2.txt
|
|
\ \ \ \ \ \ \ \ 9\ file3.txt
|
|
\ \ \ \ \ \ \ 10\ subsubdir/file4.txt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If don\[aq]t use file name encryption then the remote will look like
|
|
this \- note the \f[C]\&.bin\f[] extensions added to prevent the cloud
|
|
provider attempting to interpret the data.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ remote:path
|
|
\ \ \ \ \ \ \ 54\ file0.txt.bin
|
|
\ \ \ \ \ \ \ 57\ subdir/file3.txt.bin
|
|
\ \ \ \ \ \ \ 56\ subdir/file2.txt.bin
|
|
\ \ \ \ \ \ \ 58\ subdir/subsubdir/file4.txt.bin
|
|
\ \ \ \ \ \ \ 55\ file1.txt.bin
|
|
\f[]
|
|
.fi
|
|
.SS File name encryption modes
|
|
.PP
|
|
Here are some of the features of the file name encryption modes
|
|
.PP
|
|
Off
|
|
.IP \[bu] 2
|
|
doesn\[aq]t hide file names or directory structure
|
|
.IP \[bu] 2
|
|
allows for longer file names (~246 characters)
|
|
.IP \[bu] 2
|
|
can use sub paths and copy single files
|
|
.PP
|
|
Standard
|
|
.IP \[bu] 2
|
|
file names encrypted
|
|
.IP \[bu] 2
|
|
file names can\[aq]t be as long (~143 characters)
|
|
.IP \[bu] 2
|
|
can use sub paths and copy single files
|
|
.IP \[bu] 2
|
|
directory structure visible
|
|
.IP \[bu] 2
|
|
identical files names will have identical uploaded names
|
|
.IP \[bu] 2
|
|
can use shortcuts to shorten the directory recursion
|
|
.PP
|
|
Obfuscation
|
|
.PP
|
|
This is a simple "rotate" of the filename, with each file having a rot
|
|
distance based on the filename.
|
|
We store the distance at the beginning of the filename.
|
|
So a file called "hello" may become "53.jgnnq"
|
|
.PP
|
|
This is not a strong encryption of filenames, but it may stop automated
|
|
scanning tools from picking up on filename patterns.
|
|
As such it\[aq]s an intermediate between "off" and "standard".
|
|
The advantage is that it allows for longer path segment names.
|
|
.PP
|
|
There is a possibility with some unicode based filenames that the
|
|
obfuscation is weak and may map lower case characters to upper case
|
|
equivalents.
|
|
You can not rely on this for strong protection.
|
|
.IP \[bu] 2
|
|
file names very lightly obfuscated
|
|
.IP \[bu] 2
|
|
file names can be longer than standard encryption
|
|
.IP \[bu] 2
|
|
can use sub paths and copy single files
|
|
.IP \[bu] 2
|
|
directory structure visible
|
|
.IP \[bu] 2
|
|
identical files names will have identical uploaded names
|
|
.PP
|
|
Cloud storage systems have various limits on file name length and total
|
|
path length which you are more likely to hit using "Standard" file name
|
|
encryption.
|
|
If you keep your file names to below 156 characters in length then you
|
|
should be OK on all providers.
|
|
.PP
|
|
There may be an even more secure file name encryption mode in the future
|
|
which will address the long file name problem.
|
|
.SS Directory name encryption
|
|
.PP
|
|
Crypt offers the option of encrypting dir names or leaving them intact.
|
|
There are two options:
|
|
.PP
|
|
True
|
|
.PP
|
|
Encrypts the whole file path including directory names Example:
|
|
\f[C]1/12/123.txt\f[] is encrypted to
|
|
\f[C]p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0\f[]
|
|
.PP
|
|
False
|
|
.PP
|
|
Only encrypts file names, skips directory names Example:
|
|
\f[C]1/12/123.txt\f[] is encrypted to
|
|
\f[C]1/12/qgm4avr35m5loi1th53ato71v0\f[]
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Crypt stores modification times using the underlying remote so support
|
|
depends on that.
|
|
.PP
|
|
Hashes are not stored for crypt.
|
|
However the data integrity is protected by an extremely strong crypto
|
|
authenticator.
|
|
.PP
|
|
Note that you should use the \f[C]rclone\ cryptcheck\f[] command to
|
|
check the integrity of a crypted remote instead of
|
|
\f[C]rclone\ check\f[] which can\[aq]t check the checksums properly.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-crypt\-show\-mapping
|
|
.PP
|
|
If this flag is set then for each file that the remote is asked to list,
|
|
it will log (at level INFO) a line stating the decrypted file name and
|
|
the encrypted file name.
|
|
.PP
|
|
This is so you can work out which encrypted names are which decrypted
|
|
names just in case you need to do something with the encrypted file
|
|
names, or for debugging purposes.
|
|
.SS Backing up a crypted remote
|
|
.PP
|
|
If you wish to backup a crypted remote, it it recommended that you use
|
|
\f[C]rclone\ sync\f[] on the encrypted files, and make sure the
|
|
passwords are the same in the new encrypted remote.
|
|
.PP
|
|
This will have the following advantages
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ sync\f[] will check the checksums while copying
|
|
.IP \[bu] 2
|
|
you can use \f[C]rclone\ check\f[] between the encrypted remotes
|
|
.IP \[bu] 2
|
|
you don\[aq]t decrypt and encrypt unnecessarily
|
|
.PP
|
|
For example, let\[aq]s say you have your original remote at
|
|
\f[C]remote:\f[] with the encrypted version at \f[C]eremote:\f[] with
|
|
path \f[C]remote:crypt\f[].
|
|
You would then set up the new remote \f[C]remote2:\f[] and then the
|
|
encrypted version \f[C]eremote2:\f[] with path \f[C]remote2:crypt\f[]
|
|
using the same passwords as \f[C]eremote:\f[].
|
|
.PP
|
|
To sync the two remotes you would do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ remote:crypt\ remote2:crypt
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And to check the integrity you would do
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ check\ remote:crypt\ remote2:crypt
|
|
\f[]
|
|
.fi
|
|
.SS File formats
|
|
.SS File encryption
|
|
.PP
|
|
Files are encrypted 1:1 source file to destination object.
|
|
The file has a header and is divided into chunks.
|
|
.SS Header
|
|
.IP \[bu] 2
|
|
8 bytes magic string \f[C]RCLONE\\x00\\x00\f[]
|
|
.IP \[bu] 2
|
|
24 bytes Nonce (IV)
|
|
.PP
|
|
The initial nonce is generated from the operating systems crypto strong
|
|
random number generator.
|
|
The nonce is incremented for each chunk read making sure each nonce is
|
|
unique for each block written.
|
|
The chance of a nonce being re\-used is minuscule.
|
|
If you wrote an exabyte of data (10¹⁸ bytes) you would have a
|
|
probability of approximately 2×10⁻³² of re\-using a nonce.
|
|
.SS Chunk
|
|
.PP
|
|
Each chunk will contain 64kB of data, except for the last one which may
|
|
have less data.
|
|
The data chunk is in standard NACL secretbox format.
|
|
Secretbox uses XSalsa20 and Poly1305 to encrypt and authenticate
|
|
messages.
|
|
.PP
|
|
Each chunk contains:
|
|
.IP \[bu] 2
|
|
16 Bytes of Poly1305 authenticator
|
|
.IP \[bu] 2
|
|
1 \- 65536 bytes XSalsa20 encrypted data
|
|
.PP
|
|
64k chunk size was chosen as the best performing chunk size (the
|
|
authenticator takes too much time below this and the performance drops
|
|
off due to cache effects above this).
|
|
Note that these chunks are buffered in memory so they can\[aq]t be too
|
|
big.
|
|
.PP
|
|
This uses a 32 byte (256 bit key) key derived from the user password.
|
|
.SS Examples
|
|
.PP
|
|
1 byte file will encrypt to
|
|
.IP \[bu] 2
|
|
32 bytes header
|
|
.IP \[bu] 2
|
|
17 bytes data chunk
|
|
.PP
|
|
49 bytes total
|
|
.PP
|
|
1MB (1048576 bytes) file will encrypt to
|
|
.IP \[bu] 2
|
|
32 bytes header
|
|
.IP \[bu] 2
|
|
16 chunks of 65568 bytes
|
|
.PP
|
|
1049120 bytes total (a 0.05% overhead).
|
|
This is the overhead for big files.
|
|
.SS Name encryption
|
|
.PP
|
|
File names are encrypted segment by segment \- the path is broken up
|
|
into \f[C]/\f[] separated strings and these are encrypted individually.
|
|
.PP
|
|
File segments are padded using using PKCS#7 to a multiple of 16 bytes
|
|
before encryption.
|
|
.PP
|
|
They are then encrypted with EME using AES with 256 bit key.
|
|
EME (ECB\-Mix\-ECB) is a wide\-block encryption mode presented in the
|
|
2003 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
|
|
.PP
|
|
This makes for deterministic encryption which is what we want \- the
|
|
same filename must encrypt to the same thing otherwise we can\[aq]t find
|
|
it on the cloud storage system.
|
|
.PP
|
|
This means that
|
|
.IP \[bu] 2
|
|
filenames with the same name will encrypt the same
|
|
.IP \[bu] 2
|
|
filenames which start the same won\[aq]t have a common prefix
|
|
.PP
|
|
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
|
|
which are derived from the user password.
|
|
.PP
|
|
After encryption they are written out using a modified version of
|
|
standard \f[C]base32\f[] encoding as described in RFC4648.
|
|
The standard encoding is modified in two ways:
|
|
.IP \[bu] 2
|
|
it becomes lower case (no\-one likes upper case filenames!)
|
|
.IP \[bu] 2
|
|
we strip the padding character \f[C]=\f[]
|
|
.PP
|
|
\f[C]base32\f[] is used rather than the more efficient \f[C]base64\f[]
|
|
so rclone can be used on case insensitive remotes (eg Windows, Amazon
|
|
Drive).
|
|
.SS Key derivation
|
|
.PP
|
|
Rclone uses \f[C]scrypt\f[] with parameters \f[C]N=16384,\ r=8,\ p=1\f[]
|
|
with an optional user supplied salt (password2) to derive the 32+32+16 =
|
|
80 bytes of key material required.
|
|
If the user doesn\[aq]t supply a salt then rclone uses an internal one.
|
|
.PP
|
|
\f[C]scrypt\f[] makes it impractical to mount a dictionary attack on
|
|
rclone encrypted data.
|
|
For full protection against this you should always use a salt.
|
|
.SS Dropbox
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Dropbox paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for dropbox involves getting a token from Dropbox
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 4
|
|
Dropbox\ App\ Key\ \-\ leave\ blank\ normally.
|
|
app_key>
|
|
Dropbox\ App\ Secret\ \-\ leave\ blank\ normally.
|
|
app_secret>
|
|
Remote\ config
|
|
Please\ visit:
|
|
https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code
|
|
Enter\ the\ code:\ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
app_key\ =
|
|
app_secret\ =
|
|
token\ =\ XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your dropbox
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a dropbox directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Dropbox for business
|
|
.PP
|
|
Rclone supports Dropbox for business and Team Folders.
|
|
.PP
|
|
When using Dropbox for business \f[C]remote:\f[] and
|
|
\f[C]remote:path/to/file\f[] will refer to your personal folder.
|
|
.PP
|
|
If you wish to see Team Folders you must use a leading \f[C]/\f[] in the
|
|
path, so \f[C]rclone\ lsd\ remote:/\f[] will refer to the root and show
|
|
you all Team Folders and your User Folder.
|
|
.PP
|
|
You can then use team folders like this \f[C]remote:/TeamFolder\f[] and
|
|
\f[C]remote:/TeamFolder/path/to/file\f[].
|
|
.PP
|
|
A leading \f[C]/\f[] for a Dropbox personal account will do nothing, but
|
|
it will take an extra HTTP transaction so it should be avoided.
|
|
.SS Modified time and Hashes
|
|
.PP
|
|
Dropbox supports modified times, but the only way to set a modification
|
|
time is to re\-upload the file.
|
|
.PP
|
|
This means that if you uploaded your data with an older version of
|
|
rclone which didn\[aq]t support the v2 API and modified times, rclone
|
|
will decide to upload all your old data to fix the modification times.
|
|
If you don\[aq]t want this to happen use \f[C]\-\-size\-only\f[] or
|
|
\f[C]\-\-checksum\f[] flag to stop it.
|
|
.PP
|
|
Dropbox supports its own hash
|
|
type (https://www.dropbox.com/developers/reference/content-hash) which
|
|
is checked for all transfers.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-dropbox\-chunk\-size=SIZE
|
|
.PP
|
|
Any files larger than this will be uploaded in chunks of this size.
|
|
The default is 48MB.
|
|
The maximum is 150MB.
|
|
.PP
|
|
Note that chunks are buffered in memory (one at a time) so rclone can
|
|
deal with retries.
|
|
Setting this larger will increase the speed slightly (at most 10% for
|
|
128MB in tests) at the cost of using more memory.
|
|
It can be set smaller if you are tight on memory.
|
|
.SS Limitations
|
|
.PP
|
|
Note that Dropbox is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
There are some file names such as \f[C]thumbs.db\f[] which Dropbox
|
|
can\[aq]t store.
|
|
There is a full list of them in the "Ignored Files" section of this
|
|
document (https://www.dropbox.com/en/help/145).
|
|
Rclone will issue an error message
|
|
\f[C]File\ name\ disallowed\ \-\ not\ uploading\f[] if it attempts to
|
|
upload one of those file names, but the sync won\[aq]t fail.
|
|
.PP
|
|
If you have more than 10,000 files in a directory then
|
|
\f[C]rclone\ purge\ dropbox:dir\f[] will return the error
|
|
\f[C]Failed\ to\ purge:\ There\ are\ too\ many\ files\ involved\ in\ this\ operation\f[].
|
|
As a work\-around do an \f[C]rclone\ delete\ dropbox:dir\f[] followed by
|
|
an \f[C]rclone\ rmdir\ dropbox:dir\f[].
|
|
.SS FTP
|
|
.PP
|
|
FTP is the File Transfer Protocol.
|
|
FTP support is provided using the
|
|
github.com/jlaffaye/ftp (https://godoc.org/github.com/jlaffaye/ftp)
|
|
package.
|
|
.PP
|
|
Here is an example of making an FTP configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
An FTP remote only needs a host together with and a username and a
|
|
password.
|
|
With anonymous FTP server, you will need to use \f[C]anonymous\f[] as
|
|
username and your email address as the password.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/r/c/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ FTP\ Connection\
|
|
\ \ \ \\\ "ftp"
|
|
\ 7\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 8\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 9\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
10\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
11\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
12\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
13\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
14\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ ftp
|
|
FTP\ host\ to\ connect\ to
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Connect\ to\ ftp.example.com
|
|
\ \ \ \\\ "ftp.example.com"
|
|
host>\ ftp.example.com
|
|
FTP\ username,\ leave\ blank\ for\ current\ username,\ ncw
|
|
user>
|
|
FTP\ port,\ leave\ blank\ to\ use\ default\ (21)
|
|
port>
|
|
FTP\ password
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
y/g>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
host\ =\ ftp.example.com
|
|
user\ =\
|
|
port\ =
|
|
pass\ =\ ***\ ENCRYPTED\ ***
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all directories in the home directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:path/to/directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:path/to/directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote directory, deleting
|
|
any excess files in the directory.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
FTP does not support modified times.
|
|
Any times you see on the server will be time of upload.
|
|
.SS Checksums
|
|
.PP
|
|
FTP does not support any checksums.
|
|
.SS Limitations
|
|
.PP
|
|
Note that since FTP isn\[aq]t HTTP based the following flags don\[aq]t
|
|
work with it: \f[C]\-\-dump\-headers\f[], \f[C]\-\-dump\-bodies\f[],
|
|
\f[C]\-\-dump\-auth\f[]
|
|
.PP
|
|
Note that \f[C]\-\-timeout\f[] isn\[aq]t supported (but
|
|
\f[C]\-\-contimeout\f[] is).
|
|
.PP
|
|
Note that \f[C]\-\-bind\f[] isn\[aq]t supported.
|
|
.PP
|
|
FTP could support server side move but doesn\[aq]t yet.
|
|
.SS Google Cloud Storage
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
The initial setup for google cloud storage involves getting a token from
|
|
Google Cloud Storage which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 6
|
|
Google\ Application\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>
|
|
Google\ Application\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>
|
|
Project\ number\ optional\ \-\ needed\ only\ for\ list/create/delete\ buckets\ \-\ see\ your\ developer\ console.
|
|
project_number>\ 12345678
|
|
Service\ Account\ Credentials\ JSON\ file\ path\ \-\ needed\ only\ if\ you\ want\ use\ SA\ instead\ of\ interactive\ login.
|
|
service_account_file>
|
|
Access\ Control\ List\ for\ new\ objects.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Object\ owner\ gets\ OWNER\ access,\ and\ all\ Authenticated\ Users\ get\ READER\ access.
|
|
\ \ \ \\\ "authenticatedRead"
|
|
\ 2\ /\ Object\ owner\ gets\ OWNER\ access,\ and\ project\ team\ owners\ get\ OWNER\ access.
|
|
\ \ \ \\\ "bucketOwnerFullControl"
|
|
\ 3\ /\ Object\ owner\ gets\ OWNER\ access,\ and\ project\ team\ owners\ get\ READER\ access.
|
|
\ \ \ \\\ "bucketOwnerRead"
|
|
\ 4\ /\ Object\ owner\ gets\ OWNER\ access\ [default\ if\ left\ blank].
|
|
\ \ \ \\\ "private"
|
|
\ 5\ /\ Object\ owner\ gets\ OWNER\ access,\ and\ project\ team\ members\ get\ access\ according\ to\ their\ roles.
|
|
\ \ \ \\\ "projectPrivate"
|
|
\ 6\ /\ Object\ owner\ gets\ OWNER\ access,\ and\ all\ Users\ get\ READER\ access.
|
|
\ \ \ \\\ "publicRead"
|
|
object_acl>\ 4
|
|
Access\ Control\ List\ for\ new\ buckets.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Project\ team\ owners\ get\ OWNER\ access,\ and\ all\ Authenticated\ Users\ get\ READER\ access.
|
|
\ \ \ \\\ "authenticatedRead"
|
|
\ 2\ /\ Project\ team\ owners\ get\ OWNER\ access\ [default\ if\ left\ blank].
|
|
\ \ \ \\\ "private"
|
|
\ 3\ /\ Project\ team\ members\ get\ access\ according\ to\ their\ roles.
|
|
\ \ \ \\\ "projectPrivate"
|
|
\ 4\ /\ Project\ team\ owners\ get\ OWNER\ access,\ and\ all\ Users\ get\ READER\ access.
|
|
\ \ \ \\\ "publicRead"
|
|
\ 5\ /\ Project\ team\ owners\ get\ OWNER\ access,\ and\ all\ Users\ get\ WRITER\ access.
|
|
\ \ \ \\\ "publicReadWrite"
|
|
bucket_acl>\ 2
|
|
Location\ for\ the\ newly\ created\ buckets.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Empty\ for\ default\ location\ (US).
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ Multi\-regional\ location\ for\ Asia.
|
|
\ \ \ \\\ "asia"
|
|
\ 3\ /\ Multi\-regional\ location\ for\ Europe.
|
|
\ \ \ \\\ "eu"
|
|
\ 4\ /\ Multi\-regional\ location\ for\ United\ States.
|
|
\ \ \ \\\ "us"
|
|
\ 5\ /\ Taiwan.
|
|
\ \ \ \\\ "asia\-east1"
|
|
\ 6\ /\ Tokyo.
|
|
\ \ \ \\\ "asia\-northeast1"
|
|
\ 7\ /\ Singapore.
|
|
\ \ \ \\\ "asia\-southeast1"
|
|
\ 8\ /\ Sydney.
|
|
\ \ \ \\\ "australia\-southeast1"
|
|
\ 9\ /\ Belgium.
|
|
\ \ \ \\\ "europe\-west1"
|
|
10\ /\ London.
|
|
\ \ \ \\\ "europe\-west2"
|
|
11\ /\ Iowa.
|
|
\ \ \ \\\ "us\-central1"
|
|
12\ /\ South\ Carolina.
|
|
\ \ \ \\\ "us\-east1"
|
|
13\ /\ Northern\ Virginia.
|
|
\ \ \ \\\ "us\-east4"
|
|
14\ /\ Oregon.
|
|
\ \ \ \\\ "us\-west1"
|
|
location>\ 12
|
|
The\ storage\ class\ to\ use\ when\ storing\ objects\ in\ Google\ Cloud\ Storage.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Default
|
|
\ \ \ \\\ ""
|
|
\ 2\ /\ Multi\-regional\ storage\ class
|
|
\ \ \ \\\ "MULTI_REGIONAL"
|
|
\ 3\ /\ Regional\ storage\ class
|
|
\ \ \ \\\ "REGIONAL"
|
|
\ 4\ /\ Nearline\ storage\ class
|
|
\ \ \ \\\ "NEARLINE"
|
|
\ 5\ /\ Coldline\ storage\ class
|
|
\ \ \ \\\ "COLDLINE"
|
|
\ 6\ /\ Durable\ reduced\ availability\ storage\ class
|
|
\ \ \ \\\ "DURABLE_REDUCED_AVAILABILITY"
|
|
storage_class>\ 5
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine\ or\ Y\ didn\[aq]t\ work
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
type\ =\ google\ cloud\ storage
|
|
client_id\ =
|
|
client_secret\ =
|
|
token\ =\ {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014\-07\-17T20:49:14.929208288+01:00","Extra":null}
|
|
project_number\ =\ 12345678
|
|
object_acl\ =\ private
|
|
bucket_acl\ =\ private
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall, or use
|
|
manual mode.
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all the buckets in your project
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Service Account support
|
|
.PP
|
|
You can set up rclone with Google Cloud Storage in an unattended mode,
|
|
i.e.
|
|
not tied to a specific end\-user Google account.
|
|
This is useful when you want to synchronise files onto machines that
|
|
don\[aq]t have actively logged\-in users, for example build machines.
|
|
.PP
|
|
To get credentials for Google Cloud Platform IAM Service
|
|
Accounts (https://cloud.google.com/iam/docs/service-accounts), please
|
|
head to the Service
|
|
Account (https://console.cloud.google.com/permissions/serviceaccounts)
|
|
section of the Google Developer Console.
|
|
Service Accounts behave just like normal \f[C]User\f[] permissions in
|
|
Google Cloud Storage
|
|
ACLs (https://cloud.google.com/storage/docs/access-control), so you can
|
|
limit their access (e.g.
|
|
make them read only).
|
|
After creating an account, a JSON file containing the Service
|
|
Account\[aq]s credentials will be downloaded onto your machines.
|
|
These credentials are what rclone will use for authentication.
|
|
.PP
|
|
To use a Service Account instead of OAuth2 token flow, enter the path to
|
|
your Service Account credentials at the \f[C]service_account_file\f[]
|
|
prompt and rclone won\[aq]t use the browser based authentication flow.
|
|
If you\[aq]d rather stuff the contents of the credentials file into the
|
|
rclone config file, you can set \f[C]service_account_credentials\f[]
|
|
with the actual contents of the file instead, or set the equivalent
|
|
environment variable.
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS Modified time
|
|
.PP
|
|
Google google cloud storage stores md5sums natively and rclone stores
|
|
modification times as metadata on the object, under the "mtime" key in
|
|
RFC3339 format accurate to 1ns.
|
|
.SS Google Drive
|
|
.PP
|
|
Paths are specified as \f[C]drive:path\f[]
|
|
.PP
|
|
Drive paths may be as deep as required, eg
|
|
\f[C]drive:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for drive involves getting a token from Google drive
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/r/c/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
[snip]
|
|
10\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
[snip]
|
|
Storage>\ drive
|
|
Google\ Application\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>
|
|
Google\ Application\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>
|
|
Scope\ that\ rclone\ should\ use\ when\ requesting\ access\ from\ drive.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Full\ access\ all\ files,\ excluding\ Application\ Data\ Folder.
|
|
\ \ \ \\\ "drive"
|
|
\ 2\ /\ Read\-only\ access\ to\ file\ metadata\ and\ file\ contents.
|
|
\ \ \ \\\ "drive.readonly"
|
|
\ \ \ /\ Access\ to\ files\ created\ by\ rclone\ only.
|
|
\ 3\ |\ These\ are\ visible\ in\ the\ drive\ website.
|
|
\ \ \ |\ File\ authorization\ is\ revoked\ when\ the\ user\ deauthorizes\ the\ app.
|
|
\ \ \ \\\ "drive.file"
|
|
\ \ \ /\ Allows\ read\ and\ write\ access\ to\ the\ Application\ Data\ folder.
|
|
\ 4\ |\ This\ is\ not\ visible\ in\ the\ drive\ website.
|
|
\ \ \ \\\ "drive.appfolder"
|
|
\ \ \ /\ Allows\ read\-only\ access\ to\ file\ metadata\ but
|
|
\ 5\ |\ does\ not\ allow\ any\ access\ to\ read\ or\ download\ file\ content.
|
|
\ \ \ \\\ "drive.metadata.readonly"
|
|
scope>\ 1
|
|
ID\ of\ the\ root\ folder\ \-\ leave\ blank\ normally.\ \ Fill\ in\ to\ access\ "Computers"\ folders.\ (see\ docs).
|
|
root_folder_id>\
|
|
Service\ Account\ Credentials\ JSON\ file\ path\ \-\ needed\ only\ if\ you\ want\ use\ SA\ instead\ of\ interactive\ login.
|
|
service_account_file>
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine\ or\ Y\ didn\[aq]t\ work
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
Configure\ this\ as\ a\ team\ drive?
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ n
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
scope\ =\ drive
|
|
root_folder_id\ =\
|
|
service_account_file\ =
|
|
token\ =\ {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014\-03\-16T13:57:58.955387075Z"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Google if you use auto config mode.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall, or use
|
|
manual mode.
|
|
.PP
|
|
You can then use it like this,
|
|
.PP
|
|
List directories in top level of your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your drive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to a drive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Scopes
|
|
.PP
|
|
Rclone allows you to select which scope you would like for rclone to
|
|
use.
|
|
This changes what type of token is granted to rclone.
|
|
The scopes are defined
|
|
here. (https://developers.google.com/drive/v3/web/about-auth).
|
|
.PP
|
|
The scope are
|
|
.SS drive
|
|
.PP
|
|
This is the default scope and allows full access to all files, except
|
|
for the Application Data Folder (see below).
|
|
.PP
|
|
Choose this one if you aren\[aq]t sure.
|
|
.SS drive.readonly
|
|
.PP
|
|
This allows read only access to all files.
|
|
Files may be listed and downloaded but not uploaded, renamed or deleted.
|
|
.SS drive.file
|
|
.PP
|
|
With this scope rclone can read/view/modify only those files and folders
|
|
it creates.
|
|
.PP
|
|
So if you uploaded files to drive via the web interface (or any other
|
|
means) they will not be visible to rclone.
|
|
.PP
|
|
This can be useful if you are using rclone to backup data and you want
|
|
to be sure confidential data on your drive is not visible to rclone.
|
|
.PP
|
|
Files created with this scope are visible in the web interface.
|
|
.SS drive.appfolder
|
|
.PP
|
|
This gives rclone its own private area to store files.
|
|
Rclone will not be able to see any other files on your drive and you
|
|
won\[aq]t be able to see rclone\[aq]s files from the web interface
|
|
either.
|
|
.SS drive.metadata.readonly
|
|
.PP
|
|
This allows read only access to file names only.
|
|
It does not allow rclone to download or upload data, or rename or delete
|
|
files or directories.
|
|
.SS Root folder ID
|
|
.PP
|
|
You can set the \f[C]root_folder_id\f[] for rclone.
|
|
This is the directory (identified by its \f[C]Folder\ ID\f[]) that
|
|
rclone considers to be a the root of your drive.
|
|
.PP
|
|
Normally you will leave this blank and rclone will determine the correct
|
|
root to use itself.
|
|
.PP
|
|
However you can set this to restrict rclone to a specific folder
|
|
hierarchy or to access data within the "Computers" tab on the drive web
|
|
interface (where files from Google\[aq]s Backup and Sync desktop program
|
|
go).
|
|
.PP
|
|
In order to do this you will have to find the \f[C]Folder\ ID\f[] of the
|
|
directory you wish rclone to display.
|
|
This will be the last segment of the URL when you open the relevant
|
|
folder in the drive web interface.
|
|
.PP
|
|
So if the folder you want rclone to use has a URL which looks like
|
|
\f[C]https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh\f[]
|
|
in the browser, then you use \f[C]1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh\f[]
|
|
as the \f[C]root_folder_id\f[] in the config.
|
|
.PP
|
|
\f[B]NB\f[] folders under the "Computers" tab seem to be read only
|
|
(drive gives a 500 error) when using rclone.
|
|
.PP
|
|
There doesn\[aq]t appear to be an API to discover the folder IDs of the
|
|
"Computers" tab \- please contact us if you know otherwise!
|
|
.PP
|
|
Note also that rclone can\[aq]t access any data under the "Backups" tab
|
|
on the google drive web interface yet.
|
|
.SS Service Account support
|
|
.PP
|
|
You can set up rclone with Google Drive in an unattended mode, i.e.
|
|
not tied to a specific end\-user Google account.
|
|
This is useful when you want to synchronise files onto machines that
|
|
don\[aq]t have actively logged\-in users, for example build machines.
|
|
.PP
|
|
To use a Service Account instead of OAuth2 token flow, enter the path to
|
|
your Service Account credentials at the \f[C]service_account_file\f[]
|
|
prompt during \f[C]rclone\ config\f[] and rclone won\[aq]t use the
|
|
browser based authentication flow.
|
|
If you\[aq]d rather stuff the contents of the credentials file into the
|
|
rclone config file, you can set \f[C]service_account_credentials\f[]
|
|
with the actual contents of the file instead, or set the equivalent
|
|
environment variable.
|
|
.SS Use case \- Google Apps/G\-suite account and individual Drive
|
|
.PP
|
|
Let\[aq]s say that you are the administrator of a Google Apps (old) or
|
|
G\-suite account.
|
|
The goal is to store data on an individual\[aq]s Drive account, who IS a
|
|
member of the domain.
|
|
We\[aq]ll call the domain \f[B]example.com\f[], and the user
|
|
\f[B]foo\@example.com\f[].
|
|
.PP
|
|
There\[aq]s a few steps we need to go through to accomplish this:
|
|
.SS 1. Create a service account for example.com
|
|
.IP \[bu] 2
|
|
To create a service account and obtain its credentials, go to the Google
|
|
Developer Console (https://console.developers.google.com).
|
|
.IP \[bu] 2
|
|
You must have a project \- create one if you don\[aq]t.
|
|
.IP \[bu] 2
|
|
Then go to "IAM & admin" \-> "Service Accounts".
|
|
.IP \[bu] 2
|
|
Use the "Create Credentials" button.
|
|
Fill in "Service account name" with something that identifies your
|
|
client.
|
|
"Role" can be empty.
|
|
.IP \[bu] 2
|
|
Tick "Furnish a new private key" \- select "Key type JSON".
|
|
.IP \[bu] 2
|
|
Tick "Enable G Suite Domain\-wide Delegation".
|
|
This option makes "impersonation" possible, as documented here:
|
|
Delegating domain\-wide authority to the service
|
|
account (https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority)
|
|
.IP \[bu] 2
|
|
These credentials are what rclone will use for authentication.
|
|
If you ever need to remove access, press the "Delete service account
|
|
key" button.
|
|
.SS 2. Allowing API access to example.com Google Drive
|
|
.IP \[bu] 2
|
|
Go to example.com\[aq]s admin console
|
|
.IP \[bu] 2
|
|
Go into "Security" (or use the search bar)
|
|
.IP \[bu] 2
|
|
Select "Show more" and then "Advanced settings"
|
|
.IP \[bu] 2
|
|
Select "Manage API client access" in the "Authentication" section
|
|
.IP \[bu] 2
|
|
In the "Client Name" field enter the service account\[aq]s "Client ID"
|
|
\- this can be found in the Developer Console under "IAM & Admin" \->
|
|
"Service Accounts", then "View Client ID" for the newly created service
|
|
account.
|
|
It is a ~21 character numerical string.
|
|
.IP \[bu] 2
|
|
In the next field, "One or More API Scopes", enter
|
|
\f[C]https://www.googleapis.com/auth/drive\f[] to grant access to Google
|
|
Drive specifically.
|
|
.SS 3. Configure rclone, assuming a new install
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
|
|
n/s/q>\ n\ \ \ \ \ \ \ \ \ #\ New
|
|
name>gdrive\ \ \ \ \ \ #\ Gdrive\ is\ an\ example\ name
|
|
Storage>\ \ \ \ \ \ \ \ \ #\ Select\ the\ number\ shown\ for\ Google\ Drive
|
|
client_id>\ \ \ \ \ \ \ #\ Can\ be\ left\ blank
|
|
client_secret>\ \ \ #\ Can\ be\ left\ blank
|
|
scope>\ \ \ \ \ \ \ \ \ \ \ #\ Select\ your\ scope,\ 1\ for\ example
|
|
root_folder_id>\ \ #\ Can\ be\ left\ blank
|
|
service_account_file>\ /home/foo/myJSONfile.json\ #\ This\ is\ where\ the\ JSON\ file\ goes!
|
|
y/n>\ \ \ \ \ \ \ \ \ \ \ \ \ #\ Auto\ config,\ y
|
|
\f[]
|
|
.fi
|
|
.SS 4. Verify that it\[aq]s working
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ \-v\ \-\-drive\-impersonate\ foo\@example.com\ lsf\ gdrive:backup\f[]
|
|
.IP \[bu] 2
|
|
The arguments do:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]\-v\f[] \- verbose logging
|
|
.IP \[bu] 2
|
|
\f[C]\-\-drive\-impersonate\ foo\@example.com\f[] \- this is what does
|
|
the magic, pretending to be user foo.
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[] \- list files in a parsing friendly way
|
|
.IP \[bu] 2
|
|
\f[C]gdrive:backup\f[] \- use the remote called gdrive, work in the
|
|
folder named backup.
|
|
.RE
|
|
.SS Team drives
|
|
.PP
|
|
If you want to configure the remote to point to a Google Team Drive then
|
|
answer \f[C]y\f[] to the question
|
|
\f[C]Configure\ this\ as\ a\ team\ drive?\f[].
|
|
.PP
|
|
This will fetch the list of Team Drives from google and allow you to
|
|
configure which one you want to use.
|
|
You can also type in a team drive ID if you prefer.
|
|
.PP
|
|
For example:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Configure\ this\ as\ a\ team\ drive?
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
Fetching\ team\ drive\ list...
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Rclone\ Test
|
|
\ \ \ \\\ "xxxxxxxxxxxxxxxxxxxx"
|
|
\ 2\ /\ Rclone\ Test\ 2
|
|
\ \ \ \\\ "yyyyyyyyyyyyyyyyyyyy"
|
|
\ 3\ /\ Rclone\ Test\ 3
|
|
\ \ \ \\\ "zzzzzzzzzzzzzzzzzzzz"
|
|
Enter\ a\ Team\ Drive\ ID>\ 1
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =
|
|
client_secret\ =
|
|
token\ =\ {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014\-03\-16T13:57:58.955387075Z","Extra":null}
|
|
team_drive\ =\ xxxxxxxxxxxxxxxxxxxx
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
Google drive stores modification times accurate to 1 ms.
|
|
.SS Revisions
|
|
.PP
|
|
Google drive stores revisions of files.
|
|
When you upload a change to an existing file to google drive using
|
|
rclone it will create a new revision of that file.
|
|
.PP
|
|
Revisions follow the standard google policy which at time of writing was
|
|
.IP \[bu] 2
|
|
They are deleted after 30 days or 100 revisions (whatever comes first).
|
|
.IP \[bu] 2
|
|
They do not count towards a user storage quota.
|
|
.SS Deleting files
|
|
.PP
|
|
By default rclone will send all files to the trash when deleting files.
|
|
If deleting them permanently is required then use the
|
|
\f[C]\-\-drive\-use\-trash=false\f[] flag, or set the equivalent
|
|
environment variable.
|
|
.SS Emptying trash
|
|
.PP
|
|
If you wish to empty your trash you can use the
|
|
\f[C]rclone\ cleanup\ remote:\f[] command which will permanently delete
|
|
all your trashed files.
|
|
This command does not take any path arguments.
|
|
.SS Quota information
|
|
.PP
|
|
To view your current quota you can use the
|
|
\f[C]rclone\ about\ remote:\f[] command which will display your usage
|
|
limit (quota), the usage in Google Drive, the size of all files in the
|
|
Trash and the space used by other Google services such as Gmail.
|
|
This command does not take any path arguments.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-drive\-acknowledge\-abuse
|
|
.PP
|
|
If downloading a file returns the error
|
|
\f[C]This\ file\ has\ been\ identified\ as\ malware\ or\ spam\ and\ cannot\ be\ downloaded\f[]
|
|
with the error code \f[C]cannotDownloadAbusiveFile\f[] then supply this
|
|
flag to rclone to indicate you acknowledge the risks of downloading the
|
|
file and rclone will download it anyway.
|
|
.SS \-\-drive\-auth\-owner\-only
|
|
.PP
|
|
Only consider files owned by the authenticated user.
|
|
.SS \-\-drive\-chunk\-size=SIZE
|
|
.PP
|
|
Upload chunk size.
|
|
Must a power of 2 >= 256k.
|
|
Default value is 8 MB.
|
|
.PP
|
|
Making this larger will improve performance, but note that each chunk is
|
|
buffered in memory one per transfer.
|
|
.PP
|
|
Reducing this will reduce memory usage but decrease performance.
|
|
.SS \-\-drive\-formats
|
|
.PP
|
|
Google documents can only be exported from Google drive.
|
|
When rclone downloads a Google doc it chooses a format to download
|
|
depending upon this setting.
|
|
.PP
|
|
By default the formats are \f[C]docx,xlsx,pptx,svg\f[] which are a
|
|
sensible default for an editable document.
|
|
.PP
|
|
When choosing a format, rclone runs down the list provided in order and
|
|
chooses the first file format the doc can be exported as from the list.
|
|
If the file can\[aq]t be exported to a format on the formats list, then
|
|
rclone will choose a format from the default list.
|
|
.PP
|
|
If you prefer an archive copy then you might use
|
|
\f[C]\-\-drive\-formats\ pdf\f[], or if you prefer
|
|
openoffice/libreoffice formats you might use
|
|
\f[C]\-\-drive\-formats\ ods,odt,odp\f[].
|
|
.PP
|
|
Note that rclone adds the extension to the google doc, so if it is
|
|
calles \f[C]My\ Spreadsheet\f[] on google docs, it will be exported as
|
|
\f[C]My\ Spreadsheet.xlsx\f[] or \f[C]My\ Spreadsheet.pdf\f[] etc.
|
|
.PP
|
|
Here are the possible extensions with their corresponding mime types.
|
|
.PP
|
|
.TS
|
|
tab(@);
|
|
lw(9.7n) lw(11.7n) lw(12.6n).
|
|
T{
|
|
Extension
|
|
T}@T{
|
|
Mime Type
|
|
T}@T{
|
|
Description
|
|
T}
|
|
_
|
|
T{
|
|
csv
|
|
T}@T{
|
|
text/csv
|
|
T}@T{
|
|
Standard CSV format for Spreadsheets
|
|
T}
|
|
T{
|
|
doc
|
|
T}@T{
|
|
application/msword
|
|
T}@T{
|
|
Micosoft Office Document
|
|
T}
|
|
T{
|
|
docx
|
|
T}@T{
|
|
application/vnd.openxmlformats\-officedocument.wordprocessingml.document
|
|
T}@T{
|
|
Microsoft Office Document
|
|
T}
|
|
T{
|
|
epub
|
|
T}@T{
|
|
application/epub+zip
|
|
T}@T{
|
|
E\-book format
|
|
T}
|
|
T{
|
|
html
|
|
T}@T{
|
|
text/html
|
|
T}@T{
|
|
An HTML Document
|
|
T}
|
|
T{
|
|
jpg
|
|
T}@T{
|
|
image/jpeg
|
|
T}@T{
|
|
A JPEG Image File
|
|
T}
|
|
T{
|
|
odp
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.presentation
|
|
T}@T{
|
|
Openoffice Presentation
|
|
T}
|
|
T{
|
|
ods
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.spreadsheet
|
|
T}@T{
|
|
Openoffice Spreadsheet
|
|
T}
|
|
T{
|
|
ods
|
|
T}@T{
|
|
application/x\-vnd.oasis.opendocument.spreadsheet
|
|
T}@T{
|
|
Openoffice Spreadsheet
|
|
T}
|
|
T{
|
|
odt
|
|
T}@T{
|
|
application/vnd.oasis.opendocument.text
|
|
T}@T{
|
|
Openoffice Document
|
|
T}
|
|
T{
|
|
pdf
|
|
T}@T{
|
|
application/pdf
|
|
T}@T{
|
|
Adobe PDF Format
|
|
T}
|
|
T{
|
|
png
|
|
T}@T{
|
|
image/png
|
|
T}@T{
|
|
PNG Image Format
|
|
T}
|
|
T{
|
|
pptx
|
|
T}@T{
|
|
application/vnd.openxmlformats\-officedocument.presentationml.presentation
|
|
T}@T{
|
|
Microsoft Office Powerpoint
|
|
T}
|
|
T{
|
|
rtf
|
|
T}@T{
|
|
application/rtf
|
|
T}@T{
|
|
Rich Text Format
|
|
T}
|
|
T{
|
|
svg
|
|
T}@T{
|
|
image/svg+xml
|
|
T}@T{
|
|
Scalable Vector Graphics Format
|
|
T}
|
|
T{
|
|
tsv
|
|
T}@T{
|
|
text/tab\-separated\-values
|
|
T}@T{
|
|
Standard TSV format for spreadsheets
|
|
T}
|
|
T{
|
|
txt
|
|
T}@T{
|
|
text/plain
|
|
T}@T{
|
|
Plain Text
|
|
T}
|
|
T{
|
|
xls
|
|
T}@T{
|
|
application/vnd.ms\-excel
|
|
T}@T{
|
|
Microsoft Office Spreadsheet
|
|
T}
|
|
T{
|
|
xlsx
|
|
T}@T{
|
|
application/vnd.openxmlformats\-officedocument.spreadsheetml.sheet
|
|
T}@T{
|
|
Microsoft Office Spreadsheet
|
|
T}
|
|
T{
|
|
zip
|
|
T}@T{
|
|
application/zip
|
|
T}@T{
|
|
A ZIP file of HTML, Images CSS
|
|
T}
|
|
.TE
|
|
.SS \-\-drive\-alternate\-export
|
|
.PP
|
|
If this option is set this instructs rclone to use an alternate set of
|
|
export URLs for drive documents.
|
|
Users have reported that the official export URLs can\[aq]t export large
|
|
documents, whereas these unofficial ones can.
|
|
.PP
|
|
See rclone issue #2243 (https://github.com/ncw/rclone/issues/2243) for
|
|
background, this google drive
|
|
issue (https://issuetracker.google.com/issues/36761333) and this helpful
|
|
post (https://www.labnol.org/internet/direct-links-for-google-drive/28356/).
|
|
.SS \-\-drive\-impersonate user
|
|
.PP
|
|
When using a service account, this instructs rclone to impersonate the
|
|
user passed in.
|
|
.SS \-\-drive\-list\-chunk int
|
|
.PP
|
|
Size of listing chunk 100\-1000.
|
|
0 to disable.
|
|
(default 1000)
|
|
.SS \-\-drive\-shared\-with\-me
|
|
.PP
|
|
Instructs rclone to operate on your "Shared with me" folder (where
|
|
Google Drive lets you access the files and folders others have shared
|
|
with you).
|
|
.PP
|
|
This works both with the "list" (lsd, lsl, etc) and the "copy" commands
|
|
(copy, sync, etc), and with all other commands too.
|
|
.SS \-\-drive\-skip\-gdocs
|
|
.PP
|
|
Skip google documents in all listings.
|
|
If given, gdocs practically become invisible to rclone.
|
|
.SS \-\-drive\-trashed\-only
|
|
.PP
|
|
Only show files that are in the trash.
|
|
This will show trashed files in their original directory structure.
|
|
.SS \-\-drive\-upload\-cutoff=SIZE
|
|
.PP
|
|
File size cutoff for switching to chunked upload.
|
|
Default is 8 MB.
|
|
.SS \-\-drive\-use\-trash
|
|
.PP
|
|
Controls whether files are sent to the trash or deleted permanently.
|
|
Defaults to true, namely sending files to the trash.
|
|
Use \f[C]\-\-drive\-use\-trash=false\f[] to delete files permanently
|
|
instead.
|
|
.SS \-\-drive\-use\-created\-date
|
|
.PP
|
|
Use the file creation date in place of the modification date.
|
|
Defaults to false.
|
|
.PP
|
|
Useful when downloading data and you want the creation date used in
|
|
place of the last modified date.
|
|
.PP
|
|
\f[B]WARNING\f[]: This flag may have some unexpected consequences.
|
|
.PP
|
|
When uploading to your drive all files will be overwritten unless they
|
|
haven\[aq]t been modified since their creation.
|
|
And the inverse will occur while downloading.
|
|
This side effect can be avoided by using the \f[C]\-\-checksum\f[] flag.
|
|
.PP
|
|
This feature was implemented to retain photos capture date as recorded
|
|
by google photos.
|
|
You will first need to check the "Create a Google Photos folder" option
|
|
in your google drive settings.
|
|
You can then copy or move the photos locally and use the date the image
|
|
was taken (created) set as the modification date.
|
|
.SS Limitations
|
|
.PP
|
|
Drive has quite a lot of rate limiting.
|
|
This causes rclone to be limited to transferring about 2 files per
|
|
second only.
|
|
Individual files may be transferred much faster at 100s of MBytes/s but
|
|
lots of small files can take a long time.
|
|
.PP
|
|
Server side copies are also subject to a separate rate limit.
|
|
If you see User rate limit exceeded errors, wait at least 24 hours and
|
|
retry.
|
|
You can disable server side copies with \f[C]\-\-disable\ copy\f[] to
|
|
download and upload the files if you prefer.
|
|
.SS Limitations of Google Docs
|
|
.PP
|
|
Google docs will appear as size \-1 in \f[C]rclone\ ls\f[] and as size 0
|
|
in anything which uses the VFS layer, eg \f[C]rclone\ mount\f[],
|
|
\f[C]rclone\ serve\f[].
|
|
.PP
|
|
This is because rclone can\[aq]t find out the size of the Google docs
|
|
without downloading them.
|
|
.PP
|
|
Google docs will transfer correctly with \f[C]rclone\ sync\f[],
|
|
\f[C]rclone\ copy\f[] etc as rclone knows to ignore the size when doing
|
|
the transfer.
|
|
.PP
|
|
However an unfortunate consequence of this is that you can\[aq]t
|
|
download Google docs using \f[C]rclone\ mount\f[] \- you will get a 0
|
|
sized file.
|
|
If you try again the doc may gain its correct size and be downloadable.
|
|
.SS Duplicated files
|
|
.PP
|
|
Sometimes, for no reason I\[aq]ve been able to track down, drive will
|
|
duplicate a file that rclone uploads.
|
|
Drive unlike all the other remotes can have duplicated files.
|
|
.PP
|
|
Duplicated files cause problems with the syncing and you will see
|
|
messages in the log about duplicates.
|
|
.PP
|
|
Use \f[C]rclone\ dedupe\f[] to fix duplicated files.
|
|
.PP
|
|
Note that this isn\[aq]t just a problem with rclone, even Google Photos
|
|
on Android duplicates files on drive sometimes.
|
|
.SS Rclone appears to be re\-copying files it shouldn\[aq]t
|
|
.PP
|
|
The most likely cause of this is the duplicated file issue above \- run
|
|
\f[C]rclone\ dedupe\f[] and check your logs for duplicate object or
|
|
directory messages.
|
|
.SS Making your own client_id
|
|
.PP
|
|
When you use rclone with Google drive in its default configuration you
|
|
are using rclone\[aq]s client_id.
|
|
This is shared between all the rclone users.
|
|
There is a global rate limit on the number of queries per second that
|
|
each client_id can do set by Google.
|
|
rclone already has a high quota and I will continue to make sure it is
|
|
high enough by contacting Google.
|
|
.PP
|
|
However you might find you get better performance making your own
|
|
client_id if you are a heavy user.
|
|
Or you may not depending on exactly how Google have been raising
|
|
rclone\[aq]s rate limit.
|
|
.PP
|
|
Here is how to create your own Google Drive client ID for rclone:
|
|
.IP "1." 3
|
|
Log into the Google API Console (https://console.developers.google.com/)
|
|
with your Google account.
|
|
It doesn\[aq]t matter what Google account you use.
|
|
(It need not be the same account as the Google Drive you want to access)
|
|
.IP "2." 3
|
|
Select a project or create a new project.
|
|
.IP "3." 3
|
|
Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the then
|
|
"Google Drive API".
|
|
.IP "4." 3
|
|
Click "Credentials" in the left\-side panel (not "Create credentials",
|
|
which opens the wizard), then "Create credentials", then "OAuth client
|
|
ID".
|
|
It will prompt you to set the OAuth consent screen product name, if you
|
|
haven\[aq]t set one already.
|
|
.IP "5." 3
|
|
Choose an application type of "other", and click "Create".
|
|
(the default name is fine)
|
|
.IP "6." 3
|
|
It will show you a client ID and client secret.
|
|
Use these values in rclone config to add a new remote or edit an
|
|
existing remote.
|
|
.PP
|
|
(Thanks to \@balazer on github for these instructions.)
|
|
.SS HTTP
|
|
.PP
|
|
The HTTP remote is a read only remote for reading files of a webserver.
|
|
The webserver should provide file listings which rclone will read and
|
|
turn into a remote.
|
|
This has been tested with common webservers such as Apache/Nginx/Caddy
|
|
and will likely work with file listings from most web servers.
|
|
(If it doesn\[aq]t then please file an issue, or send a pull request!)
|
|
.PP
|
|
Paths are specified as \f[C]remote:\f[] or \f[C]remote:path/to/dir\f[].
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 7\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 8\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 9\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
10\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
11\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
12\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
13\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
14\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
15\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ http
|
|
URL\ of\ http\ host\ to\ connect\ to
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Connect\ to\ example.com
|
|
\ \ \ \\\ "https://example.com"
|
|
url>\ https://beta.rclone.org
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
url\ =\ https://beta.rclone.org
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
Current\ remotes:
|
|
|
|
Name\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Type
|
|
====\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ====
|
|
remote\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ http
|
|
|
|
e)\ Edit\ existing\ remote
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
e/n/d/r/c/s/q>\ q
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all the top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync the remote \f[C]directory\f[] to \f[C]/home/local/directory\f[],
|
|
deleting any excess files.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ remote:directory\ /home/local/directory
|
|
\f[]
|
|
.fi
|
|
.SS Read only
|
|
.PP
|
|
This remote is read only \- you can\[aq]t upload files to an HTTP
|
|
server.
|
|
.SS Modified time
|
|
.PP
|
|
Most HTTP servers store time accurate to 1 second.
|
|
.SS Checksum
|
|
.PP
|
|
No checksums are stored.
|
|
.SS Usage without a config file
|
|
.PP
|
|
Note that since only two environment variable need to be set, it is easy
|
|
to use without a config file like this.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
RCLONE_CONFIG_ZZ_TYPE=http\ RCLONE_CONFIG_ZZ_URL=https://beta.rclone.org\ rclone\ lsd\ zz:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Or if you prefer
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export\ RCLONE_CONFIG_ZZ_TYPE=http
|
|
export\ RCLONE_CONFIG_ZZ_URL=https://beta.rclone.org
|
|
rclone\ lsd\ zz:
|
|
\f[]
|
|
.fi
|
|
.SS Hubic
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[] (or \f[C]remote:\f[]
|
|
for the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:container/path/to/dir\f[].
|
|
.PP
|
|
The initial setup for Hubic involves getting a token from Hubic which
|
|
you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 8
|
|
Hubic\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>
|
|
Hubic\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =
|
|
client_secret\ =
|
|
token\ =\ {"access_token":"XXXXXX"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Hubic.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List containers in the top level of your Hubic
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your Hubic
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Hubic directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you want the directory to be visible in the official \f[I]Hubic
|
|
browser\f[], you need to copy your files to the \f[C]default\f[]
|
|
directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:default/backup
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Object\-Meta\-Mtime\f[] as floating point since the epoch
|
|
accurate to 1 ns.
|
|
.PP
|
|
This is a defacto standard (used in the official python\-swiftclient
|
|
amongst others) for storing the modification time for an object.
|
|
.PP
|
|
Note that Hubic wraps the Swift backend, so most of the properties of
|
|
are the same.
|
|
.SS Limitations
|
|
.PP
|
|
This uses the normal OpenStack Swift mechanism to refresh the Swift API
|
|
credentials and ignores the expires field returned by the Hubic API.
|
|
.PP
|
|
The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
|
|
(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
|
|
MD5SUM for these.
|
|
.SS Mega
|
|
.PP
|
|
Mega (https://mega.nz/) is a cloud storage and file hosting service
|
|
known for its security feature where all files are encrypted locally
|
|
before they are uploaded.
|
|
This prevents anyone (including employees of Mega) from accessing the
|
|
files without knowledge of the key used for encryption.
|
|
.PP
|
|
This is an rclone backend for Mega which supports the file transfer
|
|
features of Mega using the same client side encryption.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Alias\ for\ a\ existing\ remote
|
|
\ \ \ \\\ "alias"
|
|
[snip]
|
|
14\ /\ Mega
|
|
\ \ \ \\\ "mega"
|
|
[snip]
|
|
23\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ mega
|
|
User\ name
|
|
user>\ you\@example.com
|
|
Password.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
n)\ No\ leave\ this\ optional\ password\ blank
|
|
y/g/n>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
type\ =\ mega
|
|
user\ =\ you\@example.com
|
|
pass\ =\ ***\ ENCRYPTED\ ***
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your Mega
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your Mega
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an Mega directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Mega does not support modification times or hashes yet.
|
|
.SS Duplicated files
|
|
.PP
|
|
Mega can have two files with exactly the same name and path (unlike a
|
|
normal file system).
|
|
.PP
|
|
Duplicated files cause problems with the syncing and you will see
|
|
messages in the log about duplicates.
|
|
.PP
|
|
Use \f[C]rclone\ dedupe\f[] to fix duplicated files.
|
|
.SS Limitations
|
|
.PP
|
|
This backend uses the go\-mega go
|
|
library (https://github.com/t3rm1n4l/go-mega) which is an opensource go
|
|
library implementing the Mega API.
|
|
There doesn\[aq]t appear to be any documentation for the mega protocol
|
|
beyond the mega C++ SDK (https://github.com/meganz/sdk) source code so
|
|
there are likely quite a few errors still remaining in this library.
|
|
.PP
|
|
Mega allows duplicate files which may confuse rclone.
|
|
.SS Microsoft Azure Blob Storage
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[] (or \f[C]remote:\f[]
|
|
for the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:container/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making a Microsoft Azure Blob Storage
|
|
configuration.
|
|
For a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Box
|
|
\ \ \ \\\ "box"
|
|
\ 5\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 6\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 7\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 8\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 9\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
10\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
11\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
12\ /\ Microsoft\ Azure\ Blob\ Storage
|
|
\ \ \ \\\ "azureblob"
|
|
13\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
14\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
15\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
16\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
17\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ azureblob
|
|
Storage\ Account\ Name
|
|
account>\ account_name
|
|
Storage\ Account\ Key
|
|
key>\ base64encodedkey==
|
|
Endpoint\ for\ the\ service\ \-\ leave\ blank\ normally.
|
|
endpoint>\
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
account\ =\ account_name
|
|
key\ =\ base64encodedkey==
|
|
endpoint\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See all containers
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:container
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:container
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote container, deleting
|
|
any excess files in the container.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:container
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object with the
|
|
\f[C]mtime\f[] key.
|
|
It is stored using RFC3339 Format time with nanosecond precision.
|
|
The metadata is supplied during directory listings so there is no
|
|
overhead to using it.
|
|
.SS Hashes
|
|
.PP
|
|
MD5 hashes are stored with blobs.
|
|
However blobs that were uploaded in chunks only have an MD5 if the
|
|
source remote was capable of MD5 hashes, eg the local disk.
|
|
.SS Multipart uploads
|
|
.PP
|
|
Rclone supports multipart uploads with Azure Blob storage.
|
|
Files bigger than 256MB will be uploaded using chunked upload by
|
|
default.
|
|
.PP
|
|
The files will be uploaded in parallel in 4MB chunks (by default).
|
|
Note that these chunks are buffered in memory and there may be up to
|
|
\f[C]\-\-transfers\f[] of them being uploaded at once.
|
|
.PP
|
|
Files can\[aq]t be split into more than 50,000 chunks so by default, so
|
|
the largest file that can be uploaded with 4MB chunk size is 195GB.
|
|
Above this rclone will double the chunk size until it creates less than
|
|
50,000 chunks.
|
|
By default this will mean a maximum file size of 3.2TB can be uploaded.
|
|
This can be raised to 5TB using
|
|
\f[C]\-\-azureblob\-chunk\-size\ 100M\f[].
|
|
.PP
|
|
Note that rclone doesn\[aq]t commit the block list until the end of the
|
|
upload which means that there is a limit of 9.5TB of multipart uploads
|
|
in progress as Azure won\[aq]t allow more than that amount of
|
|
uncommitted blocks.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-azureblob\-upload\-cutoff=SIZE
|
|
.PP
|
|
Cutoff for switching to chunked upload \- must be <= 256MB.
|
|
The default is 256MB.
|
|
.SS \-\-azureblob\-chunk\-size=SIZE
|
|
.PP
|
|
Upload chunk size.
|
|
Default 4MB.
|
|
Note that this is stored in memory and there may be up to
|
|
\f[C]\-\-transfers\f[] chunks stored at once in memory.
|
|
This can be at most 100MB.
|
|
.SS Limitations
|
|
.PP
|
|
MD5 sums are only uploaded with chunked files if the source has an MD5
|
|
sum.
|
|
This will always be the case for a local to azure copy.
|
|
.SS Microsoft OneDrive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for OneDrive involves getting a token from Microsoft
|
|
which you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 10
|
|
Microsoft\ App\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>
|
|
Microsoft\ App\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>
|
|
Remote\ config
|
|
Choose\ OneDrive\ account\ type?
|
|
\ *\ Say\ b\ for\ a\ OneDrive\ business\ account
|
|
\ *\ Say\ p\ for\ a\ personal\ OneDrive\ account
|
|
b)\ Business
|
|
p)\ Personal
|
|
b/p>\ p
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =
|
|
client_secret\ =
|
|
token\ =\ {"access_token":"XXXXXX"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Microsoft.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your OneDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your OneDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an OneDrive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS OneDrive for Business
|
|
.PP
|
|
There is additional support for OneDrive for Business.
|
|
Select "b" when ask
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Choose\ OneDrive\ account\ type?
|
|
\ *\ Say\ b\ for\ a\ OneDrive\ business\ account
|
|
\ *\ Say\ p\ for\ a\ personal\ OneDrive\ account
|
|
b)\ Business
|
|
p)\ Personal
|
|
b/p>
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
After that rclone requires an authentication of your account.
|
|
The application will first authenticate your account, then query the
|
|
OneDrive resource URL and do a second (silent) authentication for this
|
|
resource URL.
|
|
.SS Modified time and hashes
|
|
.PP
|
|
OneDrive allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.PP
|
|
OneDrive personal supports SHA1 type hashes.
|
|
OneDrive for business and Sharepoint Server support
|
|
QuickXorHash (https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash).
|
|
.PP
|
|
For all types of OneDrive you can use the \f[C]\-\-checksum\f[] flag.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Microsoft doesn\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Microsoft\[aq]s
|
|
apps or via the OneDrive website.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-onedrive\-chunk\-size=SIZE
|
|
.PP
|
|
Above this size files will be chunked \- must be multiple of 320k.
|
|
The default is 10MB.
|
|
Note that the chunks will be buffered into memory.
|
|
.SS Limitations
|
|
.PP
|
|
Note that OneDrive is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
There are quite a few characters that can\[aq]t be in OneDrive file
|
|
names.
|
|
These can\[aq]t occur on Windows platforms, but on non\-Windows
|
|
platforms they are common.
|
|
Rclone will map these names to and from an identical looking unicode
|
|
equivalent.
|
|
For example if a file has a \f[C]?\f[] in it will be mapped to
|
|
\f[C]？\f[] instead.
|
|
.PP
|
|
The largest allowed file size is 10GiB (10,737,418,240 bytes).
|
|
.SS Versioning issue
|
|
.PP
|
|
Every change in OneDrive causes the service to create a new version.
|
|
This counts against a users quota.
|
|
For example changing the modification time of a file creates a second
|
|
version, so the file is using twice the space.
|
|
.PP
|
|
The \f[C]copy\f[] is the only rclone command affected by this as we copy
|
|
the file and then afterwards set the modification time to match the
|
|
source file.
|
|
.PP
|
|
User Weropol (https://github.com/Weropol) has found a method to disable
|
|
versioning on OneDrive
|
|
.IP "1." 3
|
|
Open the settings menu by clicking on the gear symbol at the top of the
|
|
OneDrive Business page.
|
|
.IP "2." 3
|
|
Click Site settings.
|
|
.IP "3." 3
|
|
Once on the Site settings page, navigate to Site Administration > Site
|
|
libraries and lists.
|
|
.IP "4." 3
|
|
Click Customize "Documents".
|
|
.IP "5." 3
|
|
Click General Settings > Versioning Settings.
|
|
.IP "6." 3
|
|
Under Document Version History select the option No versioning.
|
|
Note: This will disable the creation of new file versions, but will not
|
|
remove any previous versions.
|
|
Your documents are safe.
|
|
.IP "7." 3
|
|
Apply the changes by clicking OK.
|
|
.IP "8." 3
|
|
Use rclone to upload or modify files.
|
|
(I also use the \-\-no\-update\-modtime flag)
|
|
.IP "9." 3
|
|
Restore the versioning settings after using rclone.
|
|
(Optional)
|
|
.SS Troubleshooting
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Error:\ access_denied
|
|
Code:\ AADSTS65005
|
|
Description:\ Using\ application\ \[aq]rclone\[aq]\ is\ currently\ not\ supported\ for\ your\ organization\ [YOUR_ORGANIZATION]\ because\ it\ is\ in\ an\ unmanaged\ state.\ An\ administrator\ needs\ to\ claim\ ownership\ of\ the\ company\ by\ DNS\ validation\ of\ [YOUR_ORGANIZATION]\ before\ the\ application\ rclone\ can\ be\ provisioned.
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This means that rclone can\[aq]t use the OneDrive for Business API with
|
|
your account.
|
|
You can\[aq]t do much about it, maybe write an email to your admins.
|
|
.PP
|
|
However, there are other ways to interact with your OneDrive account.
|
|
Have a look at the webdav backend: https://rclone.org/webdav/#sharepoint
|
|
.SS OpenDrive
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
n)\ New\ remote
|
|
d)\ Delete\ remote
|
|
q)\ Quit\ config
|
|
e/n/d/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ OpenDrive
|
|
\ \ \ \\\ "opendrive"
|
|
11\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
12\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
13\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
14\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 10
|
|
Username
|
|
username>
|
|
Password
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
y/g>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
username\ =
|
|
password\ =\ ***\ ENCRYPTED\ ***
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List directories in top level of your OpenDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your OpenDrive
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an OpenDrive directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and MD5SUMs
|
|
.PP
|
|
OpenDrive allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
.SS Deleting files
|
|
.PP
|
|
Any files you delete with rclone will end up in the trash.
|
|
Amazon don\[aq]t provide an API to permanently delete files, nor to
|
|
empty the trash, so you will have to do that with one of Amazon\[aq]s
|
|
apps or via the OpenDrive website.
|
|
As of November 17, 2016, files are automatically deleted by Amazon from
|
|
the trash after 30 days.
|
|
.SS Limitations
|
|
.PP
|
|
Note that OpenDrive is case insensitive so you can\[aq]t have a file
|
|
called "Hello.doc" and one called "hello.doc".
|
|
.PP
|
|
There are quite a few characters that can\[aq]t be in OpenDrive file
|
|
names.
|
|
These can\[aq]t occur on Windows platforms, but on non\-Windows
|
|
platforms they are common.
|
|
Rclone will map these names to and from an identical looking unicode
|
|
equivalent.
|
|
For example if a file has a \f[C]?\f[] in it will be mapped to
|
|
\f[C]？\f[] instead.
|
|
.SS QingStor
|
|
.PP
|
|
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
|
the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:bucket/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making an QingStor configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
r)\ Rename\ remote
|
|
c)\ Copy\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/r/c/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 7\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 8\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 9\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
10\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
11\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
12\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
13\ /\ QingStor\ Object\ Storage
|
|
\ \ \ \\\ "qingstor"
|
|
14\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
15\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 13
|
|
Get\ QingStor\ credentials\ from\ runtime.\ Only\ applies\ if\ access_key_id\ and\ secret_access_key\ is\ blank.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Enter\ QingStor\ credentials\ in\ the\ next\ step
|
|
\ \ \ \\\ "false"
|
|
\ 2\ /\ Get\ QingStor\ credentials\ from\ the\ environment\ (env\ vars\ or\ IAM)
|
|
\ \ \ \\\ "true"
|
|
env_auth>\ 1
|
|
QingStor\ Access\ Key\ ID\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
access_key_id>\ access_key
|
|
QingStor\ Secret\ Access\ Key\ (password)\ \-\ leave\ blank\ for\ anonymous\ access\ or\ runtime\ credentials.
|
|
secret_access_key>\ secret_key
|
|
Enter\ a\ endpoint\ URL\ to\ connection\ QingStor\ API.
|
|
Leave\ blank\ will\ use\ the\ default\ value\ "https://qingstor.com:443"
|
|
endpoint>
|
|
Zone\ connect\ to.\ Default\ is\ "pek3a".
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ \ \ /\ The\ Beijing\ (China)\ Three\ Zone
|
|
\ 1\ |\ Needs\ location\ constraint\ pek3a.
|
|
\ \ \ \\\ "pek3a"
|
|
\ \ \ /\ The\ Shanghai\ (China)\ First\ Zone
|
|
\ 2\ |\ Needs\ location\ constraint\ sh1a.
|
|
\ \ \ \\\ "sh1a"
|
|
zone>\ 1
|
|
Number\ of\ connnection\ retry.
|
|
Leave\ blank\ will\ use\ the\ default\ value\ "3".
|
|
connection_retries>
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
env_auth\ =\ false
|
|
access_key_id\ =\ access_key
|
|
secret_access_key\ =\ secret_key
|
|
endpoint\ =
|
|
zone\ =\ pek3a
|
|
connection_retries\ =
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all buckets
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a bucket
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:bucket
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS Multipart uploads
|
|
.PP
|
|
rclone supports multipart uploads with QingStor which means that it can
|
|
upload files bigger than 5GB.
|
|
Note that files uploaded with multipart upload don\[aq]t have an MD5SUM.
|
|
.SS Buckets and Zone
|
|
.PP
|
|
With QingStor you can list buckets (\f[C]rclone\ lsd\f[]) using any
|
|
zone, but you can only access the content of a bucket from the zone it
|
|
was created in.
|
|
If you attempt to access a bucket from the wrong zone, you will get an
|
|
error,
|
|
\f[C]incorrect\ zone,\ the\ bucket\ is\ not\ in\ \[aq]XXX\[aq]\ zone\f[].
|
|
.SS Authentication
|
|
.PP
|
|
There are two ways to supply \f[C]rclone\f[] with a set of QingStor
|
|
credentials.
|
|
In order of precedence:
|
|
.IP \[bu] 2
|
|
Directly in the rclone configuration file (as configured by
|
|
\f[C]rclone\ config\f[])
|
|
.IP \[bu] 2
|
|
set \f[C]access_key_id\f[] and \f[C]secret_access_key\f[]
|
|
.IP \[bu] 2
|
|
Runtime configuration:
|
|
.IP \[bu] 2
|
|
set \f[C]env_auth\f[] to \f[C]true\f[] in the config file
|
|
.IP \[bu] 2
|
|
Exporting the following environment variables before running
|
|
\f[C]rclone\f[]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Access Key ID: \f[C]QS_ACCESS_KEY_ID\f[] or \f[C]QS_ACCESS_KEY\f[]
|
|
.IP \[bu] 2
|
|
Secret Access Key: \f[C]QS_SECRET_ACCESS_KEY\f[] or
|
|
\f[C]QS_SECRET_KEY\f[]
|
|
.RE
|
|
.SS Swift
|
|
.PP
|
|
Swift refers to Openstack Object
|
|
Storage (https://docs.openstack.org/swift/latest/).
|
|
Commercial implementations of that being:
|
|
.IP \[bu] 2
|
|
Rackspace Cloud Files (https://www.rackspace.com/cloud/files/)
|
|
.IP \[bu] 2
|
|
Memset Memstore (https://www.memset.com/cloud/storage/)
|
|
.IP \[bu] 2
|
|
OVH Object
|
|
Storage (https://www.ovh.co.uk/public-cloud/storage/object-storage/)
|
|
.IP \[bu] 2
|
|
Oracle Cloud Storage (https://cloud.oracle.com/storage-opc)
|
|
.IP \[bu] 2
|
|
IBM Bluemix Cloud ObjectStorage
|
|
Swift (https://console.bluemix.net/docs/infrastructure/objectstorage-swift/index.html)
|
|
.PP
|
|
Paths are specified as \f[C]remote:container\f[] (or \f[C]remote:\f[]
|
|
for the \f[C]lsd\f[] command.) You may put subdirectories in too, eg
|
|
\f[C]remote:container/path/to/dir\f[].
|
|
.PP
|
|
Here is an example of making a swift configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Box
|
|
\ \ \ \\\ "box"
|
|
\ 5\ /\ Cache\ a\ remote
|
|
\ \ \ \\\ "cache"
|
|
\ 6\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 7\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 8\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 9\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
10\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
11\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
12\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
13\ /\ Microsoft\ Azure\ Blob\ Storage
|
|
\ \ \ \\\ "azureblob"
|
|
14\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
15\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
16\ /\ Pcloud
|
|
\ \ \ \\\ "pcloud"
|
|
17\ /\ QingCloud\ Object\ Storage
|
|
\ \ \ \\\ "qingstor"
|
|
18\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
19\ /\ Webdav
|
|
\ \ \ \\\ "webdav"
|
|
20\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
21\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ swift
|
|
Get\ swift\ credentials\ from\ environment\ variables\ in\ standard\ OpenStack\ form.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Enter\ swift\ credentials\ in\ the\ next\ step
|
|
\ \ \ \\\ "false"
|
|
\ 2\ /\ Get\ swift\ credentials\ from\ environment\ vars.\ Leave\ other\ fields\ blank\ if\ using\ this.
|
|
\ \ \ \\\ "true"
|
|
env_auth>\ true
|
|
User\ name\ to\ log\ in\ (OS_USERNAME).
|
|
user>\
|
|
API\ key\ or\ password\ (OS_PASSWORD).
|
|
key>\
|
|
Authentication\ URL\ for\ server\ (OS_AUTH_URL).
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Rackspace\ US
|
|
\ \ \ \\\ "https://auth.api.rackspacecloud.com/v1.0"
|
|
\ 2\ /\ Rackspace\ UK
|
|
\ \ \ \\\ "https://lon.auth.api.rackspacecloud.com/v1.0"
|
|
\ 3\ /\ Rackspace\ v2
|
|
\ \ \ \\\ "https://identity.api.rackspacecloud.com/v2.0"
|
|
\ 4\ /\ Memset\ Memstore\ UK
|
|
\ \ \ \\\ "https://auth.storage.memset.com/v1.0"
|
|
\ 5\ /\ Memset\ Memstore\ UK\ v2
|
|
\ \ \ \\\ "https://auth.storage.memset.com/v2.0"
|
|
\ 6\ /\ OVH
|
|
\ \ \ \\\ "https://auth.cloud.ovh.net/v2.0"
|
|
auth>\
|
|
User\ ID\ to\ log\ in\ \-\ optional\ \-\ most\ swift\ systems\ use\ user\ and\ leave\ this\ blank\ (v3\ auth)\ (OS_USER_ID).
|
|
user_id>\
|
|
User\ domain\ \-\ optional\ (v3\ auth)\ (OS_USER_DOMAIN_NAME)
|
|
domain>\
|
|
Tenant\ name\ \-\ optional\ for\ v1\ auth,\ this\ or\ tenant_id\ required\ otherwise\ (OS_TENANT_NAME\ or\ OS_PROJECT_NAME)
|
|
tenant>\
|
|
Tenant\ ID\ \-\ optional\ for\ v1\ auth,\ this\ or\ tenant\ required\ otherwise\ (OS_TENANT_ID)
|
|
tenant_id>\
|
|
Tenant\ domain\ \-\ optional\ (v3\ auth)\ (OS_PROJECT_DOMAIN_NAME)
|
|
tenant_domain>\
|
|
Region\ name\ \-\ optional\ (OS_REGION_NAME)
|
|
region>\
|
|
Storage\ URL\ \-\ optional\ (OS_STORAGE_URL)
|
|
storage_url>\
|
|
Auth\ Token\ from\ alternate\ authentication\ \-\ optional\ (OS_AUTH_TOKEN)
|
|
auth_token>\
|
|
AuthVersion\ \-\ optional\ \-\ set\ to\ (1,2,3)\ if\ your\ auth\ URL\ has\ no\ version\ (ST_AUTH_VERSION)
|
|
auth_version>\
|
|
Endpoint\ type\ to\ choose\ from\ the\ service\ catalogue\ (OS_ENDPOINT_TYPE)
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Public\ (default,\ choose\ this\ if\ not\ sure)
|
|
\ \ \ \\\ "public"
|
|
\ 2\ /\ Internal\ (use\ internal\ service\ net)
|
|
\ \ \ \\\ "internal"
|
|
\ 3\ /\ Admin
|
|
\ \ \ \\\ "admin"
|
|
endpoint_type>\
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[test]
|
|
env_auth\ =\ true
|
|
user\ =\
|
|
key\ =\
|
|
auth\ =\
|
|
user_id\ =\
|
|
domain\ =\
|
|
tenant\ =\
|
|
tenant_id\ =\
|
|
tenant_domain\ =\
|
|
region\ =\
|
|
storage_url\ =\
|
|
auth_token\ =\
|
|
auth_version\ =\
|
|
endpoint_type\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this
|
|
.PP
|
|
See all containers
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:container
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a container
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:container
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote container, deleting
|
|
any excess files in the container.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:container
|
|
\f[]
|
|
.fi
|
|
.SS Configuration from an Openstack credentials file
|
|
.PP
|
|
An Opentstack credentials file typically looks something something like
|
|
this (without the comments)
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
export\ OS_AUTH_URL=https://a.provider.net/v2.0
|
|
export\ OS_TENANT_ID=ffffffffffffffffffffffffffffffff
|
|
export\ OS_TENANT_NAME="1234567890123456"
|
|
export\ OS_USERNAME="123abc567xy"
|
|
echo\ "Please\ enter\ your\ OpenStack\ Password:\ "
|
|
read\ \-sr\ OS_PASSWORD_INPUT
|
|
export\ OS_PASSWORD=$OS_PASSWORD_INPUT
|
|
export\ OS_REGION_NAME="SBG1"
|
|
if\ [\ \-z\ "$OS_REGION_NAME"\ ];\ then\ unset\ OS_REGION_NAME;\ fi
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The config file needs to look something like this where
|
|
\f[C]$OS_USERNAME\f[] represents the value of the \f[C]OS_USERNAME\f[]
|
|
variable \- \f[C]123abc567xy\f[] in the example above.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[remote]
|
|
type\ =\ swift
|
|
user\ =\ $OS_USERNAME
|
|
key\ =\ $OS_PASSWORD
|
|
auth\ =\ $OS_AUTH_URL
|
|
tenant\ =\ $OS_TENANT_NAME
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Note that you may (or may not) need to set \f[C]region\f[] too \- try
|
|
without first.
|
|
.SS Configuration from the environment
|
|
.PP
|
|
If you prefer you can configure rclone to use swift using a standard set
|
|
of OpenStack environment variables.
|
|
.PP
|
|
When you run through the config, make sure you choose \f[C]true\f[] for
|
|
\f[C]env_auth\f[] and leave everything else blank.
|
|
.PP
|
|
rclone will then set any empty config parameters from the environment
|
|
using standard OpenStack environment variables.
|
|
There is a list of the
|
|
variables (https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment)
|
|
in the docs for the swift library.
|
|
.SS Using an alternate authentication method
|
|
.PP
|
|
If your OpenStack installation uses a non\-standard authentication
|
|
method that might not be yet supported by rclone or the underlying swift
|
|
library, you can authenticate externally (e.g.
|
|
calling manually the \f[C]openstack\f[] commands to get a token).
|
|
Then, you just need to pass the two configuration variables
|
|
\f[C]auth_token\f[] and \f[C]storage_url\f[].
|
|
If they are both provided, the other variables are ignored.
|
|
rclone will not try to authenticate but instead assume it is already
|
|
authenticated and use these two variables to access the OpenStack
|
|
installation.
|
|
.SS Using rclone without a config file
|
|
.PP
|
|
You can use rclone with swift without a config file, if desired, like
|
|
this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
source\ openstack\-credentials\-file
|
|
export\ RCLONE_CONFIG_MYREMOTE_TYPE=swift
|
|
export\ RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true
|
|
rclone\ lsd\ myremote:
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS \-\-update and \-\-use\-server\-modtime
|
|
.PP
|
|
As noted below, the modified time is stored on metadata on the object.
|
|
It is used by default for all operations that require checking the time
|
|
a file was last updated.
|
|
It allows rclone to treat the remote more like a true filesystem, but it
|
|
is inefficient because it requires an extra API call to retrieve the
|
|
metadata.
|
|
.PP
|
|
For many operations, the time the object was last uploaded to the remote
|
|
is sufficient to determine if it is "dirty".
|
|
By using \f[C]\-\-update\f[] along with
|
|
\f[C]\-\-use\-server\-modtime\f[], you can avoid the extra API call and
|
|
simply upload files whose local modtime is newer than the time it was
|
|
last uploaded.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this cloud storage system.
|
|
.SS \-\-swift\-chunk\-size=SIZE
|
|
.PP
|
|
Above this size files will be chunked into a _segments container.
|
|
The default for this is 5GB which is its maximum value.
|
|
.SS Modified time
|
|
.PP
|
|
The modified time is stored as metadata on the object as
|
|
\f[C]X\-Object\-Meta\-Mtime\f[] as floating point since the epoch
|
|
accurate to 1 ns.
|
|
.PP
|
|
This is a defacto standard (used in the official python\-swiftclient
|
|
amongst others) for storing the modification time for an object.
|
|
.SS Limitations
|
|
.PP
|
|
The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
|
|
(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
|
|
MD5SUM for these.
|
|
.SS Troubleshooting
|
|
.SS Rclone gives Failed to create file system for "remote:": Bad Request
|
|
.PP
|
|
Due to an oddity of the underlying swift library, it gives a "Bad
|
|
Request" error rather than a more sensible error when the authentication
|
|
fails for Swift.
|
|
.PP
|
|
So this most likely means your username / password is wrong.
|
|
You can investigate further with the \f[C]\-\-dump\-bodies\f[] flag.
|
|
.PP
|
|
This may also be caused by specifying the region when you shouldn\[aq]t
|
|
have (eg OVH).
|
|
.SS Rclone gives Failed to create file system: Response didn\[aq]t have
|
|
storage storage url and auth token
|
|
.PP
|
|
This is most likely caused by forgetting to specify your tenant when
|
|
setting up a swift remote.
|
|
.SS pCloud
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
The initial setup for pCloud involves getting a token from pCloud which
|
|
you need to do in your browser.
|
|
\f[C]rclone\ config\f[] walks you through it.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Box
|
|
\ \ \ \\\ "box"
|
|
\ 5\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 6\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 7\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 8\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 9\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
10\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
11\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
12\ /\ Microsoft\ Azure\ Blob\ Storage
|
|
\ \ \ \\\ "azureblob"
|
|
13\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
14\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
15\ /\ Pcloud
|
|
\ \ \ \\\ "pcloud"
|
|
16\ /\ QingCloud\ Object\ Storage
|
|
\ \ \ \\\ "qingstor"
|
|
17\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
18\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
19\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ pcloud
|
|
Pcloud\ App\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>\
|
|
Pcloud\ App\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>\
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =\
|
|
client_secret\ =\
|
|
token\ =\ {"access_token":"XXX","token_type":"bearer","expiry":"0001\-01\-01T00:00:00Z"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from pCloud.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your pCloud
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your pCloud
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an pCloud directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
pCloud allows modification times to be set on objects accurate to 1
|
|
second.
|
|
These will be used to detect whether objects need syncing or not.
|
|
In order to set a Modification time pCloud requires the object be
|
|
re\-uploaded.
|
|
.PP
|
|
pCloud supports MD5 and SHA1 type hashes, so you can use the
|
|
\f[C]\-\-checksum\f[] flag.
|
|
.SS Deleting files
|
|
.PP
|
|
Deleted files will be moved to the trash.
|
|
Your subscription level will determine how long items stay in the trash.
|
|
\f[C]rclone\ cleanup\f[] can be used to empty the trash.
|
|
.SS SFTP
|
|
.PP
|
|
SFTP is the Secure (or SSH) File Transfer
|
|
Protocol (https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).
|
|
.PP
|
|
SFTP runs over SSH v2 and is installed as standard with most modern SSH
|
|
installations.
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[].
|
|
If the path does not begin with a \f[C]/\f[] it is relative to the home
|
|
directory of the user.
|
|
An empty path \f[C]remote:\f[] refers to the user\[aq]s home directory.
|
|
.PP
|
|
Note that some SFTP servers will need the leading \f[C]/\f[] \- Synology
|
|
is a good example of this.
|
|
.PP
|
|
Here is an example of making an SFTP configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 7\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 8\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 9\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
10\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
11\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
12\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
13\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
14\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
15\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ sftp
|
|
SSH\ host\ to\ connect\ to
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Connect\ to\ example.com
|
|
\ \ \ \\\ "example.com"
|
|
host>\ example.com
|
|
SSH\ username,\ leave\ blank\ for\ current\ username,\ ncw
|
|
user>\ sftpuser
|
|
SSH\ port,\ leave\ blank\ to\ use\ default\ (22)
|
|
port>\
|
|
SSH\ password,\ leave\ blank\ to\ use\ ssh\-agent.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
n)\ No\ leave\ this\ optional\ password\ blank
|
|
y/g/n>\ n
|
|
Path\ to\ unencrypted\ PEM\-encoded\ private\ key\ file,\ leave\ blank\ to\ use\ ssh\-agent.
|
|
key_file>\
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
host\ =\ example.com
|
|
user\ =\ sftpuser
|
|
port\ =\
|
|
pass\ =\
|
|
key_file\ =\
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This remote is called \f[C]remote\f[] and can now be used like this:
|
|
.PP
|
|
See all directories in the home directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:path/to/directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:path/to/directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote directory, deleting
|
|
any excess files in the directory.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.SS SSH Authentication
|
|
.PP
|
|
The SFTP remote supports three authentication methods:
|
|
.IP \[bu] 2
|
|
Password
|
|
.IP \[bu] 2
|
|
Key file
|
|
.IP \[bu] 2
|
|
ssh\-agent
|
|
.PP
|
|
Key files should be unencrypted PEM\-encoded private key files.
|
|
For instance \f[C]/home/$USER/.ssh/id_rsa\f[].
|
|
.PP
|
|
If you don\[aq]t specify \f[C]pass\f[] or \f[C]key_file\f[] then rclone
|
|
will attempt to contact an ssh\-agent.
|
|
.PP
|
|
If you set the \f[C]\-\-sftp\-ask\-password\f[] option, rclone will
|
|
prompt for a password when needed and no password has been configured.
|
|
.SS ssh\-agent on macOS
|
|
.PP
|
|
Note that there seem to be various problems with using an ssh\-agent on
|
|
macOS due to recent changes in the OS.
|
|
The most effective work\-around seems to be to start an ssh\-agent in
|
|
each session, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
eval\ `ssh\-agent\ \-s`\ &&\ ssh\-add\ \-A
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And then at the end of the session
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
eval\ `ssh\-agent\ \-k`
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
These commands can be used in scripts of course.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to this remote.
|
|
.SS \-\-sftp\-ask\-password
|
|
.PP
|
|
Ask for the SFTP password if needed when no password has been
|
|
configured.
|
|
.SS \-\-ssh\-path\-override
|
|
.PP
|
|
Override path used by SSH connection.
|
|
Allows checksum calculation when SFTP and SSH paths are different.
|
|
This issue affects among others Synology NAS boxes.
|
|
.PP
|
|
Shared folders can be found in directories representing volumes
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:/directory\ \-\-ssh\-path\-override\ /volume2/directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Home directory can be found in a shared folder called \f[C]homes\f[]
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:/home/directory\ \-\-ssh\-path\-override\ /volume1/homes/USER/directory
|
|
\f[]
|
|
.fi
|
|
.SS Modified time
|
|
.PP
|
|
Modified times are stored on the server to 1 second precision.
|
|
.PP
|
|
Modified times are used in syncing and are fully supported.
|
|
.PP
|
|
Some SFTP servers disable setting/modifying the file modification time
|
|
after upload (for example, certain configurations of ProFTPd with
|
|
mod_sftp).
|
|
If you are using one of these servers, you can set the option
|
|
\f[C]set_modtime\ =\ false\f[] in your RClone backend configuration to
|
|
disable this behaviour.
|
|
.SS Limitations
|
|
.PP
|
|
SFTP supports checksums if the same login has shell access and
|
|
\f[C]md5sum\f[] or \f[C]sha1sum\f[] as well as \f[C]echo\f[] are in the
|
|
remote\[aq]s PATH.
|
|
This remote checksumming (file hashing) is recommended and enabled by
|
|
default.
|
|
Disabling the checksumming may be required if you are connecting to SFTP
|
|
servers which are not under your control, and to which the execution of
|
|
remote commands is prohibited.
|
|
Set the configuration option \f[C]disable_hashcheck\f[] to \f[C]true\f[]
|
|
to disable checksumming.
|
|
.PP
|
|
Note that some SFTP servers (eg Synology) the paths are different for
|
|
SSH and SFTP so the hashes can\[aq]t be calculated properly.
|
|
For them using \f[C]disable_hashcheck\f[] is a good idea.
|
|
.PP
|
|
The only ssh agent supported under Windows is Putty\[aq]s pageant.
|
|
.PP
|
|
The Go SSH library disables the use of the aes128\-cbc cipher by
|
|
default, due to security concerns.
|
|
This can be re\-enabled on a per\-connection basis by setting the
|
|
\f[C]use_insecure_cipher\f[] setting in the configuration file to
|
|
\f[C]true\f[].
|
|
Further details on the insecurity of this cipher can be found [in this
|
|
paper] (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
|
|
.PP
|
|
SFTP isn\[aq]t supported under plan9 until this
|
|
issue (https://github.com/pkg/sftp/issues/156) is fixed.
|
|
.PP
|
|
Note that since SFTP isn\[aq]t HTTP based the following flags don\[aq]t
|
|
work with it: \f[C]\-\-dump\-headers\f[], \f[C]\-\-dump\-bodies\f[],
|
|
\f[C]\-\-dump\-auth\f[]
|
|
.PP
|
|
Note that \f[C]\-\-timeout\f[] isn\[aq]t supported (but
|
|
\f[C]\-\-contimeout\f[] is).
|
|
.SS WebDAV
|
|
.PP
|
|
Paths are specified as \f[C]remote:path\f[]
|
|
.PP
|
|
Paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
To configure the WebDAV remote you will need to have a URL for it, and a
|
|
username and password.
|
|
If you know what kind of system you are connecting to then rclone can
|
|
enable extra features.
|
|
.PP
|
|
Here is an example of how to make a remote called \f[C]remote\f[].
|
|
First run:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
\ rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
q)\ Quit\ config
|
|
n/s/q>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Box
|
|
\ \ \ \\\ "box"
|
|
\ 5\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 6\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 7\ /\ FTP\ Connection
|
|
\ \ \ \\\ "ftp"
|
|
\ 8\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 9\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
10\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
11\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
12\ /\ Microsoft\ Azure\ Blob\ Storage
|
|
\ \ \ \\\ "azureblob"
|
|
13\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
14\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
15\ /\ Pcloud
|
|
\ \ \ \\\ "pcloud"
|
|
16\ /\ QingCloud\ Object\ Storage
|
|
\ \ \ \\\ "qingstor"
|
|
17\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
18\ /\ WebDAV
|
|
\ \ \ \\\ "webdav"
|
|
19\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
20\ /\ http\ Connection
|
|
\ \ \ \\\ "http"
|
|
Storage>\ webdav
|
|
URL\ of\ http\ host\ to\ connect\ to
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Connect\ to\ example.com
|
|
\ \ \ \\\ "https://example.com"
|
|
url>\ https://example.com/remote.php/webdav/
|
|
Name\ of\ the\ WebDAV\ site/service/software\ you\ are\ using
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Nextcloud
|
|
\ \ \ \\\ "nextcloud"
|
|
\ 2\ /\ Owncloud
|
|
\ \ \ \\\ "owncloud"
|
|
\ 3\ /\ Sharepoint
|
|
\ \ \ \\\ "sharepoint"
|
|
\ 4\ /\ Other\ site/service\ or\ software
|
|
\ \ \ \\\ "other"
|
|
vendor>\ 1
|
|
User\ name
|
|
user>\ user
|
|
Password.
|
|
y)\ Yes\ type\ in\ my\ own\ password
|
|
g)\ Generate\ random\ password
|
|
n)\ No\ leave\ this\ optional\ password\ blank
|
|
y/g/n>\ y
|
|
Enter\ the\ password:
|
|
password:
|
|
Confirm\ the\ password:
|
|
password:
|
|
Remote\ config
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
url\ =\ https://example.com/remote.php/webdav/
|
|
vendor\ =\ nextcloud
|
|
user\ =\ user
|
|
pass\ =\ ***\ ENCRYPTED\ ***
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
List directories in top level of your WebDAV
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List all the files in your WebDAV
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
To copy a local directory to an WebDAV directory called backup
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ copy\ /home/source\ remote:backup
|
|
\f[]
|
|
.fi
|
|
.SS Modified time and hashes
|
|
.PP
|
|
Plain WebDAV does not support modified times.
|
|
However when used with Owncloud or Nextcloud rclone will support
|
|
modified times.
|
|
.PP
|
|
Hashes are not supported.
|
|
.SS Owncloud
|
|
.PP
|
|
Click on the settings cog in the bottom right of the page and this will
|
|
show the WebDAV URL that rclone needs in the config step.
|
|
It will look something like
|
|
\f[C]https://example.com/remote.php/webdav/\f[].
|
|
.PP
|
|
Owncloud supports modified times using the \f[C]X\-OC\-Mtime\f[] header.
|
|
.SS Nextcloud
|
|
.PP
|
|
This is configured in an identical way to Owncloud.
|
|
Note that Nextcloud does not support streaming of files (\f[C]rcat\f[])
|
|
whereas Owncloud does.
|
|
This may be
|
|
fixed (https://github.com/nextcloud/nextcloud-snap/issues/365) in the
|
|
future.
|
|
.SS Put.io
|
|
.PP
|
|
put.io can be accessed in a read only way using webdav.
|
|
.PP
|
|
Configure the \f[C]url\f[] as \f[C]https://webdav.put.io\f[] and use
|
|
your normal account username and password for \f[C]user\f[] and
|
|
\f[C]pass\f[].
|
|
Set the \f[C]vendor\f[] to \f[C]other\f[].
|
|
.PP
|
|
Your config file should end up looking like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[putio]
|
|
type\ =\ webdav
|
|
url\ =\ https://webdav.put.io
|
|
vendor\ =\ other
|
|
user\ =\ YourUserName
|
|
pass\ =\ encryptedpassword
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you are using \f[C]put.io\f[] with \f[C]rclone\ mount\f[] then use
|
|
the \f[C]\-\-read\-only\f[] flag to signal to the OS that it can\[aq]t
|
|
write to the mount.
|
|
.PP
|
|
For more help see the put.io webdav
|
|
docs (http://help.put.io/apps-and-integrations/ftp-and-webdav).
|
|
.SS Sharepoint
|
|
.PP
|
|
Can be used with Sharepoint provided by OneDrive for Business or
|
|
Office365 Education Accounts.
|
|
This feature is only needed for a few of these Accounts, mostly
|
|
Office365 Education ones.
|
|
These accounts are sometimes not verified by the domain owner
|
|
github#1975 (https://github.com/ncw/rclone/issues/1975)
|
|
.PP
|
|
This means that these accounts can\[aq]t be added using the official API
|
|
(other Accounts should work with the "onedrive" option).
|
|
However, it is possible to access them using webdav.
|
|
.PP
|
|
To use a sharepoint remote with rclone, add it like this: First, you
|
|
need to get your remote\[aq]s URL:
|
|
.IP \[bu] 2
|
|
Go here (https://onedrive.live.com/about/en-us/signin/) to open your
|
|
OneDrive or to sign in
|
|
.IP \[bu] 2
|
|
Now take a look at your address bar, the URL should look like this:
|
|
\f[C]https://[YOUR\-DOMAIN]\-my.sharepoint.com/personal/[YOUR\-EMAIL]/_layouts/15/onedrive.aspx\f[]
|
|
.PP
|
|
You\[aq]ll only need this URL upto the email address.
|
|
After that, you\[aq]ll most likely want to add "/Documents".
|
|
That subdirectory contains the actual data stored on your OneDrive.
|
|
.PP
|
|
Add the remote to rclone like this: Configure the \f[C]url\f[] as
|
|
\f[C]https://[YOUR\-DOMAIN]\-my.sharepoint.com/personal/[YOUR\-EMAIL]/Documents\f[]
|
|
and use your normal account email and password for \f[C]user\f[] and
|
|
\f[C]pass\f[].
|
|
If you have 2FA enabled, you have to generate an app password.
|
|
Set the \f[C]vendor\f[] to \f[C]sharepoint\f[].
|
|
.PP
|
|
Your config file should look like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[sharepoint]
|
|
type\ =\ webdav
|
|
url\ =\ https://[YOUR\-DOMAIN]\-my.sharepoint.com/personal/[YOUR\-EMAIL]/Documents
|
|
vendor\ =\ other
|
|
user\ =\ YourEmailAddress
|
|
pass\ =\ encryptedpassword
|
|
\f[]
|
|
.fi
|
|
.SS Yandex Disk
|
|
.PP
|
|
Yandex Disk (https://disk.yandex.com) is a cloud storage solution
|
|
created by Yandex (https://yandex.com).
|
|
.PP
|
|
Yandex paths may be as deep as required, eg
|
|
\f[C]remote:directory/subdirectory\f[].
|
|
.PP
|
|
Here is an example of making a yandex configuration.
|
|
First run
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ config
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
This will guide you through an interactive setup process:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
No\ remotes\ found\ \-\ make\ a\ new\ one
|
|
n)\ New\ remote
|
|
s)\ Set\ configuration\ password
|
|
n/s>\ n
|
|
name>\ remote
|
|
Type\ of\ storage\ to\ configure.
|
|
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
|
\ 1\ /\ Amazon\ Drive
|
|
\ \ \ \\\ "amazon\ cloud\ drive"
|
|
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
|
\ \ \ \\\ "s3"
|
|
\ 3\ /\ Backblaze\ B2
|
|
\ \ \ \\\ "b2"
|
|
\ 4\ /\ Dropbox
|
|
\ \ \ \\\ "dropbox"
|
|
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
|
\ \ \ \\\ "crypt"
|
|
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
|
\ \ \ \\\ "google\ cloud\ storage"
|
|
\ 7\ /\ Google\ Drive
|
|
\ \ \ \\\ "drive"
|
|
\ 8\ /\ Hubic
|
|
\ \ \ \\\ "hubic"
|
|
\ 9\ /\ Local\ Disk
|
|
\ \ \ \\\ "local"
|
|
10\ /\ Microsoft\ OneDrive
|
|
\ \ \ \\\ "onedrive"
|
|
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
|
\ \ \ \\\ "swift"
|
|
12\ /\ SSH/SFTP\ Connection
|
|
\ \ \ \\\ "sftp"
|
|
13\ /\ Yandex\ Disk
|
|
\ \ \ \\\ "yandex"
|
|
Storage>\ 13
|
|
Yandex\ Client\ Id\ \-\ leave\ blank\ normally.
|
|
client_id>
|
|
Yandex\ Client\ Secret\ \-\ leave\ blank\ normally.
|
|
client_secret>
|
|
Remote\ config
|
|
Use\ auto\ config?
|
|
\ *\ Say\ Y\ if\ not\ sure
|
|
\ *\ Say\ N\ if\ you\ are\ working\ on\ a\ remote\ or\ headless\ machine
|
|
y)\ Yes
|
|
n)\ No
|
|
y/n>\ y
|
|
If\ your\ browser\ doesn\[aq]t\ open\ automatically\ go\ to\ the\ following\ link:\ http://127.0.0.1:53682/auth
|
|
Log\ in\ and\ authorize\ rclone\ for\ access
|
|
Waiting\ for\ code...
|
|
Got\ code
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
[remote]
|
|
client_id\ =
|
|
client_secret\ =
|
|
token\ =\ {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","expiry":"2016\-12\-29T12:27:11.362788025Z"}
|
|
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
|
y)\ Yes\ this\ is\ OK
|
|
e)\ Edit\ this\ remote
|
|
d)\ Delete\ this\ remote
|
|
y/e/d>\ y
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for how to
|
|
set it up on a machine with no Internet browser available.
|
|
.PP
|
|
Note that rclone runs a webserver on your local machine to collect the
|
|
token as returned from Yandex Disk.
|
|
This only runs from the moment it opens your browser to the moment you
|
|
get back the verification code.
|
|
This is on \f[C]http://127.0.0.1:53682/\f[] and this it may require you
|
|
to unblock it temporarily if you are running a host firewall.
|
|
.PP
|
|
Once configured you can then use \f[C]rclone\f[] like this,
|
|
.PP
|
|
See top level directories
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ lsd\ remote:
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Make a new directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ mkdir\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
List the contents of a directory
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ ls\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Sync \f[C]/home/local/directory\f[] to the remote path, deleting any
|
|
excess files in the path.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/local/directory\ remote:directory
|
|
\f[]
|
|
.fi
|
|
.SS \-\-fast\-list
|
|
.PP
|
|
This remote supports \f[C]\-\-fast\-list\f[] which allows you to use
|
|
fewer transactions in exchange for more memory.
|
|
See the rclone docs (/docs/#fast-list) for more details.
|
|
.SS Modified time
|
|
.PP
|
|
Modified times are supported and are stored accurate to 1 ns in custom
|
|
metadata called \f[C]rclone_modified\f[] in RFC3339 with nanoseconds
|
|
format.
|
|
.SS MD5 checksums
|
|
.PP
|
|
MD5 checksums are natively supported by Yandex Disk.
|
|
.SS Emptying Trash
|
|
.PP
|
|
If you wish to empty your trash you can use the
|
|
\f[C]rclone\ cleanup\ remote:\f[] command which will permanently delete
|
|
all your trashed files.
|
|
This command does not take any path arguments.
|
|
.SS Local Filesystem
|
|
.PP
|
|
Local paths are specified as normal filesystem paths, eg
|
|
\f[C]/path/to/wherever\f[], so
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ /home/source\ /tmp/destination
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Will sync \f[C]/home/source\f[] to \f[C]/tmp/destination\f[]
|
|
.PP
|
|
These can be configured into the config file for consistencies sake, but
|
|
it is probably easier not to.
|
|
.SS Modified time
|
|
.PP
|
|
Rclone reads and writes the modified time using an accuracy determined
|
|
by the OS.
|
|
Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.
|
|
.SS Filenames
|
|
.PP
|
|
Filenames are expected to be encoded in UTF\-8 on disk.
|
|
This is the normal case for Windows and OS X.
|
|
.PP
|
|
There is a bit more uncertainty in the Linux world, but new
|
|
distributions will have UTF\-8 encoded files names.
|
|
If you are using an old Linux filesystem with non UTF\-8 file names (eg
|
|
latin1) then you can use the \f[C]convmv\f[] tool to convert the
|
|
filesystem to UTF\-8.
|
|
This tool is available in most distributions\[aq] package managers.
|
|
.PP
|
|
If an invalid (non\-UTF8) filename is read, the invalid characters will
|
|
be replaced with the unicode replacement character, \[aq]�\[aq].
|
|
\f[C]rclone\f[] will emit a debug message in this case (use \f[C]\-v\f[]
|
|
to see), eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Local\ file\ system\ at\ .:\ Replacing\ invalid\ UTF\-8\ characters\ in\ "gro\\xdf"
|
|
\f[]
|
|
.fi
|
|
.SS Long paths on Windows
|
|
.PP
|
|
Rclone handles long paths automatically, by converting all paths to long
|
|
UNC
|
|
paths (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath)
|
|
which allows paths up to 32,767 characters.
|
|
.PP
|
|
This is why you will see that your paths, for instance
|
|
\f[C]c:\\files\f[] is converted to the UNC path
|
|
\f[C]\\\\?\\c:\\files\f[] in the output, and \f[C]\\\\server\\share\f[]
|
|
is converted to \f[C]\\\\?\\UNC\\server\\share\f[].
|
|
.PP
|
|
However, in rare cases this may cause problems with buggy file system
|
|
drivers like EncFS (https://github.com/ncw/rclone/issues/261).
|
|
To disable UNC conversion globally, add this to your
|
|
\f[C]\&.rclone.conf\f[] file:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[local]
|
|
nounc\ =\ true
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you want to selectively disable UNC, you can add it to a separate
|
|
entry like this:
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
[nounc]
|
|
type\ =\ local
|
|
nounc\ =\ true
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
And use rclone like this:
|
|
.PP
|
|
\f[C]rclone\ copy\ c:\\src\ nounc:z:\\dst\f[]
|
|
.PP
|
|
This will use UNC paths on \f[C]c:\\src\f[] but not on \f[C]z:\\dst\f[].
|
|
Of course this will cause problems if the absolute path length of a file
|
|
exceeds 258 characters on z, so only use this option if you have to.
|
|
.SS Specific options
|
|
.PP
|
|
Here are the command line options specific to local storage
|
|
.SS \-\-copy\-links, \-L
|
|
.PP
|
|
Normally rclone will ignore symlinks or junction points (which behave
|
|
like symlinks under Windows).
|
|
.PP
|
|
If you supply this flag then rclone will follow the symlink and copy the
|
|
pointed to file or directory.
|
|
.PP
|
|
This flag applies to all commands.
|
|
.PP
|
|
For example, supposing you have a directory structure like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ tree\ /tmp/a
|
|
/tmp/a
|
|
├──\ b\ \->\ ../b
|
|
├──\ expected\ \->\ ../expected
|
|
├──\ one
|
|
└──\ two
|
|
\ \ \ \ └──\ three
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Then you can see the difference with and without the flag like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ ls\ /tmp/a
|
|
\ \ \ \ \ \ \ \ 6\ one
|
|
\ \ \ \ \ \ \ \ 6\ two/three
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
and
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-L\ ls\ /tmp/a
|
|
\ \ \ \ \ 4174\ expected
|
|
\ \ \ \ \ \ \ \ 6\ one
|
|
\ \ \ \ \ \ \ \ 6\ two/three
|
|
\ \ \ \ \ \ \ \ 6\ b/two
|
|
\ \ \ \ \ \ \ \ 6\ b/one
|
|
\f[]
|
|
.fi
|
|
.SS \-\-local\-no\-check\-updated
|
|
.PP
|
|
Don\[aq]t check to see if the files change during upload.
|
|
.PP
|
|
Normally rclone checks the size and modification time of files as they
|
|
are being uploaded and aborts with a message which starts
|
|
\f[C]can\[aq]t\ copy\ \-\ source\ file\ is\ being\ updated\f[] if the
|
|
file changes during upload.
|
|
.PP
|
|
However on some file systems this modification time check may fail (eg
|
|
Glusterfs #2206 (https://github.com/ncw/rclone/issues/2206)) so this
|
|
check can be disabled with this flag.
|
|
.SS \-\-local\-no\-unicode\-normalization
|
|
.PP
|
|
This flag is deprecated now.
|
|
Rclone no longer normalizes unicode file names, but it compares them
|
|
with unicode normalization in the sync routine instead.
|
|
.SS \-\-one\-file\-system, \-x
|
|
.PP
|
|
This tells rclone to stay in the filesystem specified by the root and
|
|
not to recurse into different file systems.
|
|
.PP
|
|
For example if you have a directory hierarchy like this
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
root
|
|
├──\ disk1\ \ \ \ \ \-\ disk1\ mounted\ on\ the\ root
|
|
│\ \ \ └──\ file3\ \-\ stored\ on\ disk1
|
|
├──\ disk2\ \ \ \ \ \-\ disk2\ mounted\ on\ the\ root
|
|
│\ \ \ └──\ file4\ \-\ stored\ on\ disk12
|
|
├──\ file1\ \ \ \ \ \-\ stored\ on\ the\ root\ disk
|
|
└──\ file2\ \ \ \ \ \-\ stored\ on\ the\ root\ disk
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
Using \f[C]rclone\ \-\-one\-file\-system\ copy\ root\ remote:\f[] will
|
|
only copy \f[C]file1\f[] and \f[C]file2\f[].
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ \-\-one\-file\-system\ ls\ root
|
|
\ \ \ \ \ \ \ \ 0\ file1
|
|
\ \ \ \ \ \ \ \ 0\ file2
|
|
\f[]
|
|
.fi
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
$\ rclone\ \-q\ ls\ root
|
|
\ \ \ \ \ \ \ \ 0\ disk1/file3
|
|
\ \ \ \ \ \ \ \ 0\ disk2/file4
|
|
\ \ \ \ \ \ \ \ 0\ file1
|
|
\ \ \ \ \ \ \ \ 0\ file2
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
\f[B]NB\f[] Rclone (like most unix tools such as \f[C]du\f[],
|
|
\f[C]rsync\f[] and \f[C]tar\f[]) treats a bind mount to the same device
|
|
as being on the same filesystem.
|
|
.PP
|
|
\f[B]NB\f[] This flag is only available on Unix based systems.
|
|
On systems where it isn\[aq]t supported (eg Windows) it will not appear
|
|
as an valid flag.
|
|
.SS \-\-skip\-links
|
|
.PP
|
|
This flag disables warning messages on skipped symlinks or junction
|
|
points, as you explicitly acknowledge that they should be skipped.
|
|
.SS Changelog
|
|
.IP \[bu] 2
|
|
v1.42 \- 2018\-06\-16
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New backends
|
|
.IP \[bu] 2
|
|
OpenDrive (Oliver Heyme, Jakub Karlicek, ncw)
|
|
.IP \[bu] 2
|
|
New commands
|
|
.IP \[bu] 2
|
|
deletefile command (Filip Bartodziej)
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
copy, move: Copy single files directly, don\[aq]t use
|
|
\f[C]\-\-files\-from\f[] work\-around
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes them much more efficient
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-\-max\-transfer\f[] flag to quit transferring at a
|
|
limit
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
make exit code 8 for \f[C]\-\-max\-transfer\f[] exceeded
|
|
.RE
|
|
.IP \[bu] 2
|
|
copy: copy empty source directories to destination (Ishuah Kariuki)
|
|
.IP \[bu] 2
|
|
check: Add \f[C]\-\-one\-way\f[] flag (Kasper Byrdal Nielsen)
|
|
.IP \[bu] 2
|
|
Add siginfo handler for macOS for ctrl\-T stats (kubatasiemski)
|
|
.IP \[bu] 2
|
|
rc
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
add core/gc to run a garbage collection on demand
|
|
.IP \[bu] 2
|
|
enable go profiling by default on the \f[C]\-\-rc\f[] port
|
|
.IP \[bu] 2
|
|
return error from remote on failure
|
|
.RE
|
|
.IP \[bu] 2
|
|
lsf
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-absolute\f[] flag to add a leading / onto path names
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-csv\f[] flag for compliant CSV output
|
|
.IP \[bu] 2
|
|
Add \[aq]m\[aq] format specifier to show the MimeType
|
|
.IP \[bu] 2
|
|
Implement \[aq]i\[aq] format for showing object ID
|
|
.RE
|
|
.IP \[bu] 2
|
|
lsjson
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add MimeType to the output
|
|
.IP \[bu] 2
|
|
Add ID field to output to show Object ID
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-retries\-sleep\f[] flag (Benjamin Joseph Dag)
|
|
.IP \[bu] 2
|
|
Oauth tidy up web page and error handling (Henning Surmeier)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Password prompt output with \f[C]\-\-log\-file\f[] fixed for unix (Filip
|
|
Bartodziej)
|
|
.IP \[bu] 2
|
|
Calculate ModifyWindow each time on the fly to fix various problems
|
|
(Stefan Breunig)
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Only print "File.rename error" if there actually is an error (Stefan
|
|
Breunig)
|
|
.IP \[bu] 2
|
|
Delay rename if file has open writers instead of failing outright
|
|
(Stefan Breunig)
|
|
.IP \[bu] 2
|
|
Ensure atexit gets run on interrupt
|
|
.IP \[bu] 2
|
|
macOS enhancements
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make \f[C]\-\-noappledouble\f[] \f[C]\-\-noapplexattr\f[]
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-volname\f[] flag and remove special chars from it
|
|
.IP \[bu] 2
|
|
Make Get/List/Set/Remove xattr return ENOSYS for efficiency
|
|
.IP \[bu] 2
|
|
Make \f[C]\-\-daemon\f[] work for macOS without CGO
|
|
.RE
|
|
.IP \[bu] 2
|
|
VFS
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-vfs\-read\-chunk\-size\f[] and
|
|
\f[C]\-\-vfs\-read\-chunk\-size\-limit\f[] (Fabian Möller)
|
|
.IP \[bu] 2
|
|
Fix ChangeNotify for new or changed folders (Fabian Möller)
|
|
.RE
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Fix symlink/junction point directory handling under Windows
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[] you will need to add \f[C]\-L\f[] to your command line to
|
|
copy files with reparse points
|
|
.RE
|
|
.IP \[bu] 2
|
|
Cache
|
|
.IP \[bu] 2
|
|
Add non cached dirs on notifications (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Allow root to be expired from rc (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Clean remaining empty folders from temp upload path (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Cache lists using batch writes (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Use secure websockets for HTTPS Plex addresses (John Clayton)
|
|
.IP \[bu] 2
|
|
Reconnect plex websocket on failures (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Fix panic when running without plex configs (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Fix root folder caching (Remus Bunduc)
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Check the crypted hash of files when uploading for extra data security
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Make Dropbox for business folders accessible using an initial \f[C]/\f[]
|
|
in the path
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Low level retry all operations if necessary
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-acknowledge\-abuse\f[] to download flagged files
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-alternate\-export\f[] to fix large doc export
|
|
.IP \[bu] 2
|
|
Don\[aq]t attempt to choose Team Drives when using rclone config create
|
|
.IP \[bu] 2
|
|
Fix change list polling with team drives
|
|
.IP \[bu] 2
|
|
Fix ChangeNotify for folders (Fabian Möller)
|
|
.IP \[bu] 2
|
|
Fix about (and df on a mount) for team drives
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.IP \[bu] 2
|
|
Errorhandler for onedrive for business requests (Henning Surmeier)
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Adjust upload concurrency with \f[C]\-\-s3\-upload\-concurrency\f[]
|
|
(themylogin)
|
|
.IP \[bu] 2
|
|
Fix \f[C]\-\-s3\-chunk\-size\f[] which was always using the minimum
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-ssh\-path\-override\f[] flag (Piotr Oleszczyk)
|
|
.IP \[bu] 2
|
|
Fix slow downloads for long latency connections
|
|
.IP \[bu] 2
|
|
Webdav
|
|
.IP \[bu] 2
|
|
Add workarounds for biz.mail.ru
|
|
.IP \[bu] 2
|
|
Ignore Reason\-Phrase in status line to fix 4shared (Rodrigo)
|
|
.IP \[bu] 2
|
|
Better error message generation
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.41 \- 2018\-04\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New backends
|
|
.IP \[bu] 2
|
|
Mega support added
|
|
.IP \[bu] 2
|
|
Webdav now supports SharePoint cookie authentication (hensur)
|
|
.IP \[bu] 2
|
|
New commands
|
|
.IP \[bu] 2
|
|
link: create public link to files and folders (Stefan Breunig)
|
|
.IP \[bu] 2
|
|
about: gets quota info from a remote (a\-roussos, ncw)
|
|
.IP \[bu] 2
|
|
hashsum: a generic tool for any hash to produce md5sum like output
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
lsd: Add \-R flag and fix and update docs for all ls commands
|
|
.IP \[bu] 2
|
|
ncdu: added a "refresh" key \- CTRL\-L (Keith Goldfarb)
|
|
.IP \[bu] 2
|
|
serve restic: Add append\-only mode (Steve Kriss)
|
|
.IP \[bu] 2
|
|
serve restic: Disallow overwriting files in append\-only mode (Alexander
|
|
Neumann)
|
|
.IP \[bu] 2
|
|
serve restic: Print actual listener address (Matt Holt)
|
|
.IP \[bu] 2
|
|
size: Add \-\-json flag (Matthew Holt)
|
|
.IP \[bu] 2
|
|
sync: implement \-\-ignore\-errors (Mateusz Pabian)
|
|
.IP \[bu] 2
|
|
dedupe: Add dedupe largest functionality (Richard Yang)
|
|
.IP \[bu] 2
|
|
fs: Extend SizeSuffix to include TB and PB for rclone about
|
|
.IP \[bu] 2
|
|
fs: add \-\-dump goroutines and \-\-dump openfiles for debugging
|
|
.IP \[bu] 2
|
|
rc: implement core/memstats to print internal memory usage info
|
|
.IP \[bu] 2
|
|
rc: new call rc/pid (Michael P.
|
|
Dubner)
|
|
.IP \[bu] 2
|
|
Compile
|
|
.IP \[bu] 2
|
|
Drop support for go1.6
|
|
.IP \[bu] 2
|
|
Release
|
|
.IP \[bu] 2
|
|
Fix \f[C]make\ tarball\f[] (Chih\-Hsuan Yen)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
filter: fix \-\-min\-age and \-\-max\-age together check
|
|
.IP \[bu] 2
|
|
fs: limit MaxIdleConns and MaxIdleConnsPerHost in transport
|
|
.IP \[bu] 2
|
|
lsd,lsf: make sure all times we output are in local time
|
|
.IP \[bu] 2
|
|
rc: fix setting bwlimit to unlimited
|
|
.IP \[bu] 2
|
|
rc: take note of the \-\-rc\-addr flag too as per the docs
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Use About to return the correct disk total/used/free (eg in \f[C]df\f[])
|
|
.IP \[bu] 2
|
|
Set \f[C]\-\-attr\-timeout\ default\f[] to \f[C]1s\f[] \- fixes:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone using too much memory
|
|
.IP \[bu] 2
|
|
rclone not serving files to samba
|
|
.IP \[bu] 2
|
|
excessive time listing directories
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix \f[C]df\ \-i\f[] (upstream fix)
|
|
.IP \[bu] 2
|
|
VFS
|
|
.IP \[bu] 2
|
|
Filter files \f[C]\&.\f[] and \f[C]\&..\f[] from directory listing
|
|
.IP \[bu] 2
|
|
Only make the VFS cache if \-\-vfs\-cache\-mode > Off
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Add \-\-local\-no\-check\-updated to disable updated file checks
|
|
.IP \[bu] 2
|
|
Retry remove on Windows sharing violation error
|
|
.IP \[bu] 2
|
|
Cache
|
|
.IP \[bu] 2
|
|
Flush the memory cache after close
|
|
.IP \[bu] 2
|
|
Purge file data on notification
|
|
.IP \[bu] 2
|
|
Always forget parent dir for notifications
|
|
.IP \[bu] 2
|
|
Integrate with Plex websocket
|
|
.IP \[bu] 2
|
|
Add rc cache/stats (seuffert)
|
|
.IP \[bu] 2
|
|
Add info log on notification
|
|
.IP \[bu] 2
|
|
Box
|
|
.IP \[bu] 2
|
|
Fix failure reading large directories \- parse file/directory size as
|
|
float
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Fix crypt+obfuscate on dropbox
|
|
.IP \[bu] 2
|
|
Fix repeatedly uploading the same files
|
|
.IP \[bu] 2
|
|
FTP
|
|
.IP \[bu] 2
|
|
Work around strange response from box FTP server
|
|
.IP \[bu] 2
|
|
More workarounds for FTP servers to fix mkParentDir error
|
|
.IP \[bu] 2
|
|
Fix no error on listing non\-existent directory
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Add service_account_credentials (Matt Holt)
|
|
.IP \[bu] 2
|
|
Detect bucket presence by listing it \- minimises permissions needed
|
|
.IP \[bu] 2
|
|
Ignore zero length directory markers
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Add service_account_credentials (Matt Holt)
|
|
.IP \[bu] 2
|
|
Fix directory move leaving a hardlinked directory behind
|
|
.IP \[bu] 2
|
|
Return proper google errors when Opening files
|
|
.IP \[bu] 2
|
|
When initialized with a filepath, optional features used incorrect root
|
|
path (Stefan Breunig)
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.IP \[bu] 2
|
|
Fix sync for servers which don\[aq]t return Content\-Length in HEAD
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.IP \[bu] 2
|
|
Add QuickXorHash support for OneDrive for business
|
|
.IP \[bu] 2
|
|
Fix socket leak in multipart session upload
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Look in S3 named profile files for credentials
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-s3\-disable\-checksum\f[] to disable checksum uploading
|
|
(Chris Redekop)
|
|
.IP \[bu] 2
|
|
Hierarchical configuration support (Giri Badanahatti)
|
|
.IP \[bu] 2
|
|
Add in config for all the supported S3 providers
|
|
.IP \[bu] 2
|
|
Add One Zone Infrequent Access storage class (Craig Rachel)
|
|
.IP \[bu] 2
|
|
Add \-\-use\-server\-modtime support (Peter Baumgartner)
|
|
.IP \[bu] 2
|
|
Add \-\-s3\-chunk\-size option to control multipart uploads
|
|
.IP \[bu] 2
|
|
Ignore zero length directory markers
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Update docs to match code, fix typos and clarify disable_hashcheck
|
|
prompt (Michael G.
|
|
Noll)
|
|
.IP \[bu] 2
|
|
Update docs with Synology quirks
|
|
.IP \[bu] 2
|
|
Fail soft with a debug on hash failure
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Add \-\-use\-server\-modtime support (Peter Baumgartner)
|
|
.IP \[bu] 2
|
|
Webdav
|
|
.IP \[bu] 2
|
|
Support SharePoint cookie authentication (hensur)
|
|
.IP \[bu] 2
|
|
Strip leading and trailing / off root
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.40 \- 2018\-03\-19
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New backends
|
|
.IP \[bu] 2
|
|
Alias backend to create aliases for existing remote names (Fabian
|
|
Möller)
|
|
.IP \[bu] 2
|
|
New commands
|
|
.IP \[bu] 2
|
|
\f[C]lsf\f[]: list for parsing purposes (Jakub Tasiemski)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
by default this is a simple non recursive list of files and directories
|
|
.IP \[bu] 2
|
|
it can be configured to add more info in an easy to parse way
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]serve\ restic\f[]: for serving a remote as a Restic REST endpoint
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This enables restic to use any backends that rclone can access
|
|
.IP \[bu] 2
|
|
Thanks Alexander Neumann for help, patches and review
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]rc\f[]: enable the remote control of a running rclone
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
The running rclone must be started with \-\-rc and related flags.
|
|
.IP \[bu] 2
|
|
Currently there is support for bwlimit, and flushing for mount and
|
|
cache.
|
|
.RE
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
\f[C]\-\-max\-delete\f[] flag to add a delete threshold (Bjørn Erik
|
|
Pedersen)
|
|
.IP \[bu] 2
|
|
All backends now support RangeOption for ranged Open
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]cat\f[]: Use RangeOption for limited fetches to make more efficient
|
|
.IP \[bu] 2
|
|
\f[C]cryptcheck\f[]: make reading of nonce more efficient with
|
|
RangeOption
|
|
.RE
|
|
.IP \[bu] 2
|
|
serve http/webdav/restic
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
support SSL/TLS
|
|
.IP \[bu] 2
|
|
add \f[C]\-\-user\f[] \f[C]\-\-pass\f[] and \f[C]\-\-htpasswd\f[] for
|
|
authentication
|
|
.RE
|
|
.IP \[bu] 2
|
|
\f[C]copy\f[]/\f[C]move\f[]: detect file size change during copy/move
|
|
and abort transfer (ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]cryptdecode\f[]: added option to return encrypted file names.
|
|
(ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]lsjson\f[]: add \f[C]\-\-encrypted\f[] to show encrypted name
|
|
(Jakub Tasiemski)
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-stats\-file\-name\-length\f[] to specify the printed file
|
|
name length for stats (Will Gunn)
|
|
.IP \[bu] 2
|
|
Compile
|
|
.IP \[bu] 2
|
|
Code base was shuffled and factored
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
backends moved into a backend directory
|
|
.IP \[bu] 2
|
|
large packages split up
|
|
.IP \[bu] 2
|
|
See the CONTRIBUTING.md doc for info as to what lives where now
|
|
.RE
|
|
.IP \[bu] 2
|
|
Update to using go1.10 as the default go version
|
|
.IP \[bu] 2
|
|
Implement daily full integration
|
|
tests (https://pub.rclone.org/integration-tests/)
|
|
.IP \[bu] 2
|
|
Release
|
|
.IP \[bu] 2
|
|
Include a source tarball and sign it and the binaries
|
|
.IP \[bu] 2
|
|
Sign the git tags as part of the release process
|
|
.IP \[bu] 2
|
|
Add .deb and .rpm packages as part of the build
|
|
.IP \[bu] 2
|
|
Make a beta release for all branches on the main repo (but not pull
|
|
requests)
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
config: fixes errors on non existing config by loading config file only
|
|
on first access
|
|
.IP \[bu] 2
|
|
config: retry saving the config after failure (Mateusz)
|
|
.IP \[bu] 2
|
|
sync: when using \f[C]\-\-backup\-dir\f[] don\[aq]t delete files if we
|
|
can\[aq]t set their modtime
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this fixes odd behaviour with Dropbox and \f[C]\-\-backup\-dir\f[]
|
|
.RE
|
|
.IP \[bu] 2
|
|
fshttp: fix idle timeouts for HTTP connections
|
|
.IP \[bu] 2
|
|
\f[C]serve\ http\f[]: fix serving files with : in \- fixes
|
|
.IP \[bu] 2
|
|
Fix \f[C]\-\-exclude\-if\-present\f[] to ignore directories which it
|
|
doesn\[aq]t have permission for (Iakov Davydov)
|
|
.IP \[bu] 2
|
|
Make accounting work properly with crypt and b2
|
|
.IP \[bu] 2
|
|
remove \f[C]\-\-no\-traverse\f[] flag because it is obsolete
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-attr\-timeout\f[] flag to control attribute caching in
|
|
kernel
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this now defaults to 0 which is correct but less efficient
|
|
.IP \[bu] 2
|
|
see the mount docs (/commands/rclone_mount/#attribute-caching) for more
|
|
info
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-daemon\f[] flag to allow mount to run in the background
|
|
(ishuah)
|
|
.IP \[bu] 2
|
|
Fix: Return ENOSYS rather than EIO on attempted link
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This fixes FileZilla accessing an rclone mount served over sftp.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix setting modtime twice
|
|
.IP \[bu] 2
|
|
Mount tests now run on CI for Linux (mount & cmount)/Mac/Windows
|
|
.IP \[bu] 2
|
|
Many bugs fixed in the VFS layer \- see below
|
|
.IP \[bu] 2
|
|
VFS
|
|
.IP \[bu] 2
|
|
Many fixes for \f[C]\-\-vfs\-cache\-mode\f[] writes and above
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Update cached copy if we know it has changed (fixes stale data)
|
|
.IP \[bu] 2
|
|
Clean path names before using them in the cache
|
|
.IP \[bu] 2
|
|
Disable cache cleaner if \f[C]\-\-vfs\-cache\-poll\-interval=0\f[]
|
|
.IP \[bu] 2
|
|
Fill and clean the cache immediately on startup
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix Windows opening every file when it stats the file
|
|
.IP \[bu] 2
|
|
Fix applying modtime for an open Write Handle
|
|
.IP \[bu] 2
|
|
Fix creation of files when truncating
|
|
.IP \[bu] 2
|
|
Write 0 bytes when flushing unwritten handles to avoid race conditions
|
|
in FUSE
|
|
.IP \[bu] 2
|
|
Downgrade "poll\-interval is not supported" message to Info
|
|
.IP \[bu] 2
|
|
Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Downgrade "invalid cross\-device link: trying copy" to debug
|
|
.IP \[bu] 2
|
|
Make DirMove return fs.ErrorCantDirMove to allow fallback to Copy for
|
|
cross device
|
|
.IP \[bu] 2
|
|
Fix race conditions updating the hashes
|
|
.IP \[bu] 2
|
|
Cache
|
|
.IP \[bu] 2
|
|
Add support for polling \- cache will update when remote changes on
|
|
supported backends
|
|
.IP \[bu] 2
|
|
Reduce log level for Plex api
|
|
.IP \[bu] 2
|
|
Fix dir cache issue
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-\-cache\-db\-wait\-time\f[] flag
|
|
.IP \[bu] 2
|
|
Improve efficiency with RangeOption and RangeSeek
|
|
.IP \[bu] 2
|
|
Fix dirmove with temp fs enabled
|
|
.IP \[bu] 2
|
|
Notify vfs when using temp fs
|
|
.IP \[bu] 2
|
|
Offline uploading
|
|
.IP \[bu] 2
|
|
Remote control support for path flushing
|
|
.IP \[bu] 2
|
|
Amazon cloud drive
|
|
.IP \[bu] 2
|
|
Rclone no longer has any working keys \- disable integration tests
|
|
.IP \[bu] 2
|
|
Implement DirChangeNotify to notify cache/vfs/mount of changes
|
|
.IP \[bu] 2
|
|
Azureblob
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presense if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Improve accounting for chunked uploads
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presense if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Box
|
|
.IP \[bu] 2
|
|
Improve accounting for chunked uploads
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Fix custom oauth client parameters
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presense if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Migrate to api v3 (Fabian Möller)
|
|
.IP \[bu] 2
|
|
Add scope configuration and root folder selection
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-impersonate\f[] for service accounts
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
thanks to everyone who tested, explored and contributed docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-use\-created\-date\f[] to use created date as
|
|
modified date (nbuchanan)
|
|
.IP \[bu] 2
|
|
Request the export formats only when required
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This makes rclone quicker when there are no google docs
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix finding paths with latin1 chars (a workaround for a drive bug)
|
|
.IP \[bu] 2
|
|
Fix copying of a single Google doc file
|
|
.IP \[bu] 2
|
|
Fix \f[C]\-\-drive\-auth\-owner\-only\f[] to look in all directories
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.IP \[bu] 2
|
|
Fix handling of directories with & in
|
|
.IP \[bu] 2
|
|
Onedrive
|
|
.IP \[bu] 2
|
|
Removed upload cutoff and always do session uploads
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this stops the creation of multiple versions on business onedrive
|
|
.RE
|
|
.IP \[bu] 2
|
|
Overwrite object size value with real size when reading file.
|
|
(Victor)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this fixes oddities when onedrive misreports the size of images
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.IP \[bu] 2
|
|
Remove unused chunked upload flag and code
|
|
.IP \[bu] 2
|
|
Qingstor
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presense if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Support hashes for multipart files (Chris Redekop)
|
|
.IP \[bu] 2
|
|
Initial support for IBM COS (S3) (Giri Badanahatti)
|
|
.IP \[bu] 2
|
|
Update docs to discourage use of v2 auth with CEPH and others
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presense if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix server side copy and set modtime on files with + in
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Add option to disable remote hash check command execution (Jon Fautley)
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-sftp\-ask\-password\f[] flag to prompt for password when
|
|
needed (Leo R.
|
|
Lundgren)
|
|
.IP \[bu] 2
|
|
Add \f[C]set_modtime\f[] configuration option
|
|
.IP \[bu] 2
|
|
Fix following of symlinks
|
|
.IP \[bu] 2
|
|
Fix reading config file outside of Fs setup
|
|
.IP \[bu] 2
|
|
Fix reading $USER in username fallback not $HOME
|
|
.IP \[bu] 2
|
|
Fix running under crontab \- Use correct OS way of reading username
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix refresh of authentication token
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
in v1.39 a bug was introduced which ignored new tokens \- this fixes it
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix extra HEAD transaction when uploading a new file
|
|
.IP \[bu] 2
|
|
Don\[aq]t check for bucket/container presense if listing was OK
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this makes rclone do one less request per invocation
|
|
.RE
|
|
.IP \[bu] 2
|
|
Webdav
|
|
.IP \[bu] 2
|
|
Add new time formats to support mydrive.ch and others
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.39 \- 2017\-12\-23
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New backends
|
|
.IP \[bu] 2
|
|
WebDAV
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
tested with nextcloud, owncloud, put.io and others!
|
|
.RE
|
|
.IP \[bu] 2
|
|
Pcloud
|
|
.IP \[bu] 2
|
|
cache \- wraps a cache around other backends (Remus Bunduc)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
useful in combination with mount
|
|
.IP \[bu] 2
|
|
NB this feature is in beta so use with care
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.IP \[bu] 2
|
|
serve command with subcommands:
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
serve webdav: this implements a webdav server for any rclone remote.
|
|
.IP \[bu] 2
|
|
serve http: command to serve a remote over HTTP
|
|
.RE
|
|
.IP \[bu] 2
|
|
config: add sub commands for full config file management
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
create/delete/dump/edit/file/password/providers/show/update
|
|
.RE
|
|
.IP \[bu] 2
|
|
touch: to create or update the timestamp of a file (Jakub Tasiemski)
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
curl install for rclone (Filip Bartodziej)
|
|
.IP \[bu] 2
|
|
\-\-stats now shows percentage, size, rate and ETA in condensed form
|
|
(Ishuah Kariuki)
|
|
.IP \[bu] 2
|
|
\-\-exclude\-if\-present to exclude a directory if a file is present
|
|
(Iakov Davydov)
|
|
.IP \[bu] 2
|
|
rmdirs: add \-\-leave\-root flag (lewpam)
|
|
.IP \[bu] 2
|
|
move: add \-\-delete\-empty\-src\-dirs flag to remove dirs after move
|
|
(Ishuah Kariuki)
|
|
.IP \[bu] 2
|
|
Add \-\-dump flag, introduce \-\-dump requests, responses and remove
|
|
\-\-dump\-auth, \-\-dump\-filters
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Obscure X\-Auth\-Token: from headers when dumping too
|
|
.RE
|
|
.IP \[bu] 2
|
|
Document and implement exit codes for different failure modes (Ishuah
|
|
Kariuki)
|
|
.IP \[bu] 2
|
|
Compile
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Retry lots more different types of errors to make multipart transfers
|
|
more reliable
|
|
.IP \[bu] 2
|
|
Save the config before asking for a token, fixes disappearing oauth
|
|
config
|
|
.IP \[bu] 2
|
|
Warn the user if \-\-include and \-\-exclude are used together (Ernest
|
|
Borowski)
|
|
.IP \[bu] 2
|
|
Fix duplicate files (eg on Google drive) causing spurious copies
|
|
.IP \[bu] 2
|
|
Allow trailing and leading whitespace for passwords (Jason Rose)
|
|
.IP \[bu] 2
|
|
ncdu: fix crashes on empty directories
|
|
.IP \[bu] 2
|
|
rcat: fix goroutine leak
|
|
.IP \[bu] 2
|
|
moveto/copyto: Fix to allow copying to the same name
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
\-\-vfs\-cache mode to make writes into mounts more reliable.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this requires caching files on the disk (see \-\-cache\-dir)
|
|
.IP \[bu] 2
|
|
As this is a new feature, use with care
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use sdnotify to signal systemd the mount is ready (Fabian Möller)
|
|
.IP \[bu] 2
|
|
Check if directory is not empty before mounting (Ernest Borowski)
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Add error message for cross file system moves
|
|
.IP \[bu] 2
|
|
Fix equality check for times
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Rework multipart upload
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
buffer the chunks when uploading large files so they can be retried
|
|
.IP \[bu] 2
|
|
change default chunk size to 48MB now we are buffering them in memory
|
|
.IP \[bu] 2
|
|
retry every error after the first chunk is done successfully
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix error when renaming directories
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix crash on bad authentication
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Add service account support (Tim Cooijmans)
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Make it work properly with Digital Ocean Spaces (Andrew
|
|
Starr\-Bochicchio)
|
|
.IP \[bu] 2
|
|
Fix crash if a bad listing is received
|
|
.IP \[bu] 2
|
|
Add support for ECS task IAM roles (David Minor)
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Fix multipart upload retries
|
|
.IP \[bu] 2
|
|
Fix \-\-hard\-delete to make it work 100% of the time
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Allow authentication with storage URL and auth key (Giovanni Pizzi)
|
|
.IP \[bu] 2
|
|
Add new fields for swift configuration to support IBM Bluemix Swift
|
|
(Pierre Carlson)
|
|
.IP \[bu] 2
|
|
Add OS_TENANT_ID and OS_USER_ID to config
|
|
.IP \[bu] 2
|
|
Allow configs with user id instead of user name
|
|
.IP \[bu] 2
|
|
Check if swift segments container exists before creating (John Leach)
|
|
.IP \[bu] 2
|
|
Fix memory leak in swift transfers (upstream fix)
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Add option to enable the use of aes128\-cbc cipher (Jon Fautley)
|
|
.IP \[bu] 2
|
|
Amazon cloud drive
|
|
.IP \[bu] 2
|
|
Fix download of large files failing with "Only one auth mechanism
|
|
allowed"
|
|
.IP \[bu] 2
|
|
crypt
|
|
.IP \[bu] 2
|
|
Option to encrypt directory names or leave them intact
|
|
.IP \[bu] 2
|
|
Implement DirChangeNotify (Fabian Möller)
|
|
.IP \[bu] 2
|
|
onedrive
|
|
.IP \[bu] 2
|
|
Add option to choose resourceURL during setup of OneDrive Business
|
|
account if more than one is available for user
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.38 \- 2017\-09\-30
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New backends
|
|
.IP \[bu] 2
|
|
Azure Blob Storage (thanks Andrei Dragomir)
|
|
.IP \[bu] 2
|
|
Box
|
|
.IP \[bu] 2
|
|
Onedrive for Business (thanks Oliver Heyme)
|
|
.IP \[bu] 2
|
|
QingStor from QingCloud (thanks wuyu)
|
|
.IP \[bu] 2
|
|
New commands
|
|
.IP \[bu] 2
|
|
\f[C]rcat\f[] \- read from standard input and stream upload
|
|
.IP \[bu] 2
|
|
\f[C]tree\f[] \- shows a nicely formatted recursive listing
|
|
.IP \[bu] 2
|
|
\f[C]cryptdecode\f[] \- decode crypted file names (thanks ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]config\ show\f[] \- print the config file
|
|
.IP \[bu] 2
|
|
\f[C]config\ file\f[] \- print the config file location
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Empty directories are deleted on \f[C]sync\f[]
|
|
.IP \[bu] 2
|
|
\f[C]dedupe\f[] \- implement merging of duplicate directories
|
|
.IP \[bu] 2
|
|
\f[C]check\f[] and \f[C]cryptcheck\f[] made more consistent and use less
|
|
memory
|
|
.IP \[bu] 2
|
|
\f[C]cleanup\f[] for remaining remotes (thanks ishuah)
|
|
.IP \[bu] 2
|
|
\f[C]\-\-immutable\f[] for ensuring that files don\[aq]t change (thanks
|
|
Jacob McNamee)
|
|
.IP \[bu] 2
|
|
\f[C]\-\-user\-agent\f[] option (thanks Alex McGrath Kraak)
|
|
.IP \[bu] 2
|
|
\f[C]\-\-disable\f[] flag to disable optional features
|
|
.IP \[bu] 2
|
|
\f[C]\-\-bind\f[] flag for choosing the local addr on outgoing
|
|
connections
|
|
.IP \[bu] 2
|
|
Support for zsh auto\-completion (thanks bpicode)
|
|
.IP \[bu] 2
|
|
Stop normalizing file names but do a normalized compare in \f[C]sync\f[]
|
|
.IP \[bu] 2
|
|
Compile
|
|
.IP \[bu] 2
|
|
Update to using go1.9 as the default go version
|
|
.IP \[bu] 2
|
|
Remove snapd build due to maintenance problems
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Improve retriable error detection which makes multipart uploads better
|
|
.IP \[bu] 2
|
|
Make \f[C]check\f[] obey \f[C]\-\-ignore\-size\f[]
|
|
.IP \[bu] 2
|
|
Fix bwlimit toggle in conjunction with schedules (thanks cbruegg)
|
|
.IP \[bu] 2
|
|
\f[C]config\f[] ensures newly written config is on the same mount
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Revert to copy when moving file across file system boundaries
|
|
.IP \[bu] 2
|
|
\f[C]\-\-skip\-links\f[] to suppress symlink warnings (thanks Zhiming
|
|
Wang)
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Re\-use \f[C]rcat\f[] internals to support uploads from all remotes
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Fix "entry doesn\[aq]t belong in directory" error
|
|
.IP \[bu] 2
|
|
Stop using deprecated API methods
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix server side copy to empty container with \f[C]\-\-fast\-list\f[]
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Change the default for \f[C]\-\-drive\-use\-trash\f[] to \f[C]true\f[]
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Set session token when using STS (thanks Girish Ramakrishnan)
|
|
.IP \[bu] 2
|
|
Glacier docs and error messages (thanks Jan Varho)
|
|
.IP \[bu] 2
|
|
Read 1000 (not 1024) items in dir listings to fix Wasabi
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Fix SHA1 mismatch when downloading files with no SHA1
|
|
.IP \[bu] 2
|
|
Calculate missing hashes on the fly instead of spooling
|
|
.IP \[bu] 2
|
|
\f[C]\-\-b2\-hard\-delete\f[] to permanently delete (not hide) files
|
|
(thanks John Papandriopoulos)
|
|
.IP \[bu] 2
|
|
Hubic
|
|
.IP \[bu] 2
|
|
Fix creating containers \- no longer have to use the \f[C]default\f[]
|
|
container
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Optionally configure from a standard set of OpenStack environment vars
|
|
.IP \[bu] 2
|
|
Add \f[C]endpoint_type\f[] config
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Fix bucket creation to work with limited permission users
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Implement connection pooling for multiple ssh connections
|
|
.IP \[bu] 2
|
|
Limit new connections per second
|
|
.IP \[bu] 2
|
|
Add support for MD5 and SHA1 hashes where available (thanks Christian
|
|
Brüggemann)
|
|
.IP \[bu] 2
|
|
HTTP
|
|
.IP \[bu] 2
|
|
Fix URL encoding issues
|
|
.IP \[bu] 2
|
|
Fix directories with \f[C]:\f[] in
|
|
.IP \[bu] 2
|
|
Fix panic with URL encoded content
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.37 \- 2017\-07\-22
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New backends
|
|
.IP \[bu] 2
|
|
FTP \- thanks to Antonio Messina
|
|
.IP \[bu] 2
|
|
HTTP \- thanks to Vasiliy Tolstov
|
|
.IP \[bu] 2
|
|
New commands
|
|
.IP \[bu] 2
|
|
rclone ncdu \- for exploring a remote with a text based user interface.
|
|
.IP \[bu] 2
|
|
rclone lsjson \- for listing with a machine readable output
|
|
.IP \[bu] 2
|
|
rclone dbhashsum \- to show Dropbox style hashes of files (local or
|
|
Dropbox)
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Implement \-\-fast\-list flag
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This allows remotes to list recursively if they can
|
|
.IP \[bu] 2
|
|
This uses less transactions (important if you pay for them)
|
|
.IP \[bu] 2
|
|
This may or may not be quicker
|
|
.IP \[bu] 2
|
|
This will use more memory as it has to hold the listing in memory
|
|
.IP \[bu] 2
|
|
\-\-old\-sync\-method deprecated \- the remaining uses are covered by
|
|
\-\-fast\-list
|
|
.IP \[bu] 2
|
|
This involved a major re\-write of all the listing code
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \-\-tpslimit and \-\-tpslimit\-burst to limit transactions per
|
|
second
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this is useful in conjuction with \f[C]rclone\ mount\f[] to limit
|
|
external apps
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \-\-stats\-log\-level so can see \-\-stats without \-v
|
|
.IP \[bu] 2
|
|
Print password prompts to stderr \- Hraban Luyat
|
|
.IP \[bu] 2
|
|
Warn about duplicate files when syncing
|
|
.IP \[bu] 2
|
|
Oauth improvements
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
allow auth_url and token_url to be set in the config file
|
|
.IP \[bu] 2
|
|
Print redirection URI if using own credentials.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Don\[aq]t Mkdir at the start of sync to save transactions
|
|
.IP \[bu] 2
|
|
Compile
|
|
.IP \[bu] 2
|
|
Update build to go1.8.3
|
|
.IP \[bu] 2
|
|
Require go1.6 for building rclone
|
|
.IP \[bu] 2
|
|
Compile 386 builds with "GO386=387" for maximum compatibility
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix menu selection when no remotes
|
|
.IP \[bu] 2
|
|
Config saving reworked to not kill the file if disk gets full
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete remote if name does not change while renaming
|
|
.IP \[bu] 2
|
|
moveto, copyto: report transfers and checks as per move and copy
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Add \-\-local\-no\-unicode\-normalization flag \- Bob Potter
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Now supported on Windows using cgofuse and WinFsp \- thanks to Bill
|
|
Zissimopoulos for much help
|
|
.IP \[bu] 2
|
|
Compare checksums on upload/download via FUSE
|
|
.IP \[bu] 2
|
|
Unmount when program ends with SIGINT (Ctrl+C) or SIGTERM \- Jérôme
|
|
Vizcaino
|
|
.IP \[bu] 2
|
|
On read only open of file, make open pending until first read
|
|
.IP \[bu] 2
|
|
Make \-\-read\-only reject modify operations
|
|
.IP \[bu] 2
|
|
Implement ModTime via FUSE for remotes that support it
|
|
.IP \[bu] 2
|
|
Allow modTime to be changed even before all writers are closed
|
|
.IP \[bu] 2
|
|
Fix panic on renames
|
|
.IP \[bu] 2
|
|
Fix hang on errored upload
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Report the name:root as specified by the user
|
|
.IP \[bu] 2
|
|
Add an "obfuscate" option for filename encryption \- Stephen Harris
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Fix initialization order for token renewer
|
|
.IP \[bu] 2
|
|
Remove revoked credentials, allow oauth proxy config and update docs
|
|
.IP \[bu] 2
|
|
B2
|
|
.IP \[bu] 2
|
|
Reduce minimum chunk size to 5MB
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Add team drive support
|
|
.IP \[bu] 2
|
|
Reduce bandwidth by adding fields for partial responses \- Martin
|
|
Kristensen
|
|
.IP \[bu] 2
|
|
Implement \-\-drive\-shared\-with\-me flag to view shared with me files
|
|
\- Danny Tsai
|
|
.IP \[bu] 2
|
|
Add \-\-drive\-trashed\-only to read only the files in the trash
|
|
.IP \[bu] 2
|
|
Remove obsolete \-\-drive\-full\-list
|
|
.IP \[bu] 2
|
|
Add missing seek to start on retries of chunked uploads
|
|
.IP \[bu] 2
|
|
Fix stats accounting for upload
|
|
.IP \[bu] 2
|
|
Convert / in names to a unicode equivalent (／)
|
|
.IP \[bu] 2
|
|
Poll for Google Drive changes when mounted
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.IP \[bu] 2
|
|
Fix the uploading of files with spaces
|
|
.IP \[bu] 2
|
|
Fix initialization order for token renewer
|
|
.IP \[bu] 2
|
|
Display speeds accurately when uploading \- Yoni Jah
|
|
.IP \[bu] 2
|
|
Swap to using http://localhost:53682/ as redirect URL \- Michael Ledin
|
|
.IP \[bu] 2
|
|
Retry on token expired error, reset upload body on retry \- Yoni Jah
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Add ability to specify location and storage class via config and command
|
|
line \- thanks gdm85
|
|
.IP \[bu] 2
|
|
Create container if necessary on server side copy
|
|
.IP \[bu] 2
|
|
Increase directory listing chunk to 1000 to increase performance
|
|
.IP \[bu] 2
|
|
Obtain a refresh token for GCS \- Steven Lu
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.IP \[bu] 2
|
|
Fix the name reported in log messages (was empty)
|
|
.IP \[bu] 2
|
|
Correct error return for listing empty directory
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Rewritten to use the v2 API
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now supports ModTime
|
|
.IP \[bu] 2
|
|
Can only set by uploading the file again
|
|
.IP \[bu] 2
|
|
If you uploaded with an old rclone, rclone may upload everything again
|
|
.IP \[bu] 2
|
|
Use \f[C]\-\-size\-only\f[] or \f[C]\-\-checksum\f[] to avoid this
|
|
.IP \[bu] 2
|
|
Now supports the Dropbox content hashing scheme
|
|
.IP \[bu] 2
|
|
Now supports low level retries
|
|
.RE
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Work around eventual consistency in bucket creation
|
|
.IP \[bu] 2
|
|
Create container if necessary on server side copy
|
|
.IP \[bu] 2
|
|
Add us\-east\-2 (Ohio) and eu\-west\-2 (London) S3 regions \- Zahiar
|
|
Ahmed
|
|
.IP \[bu] 2
|
|
Swift, Hubic
|
|
.IP \[bu] 2
|
|
Fix zero length directory markers showing in the subdirectory listing
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this caused lots of duplicate transfers
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix paged directory listings
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this caused duplicate directory errors
|
|
.RE
|
|
.IP \[bu] 2
|
|
Create container if necessary on server side copy
|
|
.IP \[bu] 2
|
|
Increase directory listing chunk to 1000 to increase performance
|
|
.IP \[bu] 2
|
|
Make sensible error if the user forgets the container
|
|
.IP \[bu] 2
|
|
SFTP
|
|
.IP \[bu] 2
|
|
Add support for using ssh key files
|
|
.IP \[bu] 2
|
|
Fix under Windows
|
|
.IP \[bu] 2
|
|
Fix ssh agent on Windows
|
|
.IP \[bu] 2
|
|
Adapt to latest version of library \- Igor Kharin
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.36 \- 2017\-03\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
SFTP remote (Jack Schmidt)
|
|
.IP \[bu] 2
|
|
Re\-implement sync routine to work a directory at a time reducing memory
|
|
usage
|
|
.IP \[bu] 2
|
|
Logging revamped to be more inline with rsync \- now much quieter
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\-v only shows transfers
|
|
.IP \[bu] 2
|
|
\-vv is for full debug
|
|
.IP \[bu] 2
|
|
\-\-syslog to log to syslog on capable platforms
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement \-\-backup\-dir and \-\-suffix
|
|
.IP \[bu] 2
|
|
Implement \-\-track\-renames (initial implementation by Bjørn Erik
|
|
Pedersen)
|
|
.IP \[bu] 2
|
|
Add time\-based bandwidth limits (Lukas Loesche)
|
|
.IP \[bu] 2
|
|
rclone cryptcheck: checks integrity of crypt remotes
|
|
.IP \[bu] 2
|
|
Allow all config file variables and options to be set from environment
|
|
variables
|
|
.IP \[bu] 2
|
|
Add \-\-buffer\-size parameter to control buffer size for copy
|
|
.IP \[bu] 2
|
|
Make \-\-delete\-after the default
|
|
.IP \[bu] 2
|
|
Add \-\-ignore\-checksum flag (fixed by Hisham Zarka)
|
|
.IP \[bu] 2
|
|
rclone check: Add \-\-download flag to check all the data, not just
|
|
hashes
|
|
.IP \[bu] 2
|
|
rclone cat: add \-\-head, \-\-tail, \-\-offset, \-\-count and
|
|
\-\-discard
|
|
.IP \[bu] 2
|
|
rclone config: when choosing from a list, allow the value to be entered
|
|
too
|
|
.IP \[bu] 2
|
|
rclone config: allow rename and copy of remotes
|
|
.IP \[bu] 2
|
|
rclone obscure: for generating encrypted passwords for rclone\[aq]s
|
|
config (T.C.
|
|
Ferguson)
|
|
.IP \[bu] 2
|
|
Comply with XDG Base Directory specification (Dario Giovannetti)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
this moves the default location of the config file in a backwards
|
|
compatible way
|
|
.RE
|
|
.IP \[bu] 2
|
|
Release changes
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Ubuntu snap support (Dedsec1)
|
|
.IP \[bu] 2
|
|
Compile with go 1.8
|
|
.IP \[bu] 2
|
|
MIPS/Linux big and little endian support
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix copyto copying things to the wrong place if the destination dir
|
|
didn\[aq]t exist
|
|
.IP \[bu] 2
|
|
Fix parsing of remotes in moveto and copyto
|
|
.IP \[bu] 2
|
|
Fix \-\-delete\-before deleting files on copy
|
|
.IP \[bu] 2
|
|
Fix \-\-files\-from with an empty file copying everything
|
|
.IP \[bu] 2
|
|
Fix sync: don\[aq]t update mod times if \-\-dry\-run set
|
|
.IP \[bu] 2
|
|
Fix MimeType propagation
|
|
.IP \[bu] 2
|
|
Fix filters to add ** rules to directory rules
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Implement \-L, \-\-copy\-links flag to allow rclone to follow symlinks
|
|
.IP \[bu] 2
|
|
Open files in write only mode so rclone can write to an rclone mount
|
|
.IP \[bu] 2
|
|
Fix unnormalised unicode causing problems reading directories
|
|
.IP \[bu] 2
|
|
Fix interaction between \-x flag and \-\-max\-depth
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Implement proper directory handling (mkdir, rmdir, renaming)
|
|
.IP \[bu] 2
|
|
Make include and exclude filters apply to mount
|
|
.IP \[bu] 2
|
|
Implement read and write async buffers \- control with \-\-buffer\-size
|
|
.IP \[bu] 2
|
|
Fix fsync on for directories
|
|
.IP \[bu] 2
|
|
Fix retry on network failure when reading off crypt
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Add \-\-crypt\-show\-mapping to show encrypted file mapping
|
|
.IP \[bu] 2
|
|
Fix crypt writer getting stuck in a loop
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[B]IMPORTANT\f[] this bug had the potential to cause data corruption
|
|
when
|
|
.IP \[bu] 2
|
|
reading data from a network based remote and
|
|
.IP \[bu] 2
|
|
writing to a crypt on Google Drive
|
|
.IP \[bu] 2
|
|
Use the cryptcheck command to validate your data if you are concerned
|
|
.IP \[bu] 2
|
|
If syncing two crypt remotes, sync the unencrypted remote
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Fix panics on Move (rename)
|
|
.IP \[bu] 2
|
|
Fix panic on token expiry
|
|
.IP \[bu] 2
|
|
B2
|
|
.IP \[bu] 2
|
|
Fix inconsistent listings and rclone check
|
|
.IP \[bu] 2
|
|
Fix uploading empty files with go1.8
|
|
.IP \[bu] 2
|
|
Constrain memory usage when doing multipart uploads
|
|
.IP \[bu] 2
|
|
Fix upload url not being refreshed properly
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Fix Rmdir on directories with trashed files
|
|
.IP \[bu] 2
|
|
Fix "Ignoring unknown object" when downloading
|
|
.IP \[bu] 2
|
|
Add \-\-drive\-list\-chunk
|
|
.IP \[bu] 2
|
|
Add \-\-drive\-skip\-gdocs (Károly Oláh)
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.IP \[bu] 2
|
|
Implement Move
|
|
.IP \[bu] 2
|
|
Fix Copy
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix overwrite detection in Copy
|
|
.IP \[bu] 2
|
|
Fix waitForJob to parse errors correctly
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use token renewer to stop auth errors on long uploads
|
|
.IP \[bu] 2
|
|
Fix uploading empty files with go1.8
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Fix depth 1 directory listings
|
|
.IP \[bu] 2
|
|
Yandex
|
|
.IP \[bu] 2
|
|
Fix single level directory listing
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Normalise the case for single level directory listings
|
|
.IP \[bu] 2
|
|
Fix depth 1 listing
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Added ca\-central\-1 region (Jon Yergatian)
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.35 \- 2017\-01\-02
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
moveto and copyto commands for choosing a destination name on copy/move
|
|
.IP \[bu] 2
|
|
rmdirs command to recursively delete empty directories
|
|
.IP \[bu] 2
|
|
Allow repeated \-\-include/\-\-exclude/\-\-filter options
|
|
.IP \[bu] 2
|
|
Only show transfer stats on commands which transfer stuff
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
show stats on any command using the \f[C]\-\-stats\f[] flag
|
|
.RE
|
|
.IP \[bu] 2
|
|
Allow overlapping directories in move when server side dir move is
|
|
supported
|
|
.IP \[bu] 2
|
|
Add \-\-stats\-unit option \- thanks Scott McGillivray
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix the config file being overwritten when two rclones are running
|
|
.IP \[bu] 2
|
|
Make rclone lsd obey the filters properly
|
|
.IP \[bu] 2
|
|
Fix compilation on mips
|
|
.IP \[bu] 2
|
|
Fix not transferring files that don\[aq]t differ in size
|
|
.IP \[bu] 2
|
|
Fix panic on nil retry/fatal error
|
|
.IP \[bu] 2
|
|
Mount
|
|
.IP \[bu] 2
|
|
Retry reads on error \- should help with reliability a lot
|
|
.IP \[bu] 2
|
|
Report the modification times for directories from the remote
|
|
.IP \[bu] 2
|
|
Add bandwidth accounting and limiting (fixes \-\-bwlimit)
|
|
.IP \[bu] 2
|
|
If \-\-stats provided will show stats and which files are transferring
|
|
.IP \[bu] 2
|
|
Support R/W files if truncate is set.
|
|
.IP \[bu] 2
|
|
Implement statfs interface so df works
|
|
.IP \[bu] 2
|
|
Note that write is now supported on Amazon Drive
|
|
.IP \[bu] 2
|
|
Report number of blocks in a file \- thanks Stefan Breunig
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Prevent the user pointing crypt at itself
|
|
.IP \[bu] 2
|
|
Fix failed to authenticate decrypted block errors
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
these will now return the underlying unexpected EOF instead
|
|
.RE
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Add support for server side move and directory move \- thanks Stefan
|
|
Breunig
|
|
.IP \[bu] 2
|
|
Fix nil pointer deref on size attribute
|
|
.IP \[bu] 2
|
|
B2
|
|
.IP \[bu] 2
|
|
Use new prefix and delimiter parameters in directory listings
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This makes \-\-max\-depth 1 dir listings as used in mount much faster
|
|
.RE
|
|
.IP \[bu] 2
|
|
Reauth the account while doing uploads too \- should help with token
|
|
expiry
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Make DirMove more efficient and complain about moving the root
|
|
.IP \[bu] 2
|
|
Create destination directory on Move()
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.34 \- 2016\-11\-06
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Stop single file and \f[C]\-\-files\-from\f[] operations iterating
|
|
through the source bucket.
|
|
.IP \[bu] 2
|
|
Stop removing failed upload to cloud storage remotes
|
|
.IP \[bu] 2
|
|
Make ContentType be preserved for cloud to cloud copies
|
|
.IP \[bu] 2
|
|
Add support to toggle bandwidth limits via SIGUSR2 \- thanks Marco
|
|
Paganini
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ check\f[] shows count of hashes that couldn\[aq]t be
|
|
checked
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ listremotes\f[] command
|
|
.IP \[bu] 2
|
|
Support linux/arm64 build \- thanks Fredrik Fornwall
|
|
.IP \[bu] 2
|
|
Remove \f[C]Authorization:\f[] lines from \f[C]\-\-dump\-headers\f[]
|
|
output
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Ignore files with control characters in the names
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone\ move\f[] command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Delete src files which already existed in dst
|
|
.IP \[bu] 2
|
|
Fix deletion of src file when dst file older
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix \f[C]rclone\ check\f[] on crypted file systems
|
|
.IP \[bu] 2
|
|
Make failed uploads not count as "Transferred"
|
|
.IP \[bu] 2
|
|
Make sure high level retries show with \f[C]\-q\f[]
|
|
.IP \[bu] 2
|
|
Use a vendor directory with godep for repeatable builds
|
|
.IP \[bu] 2
|
|
\f[C]rclone\ mount\f[] \- FUSE
|
|
.IP \[bu] 2
|
|
Implement FUSE mount options
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
\f[C]\-\-no\-modtime\f[], \f[C]\-\-debug\-fuse\f[],
|
|
\f[C]\-\-read\-only\f[], \f[C]\-\-allow\-non\-empty\f[],
|
|
\f[C]\-\-allow\-root\f[], \f[C]\-\-allow\-other\f[]
|
|
.IP \[bu] 2
|
|
\f[C]\-\-default\-permissions\f[], \f[C]\-\-write\-back\-cache\f[],
|
|
\f[C]\-\-max\-read\-ahead\f[], \f[C]\-\-umask\f[], \f[C]\-\-uid\f[],
|
|
\f[C]\-\-gid\f[]
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-dir\-cache\-time\f[] to control caching of directory
|
|
entries
|
|
.IP \[bu] 2
|
|
Implement seek for files opened for read (useful for video players)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
with \f[C]\-no\-seek\f[] flag to disable
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix crash on 32 bit ARM (alignment of 64 bit counter)
|
|
.IP \[bu] 2
|
|
\&...and many more internal fixes and improvements!
|
|
.IP \[bu] 2
|
|
Crypt
|
|
.IP \[bu] 2
|
|
Don\[aq]t show encrypted password in configurator to stop confusion
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
New wait for upload option \f[C]\-\-acd\-upload\-wait\-per\-gb\f[]
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
upload timeouts scale by file size and can be disabled
|
|
.RE
|
|
.IP \[bu] 2
|
|
Add 502 Bad Gateway to list of errors we retry
|
|
.IP \[bu] 2
|
|
Fix overwriting a file with a zero length file
|
|
.IP \[bu] 2
|
|
Fix ACD file size warning limit \- thanks Felix Bünemann
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Unix: implement \f[C]\-x\f[]/\f[C]\-\-one\-file\-system\f[] to stay on a
|
|
single file system
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
thanks Durval Menezes and Luiz Carlos Rumbelsperger Viana
|
|
.RE
|
|
.IP \[bu] 2
|
|
Windows: ignore the symlink bit on files
|
|
.IP \[bu] 2
|
|
Windows: Ignore directory based junction points
|
|
.IP \[bu] 2
|
|
B2
|
|
.IP \[bu] 2
|
|
Make sure each upload has at least one upload slot \- fixes strange
|
|
upload stats
|
|
.IP \[bu] 2
|
|
Fix uploads when using crypt
|
|
.IP \[bu] 2
|
|
Fix download of large files (sha1 mismatch)
|
|
.IP \[bu] 2
|
|
Return error when we try to create a bucket which someone else owns
|
|
.IP \[bu] 2
|
|
Update B2 docs with Data usage, and Crypt section \- thanks Tomasz Mazur
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Command line and config file support for
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Setting/overriding ACL \- thanks Radek Senfeld
|
|
.IP \[bu] 2
|
|
Setting storage class \- thanks Asko Tamm
|
|
.RE
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Make exponential backoff work exactly as per Google specification
|
|
.IP \[bu] 2
|
|
add \f[C]\&.epub\f[], \f[C]\&.odp\f[] and \f[C]\&.tsv\f[] as export
|
|
formats.
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Don\[aq]t read metadata for directory marker objects
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.33 \- 2016\-08\-24
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Implement encryption
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
data encrypted in NACL secretbox format
|
|
.IP \[bu] 2
|
|
with optional file name encryption
|
|
.RE
|
|
.IP \[bu] 2
|
|
New commands
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone mount \- implements FUSE mounting of remotes (EXPERIMENTAL)
|
|
.IP \[bu] 2
|
|
works on Linux, FreeBSD and OS X (need testers for the last 2!)
|
|
.IP \[bu] 2
|
|
rclone cat \- outputs remote file or files to the terminal
|
|
.IP \[bu] 2
|
|
rclone genautocomplete \- command to make a bash completion script for
|
|
rclone
|
|
.RE
|
|
.IP \[bu] 2
|
|
Editing a remote using \f[C]rclone\ config\f[] now goes through the
|
|
wizard
|
|
.IP \[bu] 2
|
|
Compile with go 1.7 \- this fixes rclone on macOS Sierra and on 386
|
|
processors
|
|
.IP \[bu] 2
|
|
Use cobra for sub commands and docs generation
|
|
.IP \[bu] 2
|
|
drive
|
|
.IP \[bu] 2
|
|
Document how to make your own client_id
|
|
.IP \[bu] 2
|
|
s3
|
|
.IP \[bu] 2
|
|
User\-configurable Amazon S3 ACL (thanks Radek Šenfeld)
|
|
.IP \[bu] 2
|
|
b2
|
|
.IP \[bu] 2
|
|
Fix stats accounting for upload \- no more jumping to 100% done
|
|
.IP \[bu] 2
|
|
On cleanup delete hide marker if it is the current file
|
|
.IP \[bu] 2
|
|
New B2 API endpoint (thanks Per Cederberg)
|
|
.IP \[bu] 2
|
|
Set maximum backoff to 5 Minutes
|
|
.IP \[bu] 2
|
|
onedrive
|
|
.IP \[bu] 2
|
|
Fix URL escaping in file names \- eg uploading files with \f[C]+\f[] in
|
|
them.
|
|
.IP \[bu] 2
|
|
amazon cloud drive
|
|
.IP \[bu] 2
|
|
Fix token expiry during large uploads
|
|
.IP \[bu] 2
|
|
Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors
|
|
.IP \[bu] 2
|
|
local
|
|
.IP \[bu] 2
|
|
Fix filenames with invalid UTF\-8 not being uploaded
|
|
.IP \[bu] 2
|
|
Fix problem with some UTF\-8 characters on OS X
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.32 \- 2016\-07\-13
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Fix upload of files large files not in root
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.31 \- 2016\-07\-13
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Reduce memory on sync by about 50%
|
|
.IP \[bu] 2
|
|
Implement \-\-no\-traverse flag to stop copy traversing the destination
|
|
remote.
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This can be used to reduce memory usage down to the smallest possible.
|
|
.IP \[bu] 2
|
|
Useful to copy a small number of files into a large destination folder.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement cleanup command for emptying trash / removing old versions of
|
|
files
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Currently B2 only
|
|
.RE
|
|
.IP \[bu] 2
|
|
Single file handling improved
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now copied with \-\-files\-from
|
|
.IP \[bu] 2
|
|
Automatically sets \-\-no\-traverse when copying a single file
|
|
.RE
|
|
.IP \[bu] 2
|
|
Info on using installing with ansible \- thanks Stefan Weichinger
|
|
.IP \[bu] 2
|
|
Implement \-\-no\-update\-modtime flag to stop rclone fixing the remote
|
|
modified times.
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix move command \- stop it running for overlapping Fses \- this was
|
|
causing data loss.
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Fix incomplete hashes \- this was causing problems for B2.
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Rename Amazon Cloud Drive to Amazon Drive \- no changes to config file
|
|
needed.
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Add support for non\-default project domain \- thanks Antonio Messina.
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Add instructions on how to use rclone with minio.
|
|
.IP \[bu] 2
|
|
Add ap\-northeast\-2 (Seoul) and ap\-south\-1 (Mumbai) regions.
|
|
.IP \[bu] 2
|
|
Skip setting the modified time for objects > 5GB as it isn\[aq]t
|
|
possible.
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Add \-\-b2\-versions flag so old versions can be listed and retreived.
|
|
.IP \[bu] 2
|
|
Treat 403 errors (eg cap exceeded) as fatal.
|
|
.IP \[bu] 2
|
|
Implement cleanup command for deleting old file versions.
|
|
.IP \[bu] 2
|
|
Make error handling compliant with B2 integrations notes.
|
|
.IP \[bu] 2
|
|
Fix handling of token expiry.
|
|
.IP \[bu] 2
|
|
Implement \-\-b2\-test\-mode to set \f[C]X\-Bz\-Test\-Mode\f[] header.
|
|
.IP \[bu] 2
|
|
Set cutoff for chunked upload to 200MB as per B2 guidelines.
|
|
.IP \[bu] 2
|
|
Make upload multi\-threaded.
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Don\[aq]t retry 461 errors.
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.30 \- 2016\-06\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Directory listing code reworked for more features and better error
|
|
reporting (thanks to Klaus Post for help).
|
|
This enables
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Directory include filtering for efficiency
|
|
.IP \[bu] 2
|
|
\-\-max\-depth parameter
|
|
.IP \[bu] 2
|
|
Better error reporting
|
|
.IP \[bu] 2
|
|
More to come
|
|
.RE
|
|
.IP \[bu] 2
|
|
Retry more errors
|
|
.IP \[bu] 2
|
|
Add \-\-ignore\-size flag \- for uploading images to onedrive
|
|
.IP \[bu] 2
|
|
Log \-v output to stdout by default
|
|
.IP \[bu] 2
|
|
Display the transfer stats in more human readable form
|
|
.IP \[bu] 2
|
|
Make 0 size files specifiable with \f[C]\-\-max\-size\ 0b\f[]
|
|
.IP \[bu] 2
|
|
Add \f[C]b\f[] suffix so we can specify bytes in \-\-bwlimit,
|
|
\-\-min\-size etc
|
|
.IP \[bu] 2
|
|
Use "password:" instead of "password>" prompt \- thanks Klaus Post and
|
|
Leigh Klotz
|
|
.IP \[bu] 2
|
|
Bug Fixes
|
|
.IP \[bu] 2
|
|
Fix retry doing one too many retries
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Fix problems with OS X and UTF\-8 characters
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Check a file exists before uploading to help with 408 Conflict errors
|
|
.IP \[bu] 2
|
|
Reauth on 401 errors \- this has been causing a lot of problems
|
|
.IP \[bu] 2
|
|
Work around spurious 403 errors
|
|
.IP \[bu] 2
|
|
Restart directory listings on error
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Check a file exists before uploading to help with duplicates
|
|
.IP \[bu] 2
|
|
Fix retry of multipart uploads
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Implement large file uploading
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Add AES256 server\-side encryption for \- thanks Justin R.
|
|
Wilson
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Make sure we don\[aq]t use conflicting content types on upload
|
|
.IP \[bu] 2
|
|
Add service account support \- thanks Michal Witkowski
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Add auth version parameter
|
|
.IP \[bu] 2
|
|
Add domain option for openstack (v3 auth) \- thanks Fabian Ruff
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.29 \- 2016\-04\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-I,\ \-\-ignore\-times\f[] for unconditional upload
|
|
.IP \[bu] 2
|
|
Improve \f[C]dedupe\f[]command
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Now removes identical copies without asking
|
|
.IP \[bu] 2
|
|
Now obeys \f[C]\-\-dry\-run\f[]
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-\-dedupe\-mode\f[] for non interactive running
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ interactive\f[] \- interactive the default.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ skip\f[] \- removes identical files then skips
|
|
anything left.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ first\f[] \- removes identical files then keeps
|
|
the first one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ newest\f[] \- removes identical files then keeps
|
|
the newest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ oldest\f[] \- removes identical files then keeps
|
|
the oldest one.
|
|
.IP \[bu] 2
|
|
\f[C]\-\-dedupe\-mode\ rename\f[] \- removes identical files then
|
|
renames the rest to be different.
|
|
.RE
|
|
.IP \[bu] 2
|
|
Bug fixes
|
|
.IP \[bu] 2
|
|
Make rclone check obey the \f[C]\-\-size\-only\f[] flag.
|
|
.IP \[bu] 2
|
|
Use "application/octet\-stream" if discovered mime type is invalid.
|
|
.IP \[bu] 2
|
|
Fix missing "quit" option when there are no remotes.
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Increase default chunk size to 8 MB \- increases upload speed of big
|
|
files
|
|
.IP \[bu] 2
|
|
Speed up directory listings and make more reliable
|
|
.IP \[bu] 2
|
|
Add missing retries for Move and DirMove \- increases reliability
|
|
.IP \[bu] 2
|
|
Preserve mime type on file update
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Enable mod time syncing
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This means that B2 will now check modification times
|
|
.IP \[bu] 2
|
|
It will upload new files to update the modification times
|
|
.IP \[bu] 2
|
|
(there isn\[aq]t an API to just set the mod time.)
|
|
.IP \[bu] 2
|
|
If you want the old behaviour use \f[C]\-\-size\-only\f[].
|
|
.RE
|
|
.IP \[bu] 2
|
|
Update API to new version
|
|
.IP \[bu] 2
|
|
Fix parsing of mod time when not in metadata
|
|
.IP \[bu] 2
|
|
Swift/Hubic
|
|
.IP \[bu] 2
|
|
Don\[aq]t return an MD5SUM for static large objects
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Fix uploading files bigger than 50GB
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.28 \- 2016\-03\-01
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Configuration file encryption \- thanks Klaus Post
|
|
.IP \[bu] 2
|
|
Improve \f[C]rclone\ config\f[] adding more help and making it easier to
|
|
understand
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-u\f[]/\f[C]\-\-update\f[] so creation times can be used
|
|
on all remotes
|
|
.IP \[bu] 2
|
|
Implement \f[C]\-\-low\-level\-retries\f[] flag
|
|
.IP \[bu] 2
|
|
Optionally disable gzip compression on downloads with
|
|
\f[C]\-\-no\-gzip\-encoding\f[]
|
|
.IP \[bu] 2
|
|
Bug fixes
|
|
.IP \[bu] 2
|
|
Don\[aq]t make directories if \f[C]\-\-dry\-run\f[] set
|
|
.IP \[bu] 2
|
|
Fix and document the \f[C]move\f[] command
|
|
.IP \[bu] 2
|
|
Fix redirecting stderr on unix\-like OSes when using
|
|
\f[C]\-\-log\-file\f[]
|
|
.IP \[bu] 2
|
|
Fix \f[C]delete\f[] command to wait until all finished \- fixes missing
|
|
deletes.
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
Use one upload URL per go routine fixes
|
|
\f[C]more\ than\ one\ upload\ using\ auth\ token\f[]
|
|
.IP \[bu] 2
|
|
Add pacing, retries and reauthentication \- fixes token expiry problems
|
|
.IP \[bu] 2
|
|
Upload without using a temporary file from local (and remotes which
|
|
support SHA1)
|
|
.IP \[bu] 2
|
|
Fix reading metadata for all files when it shouldn\[aq]t have been
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Fix listing drive documents at root
|
|
.IP \[bu] 2
|
|
Disable copy and move for Google docs
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix uploading of chunked files with non ASCII characters
|
|
.IP \[bu] 2
|
|
Allow setting of \f[C]storage_url\f[] in the config \- thanks Xavier
|
|
Lucas
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Allow IAM role and credentials from environment variables \- thanks
|
|
Brian Stengaard
|
|
.IP \[bu] 2
|
|
Allow low privilege users to use S3 (check if directory exists during
|
|
Mkdir) \- thanks Jakub Gedeon
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Retry on more things to make directory listings more reliable
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.27 \- 2016\-01\-31
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Easier headless configuration with \f[C]rclone\ authorize\f[]
|
|
.IP \[bu] 2
|
|
Add support for multiple hash types \- we now check SHA1 as well as MD5
|
|
hashes.
|
|
.IP \[bu] 2
|
|
\f[C]delete\f[] command which does obey the filters (unlike
|
|
\f[C]purge\f[])
|
|
.IP \[bu] 2
|
|
\f[C]dedupe\f[] command to deduplicate a remote.
|
|
Useful with Google Drive.
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-ignore\-existing\f[] flag to skip all files that exist on
|
|
destination.
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-delete\-before\f[], \f[C]\-\-delete\-during\f[],
|
|
\f[C]\-\-delete\-after\f[] flags.
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-memprofile\f[] flag to debug memory use.
|
|
.IP \[bu] 2
|
|
Warn the user about files with same name but different case
|
|
.IP \[bu] 2
|
|
Make \f[C]\-\-include\f[] rules add their implict exclude * at the end
|
|
of the filter list
|
|
.IP \[bu] 2
|
|
Deprecate compiling with go1.3
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Fix download of files > 10 GB
|
|
.IP \[bu] 2
|
|
Fix directory traversal ("Next token is expired") for large directory
|
|
listings
|
|
.IP \[bu] 2
|
|
Remove 409 conflict from error codes we will retry \- stops very long
|
|
pauses
|
|
.IP \[bu] 2
|
|
Backblaze B2
|
|
.IP \[bu] 2
|
|
SHA1 hashes now checked by rclone core
|
|
.IP \[bu] 2
|
|
Drive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-auth\-owner\-only\f[] to only consider files owned
|
|
by the user \- thanks Björn Harrtell
|
|
.IP \[bu] 2
|
|
Export Google documents
|
|
.IP \[bu] 2
|
|
Dropbox
|
|
.IP \[bu] 2
|
|
Make file exclusion error controllable with \-q
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix upload from unprivileged user.
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Fix updating of mod times of files with \f[C]+\f[] in.
|
|
.IP \[bu] 2
|
|
Local
|
|
.IP \[bu] 2
|
|
Add local file system option to disable UNC on Windows.
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.26 \- 2016\-01\-02
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New Features
|
|
.IP \[bu] 2
|
|
Yandex storage backend \- thank you Dmitry Burdeev ("dibu")
|
|
.IP \[bu] 2
|
|
Implement Backblaze B2 storage backend
|
|
.IP \[bu] 2
|
|
Add \-\-min\-age and \-\-max\-age flags \- thank you Adriano Aurélio
|
|
Meirelles
|
|
.IP \[bu] 2
|
|
Make ls/lsl/md5sum/size/check obey includes and excludes
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix crash in http logging
|
|
.IP \[bu] 2
|
|
Upload releases to github too
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Fix sync for chunked files
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.IP \[bu] 2
|
|
Re\-enable server side copy
|
|
.IP \[bu] 2
|
|
Don\[aq]t mask HTTP error codes with JSON decode error
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Fix corrupting Content\-Type on mod time update (thanks Joseph Spurrier)
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.25 \- 2015\-11\-14
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Implement Hubic storage system
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix deletion of some excluded files without \-\-delete\-excluded
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This could have deleted files unexpectedly on sync
|
|
.IP \[bu] 2
|
|
Always check first with \f[C]\-\-dry\-run\f[]!
|
|
.RE
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Stop SetModTime losing metadata (eg X\-Object\-Manifest)
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This could have caused data loss for files > 5GB in size
|
|
.RE
|
|
.IP \[bu] 2
|
|
Use ContentType from Object to avoid lookups in listings
|
|
.IP \[bu] 2
|
|
OneDrive
|
|
.IP \[bu] 2
|
|
disable server side copy as it seems to be broken at Microsoft
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.24 \- 2015\-11\-07
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Add support for Microsoft OneDrive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-no\-check\-certificate\f[] option to disable server
|
|
certificate verification
|
|
.IP \[bu] 2
|
|
Add async readahead buffer for faster transfer of big files
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Allow spaces in remotes and check remote names for validity at creation
|
|
time
|
|
.IP \[bu] 2
|
|
Allow \[aq]&\[aq] and disallow \[aq]:\[aq] in Windows filenames.
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Ignore directory marker objects where appropriate \- allows working with
|
|
Hubic
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the container if fs wasn\[aq]t at root
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the bucket if fs wasn\[aq]t at root
|
|
.IP \[bu] 2
|
|
Google Cloud Storage
|
|
.IP \[bu] 2
|
|
Don\[aq]t delete the bucket if fs wasn\[aq]t at root
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.23 \- 2015\-10\-03
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Implement \f[C]rclone\ size\f[] for measuring remotes
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix headless config for drive and gcs
|
|
.IP \[bu] 2
|
|
Tell the user they should try again if the webserver method failed
|
|
.IP \[bu] 2
|
|
Improve output of \f[C]\-\-dump\-headers\f[]
|
|
.IP \[bu] 2
|
|
S3
|
|
.IP \[bu] 2
|
|
Allow anonymous access to public buckets
|
|
.IP \[bu] 2
|
|
Swift
|
|
.IP \[bu] 2
|
|
Stop chunked operations logging "Failed to read info: Object Not Found"
|
|
.IP \[bu] 2
|
|
Use Content\-Length on uploads for extra reliability
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.22 \- 2015\-09\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement rsync like include and exclude flags
|
|
.IP \[bu] 2
|
|
swift
|
|
.IP \[bu] 2
|
|
Support files > 5GB \- thanks Sergey Tolmachev
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.21 \- 2015\-09\-22
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Display individual transfer progress
|
|
.IP \[bu] 2
|
|
Make lsl output times in localtime
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Fix allowing user to override credentials again in Drive, GCS and ACD
|
|
.IP \[bu] 2
|
|
Amazon Drive
|
|
.IP \[bu] 2
|
|
Implement compliant pacing scheme
|
|
.IP \[bu] 2
|
|
Google Drive
|
|
.IP \[bu] 2
|
|
Make directory reads concurrent for increased speed.
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.20 \- 2015\-09\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Amazon Drive support
|
|
.IP \[bu] 2
|
|
Oauth support redone \- fix many bugs and improve usability
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Use "golang.org/x/oauth2" as oauth libary of choice
|
|
.IP \[bu] 2
|
|
Improve oauth usability for smoother initial signup
|
|
.IP \[bu] 2
|
|
drive, googlecloudstorage: optionally use auto config for the oauth
|
|
token
|
|
.RE
|
|
.IP \[bu] 2
|
|
Implement \-\-dump\-headers and \-\-dump\-bodies debug flags
|
|
.IP \[bu] 2
|
|
Show multiple matched commands if abbreviation too short
|
|
.IP \[bu] 2
|
|
Implement server side move where possible
|
|
.IP \[bu] 2
|
|
local
|
|
.IP \[bu] 2
|
|
Always use UNC paths internally on Windows \- fixes a lot of bugs
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.IP \[bu] 2
|
|
force use of our custom transport which makes timeouts work
|
|
.IP \[bu] 2
|
|
Thanks to Klaus Post for lots of help with this release
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.19 \- 2015\-08\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
New features
|
|
.IP \[bu] 2
|
|
Server side copies for s3/swift/drive/dropbox/gcs
|
|
.IP \[bu] 2
|
|
Move command \- uses server side copies if it can
|
|
.IP \[bu] 2
|
|
Implement \-\-retries flag \- tries 3 times by default
|
|
.IP \[bu] 2
|
|
Build for plan9/amd64 and solaris/amd64 too
|
|
.IP \[bu] 2
|
|
Fixes
|
|
.IP \[bu] 2
|
|
Make a current version download with a fixed URL for scripting
|
|
.IP \[bu] 2
|
|
Ignore rmdir in limited fs rather than throwing error
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.IP \[bu] 2
|
|
Increase chunk size to improve upload speeds massively
|
|
.IP \[bu] 2
|
|
Issue an error message when trying to upload bad file name
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.18 \- 2015\-08\-17
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive
|
|
.IP \[bu] 2
|
|
Add \f[C]\-\-drive\-use\-trash\f[] flag so rclone trashes instead of
|
|
deletes
|
|
.IP \[bu] 2
|
|
Add "Forbidden to download" message for files with no downloadURL
|
|
.IP \[bu] 2
|
|
dropbox
|
|
.IP \[bu] 2
|
|
Remove datastore
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
This was deprecated and it caused a lot of problems
|
|
.IP \[bu] 2
|
|
Modification times and MD5SUMs no longer stored
|
|
.RE
|
|
.IP \[bu] 2
|
|
Fix uploading files > 2GB
|
|
.IP \[bu] 2
|
|
s3
|
|
.IP \[bu] 2
|
|
use official AWS SDK from github.com/aws/aws\-sdk\-go
|
|
.IP \[bu] 2
|
|
\f[B]NB\f[] will most likely require you to delete and recreate remote
|
|
.IP \[bu] 2
|
|
enable multipart upload which enables files > 5GB
|
|
.IP \[bu] 2
|
|
tested with Ceph / RadosGW / S3 emulation
|
|
.IP \[bu] 2
|
|
many thanks to Sam Liston and Brian Haymore at the Utah Center for High
|
|
Performance Computing (https://www.chpc.utah.edu/) for a Ceph test
|
|
account
|
|
.IP \[bu] 2
|
|
misc
|
|
.IP \[bu] 2
|
|
Show errors when reading the config file
|
|
.IP \[bu] 2
|
|
Do not print stats in quiet mode \- thanks Leonid Shalupov
|
|
.IP \[bu] 2
|
|
Add FAQ
|
|
.IP \[bu] 2
|
|
Fix created directories not obeying umask
|
|
.IP \[bu] 2
|
|
Linux installation instructions \- thanks Shimon Doodkin
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.17 \- 2015\-06\-14
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
dropbox: fix case insensitivity issues \- thanks Leonid Shalupov
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.16 \- 2015\-06\-09
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix uploading big files which was causing timeouts or panics
|
|
.IP \[bu] 2
|
|
Don\[aq]t check md5sum after download with \-\-size\-only
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.15 \- 2015\-06\-06
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Add \-\-checksum flag to only discard transfers by MD5SUM \- thanks Alex
|
|
Couper
|
|
.IP \[bu] 2
|
|
Implement \-\-size\-only flag to sync on size not checksum & modtime
|
|
.IP \[bu] 2
|
|
Expand docs and remove duplicated information
|
|
.IP \[bu] 2
|
|
Document rclone\[aq]s limitations with directories
|
|
.IP \[bu] 2
|
|
dropbox: update docs about case insensitivity
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.14 \- 2015\-05\-21
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
local: fix encoding of non utf\-8 file names \- fixes a duplicate file
|
|
problem
|
|
.IP \[bu] 2
|
|
drive: docs about rate limiting
|
|
.IP \[bu] 2
|
|
google cloud storage: Fix compile after API change in
|
|
"google.golang.org/api/storage/v1"
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.13 \- 2015\-05\-10
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Revise documentation (especially sync)
|
|
.IP \[bu] 2
|
|
Implement \-\-timeout and \-\-conntimeout
|
|
.IP \[bu] 2
|
|
s3: ignore etags from multipart uploads which aren\[aq]t md5sums
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.12 \- 2015\-03\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: Use chunked upload for files above a certain size
|
|
.IP \[bu] 2
|
|
drive: add \-\-drive\-chunk\-size and \-\-drive\-upload\-cutoff
|
|
parameters
|
|
.IP \[bu] 2
|
|
drive: switch to insert from update when a failed copy deletes the
|
|
upload
|
|
.IP \[bu] 2
|
|
core: Log duplicate files if they are detected
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.11 \- 2015\-03\-04
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
swift: add region parameter
|
|
.IP \[bu] 2
|
|
drive: fix crash on failed to update remote mtime
|
|
.IP \[bu] 2
|
|
In remote paths, change native directory separators to /
|
|
.IP \[bu] 2
|
|
Add synchronization to ls/lsl/lsd output to stop corruptions
|
|
.IP \[bu] 2
|
|
Ensure all stats/log messages to go stderr
|
|
.IP \[bu] 2
|
|
Add \-\-log\-file flag to log everything (including panics) to file
|
|
.IP \[bu] 2
|
|
Make it possible to disable stats printing with \-\-stats=0
|
|
.IP \[bu] 2
|
|
Implement \-\-bwlimit to limit data transfer bandwidth
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.10 \- 2015\-02\-12
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
s3: list an unlimited number of items
|
|
.IP \[bu] 2
|
|
Fix getting stuck in the configurator
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.09 \- 2015\-02\-07
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
windows: Stop drive letters (eg C:) getting mixed up with remotes (eg
|
|
drive:)
|
|
.IP \[bu] 2
|
|
local: Fix directory separators on Windows
|
|
.IP \[bu] 2
|
|
drive: fix rate limit exceeded errors
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.08 \- 2015\-02\-04
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: fix subdirectory listing to not list entire drive
|
|
.IP \[bu] 2
|
|
drive: Fix SetModTime
|
|
.IP \[bu] 2
|
|
dropbox: adapt code to recent library changes
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.07 \- 2014\-12\-23
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
google cloud storage: fix memory leak
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.06 \- 2014\-12\-12
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix "Couldn\[aq]t find home directory" on OSX
|
|
.IP \[bu] 2
|
|
swift: Add tenant parameter
|
|
.IP \[bu] 2
|
|
Use new location of Google API packages
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.05 \- 2014\-08\-09
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Improved tests and consequently lots of minor fixes
|
|
.IP \[bu] 2
|
|
core: Fix race detected by go race detector
|
|
.IP \[bu] 2
|
|
core: Fixes after running errcheck
|
|
.IP \[bu] 2
|
|
drive: reset root directory on Rmdir and Purge
|
|
.IP \[bu] 2
|
|
fs: Document that Purger returns error on empty directory, test and fix
|
|
.IP \[bu] 2
|
|
google cloud storage: fix ListDir on subdirectory
|
|
.IP \[bu] 2
|
|
google cloud storage: re\-read metadata in SetModTime
|
|
.IP \[bu] 2
|
|
s3: make reading metadata more reliable to work around eventual
|
|
consistency problems
|
|
.IP \[bu] 2
|
|
s3: strip trailing / from ListDir()
|
|
.IP \[bu] 2
|
|
swift: return directories without / in ListDir
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.04 \- 2014\-07\-21
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
google cloud storage: Fix crash on Update
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.03 \- 2014\-07\-20
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
swift, s3, dropbox: fix updated files being marked as corrupted
|
|
.IP \[bu] 2
|
|
Make compile with go 1.1 again
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.02 \- 2014\-07\-19
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement Dropbox remote
|
|
.IP \[bu] 2
|
|
Implement Google Cloud Storage remote
|
|
.IP \[bu] 2
|
|
Verify Md5sums and Sizes after copies
|
|
.IP \[bu] 2
|
|
Remove times from "ls" command \- lists sizes only
|
|
.IP \[bu] 2
|
|
Add add "lsl" \- lists times and sizes
|
|
.IP \[bu] 2
|
|
Add "md5sum" command
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.01 \- 2014\-07\-04
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: fix transfer of big files using up lots of memory
|
|
.RE
|
|
.IP \[bu] 2
|
|
v1.00 \- 2014\-07\-03
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: fix whole second dates
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.99 \- 2014\-06\-26
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Fix \-\-dry\-run not working
|
|
.IP \[bu] 2
|
|
Make compatible with go 1.1
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.98 \- 2014\-05\-30
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
s3: Treat missing Content\-Length as 0 for some ceph installations
|
|
.IP \[bu] 2
|
|
rclonetest: add file with a space in
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.97 \- 2014\-05\-05
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Implement copying of single files
|
|
.IP \[bu] 2
|
|
s3 & swift: support paths inside containers/buckets
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.96 \- 2014\-04\-24
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: Fix multiple files of same name being created
|
|
.IP \[bu] 2
|
|
drive: Use o.Update and fs.Put to optimise transfers
|
|
.IP \[bu] 2
|
|
Add version number, \-V and \-\-version
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.95 \- 2014\-03\-28
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
rclone.org: website, docs and graphics
|
|
.IP \[bu] 2
|
|
drive: fix path parsing
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.94 \- 2014\-03\-27
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Change remote format one last time
|
|
.IP \[bu] 2
|
|
GNU style flags
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.93 \- 2014\-03\-16
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
drive: store token in config file
|
|
.IP \[bu] 2
|
|
cross compile other versions
|
|
.IP \[bu] 2
|
|
set strict permissions on config file
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.92 \- 2014\-03\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Config fixes and \-\-config option
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.91 \- 2014\-03\-15
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Make config file
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.90 \- 2013\-06\-27
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project named rclone
|
|
.RE
|
|
.IP \[bu] 2
|
|
v0.00 \- 2012\-11\-18
|
|
.RS 2
|
|
.IP \[bu] 2
|
|
Project started
|
|
.RE
|
|
.SS Bugs and Limitations
|
|
.SS Empty directories are left behind / not created
|
|
.PP
|
|
With remotes that have a concept of directory, eg Local and Drive, empty
|
|
directories may be left behind, or not created when one was expected.
|
|
.PP
|
|
This is because rclone doesn\[aq]t have a concept of a directory \- it
|
|
only works on objects.
|
|
Most of the object storage systems can\[aq]t actually store a directory
|
|
so there is nowhere for rclone to store anything about directories.
|
|
.PP
|
|
You can work round this to some extent with the\f[C]purge\f[] command
|
|
which will delete everything under the path, \f[B]inluding\f[] empty
|
|
directories.
|
|
.PP
|
|
This may be fixed at some point in Issue
|
|
#100 (https://github.com/ncw/rclone/issues/100)
|
|
.SS Directory timestamps aren\[aq]t preserved
|
|
.PP
|
|
For the same reason as the above, rclone doesn\[aq]t have a concept of a
|
|
directory \- it only works on objects, therefore it can\[aq]t preserve
|
|
the timestamps of directories.
|
|
.SS Frequently Asked Questions
|
|
.SS Do all cloud storage systems support all rclone commands
|
|
.PP
|
|
Yes they do.
|
|
All the rclone commands (eg \f[C]sync\f[], \f[C]copy\f[] etc) will work
|
|
on all the remote storage systems.
|
|
.SS Can I copy the config from one machine to another
|
|
.PP
|
|
Sure! Rclone stores all of its config in a single file.
|
|
If you want to find this file, the simplest way is to run
|
|
\f[C]rclone\ \-h\f[] and look at the help for the \f[C]\-\-config\f[]
|
|
flag which will tell you where it is.
|
|
.PP
|
|
See the remote setup docs (https://rclone.org/remote_setup/) for more
|
|
info.
|
|
.SS How do I configure rclone on a remote / headless box with no
|
|
browser?
|
|
.PP
|
|
This has now been documented in its own remote setup
|
|
page (https://rclone.org/remote_setup/).
|
|
.SS Can rclone sync directly from drive to s3
|
|
.PP
|
|
Rclone can sync between two remote cloud storage systems just fine.
|
|
.PP
|
|
Note that it effectively downloads the file and uploads it again, so the
|
|
node running rclone would need to have lots of bandwidth.
|
|
.PP
|
|
The syncs would be incremental (on a file by file basis).
|
|
.PP
|
|
Eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
rclone\ sync\ drive:Folder\ s3:bucket
|
|
\f[]
|
|
.fi
|
|
.SS Using rclone from multiple locations at the same time
|
|
.PP
|
|
You can use rclone from multiple places at the same time if you choose
|
|
different subdirectory for the output, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Server\ A>\ rclone\ sync\ /tmp/whatever\ remote:ServerA
|
|
Server\ B>\ rclone\ sync\ /tmp/whatever\ remote:ServerB
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you sync to the same directory then you should use rclone copy
|
|
otherwise the two rclones may delete each others files, eg
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Server\ A>\ rclone\ copy\ /tmp/whatever\ remote:Backup
|
|
Server\ B>\ rclone\ copy\ /tmp/whatever\ remote:Backup
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The file names you upload from Server A and Server B should be different
|
|
in this case, otherwise some file systems (eg Drive) may make
|
|
duplicates.
|
|
.SS Why doesn\[aq]t rclone support partial transfers / binary diffs like
|
|
rsync?
|
|
.PP
|
|
Rclone stores each file you transfer as a native object on the remote
|
|
cloud storage system.
|
|
This means that you can see the files you upload as expected using
|
|
alternative access methods (eg using the Google Drive web interface).
|
|
There is a 1:1 mapping between files on your hard disk and objects
|
|
created in the cloud storage system.
|
|
.PP
|
|
Cloud storage systems (at least none I\[aq]ve come across yet) don\[aq]t
|
|
support partially uploading an object.
|
|
You can\[aq]t take an existing object, and change some bytes in the
|
|
middle of it.
|
|
.PP
|
|
It would be possible to make a sync system which stored binary diffs
|
|
instead of whole objects like rclone does, but that would break the 1:1
|
|
mapping of files on your hard disk to objects in the remote cloud
|
|
storage system.
|
|
.PP
|
|
All the cloud storage systems support partial downloads of content, so
|
|
it would be possible to make partial downloads work.
|
|
However to make this work efficiently this would require storing a
|
|
significant amount of metadata, which breaks the desired 1:1 mapping of
|
|
files to objects.
|
|
.SS Can rclone do bi\-directional sync?
|
|
.PP
|
|
No, not at present.
|
|
rclone only does uni\-directional sync from A \-> B.
|
|
It may do in the future though since it has all the primitives \- it
|
|
just requires writing the algorithm to do it.
|
|
.SS Can I use rclone with an HTTP proxy?
|
|
.PP
|
|
Yes.
|
|
rclone will use the environment variables \f[C]HTTP_PROXY\f[],
|
|
\f[C]HTTPS_PROXY\f[] and \f[C]NO_PROXY\f[], similar to cURL and other
|
|
programs.
|
|
.PP
|
|
\f[C]HTTPS_PROXY\f[] takes precedence over \f[C]HTTP_PROXY\f[] for https
|
|
requests.
|
|
.PP
|
|
The environment values may be either a complete URL or a "host[:port]",
|
|
in which case the "http" scheme is assumed.
|
|
.PP
|
|
The \f[C]NO_PROXY\f[] allows you to disable the proxy for specific
|
|
hosts.
|
|
Hosts must be comma separated, and can contain domains or parts.
|
|
For instance "foo.com" also matches "bar.foo.com".
|
|
.SS Rclone gives x509: failed to load system roots and no roots provided
|
|
error
|
|
.PP
|
|
This means that \f[C]rclone\f[] can\[aq]t file the SSL root
|
|
certificates.
|
|
Likely you are running \f[C]rclone\f[] on a NAS with a cut\-down Linux
|
|
OS, or possibly on Solaris.
|
|
.PP
|
|
Rclone (via the Go runtime) tries to load the root certificates from
|
|
these places on Linux.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
"/etc/ssl/certs/ca\-certificates.crt",\ //\ Debian/Ubuntu/Gentoo\ etc.
|
|
"/etc/pki/tls/certs/ca\-bundle.crt",\ \ \ //\ Fedora/RHEL
|
|
"/etc/ssl/ca\-bundle.pem",\ \ \ \ \ \ \ \ \ \ \ \ \ //\ OpenSUSE
|
|
"/etc/pki/tls/cacert.pem",\ \ \ \ \ \ \ \ \ \ \ \ //\ OpenELEC
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
So doing something like this should fix the problem.
|
|
It also sets the time which is important for SSL to work properly.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
mkdir\ \-p\ /etc/ssl/certs/
|
|
curl\ \-o\ /etc/ssl/certs/ca\-certificates.crt\ https://raw.githubusercontent.com/bagder/ca\-bundle/master/ca\-bundle.crt
|
|
ntpclient\ \-s\ \-h\ pool.ntp.org
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
The two environment variables \f[C]SSL_CERT_FILE\f[] and
|
|
\f[C]SSL_CERT_DIR\f[], mentioned in the x509
|
|
pacakge (https://godoc.org/crypto/x509), provide an additional way to
|
|
provide the SSL root certificates.
|
|
.PP
|
|
Note that you may need to add the \f[C]\-\-insecure\f[] option to the
|
|
\f[C]curl\f[] command line if it doesn\[aq]t work without.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
curl\ \-\-insecure\ \-o\ /etc/ssl/certs/ca\-certificates.crt\ https://raw.githubusercontent.com/bagder/ca\-bundle/master/ca\-bundle.crt
|
|
\f[]
|
|
.fi
|
|
.SS Rclone gives Failed to load config file: function not implemented
|
|
error
|
|
.PP
|
|
Likely this means that you are running rclone on Linux version not
|
|
supported by the go runtime, ie earlier than version 2.6.23.
|
|
.PP
|
|
See the system requirements section in the go install
|
|
docs (https://golang.org/doc/install) for full details.
|
|
.SS All my uploaded docx/xlsx/pptx files appear as archive/zip
|
|
.PP
|
|
This is caused by uploading these files from a Windows computer which
|
|
hasn\[aq]t got the Microsoft Office suite installed.
|
|
The easiest way to fix is to install the Word viewer and the Microsoft
|
|
Office Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
|
|
versions\[aq] file formats
|
|
.SS tcp lookup some.domain.com no such host
|
|
.PP
|
|
This happens when rclone cannot resolve a domain.
|
|
Please check that your DNS setup is generally working, e.g.
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
#\ both\ should\ print\ a\ long\ list\ of\ possible\ IP\ addresses
|
|
dig\ www.googleapis.com\ \ \ \ \ \ \ \ \ \ #\ resolve\ using\ your\ default\ DNS
|
|
dig\ www.googleapis.com\ \@8.8.8.8\ #\ resolve\ with\ Google\[aq]s\ DNS\ server
|
|
\f[]
|
|
.fi
|
|
.PP
|
|
If you are using \f[C]systemd\-resolved\f[] (default on Arch Linux),
|
|
ensure it is at version 233 or higher.
|
|
Previous releases contain a bug which causes not all domains to be
|
|
resolved properly.
|
|
.PP
|
|
Additionally with the \f[C]GODEBUG=netdns=\f[] environment variable the
|
|
Go resolver decision can be influenced.
|
|
This also allows to resolve certain issues with DNS resolution.
|
|
See the name resolution section in the go
|
|
docs (https://golang.org/pkg/net/#hdr-Name_Resolution).
|
|
.SS License
|
|
.PP
|
|
This is free software under the terms of MIT the license (check the
|
|
COPYING file included with the source code).
|
|
.IP
|
|
.nf
|
|
\f[C]
|
|
Copyright\ (C)\ 2012\ by\ Nick\ Craig\-Wood\ https://www.craig\-wood.com/nick/
|
|
|
|
Permission\ is\ hereby\ granted,\ free\ of\ charge,\ to\ any\ person\ obtaining\ a\ copy
|
|
of\ this\ software\ and\ associated\ documentation\ files\ (the\ "Software"),\ to\ deal
|
|
in\ the\ Software\ without\ restriction,\ including\ without\ limitation\ the\ rights
|
|
to\ use,\ copy,\ modify,\ merge,\ publish,\ distribute,\ sublicense,\ and/or\ sell
|
|
copies\ of\ the\ Software,\ and\ to\ permit\ persons\ to\ whom\ the\ Software\ is
|
|
furnished\ to\ do\ so,\ subject\ to\ the\ following\ conditions:
|
|
|
|
The\ above\ copyright\ notice\ and\ this\ permission\ notice\ shall\ be\ included\ in
|
|
all\ copies\ or\ substantial\ portions\ of\ the\ Software.
|
|
|
|
THE\ SOFTWARE\ IS\ PROVIDED\ "AS\ IS",\ WITHOUT\ WARRANTY\ OF\ ANY\ KIND,\ EXPRESS\ OR
|
|
IMPLIED,\ INCLUDING\ BUT\ NOT\ LIMITED\ TO\ THE\ WARRANTIES\ OF\ MERCHANTABILITY,
|
|
FITNESS\ FOR\ A\ PARTICULAR\ PURPOSE\ AND\ NONINFRINGEMENT.\ IN\ NO\ EVENT\ SHALL\ THE
|
|
AUTHORS\ OR\ COPYRIGHT\ HOLDERS\ BE\ LIABLE\ FOR\ ANY\ CLAIM,\ DAMAGES\ OR\ OTHER
|
|
LIABILITY,\ WHETHER\ IN\ AN\ ACTION\ OF\ CONTRACT,\ TORT\ OR\ OTHERWISE,\ ARISING\ FROM,
|
|
OUT\ OF\ OR\ IN\ CONNECTION\ WITH\ THE\ SOFTWARE\ OR\ THE\ USE\ OR\ OTHER\ DEALINGS\ IN
|
|
THE\ SOFTWARE.
|
|
\f[]
|
|
.fi
|
|
.SS Authors
|
|
.IP \[bu] 2
|
|
Nick Craig\-Wood <nick@craig-wood.com>
|
|
.SS Contributors
|
|
.IP \[bu] 2
|
|
Alex Couper <amcouper@gmail.com>
|
|
.IP \[bu] 2
|
|
Leonid Shalupov <leonid@shalupov.com> <shalupov@diverse.org.ru>
|
|
.IP \[bu] 2
|
|
Shimon Doodkin <helpmepro1@gmail.com>
|
|
.IP \[bu] 2
|
|
Colin Nicholson <colin@colinn.com>
|
|
.IP \[bu] 2
|
|
Klaus Post <klauspost@gmail.com>
|
|
.IP \[bu] 2
|
|
Sergey Tolmachev <tolsi.ru@gmail.com>
|
|
.IP \[bu] 2
|
|
Adriano Aurélio Meirelles <adriano@atinge.com>
|
|
.IP \[bu] 2
|
|
C.
|
|
Bess <cbess@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dmitry Burdeev <dibu28@gmail.com>
|
|
.IP \[bu] 2
|
|
Joseph Spurrier <github@josephspurrier.com>
|
|
.IP \[bu] 2
|
|
Björn Harrtell <bjorn@wololo.org>
|
|
.IP \[bu] 2
|
|
Xavier Lucas <xavier.lucas@corp.ovh.com>
|
|
.IP \[bu] 2
|
|
Werner Beroux <werner@beroux.com>
|
|
.IP \[bu] 2
|
|
Brian Stengaard <brian@stengaard.eu>
|
|
.IP \[bu] 2
|
|
Jakub Gedeon <jgedeon@sofi.com>
|
|
.IP \[bu] 2
|
|
Jim Tittsler <jwt@onjapan.net>
|
|
.IP \[bu] 2
|
|
Michal Witkowski <michal@improbable.io>
|
|
.IP \[bu] 2
|
|
Fabian Ruff <fabian.ruff@sap.com>
|
|
.IP \[bu] 2
|
|
Leigh Klotz <klotz@quixey.com>
|
|
.IP \[bu] 2
|
|
Romain Lapray <lapray.romain@gmail.com>
|
|
.IP \[bu] 2
|
|
Justin R.
|
|
Wilson <jrw972@gmail.com>
|
|
.IP \[bu] 2
|
|
Antonio Messina <antonio.s.messina@gmail.com>
|
|
.IP \[bu] 2
|
|
Stefan G.
|
|
Weichinger <office@oops.co.at>
|
|
.IP \[bu] 2
|
|
Per Cederberg <cederberg@gmail.com>
|
|
.IP \[bu] 2
|
|
Radek Šenfeld <rush@logic.cz>
|
|
.IP \[bu] 2
|
|
Fredrik Fornwall <fredrik@fornwall.net>
|
|
.IP \[bu] 2
|
|
Asko Tamm <asko@deekit.net>
|
|
.IP \[bu] 2
|
|
xor\-zz <xor@gstocco.com>
|
|
.IP \[bu] 2
|
|
Tomasz Mazur <tmazur90@gmail.com>
|
|
.IP \[bu] 2
|
|
Marco Paganini <paganini@paganini.net>
|
|
.IP \[bu] 2
|
|
Felix Bünemann <buenemann@louis.info>
|
|
.IP \[bu] 2
|
|
Durval Menezes <jmrclone@durval.com>
|
|
.IP \[bu] 2
|
|
Luiz Carlos Rumbelsperger Viana <maxd13_luiz_carlos@hotmail.com>
|
|
.IP \[bu] 2
|
|
Stefan Breunig <stefan-github@yrden.de>
|
|
.IP \[bu] 2
|
|
Alishan Ladhani <ali-l@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
0xJAKE <0xJAKE@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Thibault Molleman <thibaultmol@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Scott McGillivray <scott.mcgillivray@gmail.com>
|
|
.IP \[bu] 2
|
|
Bjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com>
|
|
.IP \[bu] 2
|
|
Lukas Loesche <lukas@mesosphere.io>
|
|
.IP \[bu] 2
|
|
emyarod <allllaboutyou@gmail.com>
|
|
.IP \[bu] 2
|
|
T.C.
|
|
Ferguson <tcf909@gmail.com>
|
|
.IP \[bu] 2
|
|
Brandur <brandur@mutelight.org>
|
|
.IP \[bu] 2
|
|
Dario Giovannetti <dev@dariogiovannetti.net>
|
|
.IP \[bu] 2
|
|
Károly Oláh <okaresz@aol.com>
|
|
.IP \[bu] 2
|
|
Jon Yergatian <jon@macfanatic.ca>
|
|
.IP \[bu] 2
|
|
Jack Schmidt <github@mowsey.org>
|
|
.IP \[bu] 2
|
|
Dedsec1 <Dedsec1@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Hisham Zarka <hzarka@gmail.com>
|
|
.IP \[bu] 2
|
|
Jérôme Vizcaino <jerome.vizcaino@gmail.com>
|
|
.IP \[bu] 2
|
|
Mike Tesch <mjt6129@rit.edu>
|
|
.IP \[bu] 2
|
|
Marvin Watson <marvwatson@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Danny Tsai <danny8376@gmail.com>
|
|
.IP \[bu] 2
|
|
Yoni Jah <yonjah+git@gmail.com> <yonjah+github@gmail.com>
|
|
.IP \[bu] 2
|
|
Stephen Harris <github@spuddy.org>
|
|
.IP \[bu] 2
|
|
Ihor Dvoretskyi <ihor.dvoretskyi@gmail.com>
|
|
.IP \[bu] 2
|
|
Jon Craton <jncraton@gmail.com>
|
|
.IP \[bu] 2
|
|
Hraban Luyat <hraban@0brg.net>
|
|
.IP \[bu] 2
|
|
Michael Ledin <mledin89@gmail.com>
|
|
.IP \[bu] 2
|
|
Martin Kristensen <me@azgul.com>
|
|
.IP \[bu] 2
|
|
Too Much IO <toomuchio@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Anisse Astier <anisse@astier.eu>
|
|
.IP \[bu] 2
|
|
Zahiar Ahmed <zahiar@live.com>
|
|
.IP \[bu] 2
|
|
Igor Kharin <igorkharin@gmail.com>
|
|
.IP \[bu] 2
|
|
Bill Zissimopoulos <billziss@navimatics.com>
|
|
.IP \[bu] 2
|
|
Bob Potter <bobby.potter@gmail.com>
|
|
.IP \[bu] 2
|
|
Steven Lu <tacticalazn@gmail.com>
|
|
.IP \[bu] 2
|
|
Sjur Fredriksen <sjurtf@ifi.uio.no>
|
|
.IP \[bu] 2
|
|
Ruwbin <hubus12345@gmail.com>
|
|
.IP \[bu] 2
|
|
Fabian Möller <fabianm88@gmail.com> <f.moeller@nynex.de>
|
|
.IP \[bu] 2
|
|
Edward Q.
|
|
Bridges <github@eqbridges.com>
|
|
.IP \[bu] 2
|
|
Vasiliy Tolstov <v.tolstov@selfip.ru>
|
|
.IP \[bu] 2
|
|
Harshavardhana <harsha@minio.io>
|
|
.IP \[bu] 2
|
|
sainaen <sainaen@gmail.com>
|
|
.IP \[bu] 2
|
|
gdm85 <gdm85@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Yaroslav Halchenko <debian@onerussian.com>
|
|
.IP \[bu] 2
|
|
John Papandriopoulos <jpap@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Zhiming Wang <zmwangx@gmail.com>
|
|
.IP \[bu] 2
|
|
Andy Pilate <cubox@cubox.me>
|
|
.IP \[bu] 2
|
|
Oliver Heyme <olihey@googlemail.com> <olihey@users.noreply.github.com>
|
|
<de8olihe@lego.com>
|
|
.IP \[bu] 2
|
|
wuyu <wuyu@yunify.com>
|
|
.IP \[bu] 2
|
|
Andrei Dragomir <adragomi@adobe.com>
|
|
.IP \[bu] 2
|
|
Christian Brüggemann <mail@cbruegg.com>
|
|
.IP \[bu] 2
|
|
Alex McGrath Kraak <amkdude@gmail.com>
|
|
.IP \[bu] 2
|
|
bpicode <bjoern.pirnay@googlemail.com>
|
|
.IP \[bu] 2
|
|
Daniel Jagszent <daniel@jagszent.de>
|
|
.IP \[bu] 2
|
|
Josiah White <thegenius2009@gmail.com>
|
|
.IP \[bu] 2
|
|
Ishuah Kariuki <kariuki@ishuah.com> <ishuah91@gmail.com>
|
|
.IP \[bu] 2
|
|
Jan Varho <jan@varho.org>
|
|
.IP \[bu] 2
|
|
Girish Ramakrishnan <girish@cloudron.io>
|
|
.IP \[bu] 2
|
|
LingMan <LingMan@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Jacob McNamee <jacobmcnamee@gmail.com>
|
|
.IP \[bu] 2
|
|
jersou <jertux@gmail.com>
|
|
.IP \[bu] 2
|
|
thierry <thierry@substantiel.fr>
|
|
.IP \[bu] 2
|
|
Simon Leinen <simon.leinen@gmail.com> <ubuntu@s3-test.novalocal>
|
|
.IP \[bu] 2
|
|
Dan Dascalescu <ddascalescu+github@gmail.com>
|
|
.IP \[bu] 2
|
|
Jason Rose <jason@jro.io>
|
|
.IP \[bu] 2
|
|
Andrew Starr\-Bochicchio <a.starr.b@gmail.com>
|
|
.IP \[bu] 2
|
|
John Leach <john@johnleach.co.uk>
|
|
.IP \[bu] 2
|
|
Corban Raun <craun@instructure.com>
|
|
.IP \[bu] 2
|
|
Pierre Carlson <mpcarl@us.ibm.com>
|
|
.IP \[bu] 2
|
|
Ernest Borowski <er.borowski@gmail.com>
|
|
.IP \[bu] 2
|
|
Remus Bunduc <remus.bunduc@gmail.com>
|
|
.IP \[bu] 2
|
|
Iakov Davydov <iakov.davydov@unil.ch> <dav05.gith@myths.ru>
|
|
.IP \[bu] 2
|
|
Jakub Tasiemski <tasiemski@gmail.com>
|
|
.IP \[bu] 2
|
|
David Minor <dminor@saymedia.com>
|
|
.IP \[bu] 2
|
|
Tim Cooijmans <cooijmans.tim@gmail.com>
|
|
.IP \[bu] 2
|
|
Laurence <liuxy6@gmail.com>
|
|
.IP \[bu] 2
|
|
Giovanni Pizzi <gio.piz@gmail.com>
|
|
.IP \[bu] 2
|
|
Filip Bartodziej <filipbartodziej@gmail.com>
|
|
.IP \[bu] 2
|
|
Jon Fautley <jon@dead.li>
|
|
.IP \[bu] 2
|
|
lewapm <32110057+lewapm@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Yassine Imounachen <yassine256@gmail.com>
|
|
.IP \[bu] 2
|
|
Chris Redekop <chris-redekop@users.noreply.github.com>
|
|
<chris.redekop@gmail.com>
|
|
.IP \[bu] 2
|
|
Jon Fautley <jon@adenoid.appstal.co.uk>
|
|
.IP \[bu] 2
|
|
Will Gunn <WillGunn@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Lucas Bremgartner <lucas@bremis.ch>
|
|
.IP \[bu] 2
|
|
Jody Frankowski <jody.frankowski@gmail.com>
|
|
.IP \[bu] 2
|
|
Andreas Roussos <arouss1980@gmail.com>
|
|
.IP \[bu] 2
|
|
nbuchanan <nbuchanan@utah.gov>
|
|
.IP \[bu] 2
|
|
Durval Menezes <rclone@durval.com>
|
|
.IP \[bu] 2
|
|
Victor <vb-github@viblo.se>
|
|
.IP \[bu] 2
|
|
Mateusz <pabian.mateusz@gmail.com>
|
|
.IP \[bu] 2
|
|
Daniel Loader <spicypixel@gmail.com>
|
|
.IP \[bu] 2
|
|
David0rk <davidork@gmail.com>
|
|
.IP \[bu] 2
|
|
Alexander Neumann <alexander@bumpern.de>
|
|
.IP \[bu] 2
|
|
Giri Badanahatti <gbadanahatti@us.ibm.com@Giris-MacBook-Pro.local>
|
|
.IP \[bu] 2
|
|
Leo R.
|
|
Lundgren <leo@finalresort.org>
|
|
.IP \[bu] 2
|
|
wolfv <wolfv6@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Dave Pedu <dave@davepedu.com>
|
|
.IP \[bu] 2
|
|
Stefan Lindblom <lindblom@spotify.com>
|
|
.IP \[bu] 2
|
|
seuffert <oliver@seuffert.biz>
|
|
.IP \[bu] 2
|
|
gbadanahatti <37121690+gbadanahatti@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Keith Goldfarb <barkofdelight@gmail.com>
|
|
.IP \[bu] 2
|
|
Steve Kriss <steve@heptio.com>
|
|
.IP \[bu] 2
|
|
Chih\-Hsuan Yen <yan12125@gmail.com>
|
|
.IP \[bu] 2
|
|
Alexander Neumann <fd0@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Matt Holt <mholt@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Eri Bastos <bastos.eri@gmail.com>
|
|
.IP \[bu] 2
|
|
Michael P.
|
|
Dubner <pywebmail@list.ru>
|
|
.IP \[bu] 2
|
|
Antoine GIRARD <sapk@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Mateusz Piotrowski <mpp302@gmail.com>
|
|
.IP \[bu] 2
|
|
Animosity022 <animosity22@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Peter Baumgartner <pete@lincolnloop.com>
|
|
.IP \[bu] 2
|
|
Craig Rachel <craig@craigrachel.com>
|
|
.IP \[bu] 2
|
|
Michael G.
|
|
Noll <miguno@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
hensur <me@hensur.de>
|
|
.IP \[bu] 2
|
|
Oliver Heyme <de8olihe@lego.com>
|
|
.IP \[bu] 2
|
|
Richard Yang <richard@yenforyang.com>
|
|
.IP \[bu] 2
|
|
Piotr Oleszczyk <piotr.oleszczyk@gmail.com>
|
|
.IP \[bu] 2
|
|
Rodrigo <rodarima@gmail.com>
|
|
.IP \[bu] 2
|
|
NoLooseEnds <NoLooseEnds@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
Jakub Karlicek <jakub@karlicek.me>
|
|
.IP \[bu] 2
|
|
John Clayton <john@codemonkeylabs.com>
|
|
.IP \[bu] 2
|
|
Kasper Byrdal Nielsen <byrdal76@gmail.com>
|
|
.IP \[bu] 2
|
|
Benjamin Joseph Dag <bjdag1234@users.noreply.github.com>
|
|
.IP \[bu] 2
|
|
themylogin <themylogin@gmail.com>
|
|
.SH Contact the rclone project
|
|
.SS Forum
|
|
.PP
|
|
Forum for general discussions and questions:
|
|
.IP \[bu] 2
|
|
https://forum.rclone.org
|
|
.SS Gitub project
|
|
.PP
|
|
The project website is at:
|
|
.IP \[bu] 2
|
|
https://github.com/ncw/rclone
|
|
.PP
|
|
There you can file bug reports, ask for help or contribute pull
|
|
requests.
|
|
.SS Google+
|
|
.PP
|
|
Rclone has a Google+ page which announcements are posted to
|
|
.IP \[bu] 2
|
|
Google+ page for general comments
|
|
.SS Twitter
|
|
.PP
|
|
You can also follow me on twitter for rclone announcements
|
|
.IP \[bu] 2
|
|
[\@njcw](https://twitter.com/njcw)
|
|
.SS Email
|
|
.PP
|
|
Or if all else fails or you want to ask something private or
|
|
confidential email Nick Craig\-Wood (mailto:nick@craig-wood.com)
|
|
.SH AUTHORS
|
|
Nick Craig\-Wood.
|