TrueCloudLab/rclone
16
0
Fork 4
Code Issues 3 Pull requests 2 Projects Releases 2 Packages Wiki Activity Actions

Version v1.53.2

Browse source
This commit is contained in:
Nick Craig-Wood 2020-10-26 13:26:58 +00:00
parent cfc5d76fca
commit 89f2d43f17
8 changed files with 1036 additions and 321 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

249
MANUAL.html generated
View file

@ -17,7 +17,7 @@
<header id="title-block-header"> <header id="title-block-header">
<h1 class="title">rclone(1) User Manual</h1> <h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p> <p class="author">Nick Craig-Wood</p>
<p class="date">Sep 13, 2020</p> <p class="date">Oct 26, 2020</p>
</header> </header>
<h1 id="rclone-syncs-your-files-to-cloud-storage">Rclone syncs your files to cloud storage</h1> <h1 id="rclone-syncs-your-files-to-cloud-storage">Rclone syncs your files to cloud storage</h1>
<p><img width="50%" src="https://rclone.org/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" ></p> <p><img width="50%" src="https://rclone.org/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" ></p>
@ -1455,7 +1455,6 @@ umount /path/to/local/mount</code></pre>
<p>--vfs-read-chunk-size will enable reading the source objects in parts. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read at the cost of an increased number of requests.</p> <p>--vfs-read-chunk-size will enable reading the source objects in parts. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read at the cost of an increased number of requests.</p>
<p>When --vfs-read-chunk-size-limit is also specified and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled for each chunk read, until the specified value is reached. A value of -1 will disable the limit and the chunk size will grow indefinitely.</p> <p>When --vfs-read-chunk-size-limit is also specified and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled for each chunk read, until the specified value is reached. A value of -1 will disable the limit and the chunk size will grow indefinitely.</p>
<p>With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.</p> <p>With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.</p>
<p>Chunked reading will only work with --vfs-cache-mode &lt; full, as the file will always be copied to the vfs cache before opening with --vfs-cache-mode full.</p>
<h2 id="vfs---virtual-file-system">VFS - Virtual File System</h2> <h2 id="vfs---virtual-file-system">VFS - Virtual File System</h2>
<p>This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.</p> <p>This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.</p>
<p>Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.</p> <p>Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.</p>
@ -3397,8 +3396,10 @@ y/n/s/!/q&gt; n</code></pre>
<p>The remote in use must support server side move or copy and you must use the same remote as the destination of the sync.</p> <p>The remote in use must support server side move or copy and you must use the same remote as the destination of the sync.</p>
<p>This is for use with files to add the suffix in the current directory or with <code>--backup-dir</code>. See <code>--backup-dir</code> for more info.</p> <p>This is for use with files to add the suffix in the current directory or with <code>--backup-dir</code>. See <code>--backup-dir</code> for more info.</p>
<p>For example</p> <p>For example</p>
<pre><code>rclone sync -i /path/to/local/file remote:current --suffix .bak</code></pre> <pre><code>rclone copy -i /path/to/local/file remote:current --suffix .bak</code></pre>
<p>will sync <code>/path/to/local</code> to <code>remote:current</code>, but for any files which would have been updated or deleted have .bak added.</p> <p>will copy <code>/path/to/local</code> to <code>remote:current</code>, but for any files which would have been updated or deleted have .bak added.</p>
<p>If using <code>rclone sync</code> with <code>--suffix</code> and without <code>--backup-dir</code> then it is recommended to put a filter rule in excluding the suffix otherwise the <code>sync</code> will delete the backup files.</p>
<pre><code>rclone sync -i /path/to/local/file remote:current --suffix .bak --exclude &quot;*.bak&quot;</code></pre>
<h3 id="suffix-keep-extension">--suffix-keep-extension</h3> <h3 id="suffix-keep-extension">--suffix-keep-extension</h3>
<p>When using <code>--suffix</code>, setting this causes rclone put the SUFFIX before the extension of the files that it backs up rather than after.</p> <p>When using <code>--suffix</code>, setting this causes rclone put the SUFFIX before the extension of the files that it backs up rather than after.</p>
<p>So let's say we had <code>--suffix -2019-01-01</code>, without the flag <code>file.txt</code> would be backed up to <code>file.txt-2019-01-01</code> and with the flag it would be backed up to <code>file-2019-01-01.txt</code>. This can be helpful to make sure the suffixed files can still be opened.</p> <p>So let's say we had <code>--suffix -2019-01-01</code>, without the flag <code>file.txt</code> would be backed up to <code>file.txt-2019-01-01</code> and with the flag it would be backed up to <code>file-2019-01-01.txt</code>. This can be helpful to make sure the suffixed files can still be opened.</p>
@ -6290,7 +6291,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--use-json-log Use json log format. --use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs). --use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata --use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.53.1&quot;) --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.53.2&quot;)
-v, --verbose count Print lots more stuff (repeat for more)</code></pre> -v, --verbose count Print lots more stuff (repeat for more)</code></pre>
<h2 id="backend-flags">Backend Flags</h2> <h2 id="backend-flags">Backend Flags</h2>
<p>These flags are available for every command. They control the backends and may be set in the config file.</p> <p>These flags are available for every command. They control the backends and may be set in the config file.</p>
@ -7178,7 +7179,7 @@ Choose a number from below, or type in your own value
/ Asia Pacific (Mumbai) / Asia Pacific (Mumbai)
13 | Needs location constraint ap-south-1. 13 | Needs location constraint ap-south-1.
\ &quot;ap-south-1&quot; \ &quot;ap-south-1&quot;
/ Asia Patific (Hong Kong) Region / Asia Pacific (Hong Kong) Region
14 | Needs location constraint ap-east-1. 14 | Needs location constraint ap-east-1.
\ &quot;ap-east-1&quot; \ &quot;ap-east-1&quot;
/ South America (Sao Paulo) Region / South America (Sao Paulo) Region
@ -7553,16 +7554,16 @@ y/e/d&gt; </code></pre>
<li>US East (Ohio) Region</li> <li>US East (Ohio) Region</li>
<li>Needs location constraint us-east-2.</li> <li>Needs location constraint us-east-2.</li>
</ul></li> </ul></li>
<li>"us-west-2"
<ul>
<li>US West (Oregon) Region</li>
<li>Needs location constraint us-west-2.</li>
</ul></li>
<li>"us-west-1" <li>"us-west-1"
<ul> <ul>
<li>US West (Northern California) Region</li> <li>US West (Northern California) Region</li>
<li>Needs location constraint us-west-1.</li> <li>Needs location constraint us-west-1.</li>
</ul></li> </ul></li>
<li>"us-west-2"
<ul>
<li>US West (Oregon) Region</li>
<li>Needs location constraint us-west-2.</li>
</ul></li>
<li>"ca-central-1" <li>"ca-central-1"
<ul> <ul>
<li>Canada (Central) Region</li> <li>Canada (Central) Region</li>
@ -7578,11 +7579,21 @@ y/e/d&gt; </code></pre>
<li>EU (London) Region</li> <li>EU (London) Region</li>
<li>Needs location constraint eu-west-2.</li> <li>Needs location constraint eu-west-2.</li>
</ul></li> </ul></li>
<li>"eu-west-3"
<ul>
<li>EU (Paris) Region</li>
<li>Needs location constraint eu-west-3.</li>
</ul></li>
<li>"eu-north-1" <li>"eu-north-1"
<ul> <ul>
<li>EU (Stockholm) Region</li> <li>EU (Stockholm) Region</li>
<li>Needs location constraint eu-north-1.</li> <li>Needs location constraint eu-north-1.</li>
</ul></li> </ul></li>
<li>"eu-south-1"
<ul>
<li>EU (Milan) Region</li>
<li>Needs location constraint eu-south-1.</li>
</ul></li>
<li>"eu-central-1" <li>"eu-central-1"
<ul> <ul>
<li>EU (Frankfurt) Region</li> <li>EU (Frankfurt) Region</li>
@ -7608,6 +7619,11 @@ y/e/d&gt; </code></pre>
<li>Asia Pacific (Seoul)</li> <li>Asia Pacific (Seoul)</li>
<li>Needs location constraint ap-northeast-2.</li> <li>Needs location constraint ap-northeast-2.</li>
</ul></li> </ul></li>
<li>"ap-northeast-3"
<ul>
<li>Asia Pacific (Osaka-Local)</li>
<li>Needs location constraint ap-northeast-3.</li>
</ul></li>
<li>"ap-south-1" <li>"ap-south-1"
<ul> <ul>
<li>Asia Pacific (Mumbai)</li> <li>Asia Pacific (Mumbai)</li>
@ -7615,7 +7631,7 @@ y/e/d&gt; </code></pre>
</ul></li> </ul></li>
<li>"ap-east-1" <li>"ap-east-1"
<ul> <ul>
<li>Asia Patific (Hong Kong) Region</li> <li>Asia Pacific (Hong Kong) Region</li>
<li>Needs location constraint ap-east-1.</li> <li>Needs location constraint ap-east-1.</li>
</ul></li> </ul></li>
<li>"sa-east-1" <li>"sa-east-1"
@ -7623,6 +7639,36 @@ y/e/d&gt; </code></pre>
<li>South America (Sao Paulo) Region</li> <li>South America (Sao Paulo) Region</li>
<li>Needs location constraint sa-east-1.</li> <li>Needs location constraint sa-east-1.</li>
</ul></li> </ul></li>
<li>"me-south-1"
<ul>
<li>Middle East (Bahrain) Region</li>
<li>Needs location constraint me-south-1.</li>
</ul></li>
<li>"af-south-1"
<ul>
<li>Africa (Cape Town) Region</li>
<li>Needs location constraint af-south-1.</li>
</ul></li>
<li>"cn-north-1"
<ul>
<li>China (Beijing) Region</li>
<li>Needs location constraint cn-north-1.</li>
</ul></li>
<li>"cn-northwest-1"
<ul>
<li>China (Ningxia) Region</li>
<li>Needs location constraint cn-northwest-1.</li>
</ul></li>
<li>"us-gov-east-1"
<ul>
<li>AWS GovCloud (US-East) Region</li>
<li>Needs location constraint us-gov-east-1.</li>
</ul></li>
<li>"us-gov-west-1"
<ul>
<li>AWS GovCloud (US) Region</li>
<li>Needs location constraint us-gov-west-1.</li>
</ul></li>
</ul></li> </ul></li>
</ul> </ul>
<h4 id="s3-region-1">--s3-region</h4> <h4 id="s3-region-1">--s3-region</h4>
@ -8202,14 +8248,14 @@ y/e/d&gt; </code></pre>
<ul> <ul>
<li>US East (Ohio) Region.</li> <li>US East (Ohio) Region.</li>
</ul></li> </ul></li>
<li>"us-west-2"
<ul>
<li>US West (Oregon) Region.</li>
</ul></li>
<li>"us-west-1" <li>"us-west-1"
<ul> <ul>
<li>US West (Northern California) Region.</li> <li>US West (Northern California) Region.</li>
</ul></li> </ul></li>
<li>"us-west-2"
<ul>
<li>US West (Oregon) Region.</li>
</ul></li>
<li>"ca-central-1" <li>"ca-central-1"
<ul> <ul>
<li>Canada (Central) Region.</li> <li>Canada (Central) Region.</li>
@ -8222,10 +8268,18 @@ y/e/d&gt; </code></pre>
<ul> <ul>
<li>EU (London) Region.</li> <li>EU (London) Region.</li>
</ul></li> </ul></li>
<li>"eu-west-3"
<ul>
<li>EU (Paris) Region.</li>
</ul></li>
<li>"eu-north-1" <li>"eu-north-1"
<ul> <ul>
<li>EU (Stockholm) Region.</li> <li>EU (Stockholm) Region.</li>
</ul></li> </ul></li>
<li>"eu-south-1"
<ul>
<li>EU (Milan) Region.</li>
</ul></li>
<li>"EU" <li>"EU"
<ul> <ul>
<li>EU Region.</li> <li>EU Region.</li>
@ -8244,20 +8298,48 @@ y/e/d&gt; </code></pre>
</ul></li> </ul></li>
<li>"ap-northeast-2" <li>"ap-northeast-2"
<ul> <ul>
<li>Asia Pacific (Seoul)</li> <li>Asia Pacific (Seoul) Region.</li>
</ul></li>
<li>"ap-northeast-3"
<ul>
<li>Asia Pacific (Osaka-Local) Region.</li>
</ul></li> </ul></li>
<li>"ap-south-1" <li>"ap-south-1"
<ul> <ul>
<li>Asia Pacific (Mumbai)</li> <li>Asia Pacific (Mumbai) Region.</li>
</ul></li> </ul></li>
<li>"ap-east-1" <li>"ap-east-1"
<ul> <ul>
<li>Asia Pacific (Hong Kong)</li> <li>Asia Pacific (Hong Kong) Region.</li>
</ul></li> </ul></li>
<li>"sa-east-1" <li>"sa-east-1"
<ul> <ul>
<li>South America (Sao Paulo) Region.</li> <li>South America (Sao Paulo) Region.</li>
</ul></li> </ul></li>
<li>"me-south-1"
<ul>
<li>Middle East (Bahrain) Region.</li>
</ul></li>
<li>"af-south-1"
<ul>
<li>Africa (Cape Town) Region.</li>
</ul></li>
<li>"cn-north-1"
<ul>
<li>China (Beijing) Region</li>
</ul></li>
<li>"cn-northwest-1"
<ul>
<li>China (Ningxia) Region.</li>
</ul></li>
<li>"us-gov-east-1"
<ul>
<li>AWS GovCloud (US-East) Region.</li>
</ul></li>
<li>"us-gov-west-1"
<ul>
<li>AWS GovCloud (US) Region.</li>
</ul></li>
</ul></li> </ul></li>
</ul> </ul>
<h4 id="s3-location-constraint-1">--s3-location-constraint</h4> <h4 id="s3-location-constraint-1">--s3-location-constraint</h4>
@ -10194,7 +10276,7 @@ y/e/d&gt; y</code></pre>
</ul> </ul>
<h3 id="limitations-3">Limitations</h3> <h3 id="limitations-3">Limitations</h3>
<p>Note that Box is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".</p> <p>Note that Box is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".</p>
<p>Box file names can't have the <code>\</code> character in. rclone maps this to and from an identical looking unicode equivalent <code>＼</code>.</p> <p>Box file names can't have the <code>\</code> character in. rclone maps this to and from an identical looking unicode equivalent <code>＼</code> (U+FF3C Fullwidth Reverse Solidus).</p>
<p>Box only supports filenames up to 255 characters in length.</p> <p>Box only supports filenames up to 255 characters in length.</p>
<h2 id="cache-beta">Cache (BETA)</h2> <h2 id="cache-beta">Cache (BETA)</h2>
<p>The <code>cache</code> remote wraps another existing remote and stores file structure and its data for long running tasks like <code>rclone mount</code>.</p> <p>The <code>cache</code> remote wraps another existing remote and stores file structure and its data for long running tasks like <code>rclone mount</code>.</p>
@ -11058,10 +11140,11 @@ y/e/d&gt; y</code></pre>
<li>Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot</li> <li>Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot</li>
</ul> </ul>
<h2 id="crypt">Crypt</h2> <h2 id="crypt">Crypt</h2>
<p>The <code>crypt</code> remote encrypts and decrypts another remote.</p> <p>Rclone <code>crypt</code> remotes encrypt and decrypt other remotes.</p>
<p>To use it first set up the underlying remote following the config instructions for that remote. You can also use a local pathname instead of a remote which will encrypt and decrypt from that directory which might be useful for encrypting onto a USB stick for example.</p> <p>To use <code>crypt</code>, first set up the underlying remote. Follow the <code>rclone config</code> instructions for that remote.</p>
<p>First check your chosen remote is working - we'll call it <code>remote:path</code> in these docs. Note that anything inside <code>remote:path</code> will be encrypted and anything outside won't. This means that if you are using a bucket based remote (eg S3, B2, swift) then you should probably put the bucket in the remote <code>s3:bucket</code>. If you just use <code>s3:</code> then rclone will make encrypted bucket names too (if using file name encryption) which may or may not be what you want.</p> <p><code>crypt</code> applied to a local pathname instead of a remote will encrypt and decrypt that directory, and can be used to encrypt USB removable drives.</p>
<p>Now configure <code>crypt</code> using <code>rclone config</code>. We will call this one <code>secret</code> to differentiate it from the <code>remote</code>.</p> <p>Before configuring the crypt remote, check the underlying remote is working. In this example the underlying remote is called <code>remote:path</code>. Anything inside <code>remote:path</code> will be encrypted and anything outside will not. In the case of an S3 based underlying remote (eg Amazon S3, B2, Swift) it is generally advisable to define a crypt remote in the underlying remote <code>s3:bucket</code>. If <code>s3:</code> alone is specified alongside file name encryption, rclone will encrypt the bucket name.</p>
<p>Configure <code>crypt</code> using <code>rclone config</code>. In this example the <code>crypt</code> remote is called <code>secret</code>, to differentiate it from the underlying <code>remote</code>.</p>
<pre><code>No remotes found - make a new one <pre><code>No remotes found - make a new one
n) New remote n) New remote
s) Set configuration password s) Set configuration password
@ -11131,21 +11214,20 @@ y) Yes this is OK
e) Edit this remote e) Edit this remote
d) Delete this remote d) Delete this remote
y/e/d&gt; y</code></pre> y/e/d&gt; y</code></pre>
<p><strong>Important</strong> The password is stored in the config file is lightly obscured so it isn't immediately obvious what it is. It is in no way secure unless you use config file encryption.</p> <p><strong>Important</strong> The crypt password stored in <code>rclone.conf</code> is lightly obscured. That only protects it from cursory inspection. It is not secure unless encryption of <code>rclone.conf</code> is specified.</p>
<p>A long passphrase is recommended, or you can use a random one.</p> <p>A long passphrase is recommended, or <code>rclone config</code> can generate a random one.</p>
<p>The obscured password is created by using AES-CTR with a static key, with the salt stored verbatim at the beginning of the obscured password. This static key is shared by between all versions of rclone.</p> <p>The obscured password is created using AES-CTR with a static key. The salt is stored verbatim at the beginning of the obscured password. This static key is shared between all versions of rclone.</p>
<p>If you reconfigure rclone with the same passwords/passphrases elsewhere it will be compatible, but the obscured version will be different due to the different salt.</p> <p>If you reconfigure rclone with the same passwords/passphrases elsewhere it will be compatible, but the obscured version will be different due to the different salt.</p>
<p>Note that rclone does not encrypt</p> <p>Rclone does not encrypt</p>
<ul> <ul>
<li>file length - this can be calculated within 16 bytes</li> <li>file length - this can be calculated within 16 bytes</li>
<li>modification time - used for syncing</li> <li>modification time - used for syncing</li>
</ul> </ul>
<h2 id="specifying-the-remote-1">Specifying the remote</h2> <h2 id="specifying-the-remote-1">Specifying the remote</h2>
<p>In normal use, make sure the remote has a <code>:</code> in. If you specify the remote without a <code>:</code> then rclone will use a local directory of that name. So if you use a remote of <code>/path/to/secret/files</code> then rclone will encrypt stuff to that directory. If you use a remote of <code>name</code> then rclone will put files in a directory called <code>name</code> in the current directory.</p> <p>In normal use, ensure the remote has a <code>:</code> in. If specified without, rclone uses a local directory of that name. For example if a remote <code>/path/to/secret/files</code> is specified, rclone encrypts content to that directory. If a remote <code>name</code> is specified, rclone targets a directory <code>name</code> in the current directory.</p>
<p>If you specify the remote as <code>remote:path/to/dir</code> then rclone will store encrypted files in <code>path/to/dir</code> on the remote. If you are using file name encryption, then when you save files to <code>secret:subdir/subfile</code> this will store them in the unencrypted path <code>path/to/dir</code> but the <code>subdir/subpath</code> bit will be encrypted.</p> <p>If remote <code>remote:path/to/dir</code> is specified, rclone stores encrypted files in <code>path/to/dir</code> on the remote. With file name encryption, files saved to <code>secret:subdir/subfile</code> are stored in the unencrypted path <code>path/to/dir</code> but the <code>subdir/subpath</code> element is encrypted.</p>
<p>Note that unless you want encrypted bucket names (which are difficult to manage because you won't know what directory they represent in web interfaces etc), you should probably specify a bucket, eg <code>remote:secretbucket</code> when using bucket based remotes such as S3, Swift, Hubic, B2, GCS.</p>
<h2 id="example">Example</h2> <h2 id="example">Example</h2>
<p>To test I made a little directory of files using "standard" file name encryption.</p> <p>Create the following file structure using "standard" file name encryption.</p>
<pre><code>plaintext/ <pre><code>plaintext/
├── file0.txt ├── file0.txt
├── file1.txt ├── file1.txt
@ -11154,7 +11236,7 @@ y/e/d&gt; y</code></pre>
├── file3.txt ├── file3.txt
└── subsubdir └── subsubdir
└── file4.txt</code></pre> └── file4.txt</code></pre>
<p>Copy these to the remote and list them back</p> <p>Copy these to the remote, and list them</p>
<pre><code>$ rclone -q copy plaintext secret: <pre><code>$ rclone -q copy plaintext secret:
$ rclone -q ls secret: $ rclone -q ls secret:
7 file1.txt 7 file1.txt
@ -11162,19 +11244,19 @@ $ rclone -q ls secret:
8 subdir/file2.txt 8 subdir/file2.txt
10 subdir/subsubdir/file4.txt 10 subdir/subsubdir/file4.txt
9 subdir/file3.txt</code></pre> 9 subdir/file3.txt</code></pre>
<p>Now see what that looked like when encrypted</p> <p>The crypt remote looks like</p>
<pre><code>$ rclone -q ls remote:path <pre><code>$ rclone -q ls remote:path
55 hagjclgavj2mbiqm6u6cnjjqcg 55 hagjclgavj2mbiqm6u6cnjjqcg
54 v05749mltvv1tf4onltun46gls 54 v05749mltvv1tf4onltun46gls
57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps</code></pre> 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps</code></pre>
<p>Note that this retains the directory structure which means you can do this</p> <p>The directory structure is preserved</p>
<pre><code>$ rclone -q ls secret:subdir <pre><code>$ rclone -q ls secret:subdir
8 file2.txt 8 file2.txt
9 file3.txt 9 file3.txt
10 subsubdir/file4.txt</code></pre> 10 subsubdir/file4.txt</code></pre>
<p>If don't use file name encryption then the remote will look like this - note the <code>.bin</code> extensions added to prevent the cloud provider attempting to interpret the data.</p> <p>Without file name encryption <code>.bin</code> extensions are added to underlying names. This prevents the cloud provider attempting to interpret file content.</p>
<pre><code>$ rclone -q ls remote:path <pre><code>$ rclone -q ls remote:path
54 file0.txt.bin 54 file0.txt.bin
57 subdir/file3.txt.bin 57 subdir/file3.txt.bin
@ -11182,7 +11264,6 @@ $ rclone -q ls secret:
58 subdir/subsubdir/file4.txt.bin 58 subdir/subsubdir/file4.txt.bin
55 file1.txt.bin</code></pre> 55 file1.txt.bin</code></pre>
<h3 id="file-name-encryption-modes">File name encryption modes</h3> <h3 id="file-name-encryption-modes">File name encryption modes</h3>
<p>Here are some of the features of the file name encryption modes</p>
<p>Off</p> <p>Off</p>
<ul> <ul>
<li>doesn't hide file names or directory structure</li> <li>doesn't hide file names or directory structure</li>
@ -11199,9 +11280,10 @@ $ rclone -q ls secret:
<li>can use shortcuts to shorten the directory recursion</li> <li>can use shortcuts to shorten the directory recursion</li>
</ul> </ul>
<p>Obfuscation</p> <p>Obfuscation</p>
<p>This is a simple "rotate" of the filename, with each file having a rot distance based on the filename. We store the distance at the beginning of the filename. So a file called "hello" may become "53.jgnnq".</p> <p>This is a simple "rotate" of the filename, with each file having a rot distance based on the filename. Rclone stores the distance at the beginning of the filename. A file called "hello" may become "53.jgnnq".</p>
<p>This is not a strong encryption of filenames, but it may stop automated scanning tools from picking up on filename patterns. As such it's an intermediate between "off" and "standard". The advantage is that it allows for longer path segment names.</p> <p>Obfuscation is not a strong encryption of filenames, but hinders automated scanning tools picking up on filename patterns. It is an intermediate between "off" and "standard" which allows for longer path segment names.</p>
<p>There is a possibility with some unicode based filenames that the obfuscation is weak and may map lower case characters to upper case equivalents. You can not rely on this for strong protection.</p> <p>There is a possibility with some unicode based filenames that the obfuscation is weak and may map lower case characters to upper case equivalents.</p>
<p>Obfuscation cannot be relied upon for strong protection.</p>
<ul> <ul>
<li>file names very lightly obfuscated</li> <li>file names very lightly obfuscated</li>
<li>file names can be longer than standard encryption</li> <li>file names can be longer than standard encryption</li>
@ -11209,8 +11291,8 @@ $ rclone -q ls secret:
<li>directory structure visible</li> <li>directory structure visible</li>
<li>identical files names will have identical uploaded names</li> <li>identical files names will have identical uploaded names</li>
</ul> </ul>
<p>Cloud storage systems have various limits on file name length and total path length which you are more likely to hit using "Standard" file name encryption. If you keep your file names to below 156 characters in length then you should be OK on all providers.</p> <p>Cloud storage systems have limits on file name length and total path length which rclone is more likely to breach using "Standard" file name encryption. Where file names are less thn 156 characters in length issues should not be encountered, irrespective of cloud storage provider.</p>
<p>There may be an even more secure file name encryption mode in the future which will address the long file name problem.</p> <p>An alternative, future rclone file name encryption mode may tolerate backend provider path length limits.</p>
<h3 id="directory-name-encryption">Directory name encryption</h3> <h3 id="directory-name-encryption">Directory name encryption</h3>
<p>Crypt offers the option of encrypting dir names or leaving them intact. There are two options:</p> <p>Crypt offers the option of encrypting dir names or leaving them intact. There are two options:</p>
<p>True</p> <p>True</p>
@ -11220,7 +11302,7 @@ $ rclone -q ls secret:
<h3 id="modified-time-and-hashes-3">Modified time and hashes</h3> <h3 id="modified-time-and-hashes-3">Modified time and hashes</h3>
<p>Crypt stores modification times using the underlying remote so support depends on that.</p> <p>Crypt stores modification times using the underlying remote so support depends on that.</p>
<p>Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.</p> <p>Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.</p>
<p>Note that you should use the <code>rclone cryptcheck</code> command to check the integrity of a crypted remote instead of <code>rclone check</code> which can't check the checksums properly.</p> <p>Use the <code>rclone cryptcheck</code> command to check the integrity of a crypted remote instead of <code>rclone check</code> which can't check the checksums properly.</p>
<h3 id="standard-options-9">Standard Options</h3> <h3 id="standard-options-9">Standard Options</h3>
<p>Here are the standard options specific to crypt (Encrypt/Decrypt a remote).</p> <p>Here are the standard options specific to crypt (Encrypt/Decrypt a remote).</p>
<h4 id="crypt-remote">--crypt-remote</h4> <h4 id="crypt-remote">--crypt-remote</h4>
@ -18470,6 +18552,87 @@ $ tree /tmp/b
<li>"error": return an error based on option value</li> <li>"error": return an error based on option value</li>
</ul> </ul>
<h1 id="changelog">Changelog</h1> <h1 id="changelog">Changelog</h1>
<h2 id="v1.53.2---2020-10-26">v1.53.2 - 2020-10-26</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.53.1...v1.53.2">See commits</a></p>
<ul>
<li>Bug Fixes
<ul>
<li>acounting
<ul>
<li>Fix incorrect speed and transferTime in core/stats (Nick Craig-Wood)</li>
<li>Stabilize display order of transfers on Windows (Nick Craig-Wood)</li>
</ul></li>
<li>operations
<ul>
<li>Fix use of --suffix without --backup-dir (Nick Craig-Wood)</li>
<li>Fix spurious "--checksum is in use but the source and destination have no hashes in common" (Nick Craig-Wood)</li>
</ul></li>
<li>build
<ul>
<li>Work around GitHub actions brew problem (Nick Craig-Wood)</li>
<li>Stop using set-env and set-path in the GitHub actions (Nick Craig-Wood)</li>
</ul></li>
</ul></li>
<li>Mount
<ul>
<li>mount2: Fix the swapped UID / GID values (Russell Cattelan)</li>
</ul></li>
<li>VFS
<ul>
<li>Detect and recover from a file being removed externally from the cache (Nick Craig-Wood)</li>
<li>Fix a deadlock vulnerability in downloaders.Close (Leo Luan)</li>
<li>Fix a race condition in retryFailedResets (Leo Luan)</li>
<li>Fix missed concurrency control between some item operations and reset (Leo Luan)</li>
<li>Add exponential backoff during ENOSPC retries (Leo Luan)</li>
<li>Add a missed update of used cache space (Leo Luan)</li>
<li>Fix --no-modtime to not attempt to set modtimes (as documented) (Nick Craig-Wood)</li>
</ul></li>
<li>Local
<ul>
<li>Fix sizes and syncing with --links option on Windows (Nick Craig-Wood)</li>
</ul></li>
<li>Chunker
<ul>
<li>Disable ListR to fix missing files on GDrive (workaround) (Ivan Andreev)</li>
<li>Fix upload over crypt (Ivan Andreev)</li>
</ul></li>
<li>Fichier
<ul>
<li>Increase maximum file size from 100GB to 300GB (gyutw)</li>
</ul></li>
<li>Jottacloud
<ul>
<li>Remove clientSecret from config when upgrading to token based authentication (buengese)</li>
<li>Avoid double url escaping of device/mountpoint (albertony)</li>
<li>Remove DirMove workaround as it's not required anymore - also (buengese)</li>
</ul></li>
<li>Mailru
<ul>
<li>Fix uploads after recent changes on server (Ivan Andreev)</li>
<li>Fix range requests after june changes on server (Ivan Andreev)</li>
<li>Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)</li>
</ul></li>
<li>Onedrive
<ul>
<li>Fix disk usage for sharepoint (Nick Craig-Wood)</li>
</ul></li>
<li>S3
<ul>
<li>Add missing regions for AWS (Anagh Kumar Baranwal)</li>
</ul></li>
<li>Seafile
<ul>
<li>Fix accessing libraries &gt; 2GB on 32 bit systems (Muffin King)</li>
</ul></li>
<li>SFTP
<ul>
<li>Always convert the checksum to lower case (buengese)</li>
</ul></li>
<li>Union
<ul>
<li>Create root directories if none exist (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.53.1---2020-09-13">v1.53.1 - 2020-09-13</h2> <h2 id="v1.53.1---2020-09-13">v1.53.1 - 2020-09-13</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1">See commits</a></p> <p><a href="https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1">See commits</a></p>
<ul> <ul>

258
MANUAL.md generated
View file

@ -1,6 +1,6 @@
% rclone(1) User Manual % rclone(1) User Manual
% Nick Craig-Wood % Nick Craig-Wood
% Sep 13, 2020 % Oct 26, 2020
# Rclone syncs your files to cloud storage # Rclone syncs your files to cloud storage
@ -2891,9 +2891,6 @@ parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
When --vfs-read-chunk-size-limit 500M is specified, the result would be When --vfs-read-chunk-size-limit 500M is specified, the result would be
0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Chunked reading will only work with --vfs-cache-mode < full, as the file will always
be copied to the vfs cache before opening with --vfs-cache-mode full.
## VFS - Virtual File System ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects This command uses the VFS layer. This adapts the cloud storage objects
@ -7033,11 +7030,17 @@ or with `--backup-dir`. See `--backup-dir` for more info.
For example For example
rclone sync -i /path/to/local/file remote:current --suffix .bak rclone copy -i /path/to/local/file remote:current --suffix .bak
will sync `/path/to/local` to `remote:current`, but for any files will copy `/path/to/local` to `remote:current`, but for any files
which would have been updated or deleted have .bak added. which would have been updated or deleted have .bak added.
If using `rclone sync` with `--suffix` and without `--backup-dir` then
it is recommended to put a filter rule in excluding the suffix
otherwise the `sync` will delete the backup files.
rclone sync -i /path/to/local/file remote:current --suffix .bak --exclude "*.bak"
### --suffix-keep-extension ### ### --suffix-keep-extension ###
When using `--suffix`, setting this causes rclone put the SUFFIX When using `--suffix`, setting this causes rclone put the SUFFIX
@ -10566,7 +10569,7 @@ These flags are available for every command.
--use-json-log Use json log format. --use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs). --use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata --use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.1") --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.2")
-v, --verbose count Print lots more stuff (repeat for more) -v, --verbose count Print lots more stuff (repeat for more)
``` ```
@ -11635,7 +11638,7 @@ Choose a number from below, or type in your own value
/ Asia Pacific (Mumbai) / Asia Pacific (Mumbai)
13 | Needs location constraint ap-south-1. 13 | Needs location constraint ap-south-1.
\ "ap-south-1" \ "ap-south-1"
/ Asia Patific (Hong Kong) Region / Asia Pacific (Hong Kong) Region
14 | Needs location constraint ap-east-1. 14 | Needs location constraint ap-east-1.
\ "ap-east-1" \ "ap-east-1"
/ South America (Sao Paulo) Region / South America (Sao Paulo) Region
@ -12041,12 +12044,12 @@ Region to connect to.
- "us-east-2" - "us-east-2"
- US East (Ohio) Region - US East (Ohio) Region
- Needs location constraint us-east-2. - Needs location constraint us-east-2.
- "us-west-2"
- US West (Oregon) Region
- Needs location constraint us-west-2.
- "us-west-1" - "us-west-1"
- US West (Northern California) Region - US West (Northern California) Region
- Needs location constraint us-west-1. - Needs location constraint us-west-1.
- "us-west-2"
- US West (Oregon) Region
- Needs location constraint us-west-2.
- "ca-central-1" - "ca-central-1"
- Canada (Central) Region - Canada (Central) Region
- Needs location constraint ca-central-1. - Needs location constraint ca-central-1.
@ -12056,9 +12059,15 @@ Region to connect to.
- "eu-west-2" - "eu-west-2"
- EU (London) Region - EU (London) Region
- Needs location constraint eu-west-2. - Needs location constraint eu-west-2.
- "eu-west-3"
- EU (Paris) Region
- Needs location constraint eu-west-3.
- "eu-north-1" - "eu-north-1"
- EU (Stockholm) Region - EU (Stockholm) Region
- Needs location constraint eu-north-1. - Needs location constraint eu-north-1.
- "eu-south-1"
- EU (Milan) Region
- Needs location constraint eu-south-1.
- "eu-central-1" - "eu-central-1"
- EU (Frankfurt) Region - EU (Frankfurt) Region
- Needs location constraint eu-central-1. - Needs location constraint eu-central-1.
@ -12074,15 +12083,36 @@ Region to connect to.
- "ap-northeast-2" - "ap-northeast-2"
- Asia Pacific (Seoul) - Asia Pacific (Seoul)
- Needs location constraint ap-northeast-2. - Needs location constraint ap-northeast-2.
- "ap-northeast-3"
- Asia Pacific (Osaka-Local)
- Needs location constraint ap-northeast-3.
- "ap-south-1" - "ap-south-1"
- Asia Pacific (Mumbai) - Asia Pacific (Mumbai)
- Needs location constraint ap-south-1. - Needs location constraint ap-south-1.
- "ap-east-1" - "ap-east-1"
- Asia Patific (Hong Kong) Region - Asia Pacific (Hong Kong) Region
- Needs location constraint ap-east-1. - Needs location constraint ap-east-1.
- "sa-east-1" - "sa-east-1"
- South America (Sao Paulo) Region - South America (Sao Paulo) Region
- Needs location constraint sa-east-1. - Needs location constraint sa-east-1.
- "me-south-1"
- Middle East (Bahrain) Region
- Needs location constraint me-south-1.
- "af-south-1"
- Africa (Cape Town) Region
- Needs location constraint af-south-1.
- "cn-north-1"
- China (Beijing) Region
- Needs location constraint cn-north-1.
- "cn-northwest-1"
- China (Ningxia) Region
- Needs location constraint cn-northwest-1.
- "us-gov-east-1"
- AWS GovCloud (US-East) Region
- Needs location constraint us-gov-east-1.
- "us-gov-west-1"
- AWS GovCloud (US) Region
- Needs location constraint us-gov-west-1.
#### --s3-region #### --s3-region
@ -12423,18 +12453,22 @@ Used when creating buckets only.
- Empty for US Region, Northern Virginia or Pacific Northwest. - Empty for US Region, Northern Virginia or Pacific Northwest.
- "us-east-2" - "us-east-2"
- US East (Ohio) Region. - US East (Ohio) Region.
- "us-west-2"
- US West (Oregon) Region.
- "us-west-1" - "us-west-1"
- US West (Northern California) Region. - US West (Northern California) Region.
- "us-west-2"
- US West (Oregon) Region.
- "ca-central-1" - "ca-central-1"
- Canada (Central) Region. - Canada (Central) Region.
- "eu-west-1" - "eu-west-1"
- EU (Ireland) Region. - EU (Ireland) Region.
- "eu-west-2" - "eu-west-2"
- EU (London) Region. - EU (London) Region.
- "eu-west-3"
- EU (Paris) Region.
- "eu-north-1" - "eu-north-1"
- EU (Stockholm) Region. - EU (Stockholm) Region.
- "eu-south-1"
- EU (Milan) Region.
- "EU" - "EU"
- EU Region. - EU Region.
- "ap-southeast-1" - "ap-southeast-1"
@ -12444,13 +12478,27 @@ Used when creating buckets only.
- "ap-northeast-1" - "ap-northeast-1"
- Asia Pacific (Tokyo) Region. - Asia Pacific (Tokyo) Region.
- "ap-northeast-2" - "ap-northeast-2"
- Asia Pacific (Seoul) - Asia Pacific (Seoul) Region.
- "ap-northeast-3"
- Asia Pacific (Osaka-Local) Region.
- "ap-south-1" - "ap-south-1"
- Asia Pacific (Mumbai) - Asia Pacific (Mumbai) Region.
- "ap-east-1" - "ap-east-1"
- Asia Pacific (Hong Kong) - Asia Pacific (Hong Kong) Region.
- "sa-east-1" - "sa-east-1"
- South America (Sao Paulo) Region. - South America (Sao Paulo) Region.
- "me-south-1"
- Middle East (Bahrain) Region.
- "af-south-1"
- Africa (Cape Town) Region.
- "cn-north-1"
- China (Beijing) Region
- "cn-northwest-1"
- China (Ningxia) Region.
- "us-gov-east-1"
- AWS GovCloud (US-East) Region.
- "us-gov-west-1"
- AWS GovCloud (US) Region.
#### --s3-location-constraint #### --s3-location-constraint
@ -14829,7 +14877,8 @@ Note that Box is case insensitive so you can't have a file called
"Hello.doc" and one called "hello.doc". "Hello.doc" and one called "hello.doc".
Box file names can't have the `\` character in. rclone maps this to Box file names can't have the `\` character in. rclone maps this to
and from an identical looking unicode equivalent `＼`. and from an identical looking unicode equivalent `＼` (U+FF3C Fullwidth
Reverse Solidus).
Box only supports filenames up to 255 characters in length. Box only supports filenames up to 255 characters in length.
@ -16111,23 +16160,26 @@ See: the [encoding section in the overview](https://rclone.org/overview/#encodin
Crypt Crypt
---------------------------------------- ----------------------------------------
The `crypt` remote encrypts and decrypts another remote. Rclone `crypt` remotes encrypt and decrypt other remotes.
To use it first set up the underlying remote following the config To use `crypt`, first set up the underlying remote. Follow the `rclone
instructions for that remote. You can also use a local pathname config` instructions for that remote.
instead of a remote which will encrypt and decrypt from that directory
which might be useful for encrypting onto a USB stick for example.
First check your chosen remote is working - we'll call it `crypt` applied to a local pathname instead of a remote will
`remote:path` in these docs. Note that anything inside `remote:path` encrypt and decrypt that directory, and can be used to encrypt USB
will be encrypted and anything outside won't. This means that if you removable drives.
are using a bucket based remote (eg S3, B2, swift) then you should
probably put the bucket in the remote `s3:bucket`. If you just use
`s3:` then rclone will make encrypted bucket names too (if using file
name encryption) which may or may not be what you want.
Now configure `crypt` using `rclone config`. We will call this one Before configuring the crypt remote, check the underlying remote is
`secret` to differentiate it from the `remote`. working. In this example the underlying remote is called `remote:path`.
Anything inside `remote:path` will be encrypted and anything outside
will not. In the case of an S3 based underlying remote (eg Amazon S3,
B2, Swift) it is generally advisable to define a crypt remote in the
underlying remote `s3:bucket`. If `s3:` alone is specified alongside
file name encryption, rclone will encrypt the bucket name.
Configure `crypt` using `rclone config`. In this example the `crypt`
remote is called `secret`, to differentiate it from the underlying
`remote`.
``` ```
No remotes found - make a new one No remotes found - make a new one
@ -16201,49 +16253,42 @@ d) Delete this remote
y/e/d> y y/e/d> y
``` ```
**Important** The password is stored in the config file is lightly **Important** The crypt password stored in `rclone.conf` is lightly
obscured so it isn't immediately obvious what it is. It is in no way obscured. That only protects it from cursory inspection. It is not
secure unless you use config file encryption. secure unless encryption of `rclone.conf` is specified.
A long passphrase is recommended, or you can use a random one. A long passphrase is recommended, or `rclone config` can generate a
random one.
The obscured password is created by using AES-CTR with a static key, with The obscured password is created using AES-CTR with a static key. The
the salt stored verbatim at the beginning of the obscured password. This salt is stored verbatim at the beginning of the obscured password. This
static key is shared by between all versions of rclone. static key is shared between all versions of rclone.
If you reconfigure rclone with the same passwords/passphrases If you reconfigure rclone with the same passwords/passphrases
elsewhere it will be compatible, but the obscured version will be different elsewhere it will be compatible, but the obscured version will be different
due to the different salt. due to the different salt.
Note that rclone does not encrypt Rclone does not encrypt
* file length - this can be calculated within 16 bytes * file length - this can be calculated within 16 bytes
* modification time - used for syncing * modification time - used for syncing
## Specifying the remote ## ## Specifying the remote ##
In normal use, make sure the remote has a `:` in. If you specify the In normal use, ensure the remote has a `:` in. If specified without,
remote without a `:` then rclone will use a local directory of that rclone uses a local directory of that name. For example if a remote
name. So if you use a remote of `/path/to/secret/files` then rclone `/path/to/secret/files` is specified, rclone encrypts content to that
will encrypt stuff to that directory. If you use a remote of `name` directory. If a remote `name` is specified, rclone targets a directory
then rclone will put files in a directory called `name` in the current `name` in the current directory.
directory.
If you specify the remote as `remote:path/to/dir` then rclone will If remote `remote:path/to/dir` is specified, rclone stores encrypted
store encrypted files in `path/to/dir` on the remote. If you are using files in `path/to/dir` on the remote. With file name encryption, files
file name encryption, then when you save files to saved to `secret:subdir/subfile` are stored in the unencrypted path
`secret:subdir/subfile` this will store them in the unencrypted path `path/to/dir` but the `subdir/subpath` element is encrypted.
`path/to/dir` but the `subdir/subpath` bit will be encrypted.
Note that unless you want encrypted bucket names (which are difficult
to manage because you won't know what directory they represent in web
interfaces etc), you should probably specify a bucket, eg
`remote:secretbucket` when using bucket based remotes such as S3,
Swift, Hubic, B2, GCS.
## Example ## ## Example ##
To test I made a little directory of files using "standard" file name Create the following file structure using "standard" file name
encryption. encryption.
``` ```
@ -16257,7 +16302,7 @@ plaintext/
└── file4.txt └── file4.txt
``` ```
Copy these to the remote and list them back Copy these to the remote, and list them
``` ```
$ rclone -q copy plaintext secret: $ rclone -q copy plaintext secret:
@ -16269,7 +16314,7 @@ $ rclone -q ls secret:
9 subdir/file3.txt 9 subdir/file3.txt
``` ```
Now see what that looked like when encrypted The crypt remote looks like
``` ```
$ rclone -q ls remote:path $ rclone -q ls remote:path
@ -16280,7 +16325,7 @@ $ rclone -q ls remote:path
56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
``` ```
Note that this retains the directory structure which means you can do this The directory structure is preserved
``` ```
$ rclone -q ls secret:subdir $ rclone -q ls secret:subdir
@ -16289,9 +16334,9 @@ $ rclone -q ls secret:subdir
10 subsubdir/file4.txt 10 subsubdir/file4.txt
``` ```
If don't use file name encryption then the remote will look like this Without file name encryption `.bin` extensions are added to underlying
- note the `.bin` extensions added to prevent the cloud provider names. This prevents the cloud provider attempting to interpret file
attempting to interpret the data. content.
``` ```
$ rclone -q ls remote:path $ rclone -q ls remote:path
@ -16304,8 +16349,6 @@ $ rclone -q ls remote:path
### File name encryption modes ### ### File name encryption modes ###
Here are some of the features of the file name encryption modes
Off Off
* doesn't hide file names or directory structure * doesn't hide file names or directory structure
@ -16324,17 +16367,19 @@ Standard
Obfuscation Obfuscation
This is a simple "rotate" of the filename, with each file having a rot This is a simple "rotate" of the filename, with each file having a rot
distance based on the filename. We store the distance at the beginning distance based on the filename. Rclone stores the distance at the
of the filename. So a file called "hello" may become "53.jgnnq". beginning of the filename. A file called "hello" may become "53.jgnnq".
This is not a strong encryption of filenames, but it may stop automated Obfuscation is not a strong encryption of filenames, but hinders
scanning tools from picking up on filename patterns. As such it's an automated scanning tools picking up on filename patterns. It is an
intermediate between "off" and "standard". The advantage is that it intermediate between "off" and "standard" which allows for longer path
allows for longer path segment names. segment names.
There is a possibility with some unicode based filenames that the There is a possibility with some unicode based filenames that the
obfuscation is weak and may map lower case characters to upper case obfuscation is weak and may map lower case characters to upper case
equivalents. You can not rely on this for strong protection. equivalents.
Obfuscation cannot be relied upon for strong protection.
* file names very lightly obfuscated * file names very lightly obfuscated
* file names can be longer than standard encryption * file names can be longer than standard encryption
@ -16342,13 +16387,14 @@ equivalents. You can not rely on this for strong protection.
* directory structure visible * directory structure visible
* identical files names will have identical uploaded names * identical files names will have identical uploaded names
Cloud storage systems have various limits on file name length and Cloud storage systems have limits on file name length and
total path length which you are more likely to hit using "Standard" total path length which rclone is more likely to breach using
file name encryption. If you keep your file names to below 156 "Standard" file name encryption. Where file names are less thn 156
characters in length then you should be OK on all providers. characters in length issues should not be encountered, irrespective of
cloud storage provider.
There may be an even more secure file name encryption mode in the An alternative, future rclone file name encryption mode may tolerate
future which will address the long file name problem. backend provider path length limits.
### Directory name encryption ### ### Directory name encryption ###
Crypt offers the option of encrypting dir names or leaving them intact. Crypt offers the option of encrypting dir names or leaving them intact.
@ -16374,10 +16420,10 @@ Example:
Crypt stores modification times using the underlying remote so support Crypt stores modification times using the underlying remote so support
depends on that. depends on that.
Hashes are not stored for crypt. However the data integrity is Hashes are not stored for crypt. However the data integrity is
protected by an extremely strong crypto authenticator. protected by an extremely strong crypto authenticator.
Note that you should use the `rclone cryptcheck` command to check the Use the `rclone cryptcheck` command to check the
integrity of a crypted remote instead of `rclone check` which can't integrity of a crypted remote instead of `rclone check` which can't
check the checksums properly. check the checksums properly.
@ -25681,6 +25727,56 @@ Options:
# Changelog # Changelog
## v1.53.2 - 2020-10-26
[See commits](https://github.com/rclone/rclone/compare/v1.53.1...v1.53.2)
* Bug Fixes
* acounting
* Fix incorrect speed and transferTime in core/stats (Nick Craig-Wood)
* Stabilize display order of transfers on Windows (Nick Craig-Wood)
* operations
* Fix use of --suffix without --backup-dir (Nick Craig-Wood)
* Fix spurious "--checksum is in use but the source and destination have no hashes in common" (Nick Craig-Wood)
* build
* Work around GitHub actions brew problem (Nick Craig-Wood)
* Stop using set-env and set-path in the GitHub actions (Nick Craig-Wood)
* Mount
* mount2: Fix the swapped UID / GID values (Russell Cattelan)
* VFS
* Detect and recover from a file being removed externally from the cache (Nick Craig-Wood)
* Fix a deadlock vulnerability in downloaders.Close (Leo Luan)
* Fix a race condition in retryFailedResets (Leo Luan)
* Fix missed concurrency control between some item operations and reset (Leo Luan)
* Add exponential backoff during ENOSPC retries (Leo Luan)
* Add a missed update of used cache space (Leo Luan)
* Fix --no-modtime to not attempt to set modtimes (as documented) (Nick Craig-Wood)
* Local
* Fix sizes and syncing with --links option on Windows (Nick Craig-Wood)
* Chunker
* Disable ListR to fix missing files on GDrive (workaround) (Ivan Andreev)
* Fix upload over crypt (Ivan Andreev)
* Fichier
* Increase maximum file size from 100GB to 300GB (gyutw)
* Jottacloud
* Remove clientSecret from config when upgrading to token based authentication (buengese)
* Avoid double url escaping of device/mountpoint (albertony)
* Remove DirMove workaround as it's not required anymore - also (buengese)
* Mailru
* Fix uploads after recent changes on server (Ivan Andreev)
* Fix range requests after june changes on server (Ivan Andreev)
* Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
* Onedrive
* Fix disk usage for sharepoint (Nick Craig-Wood)
* S3
* Add missing regions for AWS (Anagh Kumar Baranwal)
* Seafile
* Fix accessing libraries > 2GB on 32 bit systems (Muffin King)
* SFTP
* Always convert the checksum to lower case (buengese)
* Union
* Create root directories if none exist (Nick Craig-Wood)
## v1.53.1 - 2020-09-13 ## v1.53.1 - 2020-09-13
[See commits](https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1) [See commits](https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1)

270
MANUAL.txt generated
View file

@ -1,6 +1,6 @@
rclone(1) User Manual rclone(1) User Manual
Nick Craig-Wood Nick Craig-Wood
Sep 13, 2020 Oct 26, 2020
@ -2929,10 +2929,6 @@ following parts will be downloaded: 0-100M, 100M-200M, 200M-300M,
the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M,
1200M-1700M and so on. 1200M-1700M and so on.
Chunked reading will only work with --vfs-cache-mode < full, as the file
will always be copied to the vfs cache before opening with
--vfs-cache-mode full.
VFS - Virtual File System VFS - Virtual File System
@ -7150,11 +7146,17 @@ with --backup-dir. See --backup-dir for more info.
For example For example
rclone sync -i /path/to/local/file remote:current --suffix .bak rclone copy -i /path/to/local/file remote:current --suffix .bak
will sync /path/to/local to remote:current, but for any files which will copy /path/to/local to remote:current, but for any files which
would have been updated or deleted have .bak added. would have been updated or deleted have .bak added.
If using rclone sync with --suffix and without --backup-dir then it is
recommended to put a filter rule in excluding the suffix otherwise the
sync will delete the backup files.
rclone sync -i /path/to/local/file remote:current --suffix .bak --exclude "*.bak"
--suffix-keep-extension --suffix-keep-extension
When using --suffix, setting this causes rclone put the SUFFIX before When using --suffix, setting this causes rclone put the SUFFIX before
@ -10658,7 +10660,7 @@ These flags are available for every command.
--use-json-log Use json log format. --use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs). --use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata --use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.1") --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.2")
-v, --verbose count Print lots more stuff (repeat for more) -v, --verbose count Print lots more stuff (repeat for more)
@ -11706,7 +11708,7 @@ This will guide you through an interactive setup process.
/ Asia Pacific (Mumbai) / Asia Pacific (Mumbai)
13 | Needs location constraint ap-south-1. 13 | Needs location constraint ap-south-1.
\ "ap-south-1" \ "ap-south-1"
/ Asia Patific (Hong Kong) Region / Asia Pacific (Hong Kong) Region
14 | Needs location constraint ap-east-1. 14 | Needs location constraint ap-east-1.
\ "ap-east-1" \ "ap-east-1"
/ South America (Sao Paulo) Region / South America (Sao Paulo) Region
@ -12118,12 +12120,12 @@ Region to connect to.
- "us-east-2" - "us-east-2"
- US East (Ohio) Region - US East (Ohio) Region
- Needs location constraint us-east-2. - Needs location constraint us-east-2.
- "us-west-2"
- US West (Oregon) Region
- Needs location constraint us-west-2.
- "us-west-1" - "us-west-1"
- US West (Northern California) Region - US West (Northern California) Region
- Needs location constraint us-west-1. - Needs location constraint us-west-1.
- "us-west-2"
- US West (Oregon) Region
- Needs location constraint us-west-2.
- "ca-central-1" - "ca-central-1"
- Canada (Central) Region - Canada (Central) Region
- Needs location constraint ca-central-1. - Needs location constraint ca-central-1.
@ -12133,9 +12135,15 @@ Region to connect to.
- "eu-west-2" - "eu-west-2"
- EU (London) Region - EU (London) Region
- Needs location constraint eu-west-2. - Needs location constraint eu-west-2.
- "eu-west-3"
- EU (Paris) Region
- Needs location constraint eu-west-3.
- "eu-north-1" - "eu-north-1"
- EU (Stockholm) Region - EU (Stockholm) Region
- Needs location constraint eu-north-1. - Needs location constraint eu-north-1.
- "eu-south-1"
- EU (Milan) Region
- Needs location constraint eu-south-1.
- "eu-central-1" - "eu-central-1"
- EU (Frankfurt) Region - EU (Frankfurt) Region
- Needs location constraint eu-central-1. - Needs location constraint eu-central-1.
@ -12151,15 +12159,36 @@ Region to connect to.
- "ap-northeast-2" - "ap-northeast-2"
- Asia Pacific (Seoul) - Asia Pacific (Seoul)
- Needs location constraint ap-northeast-2. - Needs location constraint ap-northeast-2.
- "ap-northeast-3"
- Asia Pacific (Osaka-Local)
- Needs location constraint ap-northeast-3.
- "ap-south-1" - "ap-south-1"
- Asia Pacific (Mumbai) - Asia Pacific (Mumbai)
- Needs location constraint ap-south-1. - Needs location constraint ap-south-1.
- "ap-east-1" - "ap-east-1"
- Asia Patific (Hong Kong) Region - Asia Pacific (Hong Kong) Region
- Needs location constraint ap-east-1. - Needs location constraint ap-east-1.
- "sa-east-1" - "sa-east-1"
- South America (Sao Paulo) Region - South America (Sao Paulo) Region
- Needs location constraint sa-east-1. - Needs location constraint sa-east-1.
- "me-south-1"
- Middle East (Bahrain) Region
- Needs location constraint me-south-1.
- "af-south-1"
- Africa (Cape Town) Region
- Needs location constraint af-south-1.
- "cn-north-1"
- China (Beijing) Region
- Needs location constraint cn-north-1.
- "cn-northwest-1"
- China (Ningxia) Region
- Needs location constraint cn-northwest-1.
- "us-gov-east-1"
- AWS GovCloud (US-East) Region
- Needs location constraint us-gov-east-1.
- "us-gov-west-1"
- AWS GovCloud (US) Region
- Needs location constraint us-gov-west-1.
--s3-region --s3-region
@ -12500,18 +12529,22 @@ creating buckets only.
- Empty for US Region, Northern Virginia or Pacific Northwest. - Empty for US Region, Northern Virginia or Pacific Northwest.
- "us-east-2" - "us-east-2"
- US East (Ohio) Region. - US East (Ohio) Region.
- "us-west-2"
- US West (Oregon) Region.
- "us-west-1" - "us-west-1"
- US West (Northern California) Region. - US West (Northern California) Region.
- "us-west-2"
- US West (Oregon) Region.
- "ca-central-1" - "ca-central-1"
- Canada (Central) Region. - Canada (Central) Region.
- "eu-west-1" - "eu-west-1"
- EU (Ireland) Region. - EU (Ireland) Region.
- "eu-west-2" - "eu-west-2"
- EU (London) Region. - EU (London) Region.
- "eu-west-3"
- EU (Paris) Region.
- "eu-north-1" - "eu-north-1"
- EU (Stockholm) Region. - EU (Stockholm) Region.
- "eu-south-1"
- EU (Milan) Region.
- "EU" - "EU"
- EU Region. - EU Region.
- "ap-southeast-1" - "ap-southeast-1"
@ -12521,13 +12554,27 @@ creating buckets only.
- "ap-northeast-1" - "ap-northeast-1"
- Asia Pacific (Tokyo) Region. - Asia Pacific (Tokyo) Region.
- "ap-northeast-2" - "ap-northeast-2"
- Asia Pacific (Seoul) - Asia Pacific (Seoul) Region.
- "ap-northeast-3"
- Asia Pacific (Osaka-Local) Region.
- "ap-south-1" - "ap-south-1"
- Asia Pacific (Mumbai) - Asia Pacific (Mumbai) Region.
- "ap-east-1" - "ap-east-1"
- Asia Pacific (Hong Kong) - Asia Pacific (Hong Kong) Region.
- "sa-east-1" - "sa-east-1"
- South America (Sao Paulo) Region. - South America (Sao Paulo) Region.
- "me-south-1"
- Middle East (Bahrain) Region.
- "af-south-1"
- Africa (Cape Town) Region.
- "cn-north-1"
- China (Beijing) Region
- "cn-northwest-1"
- China (Ningxia) Region.
- "us-gov-east-1"
- AWS GovCloud (US-East) Region.
- "us-gov-west-1"
- AWS GovCloud (US) Region.
--s3-location-constraint --s3-location-constraint
@ -14845,7 +14892,8 @@ Note that Box is case insensitive so you can't have a file called
"Hello.doc" and one called "hello.doc". "Hello.doc" and one called "hello.doc".
Box file names can't have the \ character in. rclone maps this to and Box file names can't have the \ character in. rclone maps this to and
from an identical looking unicode equivalent ＼. from an identical looking unicode equivalent ＼ (U+FF3C Fullwidth
Reverse Solidus).
Box only supports filenames up to 255 characters in length. Box only supports filenames up to 255 characters in length.
@ -16132,23 +16180,24 @@ See: the encoding section in the overview for more info.
Crypt Crypt
The crypt remote encrypts and decrypts another remote. Rclone crypt remotes encrypt and decrypt other remotes.
To use it first set up the underlying remote following the config To use crypt, first set up the underlying remote. Follow the
instructions for that remote. You can also use a local pathname instead rclone config instructions for that remote.
of a remote which will encrypt and decrypt from that directory which
might be useful for encrypting onto a USB stick for example.
First check your chosen remote is working - we'll call it remote:path in crypt applied to a local pathname instead of a remote will encrypt and
these docs. Note that anything inside remote:path will be encrypted and decrypt that directory, and can be used to encrypt USB removable drives.
anything outside won't. This means that if you are using a bucket based
remote (eg S3, B2, swift) then you should probably put the bucket in the
remote s3:bucket. If you just use s3: then rclone will make encrypted
bucket names too (if using file name encryption) which may or may not be
what you want.
Now configure crypt using rclone config. We will call this one secret to Before configuring the crypt remote, check the underlying remote is
differentiate it from the remote. working. In this example the underlying remote is called remote:path.
Anything inside remote:path will be encrypted and anything outside will
not. In the case of an S3 based underlying remote (eg Amazon S3, B2,
Swift) it is generally advisable to define a crypt remote in the
underlying remote s3:bucket. If s3: alone is specified alongside file
name encryption, rclone will encrypt the bucket name.
Configure crypt using rclone config. In this example the crypt remote is
called secret, to differentiate it from the underlying remote.
No remotes found - make a new one No remotes found - make a new one
n) New remote n) New remote
@ -16220,21 +16269,22 @@ differentiate it from the remote.
d) Delete this remote d) Delete this remote
y/e/d> y y/e/d> y
IMPORTANT The password is stored in the config file is lightly obscured IMPORTANT The crypt password stored in rclone.conf is lightly obscured.
so it isn't immediately obvious what it is. It is in no way secure That only protects it from cursory inspection. It is not secure unless
unless you use config file encryption. encryption of rclone.conf is specified.
A long passphrase is recommended, or you can use a random one. A long passphrase is recommended, or rclone config can generate a random
one.
The obscured password is created by using AES-CTR with a static key, The obscured password is created using AES-CTR with a static key. The
with the salt stored verbatim at the beginning of the obscured password. salt is stored verbatim at the beginning of the obscured password. This
This static key is shared by between all versions of rclone. static key is shared between all versions of rclone.
If you reconfigure rclone with the same passwords/passphrases elsewhere If you reconfigure rclone with the same passwords/passphrases elsewhere
it will be compatible, but the obscured version will be different due to it will be compatible, but the obscured version will be different due to
the different salt. the different salt.
Note that rclone does not encrypt Rclone does not encrypt
- file length - this can be calculated within 16 bytes - file length - this can be calculated within 16 bytes
- modification time - used for syncing - modification time - used for syncing
@ -16242,28 +16292,21 @@ Note that rclone does not encrypt
Specifying the remote Specifying the remote
In normal use, make sure the remote has a : in. If you specify the In normal use, ensure the remote has a : in. If specified without,
remote without a : then rclone will use a local directory of that name. rclone uses a local directory of that name. For example if a remote
So if you use a remote of /path/to/secret/files then rclone will encrypt /path/to/secret/files is specified, rclone encrypts content to that
stuff to that directory. If you use a remote of name then rclone will directory. If a remote name is specified, rclone targets a directory
put files in a directory called name in the current directory. name in the current directory.
If you specify the remote as remote:path/to/dir then rclone will store If remote remote:path/to/dir is specified, rclone stores encrypted files
encrypted files in path/to/dir on the remote. If you are using file name in path/to/dir on the remote. With file name encryption, files saved to
encryption, then when you save files to secret:subdir/subfile this will secret:subdir/subfile are stored in the unencrypted path path/to/dir but
store them in the unencrypted path path/to/dir but the subdir/subpath the subdir/subpath element is encrypted.
bit will be encrypted.
Note that unless you want encrypted bucket names (which are difficult to
manage because you won't know what directory they represent in web
interfaces etc), you should probably specify a bucket, eg
remote:secretbucket when using bucket based remotes such as S3, Swift,
Hubic, B2, GCS.
Example Example
To test I made a little directory of files using "standard" file name Create the following file structure using "standard" file name
encryption. encryption.
plaintext/ plaintext/
@ -16275,7 +16318,7 @@ encryption.
└── subsubdir └── subsubdir
└── file4.txt └── file4.txt
Copy these to the remote and list them back Copy these to the remote, and list them
$ rclone -q copy plaintext secret: $ rclone -q copy plaintext secret:
$ rclone -q ls secret: $ rclone -q ls secret:
@ -16285,7 +16328,7 @@ Copy these to the remote and list them back
10 subdir/subsubdir/file4.txt 10 subdir/subsubdir/file4.txt
9 subdir/file3.txt 9 subdir/file3.txt
Now see what that looked like when encrypted The crypt remote looks like
$ rclone -q ls remote:path $ rclone -q ls remote:path
55 hagjclgavj2mbiqm6u6cnjjqcg 55 hagjclgavj2mbiqm6u6cnjjqcg
@ -16294,17 +16337,16 @@ Now see what that looked like when encrypted
58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
Note that this retains the directory structure which means you can do The directory structure is preserved
this
$ rclone -q ls secret:subdir $ rclone -q ls secret:subdir
8 file2.txt 8 file2.txt
9 file3.txt 9 file3.txt
10 subsubdir/file4.txt 10 subsubdir/file4.txt
If don't use file name encryption then the remote will look like this - Without file name encryption .bin extensions are added to underlying
note the .bin extensions added to prevent the cloud provider attempting names. This prevents the cloud provider attempting to interpret file
to interpret the data. content.
$ rclone -q ls remote:path $ rclone -q ls remote:path
54 file0.txt.bin 54 file0.txt.bin
@ -16315,8 +16357,6 @@ to interpret the data.
File name encryption modes File name encryption modes
Here are some of the features of the file name encryption modes
Off Off
- doesn't hide file names or directory structure - doesn't hide file names or directory structure
@ -16335,17 +16375,19 @@ Standard
Obfuscation Obfuscation
This is a simple "rotate" of the filename, with each file having a rot This is a simple "rotate" of the filename, with each file having a rot
distance based on the filename. We store the distance at the beginning distance based on the filename. Rclone stores the distance at the
of the filename. So a file called "hello" may become "53.jgnnq". beginning of the filename. A file called "hello" may become "53.jgnnq".
This is not a strong encryption of filenames, but it may stop automated Obfuscation is not a strong encryption of filenames, but hinders
scanning tools from picking up on filename patterns. As such it's an automated scanning tools picking up on filename patterns. It is an
intermediate between "off" and "standard". The advantage is that it intermediate between "off" and "standard" which allows for longer path
allows for longer path segment names. segment names.
There is a possibility with some unicode based filenames that the There is a possibility with some unicode based filenames that the
obfuscation is weak and may map lower case characters to upper case obfuscation is weak and may map lower case characters to upper case
equivalents. You can not rely on this for strong protection. equivalents.
Obfuscation cannot be relied upon for strong protection.
- file names very lightly obfuscated - file names very lightly obfuscated
- file names can be longer than standard encryption - file names can be longer than standard encryption
@ -16353,13 +16395,14 @@ equivalents. You can not rely on this for strong protection.
- directory structure visible - directory structure visible
- identical files names will have identical uploaded names - identical files names will have identical uploaded names
Cloud storage systems have various limits on file name length and total Cloud storage systems have limits on file name length and total path
path length which you are more likely to hit using "Standard" file name length which rclone is more likely to breach using "Standard" file name
encryption. If you keep your file names to below 156 characters in encryption. Where file names are less thn 156 characters in length
length then you should be OK on all providers. issues should not be encountered, irrespective of cloud storage
provider.
There may be an even more secure file name encryption mode in the future An alternative, future rclone file name encryption mode may tolerate
which will address the long file name problem. backend provider path length limits.
Directory name encryption Directory name encryption
@ -16385,9 +16428,8 @@ depends on that.
Hashes are not stored for crypt. However the data integrity is protected Hashes are not stored for crypt. However the data integrity is protected
by an extremely strong crypto authenticator. by an extremely strong crypto authenticator.
Note that you should use the rclone cryptcheck command to check the Use the rclone cryptcheck command to check the integrity of a crypted
integrity of a crypted remote instead of rclone check which can't check remote instead of rclone check which can't check the checksums properly.
the checksums properly.
Standard Options Standard Options
@ -25658,6 +25700,68 @@ Options:
CHANGELOG CHANGELOG
v1.53.2 - 2020-10-26
See commits
- Bug Fixes
- acounting
- Fix incorrect speed and transferTime in core/stats (Nick
Craig-Wood)
- Stabilize display order of transfers on Windows (Nick
Craig-Wood)
- operations
- Fix use of --suffix without --backup-dir (Nick Craig-Wood)
- Fix spurious "--checksum is in use but the source and
destination have no hashes in common" (Nick Craig-Wood)
- build
- Work around GitHub actions brew problem (Nick Craig-Wood)
- Stop using set-env and set-path in the GitHub actions (Nick
Craig-Wood)
- Mount
- mount2: Fix the swapped UID / GID values (Russell Cattelan)
- VFS
- Detect and recover from a file being removed externally from the
cache (Nick Craig-Wood)
- Fix a deadlock vulnerability in downloaders.Close (Leo Luan)
- Fix a race condition in retryFailedResets (Leo Luan)
- Fix missed concurrency control between some item operations and
reset (Leo Luan)
- Add exponential backoff during ENOSPC retries (Leo Luan)
- Add a missed update of used cache space (Leo Luan)
- Fix --no-modtime to not attempt to set modtimes (as documented)
(Nick Craig-Wood)
- Local
- Fix sizes and syncing with --links option on Windows (Nick
Craig-Wood)
- Chunker
- Disable ListR to fix missing files on GDrive (workaround) (Ivan
Andreev)
- Fix upload over crypt (Ivan Andreev)
- Fichier
- Increase maximum file size from 100GB to 300GB (gyutw)
- Jottacloud
- Remove clientSecret from config when upgrading to token based
authentication (buengese)
- Avoid double url escaping of device/mountpoint (albertony)
- Remove DirMove workaround as it's not required anymore - also
(buengese)
- Mailru
- Fix uploads after recent changes on server (Ivan Andreev)
- Fix range requests after june changes on server (Ivan Andreev)
- Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
- Onedrive
- Fix disk usage for sharepoint (Nick Craig-Wood)
- S3
- Add missing regions for AWS (Anagh Kumar Baranwal)
- Seafile
- Fix accessing libraries > 2GB on 32 bit systems (Muffin King)
- SFTP
- Always convert the checksum to lower case (buengese)
- Union
- Create root directories if none exist (Nick Craig-Wood)
v1.53.1 - 2020-09-13 v1.53.1 - 2020-09-13
See commits See commits

50
docs/content/changelog.md
View file

@ -5,6 +5,56 @@ description: "Rclone Changelog"
# Changelog # Changelog
## v1.53.2 - 2020-10-26
[See commits](https://github.com/rclone/rclone/compare/v1.53.1...v1.53.2)
* Bug Fixes
* acounting
* Fix incorrect speed and transferTime in core/stats (Nick Craig-Wood)
* Stabilize display order of transfers on Windows (Nick Craig-Wood)
* operations
* Fix use of --suffix without --backup-dir (Nick Craig-Wood)
* Fix spurious "--checksum is in use but the source and destination have no hashes in common" (Nick Craig-Wood)
* build
* Work around GitHub actions brew problem (Nick Craig-Wood)
* Stop using set-env and set-path in the GitHub actions (Nick Craig-Wood)
* Mount
* mount2: Fix the swapped UID / GID values (Russell Cattelan)
* VFS
* Detect and recover from a file being removed externally from the cache (Nick Craig-Wood)
* Fix a deadlock vulnerability in downloaders.Close (Leo Luan)
* Fix a race condition in retryFailedResets (Leo Luan)
* Fix missed concurrency control between some item operations and reset (Leo Luan)
* Add exponential backoff during ENOSPC retries (Leo Luan)
* Add a missed update of used cache space (Leo Luan)
* Fix --no-modtime to not attempt to set modtimes (as documented) (Nick Craig-Wood)
* Local
* Fix sizes and syncing with --links option on Windows (Nick Craig-Wood)
* Chunker
* Disable ListR to fix missing files on GDrive (workaround) (Ivan Andreev)
* Fix upload over crypt (Ivan Andreev)
* Fichier
* Increase maximum file size from 100GB to 300GB (gyutw)
* Jottacloud
* Remove clientSecret from config when upgrading to token based authentication (buengese)
* Avoid double url escaping of device/mountpoint (albertony)
* Remove DirMove workaround as it's not required anymore - also (buengese)
* Mailru
* Fix uploads after recent changes on server (Ivan Andreev)
* Fix range requests after june changes on server (Ivan Andreev)
* Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
* Onedrive
* Fix disk usage for sharepoint (Nick Craig-Wood)
* S3
* Add missing regions for AWS (Anagh Kumar Baranwal)
* Seafile
* Fix accessing libraries > 2GB on 32 bit systems (Muffin King)
* SFTP
* Always convert the checksum to lower case (buengese)
* Union
* Create root directories if none exist (Nick Craig-Wood)
## v1.53.1 - 2020-09-13 ## v1.53.1 - 2020-09-13
[See commits](https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1) [See commits](https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1)

3
docs/content/commands/rclone_mount.md
View file

@ -194,9 +194,6 @@ parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
When --vfs-read-chunk-size-limit 500M is specified, the result would be When --vfs-read-chunk-size-limit 500M is specified, the result would be
0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on. 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Chunked reading will only work with --vfs-cache-mode < full, as the file will always
be copied to the vfs cache before opening with --vfs-cache-mode full.
## VFS - Virtual File System ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects This command uses the VFS layer. This adapts the cloud storage objects

2
docs/content/flags.md
View file

@ -147,7 +147,7 @@ These flags are available for every command.
--use-json-log Use json log format. --use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs). --use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata --use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.1") --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.53.2")
-v, --verbose count Print lots more stuff (repeat for more) -v, --verbose count Print lots more stuff (repeat for more)
``` ```

63
docs/content/s3.md
View file

@ -545,12 +545,12 @@ Region to connect to.
- "us-east-2" - "us-east-2"
- US East (Ohio) Region - US East (Ohio) Region
- Needs location constraint us-east-2. - Needs location constraint us-east-2.
- "us-west-2"
- US West (Oregon) Region
- Needs location constraint us-west-2.
- "us-west-1" - "us-west-1"
- US West (Northern California) Region - US West (Northern California) Region
- Needs location constraint us-west-1. - Needs location constraint us-west-1.
- "us-west-2"
- US West (Oregon) Region
- Needs location constraint us-west-2.
- "ca-central-1" - "ca-central-1"
- Canada (Central) Region - Canada (Central) Region
- Needs location constraint ca-central-1. - Needs location constraint ca-central-1.
@ -560,9 +560,15 @@ Region to connect to.
- "eu-west-2" - "eu-west-2"
- EU (London) Region - EU (London) Region
- Needs location constraint eu-west-2. - Needs location constraint eu-west-2.
- "eu-west-3"
- EU (Paris) Region
- Needs location constraint eu-west-3.
- "eu-north-1" - "eu-north-1"
- EU (Stockholm) Region - EU (Stockholm) Region
- Needs location constraint eu-north-1. - Needs location constraint eu-north-1.
- "eu-south-1"
- EU (Milan) Region
- Needs location constraint eu-south-1.
- "eu-central-1" - "eu-central-1"
- EU (Frankfurt) Region - EU (Frankfurt) Region
- Needs location constraint eu-central-1. - Needs location constraint eu-central-1.
@ -578,15 +584,36 @@ Region to connect to.
- "ap-northeast-2" - "ap-northeast-2"
- Asia Pacific (Seoul) - Asia Pacific (Seoul)
- Needs location constraint ap-northeast-2. - Needs location constraint ap-northeast-2.
- "ap-northeast-3"
- Asia Pacific (Osaka-Local)
- Needs location constraint ap-northeast-3.
- "ap-south-1" - "ap-south-1"
- Asia Pacific (Mumbai) - Asia Pacific (Mumbai)
- Needs location constraint ap-south-1. - Needs location constraint ap-south-1.
- "ap-east-1" - "ap-east-1"
- Asia Patific (Hong Kong) Region - Asia Pacific (Hong Kong) Region
- Needs location constraint ap-east-1. - Needs location constraint ap-east-1.
- "sa-east-1" - "sa-east-1"
- South America (Sao Paulo) Region - South America (Sao Paulo) Region
- Needs location constraint sa-east-1. - Needs location constraint sa-east-1.
- "me-south-1"
- Middle East (Bahrain) Region
- Needs location constraint me-south-1.
- "af-south-1"
- Africa (Cape Town) Region
- Needs location constraint af-south-1.
- "cn-north-1"
- China (Beijing) Region
- Needs location constraint cn-north-1.
- "cn-northwest-1"
- China (Ningxia) Region
- Needs location constraint cn-northwest-1.
- "us-gov-east-1"
- AWS GovCloud (US-East) Region
- Needs location constraint us-gov-east-1.
- "us-gov-west-1"
- AWS GovCloud (US) Region
- Needs location constraint us-gov-west-1.
#### --s3-region #### --s3-region
@ -927,18 +954,22 @@ Used when creating buckets only.
- Empty for US Region, Northern Virginia or Pacific Northwest. - Empty for US Region, Northern Virginia or Pacific Northwest.
- "us-east-2" - "us-east-2"
- US East (Ohio) Region. - US East (Ohio) Region.
- "us-west-2"
- US West (Oregon) Region.
- "us-west-1" - "us-west-1"
- US West (Northern California) Region. - US West (Northern California) Region.
- "us-west-2"
- US West (Oregon) Region.
- "ca-central-1" - "ca-central-1"
- Canada (Central) Region. - Canada (Central) Region.
- "eu-west-1" - "eu-west-1"
- EU (Ireland) Region. - EU (Ireland) Region.
- "eu-west-2" - "eu-west-2"
- EU (London) Region. - EU (London) Region.
- "eu-west-3"
- EU (Paris) Region.
- "eu-north-1" - "eu-north-1"
- EU (Stockholm) Region. - EU (Stockholm) Region.
- "eu-south-1"
- EU (Milan) Region.
- "EU" - "EU"
- EU Region. - EU Region.
- "ap-southeast-1" - "ap-southeast-1"
@ -948,13 +979,27 @@ Used when creating buckets only.
- "ap-northeast-1" - "ap-northeast-1"
- Asia Pacific (Tokyo) Region. - Asia Pacific (Tokyo) Region.
- "ap-northeast-2" - "ap-northeast-2"
- Asia Pacific (Seoul) - Asia Pacific (Seoul) Region.
- "ap-northeast-3"
- Asia Pacific (Osaka-Local) Region.
- "ap-south-1" - "ap-south-1"
- Asia Pacific (Mumbai) - Asia Pacific (Mumbai) Region.
- "ap-east-1" - "ap-east-1"
- Asia Pacific (Hong Kong) - Asia Pacific (Hong Kong) Region.
- "sa-east-1" - "sa-east-1"
- South America (Sao Paulo) Region. - South America (Sao Paulo) Region.
- "me-south-1"
- Middle East (Bahrain) Region.
- "af-south-1"
- Africa (Cape Town) Region.
- "cn-north-1"
- China (Beijing) Region
- "cn-northwest-1"
- China (Ningxia) Region.
- "us-gov-east-1"
- AWS GovCloud (US-East) Region.
- "us-gov-west-1"
- AWS GovCloud (US) Region.
#### --s3-location-constraint #### --s3-location-constraint

462
rclone.1 generated
View file

@ -1,7 +1,7 @@
.\"t .\"t
.\" Automatically generated by Pandoc 2.5 .\" Automatically generated by Pandoc 2.5
.\" .\"
.TH "rclone" "1" "Sep 13, 2020" "User Manual" "" .TH "rclone" "1" "Oct 26, 2020" "User Manual" ""
.hy .hy
.SH Rclone syncs your files to cloud storage .SH Rclone syncs your files to cloud storage
.PP .PP
@ -3549,10 +3549,6 @@ downloaded: 0\-100M, 100M\-200M, 200M\-300M, 300M\-400M and so on.
When \-\-vfs\-read\-chunk\-size\-limit 500M is specified, the result When \-\-vfs\-read\-chunk\-size\-limit 500M is specified, the result
would be 0\-100M, 100M\-300M, 300M\-700M, 700M\-1200M, 1200M\-1700M and would be 0\-100M, 100M\-300M, 300M\-700M, 700M\-1200M, 1200M\-1700M and
so on. so on.
.PP
Chunked reading will only work with \-\-vfs\-cache\-mode < full, as the
file will always be copied to the vfs cache before opening with
\-\-vfs\-cache\-mode full.
.SS VFS \- Virtual File System .SS VFS \- Virtual File System
.PP .PP
This command uses the VFS layer. This command uses the VFS layer.
@ -8564,12 +8560,23 @@ For example
.IP .IP
.nf .nf
\f[C] \f[C]
rclone sync \-i /path/to/local/file remote:current \-\-suffix .bak rclone copy \-i /path/to/local/file remote:current \-\-suffix .bak
\f[R] \f[R]
.fi .fi
.PP .PP
will sync \f[C]/path/to/local\f[R] to \f[C]remote:current\f[R], but for will copy \f[C]/path/to/local\f[R] to \f[C]remote:current\f[R], but for
any files which would have been updated or deleted have .bak added. any files which would have been updated or deleted have .bak added.
.PP
If using \f[C]rclone sync\f[R] with \f[C]\-\-suffix\f[R] and without
\f[C]\-\-backup\-dir\f[R] then it is recommended to put a filter rule in
excluding the suffix otherwise the \f[C]sync\f[R] will delete the backup
files.
.IP
.nf
\f[C]
rclone sync \-i /path/to/local/file remote:current \-\-suffix .bak \-\-exclude \[dq]*.bak\[dq]
\f[R]
.fi
.SS \-\-suffix\-keep\-extension .SS \-\-suffix\-keep\-extension
.PP .PP
When using \f[C]\-\-suffix\f[R], setting this causes rclone put the When using \f[C]\-\-suffix\f[R], setting this causes rclone put the
@ -14147,7 +14154,7 @@ These flags are available for every command.
\-\-use\-json\-log Use json log format. \-\-use\-json\-log Use json log format.
\-\-use\-mmap Use mmap allocator (see docs). \-\-use\-mmap Use mmap allocator (see docs).
\-\-use\-server\-modtime Use server modified time instead of object metadata \-\-use\-server\-modtime Use server modified time instead of object metadata
\-\-user\-agent string Set the user\-agent to a specified string. The default is rclone/ version (default \[dq]rclone/v1.53.1\[dq]) \-\-user\-agent string Set the user\-agent to a specified string. The default is rclone/ version (default \[dq]rclone/v1.53.2\[dq])
\-v, \-\-verbose count Print lots more stuff (repeat for more) \-v, \-\-verbose count Print lots more stuff (repeat for more)
\f[R] \f[R]
.fi .fi
@ -15415,7 +15422,7 @@ Choose a number from below, or type in your own value
/ Asia Pacific (Mumbai) / Asia Pacific (Mumbai)
13 | Needs location constraint ap\-south\-1. 13 | Needs location constraint ap\-south\-1.
\[rs] \[dq]ap\-south\-1\[dq] \[rs] \[dq]ap\-south\-1\[dq]
/ Asia Patific (Hong Kong) Region / Asia Pacific (Hong Kong) Region
14 | Needs location constraint ap\-east\-1. 14 | Needs location constraint ap\-east\-1.
\[rs] \[dq]ap\-east\-1\[dq] \[rs] \[dq]ap\-east\-1\[dq]
/ South America (Sao Paulo) Region / South America (Sao Paulo) Region
@ -16005,14 +16012,6 @@ US East (Ohio) Region
Needs location constraint us\-east\-2. Needs location constraint us\-east\-2.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]us\-west\-2\[dq]
.RS 2
.IP \[bu] 2
US West (Oregon) Region
.IP \[bu] 2
Needs location constraint us\-west\-2.
.RE
.IP \[bu] 2
\[dq]us\-west\-1\[dq] \[dq]us\-west\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -16021,6 +16020,14 @@ US West (Northern California) Region
Needs location constraint us\-west\-1. Needs location constraint us\-west\-1.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]us\-west\-2\[dq]
.RS 2
.IP \[bu] 2
US West (Oregon) Region
.IP \[bu] 2
Needs location constraint us\-west\-2.
.RE
.IP \[bu] 2
\[dq]ca\-central\-1\[dq] \[dq]ca\-central\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -16045,6 +16052,14 @@ EU (London) Region
Needs location constraint eu\-west\-2. Needs location constraint eu\-west\-2.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]eu\-west\-3\[dq]
.RS 2
.IP \[bu] 2
EU (Paris) Region
.IP \[bu] 2
Needs location constraint eu\-west\-3.
.RE
.IP \[bu] 2
\[dq]eu\-north\-1\[dq] \[dq]eu\-north\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -16053,6 +16068,14 @@ EU (Stockholm) Region
Needs location constraint eu\-north\-1. Needs location constraint eu\-north\-1.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]eu\-south\-1\[dq]
.RS 2
.IP \[bu] 2
EU (Milan) Region
.IP \[bu] 2
Needs location constraint eu\-south\-1.
.RE
.IP \[bu] 2
\[dq]eu\-central\-1\[dq] \[dq]eu\-central\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -16093,6 +16116,14 @@ Asia Pacific (Seoul)
Needs location constraint ap\-northeast\-2. Needs location constraint ap\-northeast\-2.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]ap\-northeast\-3\[dq]
.RS 2
.IP \[bu] 2
Asia Pacific (Osaka\-Local)
.IP \[bu] 2
Needs location constraint ap\-northeast\-3.
.RE
.IP \[bu] 2
\[dq]ap\-south\-1\[dq] \[dq]ap\-south\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -16104,7 +16135,7 @@ Needs location constraint ap\-south\-1.
\[dq]ap\-east\-1\[dq] \[dq]ap\-east\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Asia Patific (Hong Kong) Region Asia Pacific (Hong Kong) Region
.IP \[bu] 2 .IP \[bu] 2
Needs location constraint ap\-east\-1. Needs location constraint ap\-east\-1.
.RE .RE
@ -16116,6 +16147,54 @@ South America (Sao Paulo) Region
.IP \[bu] 2 .IP \[bu] 2
Needs location constraint sa\-east\-1. Needs location constraint sa\-east\-1.
.RE .RE
.IP \[bu] 2
\[dq]me\-south\-1\[dq]
.RS 2
.IP \[bu] 2
Middle East (Bahrain) Region
.IP \[bu] 2
Needs location constraint me\-south\-1.
.RE
.IP \[bu] 2
\[dq]af\-south\-1\[dq]
.RS 2
.IP \[bu] 2
Africa (Cape Town) Region
.IP \[bu] 2
Needs location constraint af\-south\-1.
.RE
.IP \[bu] 2
\[dq]cn\-north\-1\[dq]
.RS 2
.IP \[bu] 2
China (Beijing) Region
.IP \[bu] 2
Needs location constraint cn\-north\-1.
.RE
.IP \[bu] 2
\[dq]cn\-northwest\-1\[dq]
.RS 2
.IP \[bu] 2
China (Ningxia) Region
.IP \[bu] 2
Needs location constraint cn\-northwest\-1.
.RE
.IP \[bu] 2
\[dq]us\-gov\-east\-1\[dq]
.RS 2
.IP \[bu] 2
AWS GovCloud (US\-East) Region
.IP \[bu] 2
Needs location constraint us\-gov\-east\-1.
.RE
.IP \[bu] 2
\[dq]us\-gov\-west\-1\[dq]
.RS 2
.IP \[bu] 2
AWS GovCloud (US) Region
.IP \[bu] 2
Needs location constraint us\-gov\-west\-1.
.RE
.RE .RE
.SS \-\-s3\-region .SS \-\-s3\-region
.PP .PP
@ -16978,18 +17057,18 @@ Empty for US Region, Northern Virginia or Pacific Northwest.
US East (Ohio) Region. US East (Ohio) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]us\-west\-2\[dq]
.RS 2
.IP \[bu] 2
US West (Oregon) Region.
.RE
.IP \[bu] 2
\[dq]us\-west\-1\[dq] \[dq]us\-west\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
US West (Northern California) Region. US West (Northern California) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]us\-west\-2\[dq]
.RS 2
.IP \[bu] 2
US West (Oregon) Region.
.RE
.IP \[bu] 2
\[dq]ca\-central\-1\[dq] \[dq]ca\-central\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -17008,12 +17087,24 @@ EU (Ireland) Region.
EU (London) Region. EU (London) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]eu\-west\-3\[dq]
.RS 2
.IP \[bu] 2
EU (Paris) Region.
.RE
.IP \[bu] 2
\[dq]eu\-north\-1\[dq] \[dq]eu\-north\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
EU (Stockholm) Region. EU (Stockholm) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]eu\-south\-1\[dq]
.RS 2
.IP \[bu] 2
EU (Milan) Region.
.RE
.IP \[bu] 2
\[dq]EU\[dq] \[dq]EU\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
@ -17041,19 +17132,25 @@ Asia Pacific (Tokyo) Region.
\[dq]ap\-northeast\-2\[dq] \[dq]ap\-northeast\-2\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Asia Pacific (Seoul) Asia Pacific (Seoul) Region.
.RE
.IP \[bu] 2
\[dq]ap\-northeast\-3\[dq]
.RS 2
.IP \[bu] 2
Asia Pacific (Osaka\-Local) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]ap\-south\-1\[dq] \[dq]ap\-south\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Asia Pacific (Mumbai) Asia Pacific (Mumbai) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]ap\-east\-1\[dq] \[dq]ap\-east\-1\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Asia Pacific (Hong Kong) Asia Pacific (Hong Kong) Region.
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]sa\-east\-1\[dq] \[dq]sa\-east\-1\[dq]
@ -17061,6 +17158,42 @@ Asia Pacific (Hong Kong)
.IP \[bu] 2 .IP \[bu] 2
South America (Sao Paulo) Region. South America (Sao Paulo) Region.
.RE .RE
.IP \[bu] 2
\[dq]me\-south\-1\[dq]
.RS 2
.IP \[bu] 2
Middle East (Bahrain) Region.
.RE
.IP \[bu] 2
\[dq]af\-south\-1\[dq]
.RS 2
.IP \[bu] 2
Africa (Cape Town) Region.
.RE
.IP \[bu] 2
\[dq]cn\-north\-1\[dq]
.RS 2
.IP \[bu] 2
China (Beijing) Region
.RE
.IP \[bu] 2
\[dq]cn\-northwest\-1\[dq]
.RS 2
.IP \[bu] 2
China (Ningxia) Region.
.RE
.IP \[bu] 2
\[dq]us\-gov\-east\-1\[dq]
.RS 2
.IP \[bu] 2
AWS GovCloud (US\-East) Region.
.RE
.IP \[bu] 2
\[dq]us\-gov\-west\-1\[dq]
.RS 2
.IP \[bu] 2
AWS GovCloud (US) Region.
.RE
.RE .RE
.SS \-\-s3\-location\-constraint .SS \-\-s3\-location\-constraint
.PP .PP
@ -20227,7 +20360,7 @@ Note that Box is case insensitive so you can\[aq]t have a file called
.PP .PP
Box file names can\[aq]t have the \f[C]\[rs]\f[R] character in. Box file names can\[aq]t have the \f[C]\[rs]\f[R] character in.
rclone maps this to and from an identical looking unicode equivalent rclone maps this to and from an identical looking unicode equivalent
\f[C]\[uFF3C]\f[R]. \f[C]\[uFF3C]\f[R] (U+FF3C Fullwidth Reverse Solidus).
.PP .PP
Box only supports filenames up to 255 characters in length. Box only supports filenames up to 255 characters in length.
.SS Cache (BETA) .SS Cache (BETA)
@ -21896,28 +22029,29 @@ Default:
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
.SS Crypt .SS Crypt
.PP .PP
The \f[C]crypt\f[R] remote encrypts and decrypts another remote. Rclone \f[C]crypt\f[R] remotes encrypt and decrypt other remotes.
.PP .PP
To use it first set up the underlying remote following the config To use \f[C]crypt\f[R], first set up the underlying remote.
instructions for that remote. Follow the \f[C]rclone config\f[R] instructions for that remote.
You can also use a local pathname instead of a remote which will encrypt
and decrypt from that directory which might be useful for encrypting
onto a USB stick for example.
.PP .PP
First check your chosen remote is working \- we\[aq]ll call it \f[C]crypt\f[R] applied to a local pathname instead of a remote will
\f[C]remote:path\f[R] in these docs. encrypt and decrypt that directory, and can be used to encrypt USB
Note that anything inside \f[C]remote:path\f[R] will be encrypted and removable drives.
anything outside won\[aq]t.
This means that if you are using a bucket based remote (eg S3, B2,
swift) then you should probably put the bucket in the remote
\f[C]s3:bucket\f[R].
If you just use \f[C]s3:\f[R] then rclone will make encrypted bucket
names too (if using file name encryption) which may or may not be what
you want.
.PP .PP
Now configure \f[C]crypt\f[R] using \f[C]rclone config\f[R]. Before configuring the crypt remote, check the underlying remote is
We will call this one \f[C]secret\f[R] to differentiate it from the working.
\f[C]remote\f[R]. In this example the underlying remote is called \f[C]remote:path\f[R].
Anything inside \f[C]remote:path\f[R] will be encrypted and anything
outside will not.
In the case of an S3 based underlying remote (eg Amazon S3, B2, Swift)
it is generally advisable to define a crypt remote in the underlying
remote \f[C]s3:bucket\f[R].
If \f[C]s3:\f[R] alone is specified alongside file name encryption,
rclone will encrypt the bucket name.
.PP
Configure \f[C]crypt\f[R] using \f[C]rclone config\f[R].
In this example the \f[C]crypt\f[R] remote is called \f[C]secret\f[R],
to differentiate it from the underlying \f[C]remote\f[R].
.IP .IP
.nf .nf
\f[C] \f[C]
@ -21993,51 +22127,47 @@ y/e/d> y
\f[R] \f[R]
.fi .fi
.PP .PP
\f[B]Important\f[R] The password is stored in the config file is lightly \f[B]Important\f[R] The crypt password stored in \f[C]rclone.conf\f[R]
obscured so it isn\[aq]t immediately obvious what it is. is lightly obscured.
It is in no way secure unless you use config file encryption. That only protects it from cursory inspection.
It is not secure unless encryption of \f[C]rclone.conf\f[R] is
specified.
.PP .PP
A long passphrase is recommended, or you can use a random one. A long passphrase is recommended, or \f[C]rclone config\f[R] can
generate a random one.
.PP .PP
The obscured password is created by using AES\-CTR with a static key, The obscured password is created using AES\-CTR with a static key.
with the salt stored verbatim at the beginning of the obscured password. The salt is stored verbatim at the beginning of the obscured password.
This static key is shared by between all versions of rclone. This static key is shared between all versions of rclone.
.PP .PP
If you reconfigure rclone with the same passwords/passphrases elsewhere If you reconfigure rclone with the same passwords/passphrases elsewhere
it will be compatible, but the obscured version will be different due to it will be compatible, but the obscured version will be different due to
the different salt. the different salt.
.PP .PP
Note that rclone does not encrypt Rclone does not encrypt
.IP \[bu] 2 .IP \[bu] 2
file length \- this can be calculated within 16 bytes file length \- this can be calculated within 16 bytes
.IP \[bu] 2 .IP \[bu] 2
modification time \- used for syncing modification time \- used for syncing
.SS Specifying the remote .SS Specifying the remote
.PP .PP
In normal use, make sure the remote has a \f[C]:\f[R] in. In normal use, ensure the remote has a \f[C]:\f[R] in.
If you specify the remote without a \f[C]:\f[R] then rclone will use a If specified without, rclone uses a local directory of that name.
local directory of that name. For example if a remote \f[C]/path/to/secret/files\f[R] is specified,
So if you use a remote of \f[C]/path/to/secret/files\f[R] then rclone rclone encrypts content to that directory.
will encrypt stuff to that directory. If a remote \f[C]name\f[R] is specified, rclone targets a directory
If you use a remote of \f[C]name\f[R] then rclone will put files in a \f[C]name\f[R] in the current directory.
directory called \f[C]name\f[R] in the current directory.
.PP .PP
If you specify the remote as \f[C]remote:path/to/dir\f[R] then rclone If remote \f[C]remote:path/to/dir\f[R] is specified, rclone stores
will store encrypted files in \f[C]path/to/dir\f[R] on the remote. encrypted files in \f[C]path/to/dir\f[R] on the remote.
If you are using file name encryption, then when you save files to With file name encryption, files saved to
\f[C]secret:subdir/subfile\f[R] this will store them in the unencrypted \f[C]secret:subdir/subfile\f[R] are stored in the unencrypted path
path \f[C]path/to/dir\f[R] but the \f[C]subdir/subpath\f[R] bit will be \f[C]path/to/dir\f[R] but the \f[C]subdir/subpath\f[R] element is
encrypted. encrypted.
.PP
Note that unless you want encrypted bucket names (which are difficult to
manage because you won\[aq]t know what directory they represent in web
interfaces etc), you should probably specify a bucket, eg
\f[C]remote:secretbucket\f[R] when using bucket based remotes such as
S3, Swift, Hubic, B2, GCS.
.SS Example .SS Example
.PP .PP
To test I made a little directory of files using \[dq]standard\[dq] file Create the following file structure using \[dq]standard\[dq] file name
name encryption. encryption.
.IP .IP
.nf .nf
\f[C] \f[C]
@ -22052,7 +22182,7 @@ plaintext/
\f[R] \f[R]
.fi .fi
.PP .PP
Copy these to the remote and list them back Copy these to the remote, and list them
.IP .IP
.nf .nf
\f[C] \f[C]
@ -22066,7 +22196,7 @@ $ rclone \-q ls secret:
\f[R] \f[R]
.fi .fi
.PP .PP
Now see what that looked like when encrypted The crypt remote looks like
.IP .IP
.nf .nf
\f[C] \f[C]
@ -22079,8 +22209,7 @@ $ rclone \-q ls remote:path
\f[R] \f[R]
.fi .fi
.PP .PP
Note that this retains the directory structure which means you can do The directory structure is preserved
this
.IP .IP
.nf .nf
\f[C] \f[C]
@ -22091,9 +22220,9 @@ $ rclone \-q ls secret:subdir
\f[R] \f[R]
.fi .fi
.PP .PP
If don\[aq]t use file name encryption then the remote will look like Without file name encryption \f[C].bin\f[R] extensions are added to
this \- note the \f[C].bin\f[R] extensions added to prevent the cloud underlying names.
provider attempting to interpret the data. This prevents the cloud provider attempting to interpret file content.
.IP .IP
.nf .nf
\f[C] \f[C]
@ -22107,8 +22236,6 @@ $ rclone \-q ls remote:path
.fi .fi
.SS File name encryption modes .SS File name encryption modes
.PP .PP
Here are some of the features of the file name encryption modes
.PP
Off Off
.IP \[bu] 2 .IP \[bu] 2
doesn\[aq]t hide file names or directory structure doesn\[aq]t hide file names or directory structure
@ -22135,19 +22262,19 @@ Obfuscation
.PP .PP
This is a simple \[dq]rotate\[dq] of the filename, with each file having This is a simple \[dq]rotate\[dq] of the filename, with each file having
a rot distance based on the filename. a rot distance based on the filename.
We store the distance at the beginning of the filename. Rclone stores the distance at the beginning of the filename.
So a file called \[dq]hello\[dq] may become \[dq]53.jgnnq\[dq]. A file called \[dq]hello\[dq] may become \[dq]53.jgnnq\[dq].
.PP .PP
This is not a strong encryption of filenames, but it may stop automated Obfuscation is not a strong encryption of filenames, but hinders
scanning tools from picking up on filename patterns. automated scanning tools picking up on filename patterns.
As such it\[aq]s an intermediate between \[dq]off\[dq] and It is an intermediate between \[dq]off\[dq] and \[dq]standard\[dq] which
\[dq]standard\[dq]. allows for longer path segment names.
The advantage is that it allows for longer path segment names.
.PP .PP
There is a possibility with some unicode based filenames that the There is a possibility with some unicode based filenames that the
obfuscation is weak and may map lower case characters to upper case obfuscation is weak and may map lower case characters to upper case
equivalents. equivalents.
You can not rely on this for strong protection. .PP
Obfuscation cannot be relied upon for strong protection.
.IP \[bu] 2 .IP \[bu] 2
file names very lightly obfuscated file names very lightly obfuscated
.IP \[bu] 2 .IP \[bu] 2
@ -22159,14 +22286,14 @@ directory structure visible
.IP \[bu] 2 .IP \[bu] 2
identical files names will have identical uploaded names identical files names will have identical uploaded names
.PP .PP
Cloud storage systems have various limits on file name length and total Cloud storage systems have limits on file name length and total path
path length which you are more likely to hit using \[dq]Standard\[dq] length which rclone is more likely to breach using \[dq]Standard\[dq]
file name encryption. file name encryption.
If you keep your file names to below 156 characters in length then you Where file names are less thn 156 characters in length issues should not
should be OK on all providers. be encountered, irrespective of cloud storage provider.
.PP .PP
There may be an even more secure file name encryption mode in the future An alternative, future rclone file name encryption mode may tolerate
which will address the long file name problem. backend provider path length limits.
.SS Directory name encryption .SS Directory name encryption
.PP .PP
Crypt offers the option of encrypting dir names or leaving them intact. Crypt offers the option of encrypting dir names or leaving them intact.
@ -22192,9 +22319,9 @@ Hashes are not stored for crypt.
However the data integrity is protected by an extremely strong crypto However the data integrity is protected by an extremely strong crypto
authenticator. authenticator.
.PP .PP
Note that you should use the \f[C]rclone cryptcheck\f[R] command to Use the \f[C]rclone cryptcheck\f[R] command to check the integrity of a
check the integrity of a crypted remote instead of crypted remote instead of \f[C]rclone check\f[R] which can\[aq]t check
\f[C]rclone check\f[R] which can\[aq]t check the checksums properly. the checksums properly.
.SS Standard Options .SS Standard Options
.PP .PP
Here are the standard options specific to crypt (Encrypt/Decrypt a Here are the standard options specific to crypt (Encrypt/Decrypt a
@ -34961,6 +35088,139 @@ Options:
.IP \[bu] 2 .IP \[bu] 2
\[dq]error\[dq]: return an error based on option value \[dq]error\[dq]: return an error based on option value
.SH Changelog .SH Changelog
.SS v1.53.2 \- 2020\-10\-26
.PP
See commits (https://github.com/rclone/rclone/compare/v1.53.1...v1.53.2)
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
acounting
.RS 2
.IP \[bu] 2
Fix incorrect speed and transferTime in core/stats (Nick Craig\-Wood)
.IP \[bu] 2
Stabilize display order of transfers on Windows (Nick Craig\-Wood)
.RE
.IP \[bu] 2
operations
.RS 2
.IP \[bu] 2
Fix use of \-\-suffix without \-\-backup\-dir (Nick Craig\-Wood)
.IP \[bu] 2
Fix spurious \[dq]\-\-checksum is in use but the source and destination
have no hashes in common\[dq] (Nick Craig\-Wood)
.RE
.IP \[bu] 2
build
.RS 2
.IP \[bu] 2
Work around GitHub actions brew problem (Nick Craig\-Wood)
.IP \[bu] 2
Stop using set\-env and set\-path in the GitHub actions (Nick
Craig\-Wood)
.RE
.RE
.IP \[bu] 2
Mount
.RS 2
.IP \[bu] 2
mount2: Fix the swapped UID / GID values (Russell Cattelan)
.RE
.IP \[bu] 2
VFS
.RS 2
.IP \[bu] 2
Detect and recover from a file being removed externally from the cache
(Nick Craig\-Wood)
.IP \[bu] 2
Fix a deadlock vulnerability in downloaders.Close (Leo Luan)
.IP \[bu] 2
Fix a race condition in retryFailedResets (Leo Luan)
.IP \[bu] 2
Fix missed concurrency control between some item operations and reset
(Leo Luan)
.IP \[bu] 2
Add exponential backoff during ENOSPC retries (Leo Luan)
.IP \[bu] 2
Add a missed update of used cache space (Leo Luan)
.IP \[bu] 2
Fix \-\-no\-modtime to not attempt to set modtimes (as documented) (Nick
Craig\-Wood)
.RE
.IP \[bu] 2
Local
.RS 2
.IP \[bu] 2
Fix sizes and syncing with \-\-links option on Windows (Nick
Craig\-Wood)
.RE
.IP \[bu] 2
Chunker
.RS 2
.IP \[bu] 2
Disable ListR to fix missing files on GDrive (workaround) (Ivan Andreev)
.IP \[bu] 2
Fix upload over crypt (Ivan Andreev)
.RE
.IP \[bu] 2
Fichier
.RS 2
.IP \[bu] 2
Increase maximum file size from 100GB to 300GB (gyutw)
.RE
.IP \[bu] 2
Jottacloud
.RS 2
.IP \[bu] 2
Remove clientSecret from config when upgrading to token based
authentication (buengese)
.IP \[bu] 2
Avoid double url escaping of device/mountpoint (albertony)
.IP \[bu] 2
Remove DirMove workaround as it\[aq]s not required anymore \- also
(buengese)
.RE
.IP \[bu] 2
Mailru
.RS 2
.IP \[bu] 2
Fix uploads after recent changes on server (Ivan Andreev)
.IP \[bu] 2
Fix range requests after june changes on server (Ivan Andreev)
.IP \[bu] 2
Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
.RE
.IP \[bu] 2
Onedrive
.RS 2
.IP \[bu] 2
Fix disk usage for sharepoint (Nick Craig\-Wood)
.RE
.IP \[bu] 2
S3
.RS 2
.IP \[bu] 2
Add missing regions for AWS (Anagh Kumar Baranwal)
.RE
.IP \[bu] 2
Seafile
.RS 2
.IP \[bu] 2
Fix accessing libraries > 2GB on 32 bit systems (Muffin King)
.RE
.IP \[bu] 2
SFTP
.RS 2
.IP \[bu] 2
Always convert the checksum to lower case (buengese)
.RE
.IP \[bu] 2
Union
.RS 2
.IP \[bu] 2
Create root directories if none exist (Nick Craig\-Wood)
.RE
.SS v1.53.1 \- 2020\-09\-13 .SS v1.53.1 \- 2020\-09\-13
.PP .PP
See commits (https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1) See commits (https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1)