crypt: docs: extended description

docs/content/crypt.md

@@ -8,59 +8,126 @@ description: "Encryption overlay remote"
 
 Rclone `crypt` remotes encrypt and decrypt other remotes.
 
-To use `crypt`, first set up the underlying remote. Follow the `rclone
+A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/)
-config` instructions for that remote.
+directly, but instead wraps another remote, which in turn accesses
+the storage system. This is similar to how [alias](https://rclone.org/alias/),
+[union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/)
+and a few others work. It makes the usage very flexible, as you can
+add a layer, in this case an encryption layer, on top of any other
+backend, even in multiple layers. Rclone's functionality
+can be used as with any other remote, for example you can
+[mount](https://rclone.org/commands/rclone_mount/) a crypt remote.
 
-`crypt` applied to a local pathname instead of a remote will
+Accessing a storage system through a crypt remote realizes client-side
-encrypt and decrypt that directory, and can be used to encrypt USB
+encryption, which makes it safe to keep your data in a location you do
-removable drives.
+not trust will not get compromised.
+When working against the `crypt` remote, rclone will automatically
+encrypt (before uploading) and decrypt (after downloading) on your local
+system as needed on the fly, leaving the data encrypted at rest in the
+wrapped remote. If you access the storage system using an application
+other than rclone, or access the wrapped remote directly using rclone,
+there will not be any encryption/decryption: Downloading existing content
+will just give you the encrypted (scrambled) format, and anything you
+upload will *not* become encrypted.
+
+The encryption is a secret-key encryption (also called symmetric key encryption)
+algorithm, where a password (or pass phrase) is used to generate real encryption key.
+The password can be supplied by user, or you may chose to let rclone
+generate one. It will be stored in the configuration file, in a lightly obscured form.
+If you are in an environment where you are not able to keep your configuration
+secured, you should add
+[configuration encryption](https://rclone.org/docs/#configuration-encryption)
+as protection. As long as you have this configuration file, you will be able to
+decrypt your data. Without the configuration file, as long as you remember
+the password (or keep it in a safe place), you can re-create the configuration
+and gain access to the existing data. You may also configure a corresponding
+remote in a different installation to access the same data.
+See below for guidance to [changing password](#changing-password).
+
+Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)),
+to permute the encryption key so that the same string may be encrypted in
+different ways. When configuring the crypt remote it is optional to enter a salt,
+or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.
+Normally in cryptography, the salt is stored together with the encrypted content,
+and do not have to be memorized by the user. This is not the case in rclone,
+because rclone does not store any additional information on the remotes. Use of
+custom salt is effectively a second password that must be memorized.
+
+[File content](#file-encryption) encryption is performed using
+[NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox),
+based on XSalsa20 cipher and Poly1305 for integrity.
+[Names](#name-encryption) (file- and directory names) are also encrypted
+by default, but this has some implications and is therefore
+possible to turned off.
+
+### Configuration
+
+Here is an example of how to make a remote called `secret`.
+
+To use `crypt`, first set up the underlying remote. Follow the
+`rclone config` instructions for the specific backend.
 
 Before configuring the crypt remote, check the underlying remote is
-working. In this example the underlying remote is called `remote:path`.
+working. In this example the underlying remote is called `remote`.
-Anything inside `remote:path` will be encrypted and anything outside
+We will configure a path `path` within this remote to contain the
-will not. In the case of an S3 based underlying remote (e.g. Amazon S3,
+encrypted content. Anything inside `remote:path` will be encrypted
-B2, Swift) it is generally advisable to define a crypt remote in the
+and anything outside will not.
-underlying remote `s3:bucket`. If `s3:` alone is specified alongside
-file name encryption, rclone will encrypt the bucket name.
 
 Configure `crypt` using `rclone config`. In this example the `crypt`
 remote is called `secret`, to differentiate it from the underlying
 `remote`.
 
+When you are done you can use the crypt remote named `secret` just
+as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
+and rclone will encrypt and decrypt as needed on the fly.
+If you access the wrapped remote `remote:path` directly you will bypass
+the encryption, and anything you read will be in encrypted form, and
+anything you write will be undencrypted. To avoid issues it is best to
+configure a dedicated path for encrypted content, and access it
+exclusively through a crypt remote.
+
 ```
 No remotes found - make a new one
 n) New remote
 s) Set configuration password
 q) Quit config
-n/s/q> n   
+n/s/q> n
 name> secret
 Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
 Choose a number from below, or type in your own value
 [snip]
 XX / Encrypt/Decrypt a remote
    \ "crypt"
 [snip]
 Storage> crypt
+** See help for crypt backend at: https://rclone.org/crypt/ **
+
 Remote to encrypt/decrypt.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+Normally should contain a ':' and a path, eg "myremote:path/to/dir",
 "myremote:bucket" or maybe "myremote:" (not recommended).
+Enter a string value. Press Enter for the default ("").
 remote> remote:path
 How to encrypt the filenames.
+Enter a string value. Press Enter for the default ("standard").
 Choose a number from below, or type in your own value
- 1 / Don't encrypt the file names.  Adds a ".bin" extension only.
+ 1 / Encrypt the filenames see the docs for the details.
-   \ "off"
- 2 / Encrypt the filenames see the docs for the details.
    \ "standard"
- 3 / Very simple filename obfuscation.
+ 2 / Very simple filename obfuscation.
    \ "obfuscate"
-filename_encryption> 2
+ 3 / Don't encrypt the file names.  Adds a ".bin" extension only.
+   \ "off"
+filename_encryption>
 Option to either encrypt directory names or leave them intact.
+
+NB If filename_encryption is "off" then this option will do nothing.
+Enter a boolean value (true or false). Press Enter for the default ("true").
 Choose a number from below, or type in your own value
  1 / Encrypt directory names.
    \ "true"
  2 / Don't encrypt directory names, leave them intact.
    \ "false"
-directory_name_encryption> 1
+directory_name_encryption>
 Password or pass phrase for encryption.
 y) Yes type in my own password
 g) Generate random password
@@ -73,7 +140,7 @@ Password or pass phrase for salt. Optional but recommended.
 Should be different to the previous password.
 y) Yes type in my own password
 g) Generate random password
-n) No leave this optional password blank
+n) No leave this optional password blank (default)
 y/g/n> g
 Password strength in bits.
 64 is just about memorable
@@ -81,27 +148,33 @@ Password strength in bits.
 1024 is the maximum
 Bits> 128
 Your password is: JAsJvRcgR-_veXNfy_sGmQ
-Use this password?
+Use this password? Please note that an obscured version of this
-y) Yes
+password (and not the password itself) will be stored under your
+configuration file, so keep this generated password in a safe place.
+y) Yes (default)
 n) No
-y/n> y
+y/n>
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n>
 Remote config
 --------------------
 [secret]
+type = crypt
 remote = remote:path
-filename_encryption = standard
 password = *** ENCRYPTED ***
 password2 = *** ENCRYPTED ***
 --------------------
-y) Yes this is OK
+y) Yes this is OK (default)
 e) Edit this remote
 d) Delete this remote
-y/e/d> y
+y/e/d>
 ```
 
 **Important** The crypt password stored in `rclone.conf` is lightly
 obscured. That only protects it from cursory inspection. It is not
-secure unless encryption of `rclone.conf` is specified.
+secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified.
 
 A long passphrase is recommended, or `rclone config` can generate a
 random one.
@@ -119,20 +192,80 @@ Rclone does not encrypt
   * file length - this can be calculated within 16 bytes
   * modification time - used for syncing
 
-## Specifying the remote ##
+### Specifying the remote
 
-In normal use, ensure the remote has a `:` in. If specified without,
+When configuring the remote to encrypt/decrypt, you may specify any
-rclone uses a local directory of that name.  For example if a remote
+string that rclone accepts as a source/destination of other commands.
-`/path/to/secret/files` is specified, rclone encrypts content to that
-directory.  If a remote `name` is specified, rclone targets a directory
-`name` in the current directory.
 
-If remote `remote:path/to/dir` is specified, rclone stores encrypted
+The primary use case is to specify the path into an already configured
+remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that
+data in a remote untrusted location can be stored encrypted.
+
+You may also specify a local filesystem path, such as
+`/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating
+a crypt remote pointing to such a local filesystem path, you can
+use rclone as a utility for pure local file encryption, for example
+to keep encrypted files on a removable USB drive.
+
+**Note**: A string which do not contain a `:` will by rclone be treated
+as a relative path in the local filesystem. For example, if you enter
+the name `remote` without the trailing `:`, it will be treated as
+a subdirectory of the current directory with name "remote".
+
+If a path `remote:path/to/dir` is specified, rclone stores encrypted
 files in `path/to/dir` on the remote. With file name encryption, files
 saved to `secret:subdir/subfile` are stored in the unencrypted path
 `path/to/dir` but the `subdir/subpath` element is encrypted.
 
-## Example ##
+The path you specify does not have to exist, rclone will create
+it when needed.
+
+If you intend to use the wrapped remote both directly for keeping
+unencrypted content, as well as through a crypt remote for encrypted
+content, it is recommended to point the crypt remote to a separate
+directory within the wrapped remote. If you use a bucket based storage
+system (e.g. Swift, S3, Google Compute Storage, B2, Hubic) it is generally
+advisable to wrap the crypt remote around a specific bucket (`s3:bucket`).
+If wrapping around the entire root of the storage (`s3:`), and use the
+optional file name encryption, rclone will encrypt the bucket name.
+
+### Changing password
+
+Should the password, or the configuration file containing a lightly obscured
+form of the password, be compromised, you need to re-encrypt your data with
+a new password. Since rclone uses secret-key encryption, where the encryption
+key is generated directly from the password kept on the client, it is not
+possible to change the password/key of already encrypted content. Just changing
+the password configured for an existing crypt remote means you will no longer
+able to decrypt any of the previously encrypted content. The only possibility
+is to re-upload everything via a crypt remote configured with your new password.
+
+Depending on the size of your data, your bandwith, storage quota etc, there are
+different approaches you can take:
+- If you have everything in a different location, for example on your local system,
+you could remove all of the prior encrypted files, change the password for your
+configured crypt remote (or delete and re-create the crypt configuration),
+and then re-upload everything from the alternative location.
+- If you have enough space on the storage system you can create a new crypt
+remote pointing to a separate directory on the same backend, and then use
+rclone to copy everything from the original crypt remote to the new,
+effectively decrypting everything on the fly using the old password and
+re-encrypting using the new password. When done, delete the original crypt
+remote directory and finally the rclone crypt configuration with the old password.
+All data will be streamed from the storage system and back, so you will
+get half the bandwith and be charged twice if you have upload and download quota
+on the storage system.
+
+**Note**: A security problem related to the random password generator
+was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated
+by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2
+(released 2020-10-26) are not considered secure and should be changed.
+If you made up your own password, or used rclone version older than 1.49.0 or
+newer than 1.53.2 to generate it, you are *not* affected by this issue.
+See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more
+details, and a tool you can use to check if you are affected.
+
+### Example
 
 Create the following file structure using "standard" file name
 encryption.
@@ -193,7 +326,7 @@ $ rclone -q ls remote:path
        55 file1.txt.bin
 ```
 
-### File name encryption modes ###
+### File name encryption modes
 
 Off
 
@@ -223,7 +356,7 @@ segment names.
 
 There is a possibility with some unicode based filenames that the
 obfuscation is weak and may map lower case characters to upper case
-equivalents. 
+equivalents.
 
 Obfuscation cannot be relied upon for strong protection.
 
@@ -242,7 +375,8 @@ cloud storage provider.
 An alternative, future rclone file name encryption mode may tolerate
 backend provider path length limits.
 
-### Directory name encryption ###
+### Directory name encryption
+
 Crypt offers the option of encrypting dir names or leaving them intact.
 There are two options:
 
@@ -261,7 +395,7 @@ Example:
 `1/12/qgm4avr35m5loi1th53ato71v0`
 
 
-### Modified time and hashes ###
+### Modified time and hashes
 
 Crypt stores modification times using the underlying remote so support
 depends on that.
@@ -432,7 +566,7 @@ Usage Example:
 
 {{< rem autogenerated options stop >}}
 
-## Backing up a crypted remote ##
+## Backing up a crypted remote
 
 If you wish to backup a crypted remote, it is recommended that you use
 `rclone sync` on the encrypted files, and make sure the passwords are
@@ -458,14 +592,14 @@ And to check the integrity you would do
 
     rclone check remote:crypt remote2:crypt
 
-## File formats ##
+## File formats
 
-### File encryption ###
+### File encryption
 
 Files are encrypted 1:1 source file to destination object.  The file
 has a header and is divided into chunks.
 
-#### Header ####
+#### Header
 
   * 8 bytes magic string `RCLONE\x00\x00`
   * 24 bytes Nonce (IV)
@@ -477,11 +611,11 @@ The chance of a nonce being re-used is minuscule.  If you wrote an
 exabyte of data (10¹⁸ bytes) you would have a probability of
 approximately 2×10⁻³² of re-using a nonce.
 
-#### Chunk ####
+#### Chunk
 
 Each chunk will contain 64kB of data, except for the last one which
-may have less data.  The data chunk is in standard NACL secretbox
+may have less data. The data chunk is in standard NaCl SecretBox
-format. Secretbox uses XSalsa20 and Poly1305 to encrypt and
+format. SecretBox uses XSalsa20 and Poly1305 to encrypt and
 authenticate messages.
 
 Each chunk contains:
@@ -496,7 +630,7 @@ buffered in memory so they can't be too big.
 
 This uses a 32 byte (256 bit key) key derived from the user password.
 
-#### Examples ####
+#### Examples
 
 1 byte file will encrypt to
 
@@ -513,7 +647,7 @@ This uses a 32 byte (256 bit key) key derived from the user password.
 1049120 bytes total (a 0.05% overhead).  This is the overhead for big
 files.
 
-### Name encryption ###
+### Name encryption
 
 File names are encrypted segment by segment - the path is broken up
 into `/` separated strings and these are encrypted individually.
@@ -547,7 +681,7 @@ encoding is modified in two ways:
 `base32` is used rather than the more efficient `base64` so rclone can be
 used on case insensitive remotes (e.g. Windows, Amazon Drive).
 
-### Key derivation ###
+### Key derivation
 
 Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an
 optional user supplied salt (password2) to derive the 32+32+16 = 80