Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
16 KiB
title | description |
---|---|
Crypt | Encryption overlay remote |
{{< icon "fa fa-lock" >}}Crypt
Rclone crypt
remotes encrypt and decrypt other remotes.
To use crypt
, first set up the underlying remote. Follow the rclone config
instructions for that remote.
crypt
applied to a local pathname instead of a remote will
encrypt and decrypt that directory, and can be used to encrypt USB
removable drives.
Before configuring the crypt remote, check the underlying remote is
working. In this example the underlying remote is called remote:path
.
Anything inside remote:path
will be encrypted and anything outside
will not. In the case of an S3 based underlying remote (eg Amazon S3,
B2, Swift) it is generally advisable to define a crypt remote in the
underlying remote s3:bucket
. If s3:
alone is specified alongside
file name encryption, rclone will encrypt the bucket name.
Configure crypt
using rclone config
. In this example the crypt
remote is called secret
, to differentiate it from the underlying
remote
.
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> secret
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Encrypt/Decrypt a remote
\ "crypt"
[snip]
Storage> crypt
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
remote> remote:path
How to encrypt the filenames.
Choose a number from below, or type in your own value
1 / Don't encrypt the file names. Adds a ".bin" extension only.
\ "off"
2 / Encrypt the filenames see the docs for the details.
\ "standard"
3 / Very simple filename obfuscation.
\ "obfuscate"
filename_encryption> 2
Option to either encrypt directory names or leave them intact.
Choose a number from below, or type in your own value
1 / Encrypt directory names.
\ "true"
2 / Don't encrypt directory names, leave them intact.
\ "false"
filename_encryption> 1
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:
Password or pass phrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> g
Password strength in bits.
64 is just about memorable
128 is secure
1024 is the maximum
Bits> 128
Your password is: JAsJvRcgR-_veXNfy_sGmQ
Use this password?
y) Yes
n) No
y/n> y
Remote config
--------------------
[secret]
remote = remote:path
filename_encryption = standard
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Important The crypt password stored in rclone.conf
is lightly
obscured. That only protects it from cursory inspection. It is not
secure unless encryption of rclone.conf
is specified.
A long passphrase is recommended, or rclone config
can generate a
random one.
The obscured password is created using AES-CTR with a static key. The salt is stored verbatim at the beginning of the obscured password. This static key is shared between all versions of rclone.
If you reconfigure rclone with the same passwords/passphrases elsewhere it will be compatible, but the obscured version will be different due to the different salt.
Rclone does not encrypt
- file length - this can be calculated within 16 bytes
- modification time - used for syncing
Specifying the remote
In normal use, ensure the remote has a :
in. If specified without,
rclone uses a local directory of that name. For example if a remote
/path/to/secret/files
is specified, rclone encrypts content to that
directory. If a remote name
is specified, rclone targets a directory
name
in the current directory.
If remote remote:path/to/dir
is specified, rclone stores encrypted
files in path/to/dir
on the remote. With file name encryption, files
saved to secret:subdir/subfile
are stored in the unencrypted path
path/to/dir
but the subdir/subpath
element is encrypted.
Example
Create the following file structure using "standard" file name encryption.
plaintext/
├── file0.txt
├── file1.txt
└── subdir
├── file2.txt
├── file3.txt
└── subsubdir
└── file4.txt
Copy these to the remote, and list them
$ rclone -q copy plaintext secret:
$ rclone -q ls secret:
7 file1.txt
6 file0.txt
8 subdir/file2.txt
10 subdir/subsubdir/file4.txt
9 subdir/file3.txt
The crypt remote looks like
$ rclone -q ls remote:path
55 hagjclgavj2mbiqm6u6cnjjqcg
54 v05749mltvv1tf4onltun46gls
57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
The directory structure is preserved
$ rclone -q ls secret:subdir
8 file2.txt
9 file3.txt
10 subsubdir/file4.txt
Without file name encryption .bin
extensions are added to underlying
names. This prevents the cloud provider attempting to interpret file
content.
$ rclone -q ls remote:path
54 file0.txt.bin
57 subdir/file3.txt.bin
56 subdir/file2.txt.bin
58 subdir/subsubdir/file4.txt.bin
55 file1.txt.bin
File name encryption modes
Off
- doesn't hide file names or directory structure
- allows for longer file names (~246 characters)
- can use sub paths and copy single files
Standard
- file names encrypted
- file names can't be as long (~143 characters)
- can use sub paths and copy single files
- directory structure visible
- identical files names will have identical uploaded names
- can use shortcuts to shorten the directory recursion
Obfuscation
This is a simple "rotate" of the filename, with each file having a rot distance based on the filename. Rclone stores the distance at the beginning of the filename. A file called "hello" may become "53.jgnnq".
Obfuscation is not a strong encryption of filenames, but hinders automated scanning tools picking up on filename patterns. It is an intermediate between "off" and "standard" which allows for longer path segment names.
There is a possibility with some unicode based filenames that the obfuscation is weak and may map lower case characters to upper case equivalents.
Obfuscation cannot be relied upon for strong protection.
- file names very lightly obfuscated
- file names can be longer than standard encryption
- can use sub paths and copy single files
- directory structure visible
- identical files names will have identical uploaded names
Cloud storage systems have limits on file name length and total path length which rclone is more likely to breach using "Standard" file name encryption. Where file names are less thn 156 characters in length issues should not be encountered, irrespective of cloud storage provider.
An alternative, future rclone file name encryption mode may tolerate backend provider path length limits.
Directory name encryption
Crypt offers the option of encrypting dir names or leaving them intact. There are two options:
True
Encrypts the whole file path including directory names
Example:
1/12/123.txt
is encrypted to
p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0
False
Only encrypts file names, skips directory names
Example:
1/12/123.txt
is encrypted to
1/12/qgm4avr35m5loi1th53ato71v0
Modified time and hashes
Crypt stores modification times using the underlying remote so support depends on that.
Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.
Use the rclone cryptcheck
command to check the
integrity of a crypted remote instead of rclone check
which can't
check the checksums properly.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/crypt/crypt.go then run make backenddocs" >}}
Standard Options
Here are the standard options specific to crypt (Encrypt/Decrypt a remote).
--crypt-remote
Remote to encrypt/decrypt. Normally should contain a ':' and a path, eg "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended).
- Config: remote
- Env Var: RCLONE_CRYPT_REMOTE
- Type: string
- Default: ""
--crypt-filename-encryption
How to encrypt the filenames.
- Config: filename_encryption
- Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
- Type: string
- Default: "standard"
- Examples:
- "standard"
- Encrypt the filenames see the docs for the details.
- "obfuscate"
- Very simple filename obfuscation.
- "off"
- Don't encrypt the file names. Adds a ".bin" extension only.
- "standard"
--crypt-directory-name-encryption
Option to either encrypt directory names or leave them intact.
NB If filename_encryption is "off" then this option will do nothing.
- Config: directory_name_encryption
- Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
- Type: bool
- Default: true
- Examples:
- "true"
- Encrypt directory names.
- "false"
- Don't encrypt directory names, leave them intact.
- "true"
--crypt-password
Password or pass phrase for encryption.
NB Input to this must be obscured - see rclone obscure.
- Config: password
- Env Var: RCLONE_CRYPT_PASSWORD
- Type: string
- Default: ""
--crypt-password2
Password or pass phrase for salt. Optional but recommended. Should be different to the previous password.
NB Input to this must be obscured - see rclone obscure.
- Config: password2
- Env Var: RCLONE_CRYPT_PASSWORD2
- Type: string
- Default: ""
Advanced Options
Here are the advanced options specific to crypt (Encrypt/Decrypt a remote).
--crypt-server-side-across-configs
Allow server-side operations (eg copy) to work across different crypt configs.
Normally this option is not what you want, but if you have two crypts pointing to the same backend you can use it.
This can be used, for example, to change file name encryption type without re-uploading all the data. Just make two crypt backends pointing to two different directories with the single changed parameter and use rclone move to move the files between the crypt remotes.
- Config: server_side_across_configs
- Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
- Type: bool
- Default: false
--crypt-show-mapping
For all files listed show how the names encrypt.
If this flag is set then for each file that the remote is asked to list, it will log (at level INFO) a line stating the decrypted file name and the encrypted file name.
This is so you can work out which encrypted names are which decrypted names just in case you need to do something with the encrypted file names, or for debugging purposes.
- Config: show_mapping
- Env Var: RCLONE_CRYPT_SHOW_MAPPING
- Type: bool
- Default: false
Backend commands
Here are the commands specific to the crypt backend.
Run them with
rclone backend COMMAND remote:
The help below will explain what arguments each command takes.
See the "rclone backend" command for more info on how to pass options and arguments.
These can be run on a running backend using the rc command backend/command.
encode
Encode the given filename(s)
rclone backend encode remote: [options] [<arguments>+]
This encodes the filenames given as arguments returning a list of strings of the encoded results.
Usage Example:
rclone backend encode crypt: file1 [file2...]
rclone rc backend/command command=encode fs=crypt: file1 [file2...]
decode
Decode the given filename(s)
rclone backend decode remote: [options] [<arguments>+]
This decodes the filenames given as arguments returning a list of strings of the decoded results. It will return an error if any of the inputs are invalid.
Usage Example:
rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
{{< rem autogenerated options stop >}}
Backing up a crypted remote
If you wish to backup a crypted remote, it is recommended that you use
rclone sync
on the encrypted files, and make sure the passwords are
the same in the new encrypted remote.
This will have the following advantages
rclone sync
will check the checksums while copying- you can use
rclone check
between the encrypted remotes - you don't decrypt and encrypt unnecessarily
For example, let's say you have your original remote at remote:
with
the encrypted version at eremote:
with path remote:crypt
. You
would then set up the new remote remote2:
and then the encrypted
version eremote2:
with path remote2:crypt
using the same passwords
as eremote:
.
To sync the two remotes you would do
rclone sync -i remote:crypt remote2:crypt
And to check the integrity you would do
rclone check remote:crypt remote2:crypt
File formats
File encryption
Files are encrypted 1:1 source file to destination object. The file has a header and is divided into chunks.
Header
- 8 bytes magic string
RCLONE\x00\x00
- 24 bytes Nonce (IV)
The initial nonce is generated from the operating systems crypto strong random number generator. The nonce is incremented for each chunk read making sure each nonce is unique for each block written. The chance of a nonce being re-used is minuscule. If you wrote an exabyte of data (10¹⁸ bytes) you would have a probability of approximately 2×10⁻³² of re-using a nonce.
Chunk
Each chunk will contain 64kB of data, except for the last one which may have less data. The data chunk is in standard NACL secretbox format. Secretbox uses XSalsa20 and Poly1305 to encrypt and authenticate messages.
Each chunk contains:
- 16 Bytes of Poly1305 authenticator
- 1 - 65536 bytes XSalsa20 encrypted data
64k chunk size was chosen as the best performing chunk size (the authenticator takes too much time below this and the performance drops off due to cache effects above this). Note that these chunks are buffered in memory so they can't be too big.
This uses a 32 byte (256 bit key) key derived from the user password.
Examples
1 byte file will encrypt to
- 32 bytes header
- 17 bytes data chunk
49 bytes total
1MB (1048576 bytes) file will encrypt to
- 32 bytes header
- 16 chunks of 65568 bytes
1049120 bytes total (a 0.05% overhead). This is the overhead for big files.
Name encryption
File names are encrypted segment by segment - the path is broken up
into /
separated strings and these are encrypted individually.
File segments are padded using PKCS#7 to a multiple of 16 bytes before encryption.
They are then encrypted with EME using AES with 256 bit key. EME (ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
This makes for deterministic encryption which is what we want - the same filename must encrypt to the same thing otherwise we can't find it on the cloud storage system.
This means that
- filenames with the same name will encrypt the same
- filenames which start the same won't have a common prefix
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of which are derived from the user password.
After encryption they are written out using a modified version of
standard base32
encoding as described in RFC4648. The standard
encoding is modified in two ways:
- it becomes lower case (no-one likes upper case filenames!)
- we strip the padding character
=
base32
is used rather than the more efficient base64
so rclone can be
used on case insensitive remotes (eg Windows, Amazon Drive).
Key derivation
Rclone uses scrypt
with parameters N=16384, r=8, p=1
with an
optional user supplied salt (password2) to derive the 32+32+16 = 80
bytes of key material required. If the user doesn't supply a salt
then rclone uses an internal one.
scrypt
makes it impractical to mount a dictionary attack on rclone
encrypted data. For full protection against this you should always use
a salt.