diff --git a/MANUAL.html b/MANUAL.html
index c34d361e4..b30001a19 100644
--- a/MANUAL.html
+++ b/MANUAL.html
@@ -17,7 +17,7 @@
Rclone syncs your files to cloud storage

@@ -144,6 +144,7 @@
See below for some expanded Linux / macOS instructions.
See the Usage section of the docs for how to use rclone, or run rclone -h
.
+Already installed rclone can be easily updated to the latest version using the rclone selfupdate command.
Script installation
To install rclone on Linux/macOS/BSD systems, run:
curl https://rclone.org/install.sh | sudo bash
@@ -235,7 +236,7 @@ docker run --rm \
ls ~/data/mount
kill %1
Install from source
-Make sure you have at least Go 1.12 installed. Download go if necessary. The latest release is recommended. Then
+Make sure you have at least Go go1.13 installed. Download go if necessary. The latest release is recommended. Then
git clone https://github.com/rclone/rclone.git
cd rclone
go build
@@ -257,7 +258,7 @@ go build
roles:
- rclone
-First, you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file. (See the --config
entry for how to find the config file and choose its location.)
+First, you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file. (See the --config
entry for how to find the config file and choose its location.)
The easiest way to make the config is to run rclone with the config option:
rclone config
See the following for detailed instructions for
@@ -377,13 +378,14 @@ destpath/sourcepath/two.txt
rclone sync
Make source and dest identical, modifying destination only.
Synopsis
-Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.
+Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects, see below).
Important: Since this can cause data loss, test first with the --dry-run
or the --interactive
/-i
flag.
rclone sync -i SOURCE remote:DESTINATION
-Note that files in the destination won't be deleted if there were any errors at any point.
+Note that files in the destination won't be deleted if there were any errors at any point. Duplicate objects (files with the same name, on those providers that support it) are also not yet handled.
It is always the contents of the directory that is synced, not the directory so when source:path is a directory, it's the contents of source:path that are copied, not the directory name and contents. See extended explanation in the copy
command above if unsure.
If dest:path doesn't exist, it is created and the source:path contents go there.
Note: Use the -P
/--progress
flag to view real-time transfer statistics
+Note: Use the rclone dedupe
command to deal with "Duplicate object/directory found in source/destination - ignoring" errors. See this forum post for more info.
rclone sync source:path dest:path [flags]
Options
--create-empty-src-dirs Create empty source dirs on destination after sync
@@ -647,12 +649,16 @@ rclone --dry-run --min-size 100M delete remote:path
rclone version
Show the version number.
Synopsis
-Show the version number, the go version and the architecture.
-Eg
+Show the rclone version number, the go version, the build target OS and architecture, build tags and the type of executable (static or dynamic).
+For example:
$ rclone version
-rclone v1.41
-- os/arch: linux/amd64
-- go version: go1.10
+rclone v1.54
+- os/type: linux
+- os/arch: amd64
+- go/version: go1.16
+- go/linking: static
+- go/tags: none
+Note: before rclone version 1.55 the os/type and os/arch lines were merged, and the "go/version" line was tagged as "go version".
If you supply the --check flag, then it will do an online check to compare your version with the latest release and the latest beta.
$ rclone version --check
yours: 1.42.0.6
@@ -690,7 +696,7 @@ beta: 1.42.0.5 (released 2018-06-17)
Interactively find duplicate filenames and delete/rename them.
Synopsis
By default dedupe
interactively finds files with duplicate names and offers to delete all but one or rename them to be different. This is known as deduping by name.
-Deduping by name is only useful with backends like Google Drive which can have duplicate file names. It can be run on wrapping backends (e.g. crypt) if they wrap a backend which supports duplicate file names.
+Deduping by name is only useful with a small group of backends (e.g. Google Drive, Opendrive) that can have duplicate file names. It can be run on wrapping backends (e.g. crypt) if they wrap a backend which supports duplicate file names.
However if --by-hash is passed in then dedupe will find files with duplicate hashes instead which will work on any backend which supports at least one hash. This can be used to find files with duplicate content. This is known as deduping by hash.
If deduping by name, first rclone will merge directories with the same name. It will do this iteratively until all the identically named directories have been merged.
Next, if deduping by name, for every group of duplicate file names / hashes, it will delete all but one identical files it finds without confirmation. This means that for most duplicated files the dedupe
command will not be interactive.
@@ -1060,15 +1066,16 @@ if src is directory
Copy url content to dest.
Synopsis
Download a URL's content and copy it to the destination without saving it in temporary storage.
-Setting --auto-filename will cause the file name to be retrieved from the from URL (after any redirections) and used in the destination path.
-Setting --no-clobber will prevent overwriting file on the destination if there is one with the same name.
-Setting --stdout or making the output file name "-" will cause the output to be written to standard output.
+Setting --auto-filename
will cause the file name to be retrieved from the from URL (after any redirections) and used in the destination path. With --print-filename
in addition, the resuling file name will be printed.
+Setting --no-clobber
will prevent overwriting file on the destination if there is one with the same name.
+Setting --stdout
or making the output file name -
will cause the output to be written to standard output.
rclone copyurl https://example.com dest:path [flags]
Options
- -a, --auto-filename Get the file name from the URL and use it for destination file path
- -h, --help help for copyurl
- --no-clobber Prevent overwriting file with same name
- --stdout Write the output to stdout rather than a file
+ -a, --auto-filename Get the file name from the URL and use it for destination file path
+ -h, --help help for copyurl
+ --no-clobber Prevent overwriting file with same name
+ -p, --print-filename Print the resulting name from --auto-filename
+ --stdout Write the output to stdout rather than a file
See the global flags page for global options not listed here.
SEE ALSO
@@ -1423,10 +1430,10 @@ rclone copy --files-from-raw new_files /path/to/local remote:path
On Linux and OSX, you can either run mount in foreground mode or background (daemon) mode. Mount runs in foreground mode by default, use the --daemon
flag to specify background mode. You can only run mount in foreground mode on Windows.
On Linux/macOS/FreeBSD start the mount like this, where /path/to/local/mount
is an empty existing directory:
rclone mount remote:path/to/files /path/to/local/mount
-On Windows you can start a mount in different ways. See below for details. The following examples will mount to an automatically assigned drive, to specific drive letter X:
, to path C:\path\to\nonexistent\directory
(which must be non-existent subdirectory of an existing parent directory or drive, and is not supported when mounting as a network drive), and the last example will mount as network share \\cloud\remote
and map it to an automatically assigned drive:
+On Windows you can start a mount in different ways. See below for details. The following examples will mount to an automatically assigned drive, to specific drive letter X:
, to path C:\path\parent\mount
(where parent directory or drive must exist, and mount must not exist, and is not supported when mounting as a network drive), and the last example will mount as network share \\cloud\remote
and map it to an automatically assigned drive:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
-rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files \\cloud\remote
When the program ends while in foreground mode, either via Ctrl+C or receiving a SIGINT or SIGTERM signal, the mount should be automatically stopped.
When running in background mode the user will have to stop the mount manually:
@@ -1443,10 +1450,10 @@ umount /path/to/local/mount
Mounting modes on windows
Unlike other operating systems, Microsoft Windows provides a different filesystem type for network and fixed drives. It optimises access on the assumption fixed disk drives are fast and reliable, while network drives have relatively high latency and less reliability. Some settings can also be differentiated between the two types, for example that Windows Explorer should just display icons and not create preview thumbnails for image and video files on network drives.
In most cases, rclone will mount the remote as a normal, fixed disk drive by default. However, you can also choose to mount it as a remote network drive, often described as a network share. If you mount an rclone remote using the default, fixed drive mode and experience unexpected program errors, freezes or other issues, consider mounting as a network drive instead.
-When mounting as a fixed disk drive you can either mount to an unused drive letter, or to a path - which must be non-existent subdirectory of an existing parent directory or drive. Using the special value *
will tell rclone to automatically assign the next available drive letter, starting with Z: and moving backward. Examples:
+When mounting as a fixed disk drive you can either mount to an unused drive letter, or to a path representing a non-existent subdirectory of an existing parent directory or drive. Using the special value *
will tell rclone to automatically assign the next available drive letter, starting with Z: and moving backward. Examples:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
-rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files X:
Option --volname
can be used to set a custom volume name for the mounted file system. The default is to use the remote name and path.
To mount as network drive, you can add option --network-mode
to your mount command. Mounting to a directory path is not supported in this mode, it is a limitation Windows imposes on junctions, so the remote must always be mounted to a drive letter.
@@ -1466,7 +1473,9 @@ rclone mount remote:path/to/files * --volname \\cloud\remote
The FUSE emulation layer on Windows must convert between the POSIX-based permission model used in FUSE, and the permission model used in Windows, based on access-control lists (ACL).
The mounted filesystem will normally get three entries in its access-control list (ACL), representing permissions for the POSIX permission scopes: Owner, group and others. By default, the owner and group will be taken from the current user, and the built-in group "Everyone" will be used to represent others. The user/group can be customized with FUSE options "UserName" and "GroupName", e.g. -o UserName=user123 -o GroupName="Authenticated Users"
.
The permissions on each entry will be set according to options --dir-perms
and --file-perms
, which takes a value in traditional numeric notation, where the default corresponds to --file-perms 0666 --dir-perms 0777
.
-Note that the mapping of permissions is not always trivial, and the result you see in Windows Explorer may not be exactly like you expected. For example, when setting a value that includes write access, this will be mapped to individual permissions "write attributes", "write data" and "append data", but not "write extended attributes" (WinFsp does not support extended attributes, see this). Windows will then show this as basic permission "Special" instead of "Write", because "Write" includes the "write extended attributes" permission.
+Note that the mapping of permissions is not always trivial, and the result you see in Windows Explorer may not be exactly like you expected. For example, when setting a value that includes write access, this will be mapped to individual permissions "write attributes", "write data" and "append data", but not "write extended attributes". Windows will then show this as basic permission "Special" instead of "Write", because "Write" includes the "write extended attributes" permission.
+If you set POSIX permissions for only allowing access to the owner, using --file-perms 0600 --dir-perms 0700
, the user group and the built-in "Everyone" group will still be given some special permissions, such as "read attributes" and "read permissions", in Windows. This is done for compatibility reasons, e.g. to allow users without additional permissions to be able to read basic metadata about files like in UNIX. One case that may arise is that other programs (incorrectly) interprets this as the file being accessible by everyone. For example an SSH client may warn about "unprotected private key file".
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity", that allows the complete specification of file security descriptors using SDDL. With this you can work around issues such as the mentioned "unprotected private key file" by specifying -o FileSecurity="D:P(A;;FA;;;OW)"
, for file all access (FA) to the owner (OW).
Windows caveats
Note that drives created as Administrator are not visible by other accounts (including the account that was elevated as Administrator). So if you start a Windows drive from an Administrative Command Prompt and then try to access the same drive from Explorer (which does not run as Administrator), you will not be able to see the new drive.
The easiest way around this is to start the drive from a normal command prompt. It is also possible to start a drive from the SYSTEM account (using the WinFsp.Launcher infrastructure) which creates drives accessible for everyone on the system or alternatively using the nssm service manager.
@@ -1475,7 +1484,7 @@ rclone mount remote:path/to/files * --volname \\cloud\remote
The bucket based remotes (e.g. Swift, S3, Google Compute Storage, B2, Hubic) do not support the concept of empty directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.
Only supported on Linux, FreeBSD, OS X and Windows at the moment.
rclone mount vs rclone sync/copy
-File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone mount can't use retries in the same way without making local copies of the uploads. Look at the VFS file caching section for solutions to make mount more reliable.
+File systems expect things to be 100% reliable, whereas cloud storage systems are a long way from 100% reliable. The rclone sync/copy commands cope with this with lots of retries. However rclone mount can't use retries in the same way without making local copies of the uploads. Look at the VFS File Caching for solutions to make mount more reliable.
Attribute caching
You can use the flag --attr-timeout
to set the time the kernel caches the attributes (size, modification time, etc.) for directory entries.
The default is 1s
which caches files just long enough to avoid too many callbacks to rclone from the kernel.
@@ -1526,6 +1535,7 @@ rclone mount remote:path/to/files * --volname \\cloud\remote
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back second. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
note that the cache may exceed this size for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache.
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -1583,6 +1593,9 @@ rclone mount remote:path/to/files * --volname \\cloud\remote
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on mounted file system. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system mounted by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
+WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone mount remote:path /path/to/mountpoint [flags]
Options
--allow-non-empty Allow mounting over a non-empty directory. Not supported on Windows.
@@ -1621,6 +1634,7 @@ rclone mount remote:path/to/files * --volname \\cloud\remote
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
--volname string Set the volume name. Supported on Windows and OSX only.
@@ -1785,17 +1799,42 @@ ffmpeg - | rclone rcat remote:path/to/file
- rclone - Show help for rclone commands, flags and backends.
+rclone selfupdate
+Update the rclone binary.
+Synopsis
+This command downloads the latest release of rclone and replaces the currently running binary. The download is verified with a hashsum and cryptographically signed signature.
+If used without flags (or with implied --stable
flag), this command will install the latest stable release. However, some issues may be fixed (or features added) only in the latest beta release. In such cases you should run the command with the --beta
flag, i.e. rclone selfupdate --beta
. You can check in advance what version would be installed by adding the --check
flag, then repeat the command without it when you are satisfied.
+Sometimes the rclone team may recommend you a concrete beta or stable rclone release to troubleshoot your issue or add a bleeding edge feature. The --version VER
flag, if given, will update to the concrete version instead of the latest one. If you omit micro version from VER
(for example 1.53
), the latest matching micro version will be used.
+Upon successful update rclone will print a message that contains a previous version number. You will need it if you later decide to revert your update for some reason. Then you'll have to note the previous version and run the following command: rclone selfupdate [--beta] OLDVER
. If the old version contains only dots and digits (for example v1.54.0
) then it's a stable release so you won't need the --beta
flag. Beta releases have an additional information similar to v1.54.0-beta.5111.06f1c0c61
. (if you are a developer and use a locally built rclone, the version number will end with -DEV
, you will have to rebuild it as it obvisously can't be distributed).
+If you previously installed rclone via a package manager, the package may include local documentation or configure services. You may wish to update with the flag --package deb
or --package rpm
(whichever is correct for your OS) to update these too. This command with the default --package zip
will update only the rclone executable so the local manual may become inaccurate after it.
+The rclone mount
command (https://rclone.org/commands/rclone_mount/) may or may not support extended FUSE options depending on the build and OS. selfupdate
will refuse to update if the capability would be discarded.
+Note: Windows forbids deletion of a currently running executable so this command will rename the old executable to 'rclone.old.exe' upon success.
+Please note that this command was not available before rclone version 1.55. If it fails for you with the message unknown command "selfupdate"
then you will need to update manually following the install instructions located at https://rclone.org/install/
+rclone selfupdate [flags]
+Options
+ --beta Install beta release.
+ --check Check for latest release, do not download.
+ -h, --help help for selfupdate
+ --output string Save the downloaded binary at a given path (default: replace running binary)
+ --package string Package format: zip|deb|rpm (default: zip)
+ --stable Install stable release (this is the default)
+ --version string Install the given rclone version (default: latest)
+See the global flags page for global options not listed here.
+SEE ALSO
+
+- rclone - Show help for rclone commands, flags and backends.
+
rclone serve
Serve a remote over a protocol.
-Synopsis
+Synopsis
rclone serve is used to serve a remote over a given protocol. This command requires the use of a subcommand to specify the protocol, e.g.
rclone serve http remote:
Each subcommand has its own options which you can see in their help.
rclone serve <protocol> [opts] <remote> [flags]
-Options
+Options
-h, --help help for serve
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
- rclone serve dlna - Serve remote:path over DLNA
@@ -1807,7 +1846,7 @@ ffmpeg - | rclone rcat remote:path/to/file
rclone serve dlna
Serve remote:path over DLNA
-Synopsis
+Synopsis
rclone serve dlna is a DLNA media server for media stored in an rclone remote. Many devices, such as the Xbox and PlayStation, can automatically discover this server in the LAN and play audio/video from it. VLC is also supported. Service discovery uses UDP multicast packets (SSDP) and will thus only work on LANs.
Rclone will list all files present in the remote, without filtering based on media formats or file extensions. Additionally, there is no media transcoding support. This means that some players might show files that they are not able to play back correctly.
Server options
@@ -1848,6 +1887,7 @@ ffmpeg - | rclone rcat remote:path/to/file
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back second. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
note that the cache may exceed this size for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache.
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -1905,8 +1945,11 @@ ffmpeg - | rclone rcat remote:path/to/file
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on mounted file system. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system mounted by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
+WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve dlna remote:path [flags]
-Options
+Options
--addr string ip:port or :port to bind the DLNA http server to. (default ":7879")
--dir-cache-time duration Time to cache directory entries for. (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
@@ -1931,16 +1974,17 @@ ffmpeg - | rclone rcat remote:path/to/file
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
rclone serve ftp
Serve remote:path over FTP.
-Synopsis
+Synopsis
rclone serve ftp implements a basic ftp server to serve the remote over FTP protocol. This can be viewed with a ftp client or you can make a remote of type ftp to read and write it.
Server options
Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
@@ -1982,6 +2026,7 @@ ffmpeg - | rclone rcat remote:path/to/file
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back second. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
note that the cache may exceed this size for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache.
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -2039,6 +2084,9 @@ ffmpeg - | rclone rcat remote:path/to/file
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on mounted file system. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system mounted by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
+WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
Auth Proxy
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
@@ -2070,7 +2118,7 @@ ffmpeg - | rclone rcat remote:path/to/file
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
rclone serve ftp remote:path [flags]
-Options
+Options
--addr string IPaddress:Port or :Port to bind server to. (default "localhost:2121")
--auth-proxy string A program to use to create the backend from the auth.
--cert string TLS PEM key (concatenation of certificate and CA certificate)
@@ -2100,16 +2148,17 @@ ffmpeg - | rclone rcat remote:path/to/file
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
rclone serve http
Serve the remote over HTTP.
-Synopsis
+Synopsis
rclone serve http implements a basic web server to serve the remote over HTTP. This can be viewed in a web browser or you can make a remote of type http read from it.
You can use the filter flags (e.g. --include, --exclude) to control what is served.
The server will log errors. Use -v to see access logs.
@@ -2246,6 +2295,7 @@ htpasswd -B htpasswd anotherUser
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back second. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
note that the cache may exceed this size for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache.
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -2303,8 +2353,11 @@ htpasswd -B htpasswd anotherUser
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on mounted file system. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system mounted by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
+WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve http remote:path [flags]
-Options
+Options
--addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080")
--baseurl string Prefix for URLs - leave blank for root.
--cert string SSL PEM key (concatenation of certificate and CA certificate)
@@ -2339,16 +2392,17 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
rclone serve restic
Serve the remote for restic's REST API.
-Synopsis
+Synopsis
rclone serve restic implements restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.
Restic is a command line program for doing backups.
The server will log errors. Use -v to see access logs.
@@ -2488,7 +2542,7 @@ htpasswd -B htpasswd anotherUser
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.
--cert should be either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.
rclone serve restic remote:path [flags]
-Options
+Options
--addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080")
--append-only disallow deletion of repository data
--baseurl string Prefix for URLs - leave blank for root.
@@ -2508,13 +2562,13 @@ htpasswd -B htpasswd anotherUser
--template string User Specified Template.
--user string User name for authentication.
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
rclone serve sftp
Serve the remote over SFTP.
-Synopsis
+Synopsis
rclone serve sftp implements an SFTP server to serve the remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.
You can use the filter flags (e.g. --include, --exclude) to control what is served.
The server will log errors. Use -v to see access logs.
@@ -2558,6 +2612,7 @@ htpasswd -B htpasswd anotherUser
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back second. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
note that the cache may exceed this size for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache.
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -2615,6 +2670,9 @@ htpasswd -B htpasswd anotherUser
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on mounted file system. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system mounted by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
+WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
Auth Proxy
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
@@ -2646,7 +2704,7 @@ htpasswd -B htpasswd anotherUser
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
rclone serve sftp remote:path [flags]
-Options
+Options
--addr string IPaddress:Port or :Port to bind server to. (default "localhost:2022")
--auth-proxy string A program to use to create the backend from the auth.
--authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
@@ -2675,16 +2733,17 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
rclone serve webdav
Serve remote:path over webdav.
-Synopsis
+Synopsis
rclone serve webdav implements a basic webdav server to serve the remote over HTTP via the webdav protocol. This can be viewed with a webdav client, through a web browser, or you can make a remote of type webdav to read and write it.
Webdav options
--etag-hash
@@ -2823,6 +2882,7 @@ htpasswd -B htpasswd anotherUser
The cache has 4 different modes selected by --vfs-cache-mode
. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back second. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size
note that the cache may exceed this size for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval
. Secondly because open files cannot be evicted from the cache.
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off
. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir
. You don't need to worry about this if the remotes in use don't overlap.
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -2880,6 +2940,9 @@ htpasswd -B htpasswd anotherUser
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on mounted file system. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system mounted by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df
on the filesystem, then pass the flag --vfs-used-is-size
to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size
and compute the total used space itself.
+WARNING. Contrary to rclone size
, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
Auth Proxy
If you supply the parameter --auth-proxy /path/to/program
then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
PLEASE NOTE: --auth-proxy
and --authorized-keys
cannot be used together, if --auth-proxy
is set the authorized keys option will be ignored.
@@ -2911,7 +2974,7 @@ htpasswd -B htpasswd anotherUser
Note that an internal cache is keyed on user
so only use that for configuration, don't use pass
or public_key
. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
rclone serve webdav remote:path [flags]
-Options
+Options
--addr string IPaddress:Port or :Port to bind server to. (default "localhost:8080")
--auth-proxy string A program to use to create the backend from the auth.
--baseurl string Prefix for URLs - leave blank for root.
@@ -2949,16 +3012,17 @@ htpasswd -B htpasswd anotherUser
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
rclone settier
Changes storage class/tier of objects in remote.
-Synopsis
+Synopsis
rclone settier changes storage tier or class at remote if supported. Few cloud storage services provides different storage classes on objects, for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
Note that, certain tier changes make objects not available to access immediately. For example tiering to archive in azure blob storage makes objects in frozen state, user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object inaccessible.true
You can use it to tier single object
@@ -2968,16 +3032,96 @@ htpasswd -B htpasswd anotherUser
Or just provide remote directory and all files in directory will be tiered
rclone settier tier remote:path/dir
rclone settier tier remote:path [flags]
-Options
+Options
-h, --help help for settier
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
+rclone test
+Run a test command
+Synopsis
+Rclone test is used to run test commands.
+Select which test comand you want with the subcommand, eg
+rclone test memory remote:
+Each subcommand has its own options which you can see in their help.
+NB Be careful running these commands, they may do strange things so reading their documentation first is recommended.
+Options
+ -h, --help help for test
+See the global flags page for global options not listed here.
+SEE ALSO
+
+rclone test histogram
+Makes a histogram of file name characters.
+Synopsis
+This command outputs JSON which shows the histogram of characters used in filenames in the remote:path specified.
+The data doesn't contain any identifying information but is useful for the rclone developers when developing filename compression.
+rclone test histogram [remote:path] [flags]
+Options
+ -h, --help help for histogram
+See the global flags page for global options not listed here.
+SEE ALSO
+
+rclone test info
+Discovers file name or other limitations for paths.
+Synopsis
+rclone info discovers what filenames and upload methods are possible to write to the paths passed in and how long they can be. It can take some time. It will write test files into the remote:path passed in. It outputs a bit of go code for each one.
+NB this can create undeletable files and other hazards - use with care
+rclone test info [remote:path]+ [flags]
+Options
+ --all Run all tests.
+ --check-control Check control characters.
+ --check-length Check max filename length.
+ --check-normalization Check UTF-8 Normalization.
+ --check-streaming Check uploads with indeterminate file size.
+ -h, --help help for info
+ --upload-wait duration Wait after writing a file.
+ --write-json string Write results to file.
+See the global flags page for global options not listed here.
+SEE ALSO
+
+rclone test makefiles
+Make a random file hierarchy in
+
+rclone test makefiles <dir> [flags]
+Options
+ --files int Number of files to create (default 1000)
+ --files-per-directory int Average number of files per directory (default 10)
+ -h, --help help for makefiles
+ --max-file-size SizeSuffix Maximum size of files to create (default 100)
+ --max-name-length int Maximum size of file names (default 12)
+ --min-file-size SizeSuffix Minimum size of file to create
+ --min-name-length int Minimum size of file names (default 4)
+See the global flags page for global options not listed here.
+SEE ALSO
+
+rclone test memory
+Load all the objects at remote:path into memory and report memory stats.
+rclone test memory remote:path [flags]
+Options
+ -h, --help help for memory
+See the global flags page for global options not listed here.
+SEE ALSO
+
rclone touch
Create new file or change file modification time.
-Synopsis
+Synopsis
Set the modification time on object(s) as specified by remote:path to have the current time.
If remote:path does not exist then a zero sized object will be created unless the --no-create flag is provided.
If --timestamp is used then it will set the modification time to that time instead of the current time. Times may be specified as one of:
@@ -2988,19 +3132,19 @@ htpasswd -B htpasswd anotherUser
Note that --timestamp is in UTC if you want local time then add the --localtime flag.
rclone touch remote:path [flags]
-Options
+Options
-h, --help help for touch
--localtime Use localtime for timestamp, not UTC.
-C, --no-create Do not create the file if it does not exist.
-t, --timestamp string Use specified time instead of the current time of day.
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
rclone tree
List the contents of the remote in a tree like fashion.
-Synopsis
+Synopsis
rclone tree lists the contents of a remote in a similar way to the unix tree command.
For example
$ rclone tree remote:path
@@ -3016,7 +3160,7 @@ htpasswd -B htpasswd anotherUser
You can use any of the filtering options with the tree command (e.g. --include and --exclude). You can also use --fast-list.
The tree command has many options for controlling the listing which are compatible with the tree command. Note that not all of them have short options as they conflict with rclone's short options.
rclone tree remote:path [flags]
-Options
+Options
-a, --all All files are listed (list . files too).
-C, --color Turn colorization on always.
-d, --dirs-only List directories only.
@@ -3039,7 +3183,7 @@ htpasswd -B htpasswd anotherUser
-U, --unsorted Leave files unsorted.
--version Sort files alphanumerically by version.
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
@@ -3057,7 +3201,7 @@ htpasswd -B htpasswd anotherUser
The syntax of the paths passed to the rclone command are as follows.
/path/to/dir
This refers to the local file system.
-On Windows only \
may be used instead of /
in local paths only, non local paths must use /
.
+On Windows \
may be used instead of /
in local paths only, non local paths must use /
. See local filesystem documentation for more about Windows-specific paths.
These paths needn't start with a leading /
- if they don't then they will be relative to the current directory.
remote:path/to/dir
This refers to a directory path/to/dir
on remote:
as defined in the config file (configured with rclone config
).
@@ -3074,6 +3218,47 @@ htpasswd -B htpasswd anotherUser
To copy files and directories in https://example.com/path/to/dir
to /tmp/dir
.
rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir
To copy files and directories from example.com
in the relative directory path/to/dir
to /tmp/dir
using sftp.
+Connection strings
+The above examples can also be written using a connection string syntax, so instead of providing the arguments as command line parameters --http-url https://pub.rclone.org
they are provided as part of the remote specification as a kind of connection string.
+rclone lsd ":http,url='https://pub.rclone.org':"
+rclone lsf ":http,url='https://example.com':path/to/dir"
+rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
+These can apply to modify existing remotes as well as create new remotes with the on the fly syntax. This example is equivalent to adding the --drive-shared-with-me
parameter to the remote gdrive:
.
+rclone lsf "gdrive,shared_with_me:path/to/dir"
+The major advantage to using the connection string style syntax is that it only applies the the remote, not to all the remotes of that type of the command line. A common confusion is this attempt to copy a file shared on google drive to the normal drive which does not work because the --drive-shared-with-me
flag applies to both the source and the destination.
+rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
+However using the connection string syntax, this does work.
+rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
+The connection strings have the following syntax
+remote,parameter=value,parameter2=value2:path/to/dir
+:backend,parameter=value,parameter2=value2:path/to/dir
+If the parameter
has a :
or ,
then it must be placed in quotes "
or '
, so
+remote,parameter="colon:value",parameter2="comma,value":path/to/dir
+:backend,parameter='colon:value',parameter2='comma,value':path/to/dir
+If a quoted value needs to include that quote, then it should be doubled, so
+remote,parameter="with""quote",parameter2='with''quote':path/to/dir
+This will make parameter
be with"quote
and parameter2
be with'quote
.
+If you leave off the =parameter
then rclone will substitute =true
which works very well with flags. For example to use s3 configured in the environment you could use:
+rclone lsd :s3,env_auth:
+Which is equivalent to
+rclone lsd :s3,env_auth=true:
+Note that on the command line you might need to surround these connection strings with "
or '
to stop the shell interpreting any special characters within them.
+If you are a shell master then you'll know which strings are OK and which aren't, but if you aren't sure then enclose them in "
and use '
as the inside quote. This syntax works on all OSes.
+rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+On Linux/macOS some characters are still interpreted inside "
strings in the shell (notably \
and $
and "
) so if your strings contain those you can swap the roles of "
and '
thus. (This syntax does not work on Windows.)
+rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
+Connection strings, config and logging
+If you supply extra configuration to a backend by command line flag, environment variable or connection string then rclone will add a suffix based on the hash of the config to the name of the remote, eg
+rclone -vv lsf --s3-chunk-size 20M s3:
+Has the log message
+DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
+This is so rclone can tell the modified remote apart from the unmodified remote when caching the backends.
+This should only be noticeable in the logs.
+This means that on the fly backends such as
+rclone -vv lsf :s3,env_auth:
+Will get their own names
+DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
Valid remote names
- Remote names may only contain 0-9, A-Z ,a-z ,_ , - and space.
@@ -3111,7 +3296,7 @@ htpasswd -B htpasswd anotherUser
This can be used when scripting to make aged backups efficiently, e.g.
rclone sync -i remote:current-backup remote:previous-backup
rclone sync -i /path/to/files remote:current-backup
-Options
+Options
Rclone has a number of options to control its behaviour.
Options that take parameters can have the values passed in two ways, --option=value
or --option value
. However boolean (true/false) options behave slightly differently to the other options in that --boolean
sets the option to true
and the absence of the flag sets it to false
. It is also possible to specify --boolean=false
or --boolean=true
. Note that --boolean false
is not valid - this is parsed as --boolean
and the false
is parsed as an extra command line argument for rclone.
Options which use TIME use the go time parser. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
@@ -3130,10 +3315,10 @@ rclone sync -i /path/to/files remote:current-backup
--bwlimit=BANDWIDTH_SPEC
This option controls the bandwidth limit. For example
--bwlimit 10M
-would mean limit the upload and download bandwidth to 10 Mbyte/s. NB this is bytes per second not bits per second. To use a single limit, specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The default is 0
which means to not limit bandwidth.
+would mean limit the upload and download bandwidth to 10 MByte/s. NB this is bytes per second not bits per second. To use a single limit, specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The default is 0
which means to not limit bandwidth.
The upload and download bandwidth can be specified seperately, as --bwlimit UP:DOWN
, so
--bwlimit 10M:100k
-would mean limit the upload bandwidth to 10 Mbyte/s and the download bandwidth to 100 kByte/s. Either limit can be "off" meaning no limit, so to just limit the upload bandwidth you would use
+would mean limit the upload bandwidth to 10 MByte/s and the download bandwidth to 100 kByte/s. Either limit can be "off" meaning no limit, so to just limit the upload bandwidth you would use
--bwlimit 10M:off
this would limit the upload bandwidth to 10MByte/s but the download bandwidth would be unlimited.
When specified as above the bandwidth limits last for the duration of run of the rclone binary.
@@ -3145,10 +3330,10 @@ rclone sync -i /path/to/files remote:current-backup
An example of a typical timetable to avoid link saturation during daytime working hours could be:
--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"
-In this example, the transfer bandwidth will be set to 512kBytes/sec at 8am every day. At noon, it will rise to 10Mbytes/s, and drop back to 512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it will be completely disabled (full speed). Anything between 11pm and 8am will remain unlimited.
+In this example, the transfer bandwidth will be set to 512kBytes/sec at 8am every day. At noon, it will rise to 10MByte/s, and drop back to 512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to 30MByte/s, and at 11pm it will be completely disabled (full speed). Anything between 11pm and 8am will remain unlimited.
An example of timetable with WEEKDAY
could be:
--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"
-It means that, the transfer bandwidth will be set to 512kBytes/sec on Monday. It will rise to 10Mbytes/s before the end of Friday. At 10:00 on Saturday it will be set to 1Mbyte/s. From 20:00 on Sunday it will be unlimited.
+It means that, the transfer bandwidth will be set to 512kBytes/sec on Monday. It will rise to 10MByte/s before the end of Friday. At 10:00 on Saturday it will be set to 1MByte/s. From 20:00 on Sunday it will be unlimited.
Timeslots without WEEKDAY
are extended to the whole week. So this example:
--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"
Is equivalent to this:
@@ -3189,11 +3374,21 @@ rclone sync -i /path/to/files remote:current-backup
You must use the same remote as the destination of the sync. The compare directory must not overlap the destination directory.
See --copy-dest
and --backup-dir
.
--config=CONFIG_FILE
-Specify the location of the rclone config file.
+Specify the location of the rclone configuration file.
Normally the config file is in your home directory as a file called .config/rclone/rclone.conf
(or .rclone.conf
if created with an older version). If $XDG_CONFIG_HOME
is set it will be at $XDG_CONFIG_HOME/rclone/rclone.conf
.
If there is a file rclone.conf
in the same directory as the rclone executable it will be preferred. This file must be created manually for Rclone to use it, it will never be created automatically.
If you run rclone config file
you will see where the default location is for you.
Use this flag to override the config location, e.g. rclone --config=".myconfig" .config
.
+If the location is set to empty string ""
or the special value /notfound
, or the os null device represented by value NUL
on Windows and /dev/null
on Unix systems, then rclone will keep the config file in memory only.
+The file format is basic INI: Sections of text, led by a [section]
header and followed by key=value
entries on separate lines. In rclone each remote is represented by its own section, where the section name defines the name of the remote. Options are specified as the key=value
entries, where the key is the option name without the --backend-
prefix, in lowercase and with _
instead of -
. E.g. option --mega-hard-delete
corresponds to key hard_delete
. Only backend options can be specified. A special, and required, key type
identifies the storage system, where the value is the internal lowercase name as returned by command rclone help backends
. Comments are indicated by ;
or #
at the beginning of a line.
+Example:
+[megaremote]
+type = mega
+user = you@example.com
+pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
+Note that passwords are in obscured form. Also, many storage systems uses token-based authentication instead of passwords, and this requires additional steps. It is easier, and safer, to use the interactive command rclone config
instead of manually editing the configuration file.
+The configuration file will typically contain login information, and should therefore have restricted permissions so that only the current user can read it. Rclone tries to ensure this when it writes the file. You may also choose to encrypt the file.
+When token-based authentication are used, the configuration file must be writable, because rclone needs to update the tokens inside it.
--contimeout=TIME
Set the connection timeout. This should be in go time format which looks like 5s
for 5 seconds, 10m
for 10 minutes, or 3h30m
.
The connection timeout is the amount of time rclone will wait for a connection to go through to a remote object storage system. It is 1m
by default.
@@ -3211,6 +3406,12 @@ rclone sync -i /path/to/files remote:current-backup
--disable help
See the overview features and optional features to get an idea of which feature does what.
This flag can be useful for debugging and in exceptional circumstances (e.g. Google Drive limiting the total volume of Server Side Copies to 100GB/day).
+--dscp VALUE
+Specify a DSCP value or name to use in connections. This could help QoS system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are allowed.
+See the description of differentiated services to get an idea of this field. Setting this to 1 (LE) to identify the flow to SCAVENGER class can avoid occupying too much bandwidth in a network with DiffServ support (RFC 8622).
+For example, if you configured QoS on router to handle LE properly. Running:
+rclone copy --dscp LE from:/from to:/to
+would make the priority lower than usual internet flows.
-n, --dry-run
Do a trial run with no permanent changes. Use this to see what rclone would do without actually doing it. Useful when setting up the sync
command which deletes files in the destination.
--expect-continue-timeout=TIME
@@ -3221,6 +3422,12 @@ rclone sync -i /path/to/files remote:current-backup
By default, rclone will exit with return code 0 if there were no errors.
This option allows rclone to return exit code 9 if no files were transferred between the source and destination. This allows using rclone in scripts, and triggering follow-on actions if data was copied, or skipping if not.
NB: Enabling this option turns a usually non-fatal error into a potentially fatal one - please check and adjust your scripts accordingly!
+--fs-cache-expire-duration=TIME
+When using rclone via the API rclone caches created remotes for 5 minutes by default in the "fs cache". This means that if you do repeated actions on the same remote then rclone won't have to build it again from scratch, which makes it more efficient.
+This flag sets the time that the remotes are cached for. If you set it to 0
(or negative) then rclone won't cache the remotes at all.
+Note that if you use some flags, eg --backup-dir
and if this is set to 0
rclone may build two remotes (one for the source or destination and one for the --backup-dir
where it may have only built one before.
+--fs-cache-expire-interval=TIME
+This controls how often rclone checks for cached remotes to expire. See the --fs-cache-expire-duration
documentation above for more info. The default is 60s, set to 0 to disable expiry.
Add an HTTP header for all transactions. The flag can be repeated to add multiple headers.
If you want to add headers only for uploads use --header-upload
and if you want to add headers only for downloads use --header-download
.
@@ -3574,7 +3781,7 @@ y/n/s/!/q> n
This option defaults to false
.
This should be used only for testing.
Configuration Encryption
-Your configuration file contains information for logging in to your cloud services. This means that you should keep your .rclone.conf
file in a secure location.
+Your configuration file contains information for logging in to your cloud services. This means that you should keep your rclone.conf
file in a secure location.
If you are in an environment where that isn't possible, you can add a password to your configuration. This means that you will have to supply the password every time you start rclone.
To add a password to your rclone configuration, execute rclone config
.
>rclone config
@@ -3619,6 +3826,7 @@ export RCLONE_CONFIG_PASS
export RCLONE_PASSWORD_COMMAND="pass rclone/config"
If the passwordstore
password manager holds the password for the rclone configuration, using the script method means the password is primarily protected by the passwordstore
system, and is never embedded in the clear in scripts, nor available for examination using the standard commands available. It is quite possible with long running rclone sessions for copies of passwords to be innocently captured in log files or terminal scroll buffers, etc. Using the script method of supplying the password enhances the security of the config password considerably.
If you are running rclone inside a script, unless you are using the --password-command
method, you might want to disable password prompts. To do that, pass the parameter --ask-password=false
to rclone. This will make rclone fail instead of asking for a password if RCLONE_CONFIG_PASS
doesn't contain a valid password, and --password-command
has not been supplied.
+Some rclone commands, such as genautocomplete
, do not require configuration. Nevertheless, rclone will read any configuration file found according to the rules described above. If an encrypted configuration file is found, this means you will be prompted for password (unless using --password-command
). To avoid this, you can bypass the loading of the configuration file by overriding the location with an empty string ""
or the special value /notfound
, or the os null device represented by value NUL
on Windows and /dev/null
on Unix systems (before rclone version 1.55 only this null device alternative was supported). E.g. rclone --config="" genautocomplete bash
.
Developer options
These options are useful when developing or debugging rclone. There are also some more remote specific options which aren't documented here which are used for testing. These start with remote name e.g. --drive-test-option
- see the docs for the remote in question.
--cpuprofile=FILE
@@ -3703,14 +3911,14 @@ export RCLONE_CONFIG_PASS
Environment Variables
Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.
-Options
+Options
Every option in rclone can have its default set by environment variable.
To find the name of the environment variable, first, take the long option name, strip the leading --
, change -
to _
, make upper case and prepend RCLONE_
.
For example, to always set --stats 5s
, set the environment variable RCLONE_STATS=5s
. If you set stats on the command line this will override the environment variable setting.
Or to always use the trash in drive --drive-use-trash
, set RCLONE_DRIVE_USE_TRASH=true
.
The same parser is used for the options and the environment variables so they take exactly the same form.
Config file
-You can set defaults for values in the config file on an individual remote basis. If you want to use this feature, you will need to discover the name of the config items that you want. The easiest way is to run through rclone config
by hand, then look in the config file to see what the values are (the config file can be found by looking at the help for --config
in rclone help
).
+You can set defaults for values in the config file on an individual remote basis. The names of the config items are documented in the page for each backend.
To find the name of the environment variable, you need to set, take RCLONE_CONFIG_
+ name of remote + _
+ name of config file option and make it all uppercase.
For example, to configure an S3 remote named mys3:
without a config file (using unix ways of setting environment variables):
$ export RCLONE_CONFIG_MYS3_TYPE=s3
@@ -3721,6 +3929,8 @@ $ rclone lsd MYS3:
$ rclone listremotes | grep mys3
mys3:
Note that if you want to create a remote using environment variables you must create the ..._TYPE
variable as above.
+Note also that now rclone has connectionstrings, it is probably easier to use those instead which makes the above example
+rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
Precedence
The various different methods of backend configuration are read in this order and the first one with a value is used.
@@ -3842,6 +4052,18 @@ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]
/file.jpg - matches "file.jpg" in the root directory of the remote
- doesn't match "afile.jpg"
- doesn't match "directory/file.jpg"
+The top level of the remote may not be the top level of the drive.
+E.g. for a Microsoft Windows local directory structure
+
F:
+├── bkp
+├── data
+│ ├── excl
+│ │ ├── 123.jpg
+│ │ └── 456.jpg
+│ ├── incl
+│ │ └── document.pdf
+To copy the contents of folder data
into folder bkp
excluding the contents of subfolder excl
the following command treats F:\data
and F:\bkp
as top level for filtering.
+rclone copy F:\data\ F:\bkp\ --exclude=/excl/**
Important Use /
in path/file name patterns and not \
even if running on Microsoft Windows.
Simple patterns are case sensitive unless the --ignore-case
flag is used.
Without --ignore-case
(default)
@@ -3851,7 +4073,7 @@ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]
potato - matches "potato"
- matches "POTATO"
How filter rules are applied to files
-Rclone path / file name filters are made up of one or more of the following flags:
+Rclone path/file name filters are made up of one or more of the following flags:
--include
--include-from
@@ -3865,7 +4087,7 @@ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]
All flags of the same type are processed together in the order above, regardless of what order the different types of flags are included on the command line.
Multiple instances of the same flag are processed from left to right according to their position in the command line.
To mix up the order of processing includes and excludes use --filter...
flags.
-Within --include-from
, --exclude-from
and --filter-from
flags rules are processed from top to bottom of the referenced file..
+Within --include-from
, --exclude-from
and --filter-from
flags rules are processed from top to bottom of the referenced file.
If there is an --include
or --include-from
flag specified, rclone implies a - **
rule which it adds to the bottom of the internal rule list. Specifying a +
rule with a --filter...
flag does not imply that rule.
Each path/file name passed through rclone is matched against the combined filter list. At first match to a rule the path/file name is included or excluded and no further filter rules are processed for that path/file.
If rclone does not find a match, after testing against all rules (including the implied rule if appropriate), the path/file name is included.
@@ -3873,14 +4095,14 @@ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]
--files-from
and --files-from-raw
flags over-ride and cannot be combined with other filter options.
To see the internal combined rule list, in regular expression form, for a command add the --dump filters
flag. Running an rclone command with --dump filters
and -vv
flags lists the internal filter elements and shows how they are applied to each source path/file. There is not currently a means provided to pass regular expression filter options into rclone directly though character class filter rules contain character classes. Go regular expression reference
How filter rules are applied to directories
-Rclone commands filter, and are applied to, path/file names not directories. The entire contents of a directory can be matched to a filter by the pattern directory/*
or recursively by directory/**
.
+Rclone commands are applied to path/file names not directories. The entire contents of a directory can be matched to a filter by the pattern directory/*
or recursively by directory/**
.
Directory filter rules are defined with a closing /
separator.
E.g. /directory/subdirectory/
is an rclone directory filter rule.
Rclone commands can use directory filter rules to determine whether they recurse into subdirectories. This potentially optimises access to a remote by avoiding listing unnecessary directories. Whether optimisation is desirable depends on the specific filter rules and source remote content.
-Optimisation occurs if either:
+Directory recursion optimisation occurs if either:
-A source remote does not support the rclone ListR
primitive. local
, sftp
, Microsoft OneDrive
and WebDav
do not support ListR
. Google Drive and most bucket type storage do. Full list
-On other remotes, if the rclone command is not naturally recursive, provided it is not run with the --fast-list
flag. ls
, lsf -R
and size
are recursive but sync
, copy
and move
are not.
+A source remote does not support the rclone ListR
primitive. local, sftp, Microsoft OneDrive and WebDav do not support ListR
. Google Drive and most bucket type storage do. Full list
+On other remotes (those that support ListR
), if the rclone command is not naturally recursive, and provided it is not run with the --fast-list
flag. ls
, lsf -R
and size
are naturally recursive but sync
, copy
and move
are not.
Whenever the --disable ListR
flag is applied to an rclone command.
Rclone commands imply directory filter rules from path/file filter rules. To view the directory filter rules rclone has implied for a command specify the --dump filters
flag.
@@ -3889,14 +4111,19 @@ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]
Rclone implies the directory include rule
/a/
Directory filter rules specified in an rclone command can limit the scope of an rclone command but path/file filters still have to be specified.
-E.g. rclone ls remote: --include /directory/
will not match any files. Because it is an --include
option the --exclude **
rule is implied, and the \directory\
pattern serves only to optimise access to the remote by ignoring everything outside of that directory.
+E.g. rclone ls remote: --include /directory/
will not match any files. Because it is an --include
option the --exclude **
rule is implied, and the /directory/
pattern serves only to optimise access to the remote by ignoring everything outside of that directory.
E.g. rclone ls remote: --filter-from filter-list.txt
with a file filter-list.txt
:
- /dir1/
- /dir2/
+ *.pdf
- **
-All files in directories dir1
or dir2
or their subdirectories are completely excluded from the listing. Only files of suffix 'pdf
in the root of remote:
or its subdirectories are listed. The - **
rule prevents listing of any path/files not previously matched by the rules above.
+All files in directories dir1
or dir2
or their subdirectories are completely excluded from the listing. Only files of suffix pdf
in the root of remote:
or its subdirectories are listed. The - **
rule prevents listing of any path/files not previously matched by the rules above.
Option exclude-if-present
creates a directory exclude rule based on the presence of a file in a directory and takes precedence over other rclone directory filter rules.
+When using pattern list syntax, if a pattern item contains either /
or **
, then rclone will not able to imply a directory filter rule from this pattern list.
+E.g. for an include rule
+{dir1/**,dir2/**}
+Rclone will match files below directories dir1
or dir2
only, but will not be able to use this filter to exclude a directory dir3
from being traversed.
+Directory recursion optimisation may affect performance, but normally not the result. One exception to this is sync operations with option --create-empty-src-dirs
, where any traversed empty directories will be created. With the pattern list example {dir1/**,dir2/**}
above, this would create an empty directory dir3
on destination (when it exists on source). Changing the filter to {dir1,dir2}/**
, or splitting it into two include rules --include dir1/** --include dir2/**
, will match the same files while also filtering directories, with the result that an empty directory dir3
will no longer be created.
--exclude
- Exclude files matching pattern
Excludes path/file names from an rclone command based on a single exclude rule.
This flag can be repeated. See above for the order filter flags are processed in.
@@ -3904,7 +4131,7 @@ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]
--exclude
has no effect when combined with --files-from
or --files-from-raw
flags.
E.g. rclone ls remote: --exclude *.bak
excludes all .bak files from listing.
E.g. rclone size remote: "--exclude /dir/**"
returns the total size of all files on remote:
excluding those in root directory dir
and sub directories.
-E.g. on Microsoft Windows rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"
lists the files in remote:
with [JP]
or [KR]
or [HK]
in their name. The single quotes prevent the shell from interpreting the \
characters. The \
characters escape the [
and ]
so ran clone filter treats them literally rather than as a character-range. The {
and }
define an rclone pattern list. For other operating systems single quotes are required ie rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'
+E.g. on Microsoft Windows rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"
lists the files in remote:
with [JP]
or [KR]
or [HK]
in their name. Quotes prevent the shell from interpreting the \
characters.\
characters escape the [
and ]
so an rclone filter treats them literally rather than as a character-range. The {
and }
define an rclone pattern list. For other operating systems single quotes are required ie rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'
--exclude-from
- Read exclude patterns from file
Excludes path/file names from an rclone command based on rules in a named file. The file contains a list of remarks and pattern rules.
For an example exclude-file.txt
:
@@ -3952,7 +4179,7 @@ file2.avi
Adds path/file names to an rclone command based on rules in a named file. The file contains a list of remarks and pattern rules. Include rules start with +
and exclude rules with -
. !
clears existing rules. Rules are processed in the order they are defined.
This flag can be repeated. See above for the order filter flags are processed in.
Arrange the order of filter rules with the most restrictive first and work down.
-E.g. For filter-file.txt
:
+E.g. for filter-file.txt
:
# a sample filter rule file
- secret*.jpg
+ *.jpg
@@ -4007,11 +4234,11 @@ user2/prefect
/home/user1/42 → remote:backup/user1/important
/home/user1/dir/ford → remote:backup/user1/dir/file
/home/user2/prefect → remote:backup/user2/stuff
-Alternatively if /
is chosen as root files-from.txt
would be:
+Alternatively if /
is chosen as root files-from.txt
will be:
/home/user1/42
/home/user1/dir/ford
/home/user2/prefect
-The copy command would be:
+The copy command will be:
rclone copy --files-from files-from.txt / remote:backup
Then there will be an extra home
directory on the remote:
/home/user1/42 → remote:backup/home/user1/42
@@ -4053,10 +4280,10 @@ user2/prefect
--max-age
can also be specified as an absolute time in the following formats:
-- RFC3339 - e.g. "2006-01-02T15:04:05Z07:00"
-- ISO8601 Date and time, local timezone - "2006-01-02T15:04:05"
-- ISO8601 Date and time, local timezone - "2006-01-02 15:04:05"
-- ISO8601 Date - "2006-01-02" (YYYY-MM-DD)
+- RFC3339 - e.g.
2006-01-02T15:04:05Z
or 2006-01-02T15:04:05+07:00
+- ISO8601 Date and time, local timezone -
2006-01-02T15:04:05
+- ISO8601 Date and time, local timezone -
2006-01-02 15:04:05
+- ISO8601 Date -
2006-01-02
(YYYY-MM-DD)
--max-age
applies only to files and not to directories.
E.g. rclone ls remote: --max-age 2d
lists files on remote:
of 2 days old or less.
@@ -4067,7 +4294,7 @@ user2/prefect
Other flags
--delete-excluded
- Delete files on dest excluded from sync
Important this flag is dangerous to your data - use with --dry-run
and -v
first.
-In conjunction with rclone sync
the `--delete-excluded deletes any files on the destination which are excluded from the command.
+In conjunction with rclone sync
, --delete-excluded
deletes any files on the destination which are excluded from the command.
E.g. the scope of rclone sync -i A: B:
can be restricted:
rclone --min-size 50k --delete-excluded sync A: B:
All files on B:
which are less than 50 kBytes are deleted because they are excluded from the rclone sync command.
@@ -4085,7 +4312,7 @@ dir1/dir2/dir3/.ignore
The command rclone ls --exclude-if-present .ignore dir1
does not list dir3
, file3
or .ignore
.
--exclude-if-present
can only be used once in an rclone command.
Common pitfalls
-The most frequent filter support issues on the rclone forum are:
+The most frequent filter support issues on the rclone forum are:
- Not using paths relative to the root of the remote
- Not using
/
to match from the root of a remote
@@ -4298,6 +4525,32 @@ dir1/dir2/dir3/.ignore
2
]
}
+Setting config flags with _config
+If you wish to set config (the equivalent of the global flags) for the duration of an rc call only then pass in the _config
parameter.
+This should be in the same format as the config
key returned by options/get.
+For example, if you wished to run a sync with the --checksum
parameter, you would pass this parameter in your JSON blob.
+
"_config":{"CheckSum": true}
+If using rclone rc
this could be passed as
+rclone rc operations/sync ... _config='{"CheckSum": true}'
+Any config parameters you don't set will inherit the global defaults which were set with command line flags or environment variables.
+Note that it is possible to set some values as strings or integers - see data types for more info. Here is an example setting the equivalent of --buffer-size
in string or integer format.
+"_config":{"BufferSize": "42M"}
+"_config":{"BufferSize": 44040192}
+If you wish to check the _config
assignment has worked properly then calling options/local
will show what the value got set to.
+Setting filter flags with _filter
+If you wish to set filters for the duration of an rc call only then pass in the _filter
parameter.
+This should be in the same format as the filter
key returned by options/get.
+For example, if you wished to run a sync with these flags
+--max-size 1M --max-age 42s --include "a" --include "b"
+you would pass this parameter in your JSON blob.
+"_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}
+If using rclone rc
this could be passed as
+rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}'
+Any filter parameters you don't set will inherit the global defaults which were set with command line flags or environment variables.
+Note that it is possible to set some values as strings or integers - see data types for more info. Here is an example setting the equivalent of --buffer-size
in string or integer format.
+"_filter":{"MinSize": "42M"}
+"_filter":{"MinSize": 44040192}
+If you wish to check the _filter
assignment has worked properly then calling options/local
will show what the value got set to.
Assigning operations to groups with _group = value
Each rc call has its own stats group for tracking its metrics. By default grouping is done by the composite group name from prefix job/
and id of the job like so job/1
.
If _group
has a value then stats for that request will be grouped under that value. This allows caller to group stats under their own name.
@@ -4307,6 +4560,41 @@ dir1/dir2/dir3/.ignore
"speed": 12345
...
}
+Data types
+When the API returns types, these will mostly be straight forward integer, string or boolean types.
+However some of the types returned by the options/get call and taken by the options/set calls as well as the vfsOpt
, mountOpt
and the _config
parameters.
+
+Duration
- these are returned as an integer duration in nanoseconds. They may be set as an integer, or they may be set with time string, eg "5s". See the options section for more info.
+Size
- these are returned as an integer number of bytes. They may be set as an integer or they may be set with a size suffix string, eg "10M". See the options section for more info.
+- Enumerated type (such as
CutoffMode
, DumpFlags
, LogLevel
, VfsCacheMode
- these will be returned as an integer and may be set as an integer but more conveniently they can be set as a string, eg "HARD" for CutoffMode
or DEBUG
for LogLevel
.
+BandwidthSpec
- this will be set and returned as a string, eg "1M".
+
+Specifying remotes to work on
+Remotes are specified with the fs=
, srcFs=
, dstFs=
parameters depending on the command being used.
+The parameters can be a string as per the rest of rclone, eg s3:bucket/path
or :sftp:/my/dir
. They can also be specified as JSON blobs.
+If specifyng a JSON blob it should be a object mapping strings to strings. These values will be used to configure the remote. There are 3 special values which may be set:
+
+type
- set to type
to specify a remote called :type:
+_name
- set to name
to specify a remote called name:
+_root
- sets the root of the remote - may be empty
+
+One of _name
or type
should normally be set. If the local
backend is desired then type
should be set to local
. If _root
isn't specified then it defaults to the root of the remote.
+For example this JSON is equivalent to remote:/tmp
+{
+ "_name": "remote",
+ "_path": "/tmp"
+}
+And this is equivalent to :sftp,host='example.com':/tmp
+{
+ "type": "sftp",
+ "host": "example.com",
+ "_path": "/tmp"
+}
+And this is equivalent to /tmp/dir
+{
+ type = "local",
+ _ path = "/tmp/dir"
+}
Supported commands
backend/command: Runs a backend command.
This takes the following parameters
@@ -4525,18 +4813,22 @@ OR
Returns the following values:
{
- "speed": average speed in bytes/sec since start of the process,
- "bytes": total transferred bytes since the start of the process,
+ "bytes": total transferred bytes since the start of the group,
+ "checks": number of files checked,
+ "deletes" : number of files deleted,
+ "elapsedTime": time in floating point seconds since rclone was started,
"errors": number of errors,
- "fatalError": whether there has been at least one FatalError,
- "retryError": whether there has been at least one non-NoRetryError,
- "checks": number of checked files,
- "transfers": number of transferred files,
- "deletes" : number of deleted files,
- "renames" : number of renamed files,
+ "eta": estimated time in seconds until the group completes,
+ "fatalError": boolean whether there has been at least one fatal error,
+ "lastError": last error string,
+ "renames" : number of files renamed,
+ "retryError": boolean showing whether there has been at least one non-NoRetryError,
+ "speed": average speed in bytes/sec since start of the group,
+ "totalBytes": total number of bytes in the group,
+ "totalChecks": total number of checks in the group,
+ "totalTransfers": total number of transfers in the group,
"transferTime" : total time spent on running jobs,
- "elapsedTime": time in seconds since the start of the process,
- "lastError": last occurred error,
+ "transfers": number of transferred files,
"transferring": an array of currently active file transfers:
[
{
@@ -4599,6 +4891,8 @@ OR
os - OS in use as according to Go
arch - cpu architecture in use according to Go
goVersion - version of Go runtime in use
+linking - type of rclone executable (static or dynamic)
+goTags - space separated build tags or "none"
debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling.
SetBlockProfileRate controls the fraction of goroutine blocking events that are reported in the blocking profile. The profiler aims to sample an average of one blocking event per rate nanoseconds spent blocked.
@@ -4622,6 +4916,14 @@ OR
+fscache/clear: Clear the Fs cache.
+This clears the fs cache. This is where remotes created from backends are cached for a short while to make repeated rc calls more efficient.
+If you change the parameters of a backend then you may want to call this to clear an existing remote out of the cache before re-creating it.
+Authentication is required for this call.
+fscache/entries: Returns the number of entries in the fs cache.
+This returns the number of entries in the fs cache.
+Returns - entries - number of items in the cache
+Authentication is required for this call.
job/list: Lists the IDs of the running jobs
Parameters - None
Results
@@ -4902,8 +5204,14 @@ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"Cache
Authentication is required for this call.
options/blocks: List all the option blocks
Returns - options - a list of the options block names
-options/get: Get all the options
+options/get: Get all the global options
Returns an object where keys are option block names and values are an object with the current option values in.
+Note that these are the global options which are unaffected by use of the _config and _filter parameters. If you wish to read the parameters set in _config then use options/config and for _filter use options/filter.
+This shows the internal names of the option within rclone which should map to the external options very easily with a few exceptions.
+options/local: Get the currently active config for this call
+Returns an object with the keys "config" and "filter". The "config" key contains the local config and the "filter" key contains the local filters.
+Note that these are the local options specific to this rc call. If _config was not supplied then they will be the global options. Likewise with "_filter".
+This call is mostly useful for seeing if _config and _filter passing is working.
This shows the internal names of the option within rclone which should map to the external options very easily with a few exceptions.
options/set: Set an option
Parameters
@@ -4916,12 +5224,13 @@ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"Cache
Repeated as often as required.
Only supply the options you wish to change. If an option is unknown it will be silently ignored. Not all options will have an effect when changed like this.
For example:
-This sets DEBUG level logs (-vv)
-rclone rc options/set --json '{"main": {"LogLevel": 8}}'
+This sets DEBUG level logs (-vv) (these can be set by number or string)
+rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'
+rclone rc options/set --json '{"main": {"LogLevel": 8}}'
And this sets INFO level logs (-v)
-rclone rc options/set --json '{"main": {"LogLevel": 7}}'
+rclone rc options/set --json '{"main": {"LogLevel": "INFO"}}'
And this sets NOTICE level logs (normal without -v)
-rclone rc options/set --json '{"main": {"LogLevel": 6}}'
+rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}'
pluginsctl/addPlugin: Add a plugin using url
used for adding a plugin to the webgui
This takes the following parameters
@@ -4997,6 +5306,7 @@ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"Cache
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
See the copy command command for more information on the above.
Authentication is required for this call.
@@ -5005,6 +5315,7 @@ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"Cache
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
- deleteEmptySrcDirs - delete empty src directories if set
See the move command command for more information on the above.
@@ -5014,6 +5325,7 @@ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"Cache
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
See the sync command command for more information on the above.
Authentication is required for this call.
@@ -5867,13 +6179,13 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
1Fichier |
No |
+Yes |
+Yes |
No |
No |
No |
No |
-No |
-No |
-No |
+Yes |
No |
Yes |
@@ -5883,10 +6195,10 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
Yes |
Yes |
-No #575 |
No |
No |
-No #2178 |
+No |
+No |
No |
Yes |
@@ -5899,7 +6211,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
Yes |
Yes |
Yes |
-No #2178 |
+Yes |
No |
No |
@@ -5926,7 +6238,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
Yes |
Yes |
-No |
+Yes |
Yes |
@@ -5948,7 +6260,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
Yes |
Yes |
Yes |
-No #575 |
+No |
No |
Yes |
Yes |
@@ -5961,7 +6273,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
Yes |
Yes |
Yes |
-No |
+Yes |
No |
No |
No |
@@ -5977,7 +6289,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
No |
Yes |
-No #2178 |
+No |
No |
Yes |
@@ -5990,7 +6302,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
Yes |
Yes |
-No #2178 |
+No |
No |
No |
@@ -6042,7 +6354,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
No |
No |
-No #2178 |
+No |
No |
Yes |
@@ -6055,7 +6367,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
Yes |
Yes |
-No #2178 |
+No |
Yes |
No |
@@ -6094,7 +6406,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
Yes |
No |
No |
-No #2178 |
+Yes |
Yes |
Yes |
@@ -6120,7 +6432,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
Yes |
Yes |
-No #2178 |
+No |
No |
No |
@@ -6159,7 +6471,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
Yes |
Yes |
-No #2178 |
+No |
Yes |
No |
@@ -6198,7 +6510,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
Yes |
No |
Yes |
-No #2178 |
+No |
Yes |
Yes |
@@ -6211,7 +6523,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
Yes |
Yes |
No |
-No #2178 |
+No |
No |
No |
@@ -6237,7 +6549,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
No |
Yes |
-No #2178 |
+No |
Yes |
Yes |
@@ -6276,7 +6588,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
No |
No |
Yes ‡ |
-No #2178 |
+No |
Yes |
Yes |
@@ -6368,10 +6680,10 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
-c, --checksum Skip based on checksum (if available) & size, not mod-time & size
--client-cert string Client SSL certificate (PEM) for mutual TLS auth
--client-key string Client SSL private key (PEM) for mutual TLS auth
- --compare-dest string Include additional server-side path during comparison.
+ --compare-dest stringArray Include additional comma separated server-side paths during comparison.
--config string Config file. (default "$HOME/.config/rclone/rclone.conf")
--contimeout duration Connect timeout (default 1m0s)
- --copy-dest string Implies --compare-dest but also copies files from path into destination.
+ --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination.
--cpuprofile string Write cpu profile to file
--cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
--delete-after When synchronizing, delete files on destination after transferring (default)
@@ -6380,6 +6692,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
-n, --dry-run Do a trial run with no permanent changes
+ --dscp string Set DSCP value to connections. Can be value or names, eg. CS1, LE, DF, AF21.
--dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
--dump-headers Dump HTTP headers - may contain sensitive info
@@ -6393,6 +6706,8 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
-f, --filter stringArray Add a file-filtering rule
--filter-from stringArray Read filtering patterns from a file (use - to read from stdin)
+ --fs-cache-expire-duration duration cache remotes for this long (0 to disable caching) (default 5m0s)
+ --fs-cache-expire-interval duration interval to check for expired remotes (default 1m0s)
--header stringArray Set HTTP header for all transactions
--header-download stringArray Set HTTP header for download transactions
--header-upload stringArray Set HTTP header for upload transactions
@@ -6491,7 +6806,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
- --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.54.0")
+ --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
-v, --verbose count Print lots more stuff (repeat for more)
Backend Flags
These flags are available for every command. They control the backends and may be set in the config file.
@@ -6518,6 +6833,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_object_id or msi_mi_res_id specified.
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_object_id specified.
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_mi_res_id specified.
+ --azureblob-public-access string Public access level of a container: blob, container.
--azureblob-sas-url string SAS URL for container level access only
--azureblob-service-principal-file string Path to file containing credentials for use with a service principal.
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256MB). (Deprecated)
@@ -6581,6 +6897,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
-L, --copy-links Follow symlinks and copy the pointed to item.
--crypt-directory-name-encryption Option to either encrypt directory names or leave them intact. (default true)
--crypt-filename-encryption string How to encrypt the filenames. (default "standard")
+ --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted.
--crypt-password string Password or pass phrase for encryption. (obscured)
--crypt-password2 string Password or pass phrase for salt. Optional but recommended. (obscured)
--crypt-remote string Remote to encrypt/decrypt.
@@ -6616,7 +6933,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--drive-starred-only Only show files that are starred.
--drive-stop-on-download-limit Make download limit errors be fatal
--drive-stop-on-upload-limit Make upload limit errors be fatal
- --drive-team-drive string ID of the Team Drive
+ --drive-team-drive string ID of the Shared Drive (Team Drive)
--drive-token string OAuth Access Token as a JSON blob.
--drive-token-url string Token server url.
--drive-trashed-only Only show files that are in the trash.
@@ -6645,12 +6962,14 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--filefabric-token-expiry string Token expiry time
--filefabric-url string URL of the Enterprise File Fabric to connect to
--filefabric-version string Version read from the file fabric
+ --ftp-close-timeout Duration Maximum time to wait for a response to close. (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
--ftp-disable-epsv Disable using EPSV even if server advertises support
--ftp-disable-mlsd Disable using MLSD even if server advertises support
--ftp-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot)
--ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
--ftp-host string FTP host to connect to
+ --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--ftp-no-check-certificate Do not verify the TLS certificate of the server
--ftp-pass string FTP password (obscured)
--ftp-port string FTP port, leave blank to use default (21)
@@ -6712,6 +7031,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--local-case-sensitive Force the filesystem to report itself as case sensitive.
--local-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot)
--local-no-check-updated Don't check to see if the files change during upload
+ --local-no-preallocate Disable preallocation of disk space for transferred files
--local-no-set-modtime Disable setting modtime
--local-no-sparse Disable sparse files for multi-thread downloads
--local-no-unicode-normalization Don't apply unicode normalization to paths and filenames (Deprecated)
@@ -6816,8 +7136,10 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--seafile-url string URL of seafile host to connect to
--seafile-user string User name (usually email address)
--sftp-ask-password Allow asking for SFTP password when needed.
+ --sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available.
--sftp-host string SSH host to connect to
+ --sftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--sftp-key-file string Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.
--sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file. (obscured)
--sftp-key-pem string Raw PEM-encoded private key, If specified, will override key_file parameter.
@@ -6887,9 +7209,10 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--union-upstreams string List of space separated upstreams.
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
--webdav-bearer-token-command string Command to run to get a bearer token
+ --webdav-encoding string This sets the encoding for the backend.
--webdav-pass string Password. (obscured)
--webdav-url string URL of http host to connect to
- --webdav-user string User name
+ --webdav-user string User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-auth-url string Auth server URL.
--yandex-client-id string OAuth Client Id
@@ -6897,8 +7220,13 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--yandex-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot)
--yandex-token string OAuth Access Token as a JSON blob.
--yandex-token-url string Token server url.
+ --zoho-auth-url string Auth server URL.
+ --zoho-client-id string OAuth Client Id
+ --zoho-client-secret string OAuth Client Secret
--zoho-encoding MultiEncoder This sets the encoding for the backend. (default Del,Ctl,InvalidUtf8)
- --zoho-region string Zoho region to connect to. You'll have to use the region you organization is registered in.
+ --zoho-region string Zoho region to connect to. You'll have to use the region you organization is registered in.
+ --zoho-token string OAuth Access Token as a JSON blob.
+ --zoho-token-url string Token server url.
1Fichier
This is a backend for the 1fichier cloud storage service. Note that a Premium subscription is required to use the API.
Paths are specified as remote:path
@@ -7575,6 +7903,10 @@ y/e/d>
rclone copy --min-age 24h --no-traverse /path/to/source s3:bucket
You'd then do a full rclone sync
less often.
Note that --fast-list
isn't required in the top-up sync.
+Avoiding HEAD requests after PUT
+By default rclone will HEAD every object it uploads. It does this to check the object got uploaded correctly.
+You can disable this with the --s3-no-head option - see there for more details.
+Setting this flag increases the chance for undetected upload failures.
Hashes
For small objects which weren't uploaded as multipart uploads (objects sized below --s3-upload-cutoff
if uploaded with rclone) rclone uses the ETag:
header as an MD5 checksum.
However for objects which were uploaded as multipart uploads or with server side encryption (SSE-AWS or SSE-C) the ETag
header is no longer the MD5 sum of the data, so rclone adds an additional piece of metadata X-Amz-Meta-Md5chksum
which is a base64 encoded MD5 hash (in the same format as is required for Content-MD5
).
@@ -10735,7 +11067,7 @@ chunk_total_size = 10G
Certificate Validation
When the Plex server is configured to only accept secure connections, it is possible to use .plex.direct
URLs to ensure certificate validation succeeds. These URLs are used by Plex internally to connect to the Plex server securely.
The format for these URLs is the following:
-https://ip-with-dots-replaced.server-hash.plex.direct:32400/
+https://ip-with-dots-replaced.server-hash.plex.direct:32400/
The ip-with-dots-replaced
part can be any IPv4 address, where the dots have been replaced with dashes, e.g. 127.0.0.1
becomes 127-0-0-1
.
To get the server-hash
part, the easiest way is to visit
https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
@@ -11096,6 +11428,7 @@ y/e/d> y
The default chunk name format is *.rclone_chunk.###
, hence by default chunk names are BIG_FILE_NAME.rclone_chunk.001
, BIG_FILE_NAME.rclone_chunk.002
etc. You can configure another name format using the name_format
configuration file option. The format uses asterisk *
as a placeholder for the base file name and one or more consecutive hash characters #
as a placeholder for sequential chunk number. There must be one and only one asterisk. The number of consecutive hash characters defines the minimum length of a string representing a chunk number. If decimal chunk number has less digits than the number of hashes, it is left-padded by zeros. If the decimal string is longer, it is left intact. By default numbering starts from 1 but there is another option that allows user to start from 0, e.g. for compatibility with legacy software.
For example, if name format is big_*-##.part
and original file name is data.txt
and numbering starts from 0, then the first chunk will be named big_data.txt-00.part
, the 99th chunk will be big_data.txt-98.part
and the 302nd chunk will become big_data.txt-301.part
.
Note that list
assembles composite directory entries only when chunk names match the configured format and treats non-conforming file names as normal non-chunked files.
+When using norename
transactions, chunk names will additionally have a unique file version suffix. For example, BIG_FILE_NAME.rclone_chunk.001_bp562k
.
Besides data chunks chunker will by default create metadata object for a composite file. The object is named after the original file. Chunker allows user to disable metadata completely (the none
format). Note that metadata is normally not created for files smaller than the configured chunk size. This may change in future rclone releases.
@@ -11106,6 +11439,7 @@ y/e/d> y
nchunks
- number of data chunks in file
md5
- MD5 hashsum of composite file (if present)
sha1
- SHA1 hashsum (if present)
+txn
- identifies current version of the file
There is no field for composite file name as it's simply equal to the name of meta object on the wrapped remote. Please refer to respective sections for details on hashsums and modified time handling.
@@ -11119,7 +11453,7 @@ y/e/d> y
Modified time
Chunker stores modification times using the wrapped remote so support depends on that. For a small non-chunked file the chunker overlay simply manipulates modification time of the wrapped remote file. For a composite file with metadata chunker will get and set modification time of the metadata object on the wrapped remote. If file is chunked but metadata format is none
then chunker will use modification time of the first data chunk.
Migrations
-The idiomatic way to migrate to a different chunk size, hash type or chunk naming scheme is to:
+The idiomatic way to migrate to a different chunk size, hash type, transaction style or chunk naming scheme is to:
- Collect all your chunked files under a directory and have your chunker remote point to it.
- Create another directory (most probably on the same cloud storage) and configure a new remote with desired metadata format, hash type, chunk naming etc.
@@ -11134,6 +11468,7 @@ y/e/d> y
Chunker will not automatically rename existing chunks when you run rclone config
on a live remote and change the chunk name format. Beware that in result of this some files which have been treated as chunks before the change can pop up in directory listings as normal files and vice versa. The same warning holds for the chunk size. If you desperately need to change critical chunking settings, you should run data migration as described above.
If wrapped remote is case insensitive, the chunker overlay will inherit that property (so you can't have a file called "Hello.doc" and "hello.doc" in the same directory).
Chunker included in rclone releases up to v1.54
can sometimes fail to detect metadata produced by recent versions of rclone. We recommend users to keep rclone up-to-date to avoid data corruption.
+Changing transactions
is dangerous and requires explicit migration.
Standard Options
Here are the standard options specific to chunker (Transparently chunk/split large files).
--chunker-remote
@@ -11248,6 +11583,36 @@ y/e/d> y
+--chunker-transactions
+Choose how chunker should handle temporary files during transactions.
+
+- Config: transactions
+- Env Var: RCLONE_CHUNKER_TRANSACTIONS
+- Type: string
+- Default: "rename"
+- Examples:
+
+- "rename"
+
+- Rename temporary files after a successful transaction.
+
+- "norename"
+
+- Leave temporary file names and write transaction ID to metadata file.
+- Metadata is required for no rename transactions (meta format cannot be "none").
+- If you are using norename transactions you should be careful not to downgrade Rclone
+- as older versions of Rclone don't support this transaction style and will misinterpret
+- files manipulated by norename transactions.
+- This method is EXPERIMENTAL, don't use on production systems.
+
+- "auto"
+
+- Rename or norename will be used depending on capabilities of the backend.
+- If meta format is set to "none", rename transactions will always be used.
+- This method is EXPERIMENTAL, don't use on production systems.
+
+
+
Citrix ShareFile
Citrix ShareFile is a secure file sharing and transfer service aimed as business.
The initial setup for Citrix ShareFile involves getting a token from Citrix ShareFile which you can in your browser. rclone config
walks you through it.
@@ -11491,7 +11856,7 @@ y/e/d> y
To use crypt
, first set up the underlying remote. Follow the rclone config
instructions for the specific backend.
Before configuring the crypt remote, check the underlying remote is working. In this example the underlying remote is called remote
. We will configure a path path
within this remote to contain the encrypted content. Anything inside remote:path
will be encrypted and anything outside will not.
Configure crypt
using rclone config
. In this example the crypt
remote is called secret
, to differentiate it from the underlying remote
.
-When you are done you can use the crypt remote named secret
just as you would with any other remote, e.g. rclone copy D:\docs secret:\docs
, and rclone will encrypt and decrypt as needed on the fly. If you access the wrapped remote remote:path
directly you will bypass the encryption, and anything you read will be in encrypted form, and anything you write will be undencrypted. To avoid issues it is best to configure a dedicated path for encrypted content, and access it exclusively through a crypt remote.
+When you are done you can use the crypt remote named secret
just as you would with any other remote, e.g. rclone copy D:\docs secret:\docs
, and rclone will encrypt and decrypt as needed on the fly. If you access the wrapped remote remote:path
directly you will bypass the encryption, and anything you read will be in encrypted form, and anything you write will be unencrypted. To avoid issues it is best to configure a dedicated path for encrypted content, and access it exclusively through a crypt remote.
No remotes found - make a new one
n) New remote
s) Set configuration password
@@ -11766,6 +12131,25 @@ $ rclone -q ls secret:
Type: bool
Default: false
+--crypt-no-data-encryption
+Option to either encrypt file data or leave it unencrypted.
+
+- Config: no_data_encryption
+- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Type: bool
+- Default: false
+- Examples:
+
+- "true"
+
+- Don't encrypt file data, leave it unencrypted.
+
+- "false"
+
+
+
Backend commands
Here are the commands specific to the crypt backend.
Run them with
@@ -11851,7 +12235,7 @@ rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile
Key derivation
Rclone uses scrypt
with parameters N=16384, r=8, p=1
with an optional user supplied salt (password2) to derive the 32+32+16 = 80 bytes of key material required. If the user doesn't supply a salt then rclone uses an internal one.
scrypt
makes it impractical to mount a dictionary attack on rclone encrypted data. For full protection against this you should always use a salt.
-SEE ALSO
+SEE ALSO
@@ -12125,6 +12509,9 @@ y/e/d> y
--dropbox-impersonate
Impersonate this user when using a business account.
+Note that if you want to use impersonate, you should make sure this flag is set when running "rclone config" as this will cause rclone to request the "members.read" scope which it won't normally. This is needed to lookup a members email address into the internal ID that dropbox uses in the API.
+Using the "members.read" scope will require a Dropbox Team Admin to approve during the OAuth flow.
+You will have to use your own App (setting your own client_id and client_secret) to use this option as currently rclone's default set of permissions doesn't include "members.read". This can be added once v1.55 or later is in use everywhere.
- Config: impersonate
- Env Var: RCLONE_DROPBOX_IMPERSONATE
@@ -12428,7 +12815,7 @@ y/e/d> y
Example without a config file
rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=`rclone obscure dummy`
Implicit TLS
-Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to be enabled in the FTP backend config for the remote, or with [--ftp-tls]{#ftp-tls}
. The default FTPS port is 990
, not 21
and can be set with [--ftp-port]{#ftp-port}
.
+Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to be enabled in the FTP backend config for the remote, or with --ftp-tls
. The default FTPS port is 990
, not 21
and can be set with --ftp-port
.
Standard Options
Here are the standard options specific to ftp (FTP Connection).
--ftp-host
@@ -12521,6 +12908,24 @@ y/e/d> y
Type: bool
Default: false
+--ftp-idle-timeout
+Max time before closing idle connections
+If no connections have been returned to the connection pool in the time given, rclone will empty the connection pool.
+Set to 0 to keep connections indefinitely.
+
+- Config: idle_timeout
+- Env Var: RCLONE_FTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
+--ftp-close-timeout
+Maximum time to wait for a response to close.
+
+- Config: close_timeout
+- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
--ftp-encoding
This sets the encoding for the backend.
See: the encoding section in the overview for more info.
@@ -13132,7 +13537,7 @@ If your browser doesn't open automatically go to the following link: http://
Log in and authorize rclone for access
Waiting for code...
Got code
-Configure this as a team drive?
+Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> n
@@ -13233,15 +13638,15 @@ y/n> # Auto config, y
Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using --drive-impersonate
, do this instead: - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1 - use rclone without specifying the --drive-impersonate
option, like this: rclone -v foo@example.com lsf gdrive:backup
-Team drives
-If you want to configure the remote to point to a Google Team Drive then answer y
to the question Configure this as a team drive?
.
-This will fetch the list of Team Drives from google and allow you to configure which one you want to use. You can also type in a team drive ID if you prefer.
+Shared drives (team drives)
+If you want to configure the remote to point to a Google Shared Drive (previously known as Team Drives) then answer y
to the question Configure this as a Shared Drive (Team Drive)?
.
+This will fetch the list of Shared Drives from google and allow you to configure which one you want to use. You can also type in a Shared Drive ID if you prefer.
For example:
-Configure this as a team drive?
+Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> y
-Fetching team drive list...
+Fetching Shared Drive list...
Choose a number from below, or type in your own value
1 / Rclone Test
\ "xxxxxxxxxxxxxxxxxxxx"
@@ -13249,7 +13654,7 @@ Choose a number from below, or type in your own value
\ "yyyyyyyyyyyyyyyyyyyy"
3 / Rclone Test 3
\ "zzzzzzzzzzzzzzzzzzzz"
-Enter a Team Drive ID> 1
+Enter a Shared Drive ID> 1
--------------------
[remote]
client_id =
@@ -13659,7 +14064,7 @@ trashed=false and 'c' in parents
Default: ""
--drive-team-drive
-ID of the Team Drive
+ID of the Shared Drive (Team Drive)
- Config: team_drive
- Env Var: RCLONE_DRIVE_TEAM_DRIVE
@@ -13972,9 +14377,9 @@ rclone backend shortcut drive: source_item -o target=drive2: destination_shortcu
- "target": optional target remote for the shortcut destination
drives
-List the shared drives available to this account
+List the Shared Drives available to this account
rclone backend drives remote: [options] [<arguments>+]
-This command lists the shared drives (teamdrives) available to this account.
+This command lists the Shared Drives (Team Drives) available to this account.
Usage:
rclone backend drives drive:
This will return a JSON list of objects like this
@@ -15104,7 +15509,7 @@ y/e/d> y
Limitations
Note that Koofr is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".
Mail.ru Cloud
-Mail.ru Cloud is a cloud storage provided by a Russian internet company Mail.Ru Group. The official desktop client is Disk-O:, available only on Windows. (Please note that official sites are in Russian)
+Mail.ru Cloud is a cloud storage provided by a Russian internet company Mail.Ru Group. The official desktop client is Disk-O:, available on Windows and Mac OS.
Currently it is recommended to disable 2FA on Mail.ru accounts intended for rclone until it gets eventually implemented.
Features highlights
@@ -15864,6 +16269,29 @@ container/
- Type: MultiEncoder
- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
+--azureblob-public-access
+Public access level of a container: blob, container.
+
+- Config: public_access
+- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
+- Type: string
+- Default: ""
+- Examples:
+
+- ""
+
+- The container and its blobs can be accessed only with an authorized request. It's a default value
+
+- "blob"
+
+- Blob data within this container can be read via anonymous request.
+
+- "container"
+
+- Allow full public read access for container and blob data.
+
+
+
Limitations
MD5 sums are only uploaded with chunked files if the source has an MD5 sum. This will always be the case for a local to azure copy.
rclone about
is not supported by the Microsoft Azure Blob storage backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs
(most free space) as a member of an rclone union remote.
@@ -18168,12 +18596,35 @@ known_hosts_file = ~/.ssh/known_hosts
Type: bool
Default: false
+--sftp-disable-concurrent-reads
+If set don't use concurrent reads
+Normally concurrent reads are safe to use and not using them will degrade performance, so this option is disabled by default.
+Some servers limit the amount number of times a file can be downloaded. Using concurrent reads can trigger this limit, so if you have a server which returns
+Failed to copy: file does not exist
+Then you may need to enable this flag.
+If concurrent reads are disabled, the use_fstat option is ignored.
+
+- Config: disable_concurrent_reads
+- Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
+- Type: bool
+- Default: false
+
+--sftp-idle-timeout
+Max time before closing idle connections
+If no connections have been returned to the connection pool in the time given, rclone will empty the connection pool.
+Set to 0 to keep connections indefinitely.
+
+- Config: idle_timeout
+- Env Var: RCLONE_SFTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
Limitations
SFTP supports checksums if the same login has shell access and md5sum
or sha1sum
as well as echo
are in the remote's PATH. This remote checksumming (file hashing) is recommended and enabled by default. Disabling the checksumming may be required if you are connecting to SFTP servers which are not under your control, and to which the execution of remote commands is prohibited. Set the configuration option disable_hashcheck
to true
to disable checksumming.
SFTP also supports about
if the same login has shell access and df
are in the remote's PATH. about
will return the total space, free space, and used space on the remote for the disk of the specified path on the remote or, if not set, the disk of the root on the remote. about
will fail if it does not have shell access or if df
is not in the remote's PATH.
Note that some SFTP servers (e.g. Synology) the paths are different for SSH and SFTP so the hashes can't be calculated properly. For them using disable_hashcheck
is a good idea.
The only ssh agent supported under Windows is Putty's pageant.
-The Go SSH library disables the use of the aes128-cbc cipher by default, due to security concerns. This can be re-enabled on a per-connection basis by setting the use_insecure_cipher
setting in the configuration file to true
. Further details on the insecurity of this cipher can be found [in this paper] (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
+The Go SSH library disables the use of the aes128-cbc cipher by default, due to security concerns. This can be re-enabled on a per-connection basis by setting the use_insecure_cipher
setting in the configuration file to true
. Further details on the insecurity of this cipher can be found in this paper.
SFTP isn't supported under plan9 until this issue is fixed.
Note that since SFTP isn't HTTP based the following flags don't work with it: --dump-headers
, --dump-bodies
, --dump-auth
Note that --timeout
isn't supported (but --contimeout
is).
@@ -18887,9 +19338,11 @@ Choose a number from below, or type in your own value
\ "nextcloud"
2 / Owncloud
\ "owncloud"
- 3 / Sharepoint
+ 3 / Sharepoint Online, authenticated by Microsoft account.
\ "sharepoint"
- 4 / Other site/service or software
+ 4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
+ \ "sharepoint-ntlm"
+ 5 / Other site/service or software
\ "other"
vendor> 1
User name
@@ -18965,7 +19418,11 @@ y/e/d> y
"sharepoint"
-- Sharepoint
+- Sharepoint Online, authenticated by Microsoft account.
+
+"sharepoint-ntlm"
+
+- Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
"other"
@@ -18974,7 +19431,7 @@ y/e/d> y
--webdav-user
-User name
+User name. In case NTLM authentication is used, the username should be in the format 'Domain'.
- Config: user
- Env Var: RCLONE_WEBDAV_USER
@@ -19008,6 +19465,16 @@ y/e/d> y
- Type: string
- Default: ""
+--webdav-encoding
+This sets the encoding for the backend.
+See: the encoding section in the overview for more info.
+Default encoding is Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8 for sharepoint-ntlm or identity otherwise.
+
+- Config: encoding
+- Env Var: RCLONE_WEBDAV_ENCODING
+- Type: string
+- Default: ""
+
Provider notes
See below for notes on specific providers.
Owncloud
@@ -19015,7 +19482,7 @@ y/e/d> y
Owncloud supports modified times using the X-OC-Mtime
header.
Nextcloud
This is configured in an identical way to Owncloud. Note that Nextcloud initially did not support streaming of files (rcat
) whereas Owncloud did, but this seems to be fixed as of 2020-11-27 (tested with rclone v1.53.1 and Nextcloud Server v19).
-Sharepoint
+Sharepoint Online
Rclone can be used with Sharepoint provided by OneDrive for Business or Office365 Education Accounts. This feature is only needed for a few of these Accounts, mostly Office365 Education ones. These accounts are sometimes not verified by the domain owner github#1975
This means that these accounts can't be added using the official API (other Accounts should work with the "onedrive" option). However, it is possible to access them using webdav.
To use a sharepoint remote with rclone, add it like this: First, you need to get your remote's URL:
@@ -19029,9 +19496,23 @@ y/e/d> y
[sharepoint]
type = webdav
url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents
-vendor = other
+vendor = sharepoint
user = YourEmailAddress
pass = encryptedpassword
+Sharepoint with NTLM Authentication
+Use this option in case your (hosted) Sharepoint is not tied to OneDrive accounts and uses NTLM authentication.
+To get the url
configuration, similarly to the above, first navigate to the desired directory in your browser to get the URL, then strip everything after the name of the opened directory.
+Example: If the URL is: https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx
+The configuration to use would be: https://example.sharepoint.com/sites/12345/Documents
+Set the vendor
to sharepoint-ntlm
.
+NTLM uses domain and user name combination for authentication, set user
to DOMAIN\username
.
+Your config file should look like this:
+[sharepoint]
+type = webdav
+url = https://[YOUR-DOMAIN]/some-path-to/Documents
+vendor = sharepoint-ntlm
+user = DOMAIN\user
+pass = encryptedpassword
Required Flags for SharePoint
As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer.
For Rclone calls copying files (especially Office files such as .docx, .xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should append these flags to ensure Rclone uses the "Last Modified" datetime property to compare your documents:
@@ -19246,7 +19727,7 @@ e) Edit this remote
d) Delete this remote
y/e/d>
See the remote setup docs for how to set it up on a machine with no Internet browser available.
-Note that rclone runs a webserver on your local machine to collect the token as returned from Zoho Workdrive. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/
and this it may require you to unblock it temporarily if you are running a host firewall.
+Rclone runs a webserver on your local computer to collect the authorization token from Zoho Workdrive. This is only from the moment your browser is opened until the token is returned. The webserver runs on http://127.0.0.1:53682/
. If local port 53682
is protected by a firewall you may need to temporarily unblock the firewall to complete authorization.
Once configured you can then use rclone
like this,
See top level directories
rclone lsd remote:
@@ -19267,6 +19748,22 @@ y/e/d>
Only control characters and invalid UTF-8 are replaced. In addition most Unicode full-width characters are not supported at all and will be removed from filenames during upload.
Standard Options
Here are the standard options specific to zoho (Zoho).
+--zoho-client-id
+OAuth Client Id Leave blank normally.
+
+- Config: client_id
+- Env Var: RCLONE_ZOHO_CLIENT_ID
+- Type: string
+- Default: ""
+
+--zoho-client-secret
+OAuth Client Secret Leave blank normally.
+
+- Config: client_secret
+- Env Var: RCLONE_ZOHO_CLIENT_SECRET
+- Type: string
+- Default: ""
+
--zoho-region
Zoho region to connect to. You'll have to use the region you organization is registered in.
@@ -19296,6 +19793,30 @@ y/e/d>
Advanced Options
Here are the advanced options specific to zoho (Zoho).
+--zoho-token
+OAuth Access Token as a JSON blob.
+
+- Config: token
+- Env Var: RCLONE_ZOHO_TOKEN
+- Type: string
+- Default: ""
+
+--zoho-auth-url
+Auth server URL. Leave blank to use the provider defaults.
+
+- Config: auth_url
+- Env Var: RCLONE_ZOHO_AUTH_URL
+- Type: string
+- Default: ""
+
+--zoho-token-url
+Token server url. Leave blank to use the provider defaults.
+
+- Config: token_url
+- Env Var: RCLONE_ZOHO_TOKEN_URL
+- Type: string
+- Default: ""
+
--zoho-encoding
This sets the encoding for the backend.
See: the encoding section in the overview for more info.
@@ -19308,8 +19829,8 @@ y/e/d>
Local Filesystem
Local paths are specified as normal filesystem paths, e.g. /path/to/wherever
, so
rclone sync -i /home/source /tmp/destination
-Will sync /home/source
to /tmp/destination
-These can be configured into the config file for consistencies sake, but it is probably easier not to.
+Will sync /home/source
to /tmp/destination
.
+For consistencies sake one can also configure a remote of type local
in the config file, and access the local filesystem using rclone remote paths, e.g. remote:path/to/wherever
, but it is probably easier not to.
Modified time
Rclone reads and writes the modified time using an accuracy determined by the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.
Filenames
@@ -19580,9 +20101,12 @@ y/e/d>
Invalid UTF-8 bytes will also be replaced, as they can't be converted to UTF-16.
-Long paths on Windows
-Rclone handles long paths automatically, by converting all paths to long UNC paths which allows paths up to 32,767 characters.
-This is why you will see that your paths, for instance c:\files
is converted to the UNC path \\?\c:\files
in the output, and \\server\share
is converted to \\?\UNC\server\share
.
+Paths on Windows
+On Windows there are many ways of specifying a path to a file system resource. Both absolute paths like C:\path\to\wherever
, and relative paths like ..\wherever
can be used, and path separator can be either \
(as in C:\path\to\wherever
) or /
(as in C:/path/to/wherever
). Length of these paths are limited to 259 characters for files and 247 characters for directories, but there is an alternative extended-length path format increasing the limit to (approximately) 32,767 characters. This format requires absolute paths and the use of prefix \\?\
, e.g. \\?\D:\some\very\long\path
. For convenience rclone will automatically convert regular paths into the corresponding extended-length paths, so in most cases you do not have to worry about this (read more below).
+Note that Windows supports using the same prefix \\?\
to specify path to volumes identified by their GUID, e.g. \\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path
. This is not supported in rclone, due to an issue in go.
+Long paths
+Rclone handles long paths automatically, by converting all paths to extended-length path format, which allows paths up to 32,767 characters.
+This conversion will ensure paths are absolute and prefix them with the \\?\
. This is why you will see that your paths, for instance .\files
is shown as path \\?\C:\files
in the output, and \\server\share
as \\?\UNC\server\share
.
However, in rare cases this may cause problems with buggy file system drivers like EncFS. To disable UNC conversion globally, add this to your .rclone.conf
file:
[local]
nounc = true
@@ -19592,7 +20116,7 @@ type = local
nounc = true
And use rclone like this:
rclone copy c:\src nounc:z:\dst
-This will use UNC paths on c:\src
but not on z:\dst
. Of course this will cause problems if the absolute path length of a file exceeds 258 characters on z, so only use this option if you have to.
+This will use UNC paths on c:\src
but not on z:\dst
. Of course this will cause problems if the absolute path length of a file exceeds 259 characters on z, so only use this option if you have to.
Symlinks / Junction points
Normally rclone will ignore symlinks or junction points (which behave like symlinks under Windows).
If you supply --copy-links
or -L
then rclone will follow the symlink and copy the pointed to file or directory. Note that this flag is incompatible with -links
/ -l
.
@@ -19675,8 +20199,8 @@ $ tree /tmp/b
0 file2
NB Rclone (like most unix tools such as du
, rsync
and tar
) treats a bind mount to the same device as being on the same filesystem.
NB This flag is only available on Unix based systems. On systems where it isn't supported (e.g. Windows) it will be ignored.
-Standard Options
-Here are the standard options specific to local (Local Disk).
+Advanced Options
+Here are the advanced options specific to local (Local Disk).
--local-nounc
Disable UNC (long path names) conversion on Windows
@@ -19692,8 +20216,6 @@ $ tree /tmp/b
-Advanced Options
-Here are the advanced options specific to local (Local Disk).
--copy-links / -L
Follow symlinks and copy the pointed to item.
@@ -19780,6 +20302,15 @@ $ tree /tmp/b
- Type: bool
- Default: false
+--local-no-preallocate
+Disable preallocation of disk space for transferred files
+Preallocation of disk space helps prevent filesystem fragmentation. However, some virtual filesystem layers (such as Google Drive File Stream) may incorrectly set the actual file size equal to the preallocated space, causing checksum and file size checks to fail. Use this flag to disable preallocation.
+
+- Config: no_preallocate
+- Env Var: RCLONE_LOCAL_NO_PREALLOCATE
+- Type: bool
+- Default: false
+
--local-no-sparse
Disable sparse files for multi-thread downloads
On Windows platforms rclone will make sparse files when doing multi-thread downloads. This avoids long pauses on large files where the OS zeros the file. However sparse files may be undesirable as they cause disk fragmentation and can be slow to work with.
@@ -19824,12 +20355,289 @@ $ tree /tmp/b
"error": return an error based on option value
Changelog
+v1.55.0 - 2021-03-31
+See commits
+
+- New commands
+
+- selfupdate (Ivan Andreev)
+
+- Allows rclone to update itself in-place or via a package (using
--package
flag)
+- Reads cryptographically signed signatures for non beta releases
+- Works on all OSes.
+
+- test - these are test commands - use with care!
+
+histogram
- Makes a histogram of file name characters.
+info
- Discovers file name or other limitations for paths.
+makefiles
- Make a random file hierarchy for testing.
+memory
- Load all the objects at remote:path into memory and report memory stats.
+
+
+- New Features
+
+- Connection strings
+
+- Config parameters can now be passed as part of the remote name as a connection string.
+- For example to do the equivalent of
--drive-shared-with-me
use drive,shared_with_me:
+- Make sure we don't save on the fly remote config to the config file (Nick Craig-Wood)
+- Make sure backends with additional config have a different name for caching (Nick Craig-Wood)
+- This work was sponsored by CERN, through the CS3MESH4EOSC Project.
+
+- CS3MESH4EOSC has received funding from the European Union’s Horizon 2020
+- research and innovation programme under Grant Agreement no. 863353.
+
+
+- build
+
+- Update go build version to go1.16 and raise minimum go version to go1.13 (Nick Craig-Wood)
+- Make a macOS ARM64 build to support Apple Silicon (Nick Craig-Wood)
+- Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood)
+- Use
GO386=softfloat
instead of deprecated GO386=387
for 386 builds (Nick Craig-Wood)
+- Disable IOS builds for the time being (Nick Craig-Wood)
+- Androids builds made with up to date NDK (x0b)
+- Add an rclone user to the Docker image but don't use it by default (cynthia kwok)
+
+- dedupe: Make largest directory primary to minimize data moved (Saksham Khanna)
+- config
+
+- Wrap config library in an interface (Fionera)
+- Make config file system pluggable (Nick Craig-Wood)
+--config ""
or "/notfound"
for in memory config only (Nick Craig-Wood)
+- Clear fs cache of stale entries when altering config (Nick Craig-Wood)
+
+- copyurl: Add option to print resulting auto-filename (albertony)
+- delete: Make
--rmdirs
obey the filters (Nick Craig-Wood)
+- docs - many fixes and reworks from edwardxml, albertony, pvalls, Ivan Andreev, Evan Harris, buengese, Alexey Tabakman
+- encoder/filename - add SCSU as tables (Klaus Post)
+- Add multiple paths support to
--compare-dest
and --copy-dest
flag (K265)
+- filter: Make
--exclude "dir/"
equivalent to --exclude "dir/**"
(Nick Craig-Wood)
+- fshttp: Add DSCP support with
--dscp
for QoS with differentiated services (Max Sum)
+- lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood)
+- lib/file
+
+- Make pre-allocate detect disk full errors and return them (Nick Craig-Wood)
+- Don't run preallocate concurrently (Nick Craig-Wood)
+- Retry preallocate on EINTR (Nick Craig-Wood)
+
+- operations: Made copy and sync operations obey a RetryAfterError (Ankur Gupta)
+- rc
+
+- Add string alternatives for setting options over the rc (Nick Craig-Wood)
+- Add
options/local
to see the options configured in the context (Nick Craig-Wood)
+- Add
_config
parameter to set global config for just this rc call (Nick Craig-Wood)
+- Implement passing filter config with
_filter
parameter (Nick Craig-Wood)
+- Add
fscache/clear
and fscache/entries
to control the fs cache (Nick Craig-Wood)
+- Avoid +Inf value for speed in
core/stats
(albertony)
+- Add a full set of stats to
core/stats
(Nick Craig-Wood)
+- Allow
fs=
params to be a JSON blob (Nick Craig-Wood)
+
+- rcd: Added systemd notification during the
rclone rcd
command. (Naveen Honest Raj)
+- rmdirs: Make
--rmdirs
obey the filters (Nick Craig-Wood)
+- version: Show build tags and type of executable (Ivan Andreev)
+
+- Bug Fixes
+
+- install.sh: make it fail on download errors (Ivan Andreev)
+- Fix excessive retries missing
--max-duration
timeout (Nick Craig-Wood)
+- Fix crash when
--low-level-retries=0
(Nick Craig-Wood)
+- Fix failed token refresh on mounts created via the rc (Nick Craig-Wood)
+- fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood)
+- lib/atexit
+
+- Unregister interrupt handler once it has fired so users can interrupt again (Nick Craig-Wood)
+- Fix occasional failure to unmount with CTRL-C (Nick Craig-Wood)
+- Fix deadlock calling Finalise while Run is running (Nick Craig-Wood)
+
+- lib/rest: Fix multipart uploads not stopping on context cancel (Nick Craig-Wood)
+
+- Mount
+
+- Allow mounting to root directory on windows (albertony)
+- Improved handling of relative paths on windows (albertony)
+- Fix unicode issues with accented characters on macOS (Nick Craig-Wood)
+- Docs: document the new FileSecurity option in WinFsp 2021 (albertony)
+- Docs: add note about volume path syntax on windows (albertony)
+- Fix caching of old directories after renaming them (Nick Craig-Wood)
+- Update cgofuse to the latest version to bring in macfuse 4 fix (Nick Craig-Wood)
+
+- VFS
+
+--vfs-used-is-size
to report used space using recursive scan (tYYGH)
+- Don't set modification time if it was already correct (Nick Craig-Wood)
+- Fix Create causing windows explorer to truncate files on CTRL-C CTRL-V (Nick Craig-Wood)
+- Fix modtimes not updating when writing via cache (Nick Craig-Wood)
+- Fix modtimes changing by fractional seconds after upload (Nick Craig-Wood)
+- Fix modtime set if
--vfs-cache-mode writes
/full
and no write (Nick Craig-Wood)
+- Rename files in cache and cancel uploads on directory rename (Nick Craig-Wood)
+- Fix directory renaming by renaming dirs cached in memory (Nick Craig-Wood)
+
+- Local
+
+- Add flag
--local-no-preallocate
(David Sze)
+- Make
nounc
an advanced option except on Windows (albertony)
+- Don't ignore preallocate disk full errors (Nick Craig-Wood)
+
+- Cache
+
+- Add
--fs-cache-expire-duration
to control the fs cache (Nick Craig-Wood)
+
+- Crypt
+
+- Add option to not encrypt data (Vesnyx)
+- Log hash ok on upload (albertony)
+
+- Azure Blob
+
+- Add container public access level support. (Manish Kumar)
+
+- B2
+
+- Fix HTML files downloaded via cloudflare (Nick Craig-Wood)
+
+- Box
+
+- Fix transfers getting stuck on token expiry after API change (Nick Craig-Wood)
+
+- Chunker
+
+- Partially implement no-rename transactions (Maxwell Calman)
+
+- Drive
+
+- Don't stop server side copy if couldn't read description (Nick Craig-Wood)
+- Pass context on to drive SDK - to help with cancellation (Nick Craig-Wood)
+
+- Dropbox
+
+- Add polling for changes support (Robert Thomas)
+- Make
--timeout 0
work properly (Nick Craig-Wood)
+- Raise priority of rate limited message to INFO to make it more noticeable (Nick Craig-Wood)
+
+- Fichier
+
+- Implement copy & move (buengese)
+- Implement public link (buengese)
+
+- FTP
+
+- Implement Shutdown method (Nick Craig-Wood)
+- Close idle connections after
--ftp-idle-timeout
(1m by default) (Nick Craig-Wood)
+- Make
--timeout 0
work properly (Nick Craig-Wood)
+- Add
--ftp-close-timeout
flag for use with awkward ftp servers (Nick Craig-Wood)
+- Retry connections and logins on 421 errors (Nick Craig-Wood)
+
+- Hdfs
+
+- Fix permissions for when directory is created (Lucas Messenger)
+
+- Onedrive
+
+- Make
--timeout 0
work properly (Nick Craig-Wood)
+
+- S3
+
+- Fix
--s3-profile
which wasn't working (Nick Craig-Wood)
+
+- SFTP
+
+- Close idle connections after
--sftp-idle-timeout
(1m by default) (Nick Craig-Wood)
+- Fix "file not found" errors for read once servers (Nick Craig-Wood)
+- Fix SetModTime stat failed: object not found with
--sftp-set-modtime=false
(Nick Craig-Wood)
+
+- Swift
+
+- Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood)
+- Implement copying large objects (nguyenhuuluan434)
+
+- Union
+
+- Fix crash when using epff policy (Nick Craig-Wood)
+- Fix union attempting to update files on a read only file system (Nick Craig-Wood)
+- Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick Craig-Wood)
+- Fix initialisation broken in refactor (Nick Craig-Wood)
+
+- WebDAV
+
+- Add support for sharepoint with NTLM authentication (Rauno Ots)
+- Make sharepoint-ntlm docs more consistent (Alex Chen)
+- Improve terminology in sharepoint-ntlm docs (Ivan Andreev)
+- Disable HTTP/2 for NTLM authentication (georne)
+- Fix sharepoint-ntlm error 401 for parallel actions (Ivan Andreev)
+- Check that purged directory really exists (Ivan Andreev)
+
+- Yandex
+
+- Make
--timeout 0
work properly (Nick Craig-Wood)
+
+- Zoho
+
+- Replace client id - you will need to
rclone config reconnect
after this (buengese)
+- Add forgotten setupRegion() to NewFs - this finally fixes regions other than EU (buengese)
+
+
+v1.54.1 - 2021-03-08
+See commits
+
+- Bug Fixes
+
+- accounting: Fix --bwlimit when up or down is off (Nick Craig-Wood)
+- docs
+
+- Fix nesting of brackets and backticks in ftp docs (edwardxml)
+- Fix broken link in sftp page (edwardxml)
+- Fix typo in crypt.md (Romeo Kienzler)
+- Changelog: Correct link to digitalis.io (Alex JOST)
+- Replace #file-caching with #vfs-file-caching (Miron Veryanskiy)
+- Convert bogus example link to code (edwardxml)
+- Remove dead link from rc.md (edwardxml)
+
+- rc: Sync,copy,move: document createEmptySrcDirs parameter (Nick Craig-Wood)
+- lsjson: Fix unterminated JSON in the presence of errors (Nick Craig-Wood)
+
+- Mount
+
+- Fix mount dropping on macOS by setting --daemon-timeout 10m (Nick Craig-Wood)
+
+- VFS
+
+- Document simultaneous usage with the same cache shouldn't be used (Nick Craig-Wood)
+
+- B2
+
+- Automatically raise upload cutoff to avoid spurious error (Nick Craig-Wood)
+- Fix failed to create file system with application key limited to a prefix (Nick Craig-Wood)
+
+- Drive
+
+- Refer to Shared Drives instead of Team Drives (Nick Craig-Wood)
+
+- Dropbox
+
+- Add scopes to oauth request and optionally "members.read" (Nick Craig-Wood)
+
+- S3
+
+- Fix failed to create file system with folder level permissions policy (Nick Craig-Wood)
+- Fix Wasabi HEAD requests returning stale data by using only 1 transport (Nick Craig-Wood)
+- Fix shared_credentials_file auth (Dmitry Chepurovskiy)
+- Add --s3-no-head to reducing costs docs (Nick Craig-Wood)
+
+- Union
+
+- Fix mkdir at root with remote:/ (Nick Craig-Wood)
+
+- Zoho
+
+- Fix custom client id's (buengese)
+
+
v1.54.0 - 2021-02-02
See commits
- New backends
-- Compression remote (experimental)(buengese)
+- Compression remote (experimental) (buengese)
- Enterprise File Fabric (Nick Craig-Wood)
- This work was sponsored by Storage Made Easy
@@ -19842,8 +20650,8 @@ $ tree /tmp/b
- Deglobalise the config (Nick Craig-Wood)
- Global config now read from the context
-- Global config can be passed into the rc
-- This work was sponsored by Digitalis
+- This will enable passing of global config via the rc
+- This work was sponsored by Digitalis
- Add
--bwlimit
for upload and download (Nick Craig-Wood)
@@ -19851,48 +20659,38 @@ $ tree /tmp/b
- Enhance systemd integration (Hekmon)
-- log level identification
-- manual activation with flag
-- automatic systemd launch detection
+- log level identification, manual activation with flag, automatic systemd launch detection
- Don't compile systemd log integration for non unix systems (Benjamin Gustin)
-- Add a download flag to hashsum and related commands to force rclone to download and hash files locally (lostheli)
+- Add a
--download
flag to md5sum/sha1sum/hashsum to force rclone to download and hash files locally (lostheli)
+- Add
--progress-terminal-title
to print ETA to terminal title (LaSombra)
+- Make backend env vars show in help as the defaults for backend flags (Nick Craig-Wood)
- build
- Raise minimum go version to go1.12 (Nick Craig-Wood)
-- check
-
-- Make the error count match up in the log message (Nick Craig-Wood)
-
-- cmd
-
-- Add --progress-terminal-title to print ETA to terminal title (LaSombra)
-- Make backend env vars show in help as the defaults for backend flags (Nick Craig-Wood)
-
- dedupe
-- Add --by-hash to dedupe on hash not file name (Nick Craig-Wood)
-- Add --dedupe-mode list to just list dupes, changing nothing (Nick Craig-Wood)
+- Add
--by-hash
to dedupe on content hash not file name (Nick Craig-Wood)
+- Add
--dedupe-mode list
to just list dupes, changing nothing (Nick Craig-Wood)
- Add warning if used on a remote which can't have duplicate names (Nick Craig-Wood)
-- flags: Improve error message when reading environment vars (Nick Craig-Wood)
- fs
- Add Shutdown optional method for backends (Nick Craig-Wood)
-- When using --files-from check files concurrently (zhucan)
-- Accumulate stats when using --dry-run (Ingo Weiss)
-- Always show stats when using --dry-run or --interactive (Nick Craig-Wood)
-- Add support for flag --no-console on windows to hide the console window (albertony)
+- When using
--files-from
check files concurrently (zhucan)
+- Accumulate stats when using
--dry-run
(Ingo Weiss)
+- Always show stats when using
--dry-run
or --interactive
(Nick Craig-Wood)
+- Add support for flag
--no-console
on windows to hide the console window (albertony)
- genautocomplete: Add support to output to stdout (Ingo)
- ncdu
- Highlight read errors instead of aborting (Claudio Bantaloukas)
- Add sort by average size in directory (Adam Plánský)
-- Add toggle option for average size in directory - key 'a' (Adam Plánský)
+- Add toggle option for average s3ize in directory - key 'a' (Adam Plánský)
- Add empty folder flag into ncdu browser (Adam Plánský)
-- Add ! (errror) and . (unreadable) file flags to go with e (empty) (Nick Craig-Wood)
+- Add
!
(errror) and .
(unreadable) file flags to go with e
(empty) (Nick Craig-Wood)
- obscure: Make
rclone osbcure -
ignore newline at end of line (Nick Craig-Wood)
- operations
@@ -19919,36 +20717,32 @@ $ tree /tmp/b
- Bug Fixes
-- build
-
-- Explicitly set ARM version to fix build (Nick Craig-Wood)
-- Don't explicitly set ARM version to fix ARMv5 build (Nick Craig-Wood)
-- Fix nfpm install (Nick Craig-Wood)
-- Fix docker build by upgrading ilteoood/docker_buildx (Nick Craig-Wood)
-- Temporary fix for Windows build errors (Ivan Andreev)
-
- fs
- Fix nil pointer on copy & move operations directly to remote (Anagh Kumar Baranwal)
- Fix parsing of .. when joining remotes (Nick Craig-Wood)
-- log: Fix enabling systemd logging when using --log-file (Nick Craig-Wood)
-- move: Fix data loss when moving the same object (Nick Craig-Wood)
+- log: Fix enabling systemd logging when using
--log-file
(Nick Craig-Wood)
+- check
+
+- Make the error count match up in the log message (Nick Craig-Wood)
+
+- move: Fix data loss when source and destination are the same object (Nick Craig-Wood)
- operations
-- Fix --cutof-mode hard not cutting off immediately (Nick Craig-Wood)
-- Fix --immutable error message (Nick Craig-Wood)
+- Fix
--cutof-mode
hard not cutting off immediately (Nick Craig-Wood)
+- Fix
--immutable
error message (Nick Craig-Wood)
- sync
-- Fix --cutoff-mode soft & cautious so it doesn't end the transfer early (Nick Craig-Wood)
-- Fix --immutable errors retrying many times (Nick Craig-Wood)
+- Fix
--cutoff-mode
soft & cautious so it doesn't end the transfer early (Nick Craig-Wood)
+- Fix
--immutable
errors retrying many times (Nick Craig-Wood)
- Docs
- Many fixes and a rewrite of the filtering docs (edwardxml)
-- Many spelling and grammar problems (Josh Soref)
+- Many spelling and grammar fixes (Josh Soref)
- Doc fixes for commands delete, purge, rmdir, rmdirs and mount (albertony)
- And thanks to these people for many doc fixes too numerous to list
@@ -19963,13 +20757,11 @@ $ tree /tmp/b
- Update systemd status with cache stats (Hekmon)
- Disable bazil/fuse based mount on macOS (Nick Craig-Wood)
-- Make mount be cmount under macOS (Nick Craig-Wood)
+- Make
rclone mount
actually run rclone cmount
under macOS (Nick Craig-Wood)
- Implement mknod to make NFS file creation work (Nick Craig-Wood)
- Make sure we don't call umount more than once (Nick Craig-Wood)
-- Don't call host.Umount if a signal has been received (Nick Craig-Wood)
- More user friendly mounting as network drive on windows (albertony)
-- Cleanup OS specific option handling and documentation (albertony)
- Detect if uid or gid are set in same option string: -o uid=123,gid=456 (albertony)
- Don't attempt to unmount if fs has been destroyed already (Nick Craig-Wood)
@@ -19977,32 +20769,34 @@ $ tree /tmp/b
- Fix virtual entries causing deleted files to still appear (Nick Craig-Wood)
- Fix "file already exists" error for stale cache files (Nick Craig-Wood)
-- Fix file leaks with --vfs-cache-mode full and --buffer-size 0 (Nick Craig-Wood)
+- Fix file leaks with
--vfs-cache-mode
full and --buffer-size 0
(Nick Craig-Wood)
- Fix invalid cache path on windows when using :backend: as remote (albertony)
- Local
- Continue listing files/folders when a circular symlink is detected (Manish Gupta)
-- New flag --local-zero-size-links to fix sync on some virtual filesystems (Riccardo Iaconelli)
+- New flag
--local-zero-size-links
to fix sync on some virtual filesystems (Riccardo Iaconelli)
- Azure Blob
- Add support for service principals (James Lim)
-- Utilize streaming capabilities (Denis Neuling)
-- Update SDK to v0.13.0 and fix API breakage (Nick Craig-Wood, Mitsuo Heijo)
+- Add support for managed identities (Brad Ackerman)
+- Add examples for access tier (Bob Pusateri)
+- Utilize the streaming capabilities from the SDK for multipart uploads (Denis Neuling)
- Fix setting of mime types (Nick Craig-Wood)
- Fix crash when listing outside a SAS URL's root (Nick Craig-Wood)
-- Delete archive tier blobs before update if --azureblob-archive-tier-delete (Nick Craig-Wood)
-- Add support for managed identities (Brad Ackerman)
+- Delete archive tier blobs before update if
--azureblob-archive-tier-delete
(Nick Craig-Wood)
- Fix crash on startup (Nick Craig-Wood)
-- Add examples for access tier (Bob Pusateri)
-- Fix memory usage by upgrading the SDK and implementing a TransferManager (Nick Craig-Wood)
+- Fix memory usage by upgrading the SDK to v0.13.0 and implementing a TransferManager (Nick Craig-Wood)
- Require go1.14+ to compile due to SDK changes (Nick Craig-Wood)
- B2
-- Make NewObject use less expensive API calls (Nick Craig-Wood)
-- Fixed possible crash when accessing Backblaze b2 remote (lluuaapp)
+- Make NewObject use less expensive API calls (Nick Craig-Wood)
+
+- This will improve
--files-from
and restic serve
in particular
+
+- Fixed crash on an empty file name (lluuaapp)
- Box
@@ -20012,12 +20806,12 @@ $ tree /tmp/b
- Chunker
- Skip long local hashing, hash in-transit (fixes) (Ivan Andreev)
-- Set Features.ReadMimeType=false as Object.MimeType not supported (Nick Craig-Wood)
+- Set Features ReadMimeType to false as Object.MimeType not supported (Nick Craig-Wood)
- Fix case-insensitive NewObject, test metadata detection (Ivan Andreev)
- Drive
-- Implement "rclone backend copyid" command for copying files by ID (Nick Craig-Wood)
+- Implement
rclone backend copyid
command for copying files by ID (Nick Craig-Wood)
- Added flag
--drive-stop-on-download-limit
to stop transfers when the download limit is exceeded (Anagh Kumar Baranwal)
- Implement CleanUp workaround for team drives (buengese)
- Allow shortcut resolution and creation to be retried (Nick Craig-Wood)
@@ -20027,45 +20821,44 @@ $ tree /tmp/b
- Dropbox
- Add support for viewing shared files and folders (buengese)
-- Implement IDer (buengese)
-- Set Features.ReadMimeType=false as Object.MimeType not supported (Nick Craig-Wood)
-- Tidy repeated error message (Nick Craig-Wood)
+- Enable short lived access tokens (Nick Craig-Wood)
+- Implement IDer on Objects so
rclone lsf
etc can read the IDs (buengese)
+- Set Features ReadMimeType to false as Object.MimeType not supported (Nick Craig-Wood)
- Make malformed_path errors from too long files not retriable (Nick Craig-Wood)
- Test file name length before upload to fix upload loop (Nick Craig-Wood)
-- Enable short lived access tokens (Nick Craig-Wood)
- Fichier
-- Set Features.ReadMimeType=true as Object.MimeType is supported (Nick Craig-Wood)
+- Set Features ReadMimeType to true as Object.MimeType is supported (Nick Craig-Wood)
- FTP
-- Add --ftp-disable-msld option to ignore MLSD for really old servers (Nick Craig-Wood)
-- Make --tpslimit apply (Nick Craig-Wood)
+- Add
--ftp-disable-msld
option to ignore MLSD for really old servers (Nick Craig-Wood)
+- Make
--tpslimit apply
(Nick Craig-Wood)
- Google Cloud Storage
- Storage class object header support (Laurens Janssen)
- Fix anonymous client to use rclone's HTTP client (Nick Craig-Wood)
-- Fix Entry doesn't belong in directory "" (same as directory) - ignoring (Nick Craig-Wood)
+- Fix
Entry doesn't belong in directory "" (same as directory) - ignoring
(Nick Craig-Wood)
- Googlephotos
-- New flag --gphotos-include-archived (Nicolas Rueff)
+- New flag
--gphotos-include-archived
to show archived photos as well (Nicolas Rueff)
- Jottacloud
-- Don't erroniously report support for writing mime types (buengese)
-- Add support for Telia Cloud (#4930) (Patrik Nordlén)
+- Don't erroneously report support for writing mime types (buengese)
+- Add support for Telia Cloud (Patrik Nordlén)
- Mailru
+- Accept special folders eg camera-upload (Ivan Andreev)
+- Avoid prehashing of large local files (Ivan Andreev)
- Fix uploads after recent changes on server (Ivan Andreev)
- Fix range requests after June 2020 changes on server (Ivan Andreev)
- Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
- Remove deprecated protocol quirks (Ivan Andreev)
-- Accept special folders eg camera-upload (Ivan Andreev)
-- Avoid prehashing of large local files (Ivan Andreev)
- Memory
@@ -20073,14 +20866,14 @@ $ tree /tmp/b
- Onedrive
-- Add support for china region operated by 21vianet and other regional suppliers (#4963) (NyaMisty)
+- Add support for China region operated by 21vianet and other regional suppliers (NyaMisty)
- Warn on gateway timeout errors (Nick Craig-Wood)
-- Fall back to normal copy if server-side copy unavailable (#4903) (Alex Chen)
+- Fall back to normal copy if server-side copy unavailable (Alex Chen)
- Fix server-side copy completely disabled on OneDrive for Business (Cnly)
-- (business only) workaround to replace existing file on server-side copy (#4904) (Alex Chen)
+- (business only) workaround to replace existing file on server-side copy (Alex Chen)
- Enhance link creation with expiry, scope, type and password (Nick Craig-Wood)
-- Remove % and # from the set of encoded characters (#4909) (Alex Chen)
-- Support addressing site by server-relative URL (#4761) (kice)
+- Remove % and # from the set of encoded characters (Alex Chen)
+- Support addressing site by server-relative URL (kice)
- Opendrive
@@ -20102,14 +20895,16 @@ $ tree /tmp/b
- S3
- Added
--s3-disable-http2
to disable http/2 (Anagh Kumar Baranwal)
-- Complete SSE-C implementation (Nick Craig-Wood)
+- Complete SSE-C implementation (Nick Craig-Wood)
+
- Fix hashes on small files with AWS:KMS and SSE-C (Nick Craig-Wood)
- Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C (Nick Craig-Wood)
+
+- Add
--s3-no-head parameter
to minimise transactions on upload (Nick Craig-Wood)
- Update docs with a Reducing Costs section (Nick Craig-Wood)
- Added error handling for error code 429 indicating too many requests (Anagh Kumar Baranwal)
- Add requester pays option (kelv)
- Fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch' (Louis Koo)
-- Add --s3-no-head parameter to minimise transactions on upload (Nick Craig-Wood)
- SFTP
@@ -20119,8 +20914,8 @@ $ tree /tmp/b
- Remember entered password in AskPass mode (Stephen Harris)
- Implement Shutdown method (Nick Craig-Wood)
- Implement keyboard interactive authentication (Nick Craig-Wood)
-- Make --tpslimit apply (Nick Craig-Wood)
-- Implement --sftp-use-fstat (Nick Craig-Wood)
+- Make
--tpslimit
apply (Nick Craig-Wood)
+- Implement
--sftp-use-fstat
for unusual SFTP servers (Nick Craig-Wood)
- Sugarsync
@@ -20130,7 +20925,7 @@ $ tree /tmp/b
- Swift
- Fix deletion of parts of Static Large Object (SLO) (Nguyễn Hữu Luân)
-- Ensure partially uploaded large files are uploaded unless --swift-leave-parts-on-error (Nguyễn Hữu Luân)
+- Ensure partially uploaded large files are uploaded unless
--swift-leave-parts-on-error
(Nguyễn Hữu Luân)
- Tardigrade
@@ -20142,7 +20937,7 @@ $ tree /tmp/b
- Yandex
-- Set Features.WriteMimeType=false as Yandex ignores mime types (Nick Craig-Wood)
+- Set Features WriteMimeType to false as Yandex ignores mime types (Nick Craig-Wood)
v1.53.4 - 2021-01-20
@@ -25029,6 +25824,28 @@ THE SOFTWARE.
- Nicolas Rueff nicolas@rueff.fr
- Pau Rodriguez-Estivill prodrigestivill@gmail.com
- Bob Pusateri BobPusateri@users.noreply.github.com
+- Alex JOST 25005220+dimejo@users.noreply.github.com
+- Alexey Tabakman samosad.ru@gmail.com
+- David Sze sze.david@gmail.com
+- cynthia kwok cynthia.m.kwok@gmail.com
+- Ankur Gupta agupta@egnyte.com
+- Miron Veryanskiy MironVeryanskiy@gmail.com
+- K265 k.265@qq.com
+- Vesnyx Vesnyx@users.noreply.github.com
+- Dmitry Chepurovskiy me@dm3ch.net
+- Rauno Ots rauno.ots@cgi.com
+- Georg Neugschwandtner georg.neugschwandtner@gmx.net
+- pvalls polvallsrue@gmail.com
+- Robert Thomas 31854736+wolveix@users.noreply.github.com
+- Romeo Kienzler romeo.kienzler@gmail.com
+- tYYGH tYYGH@users.noreply.github.com
+- georne 77802995+georne@users.noreply.github.com
+- Maxwell Calman mcalman@MacBook-Pro.local
+- Naveen Honest Raj naveendurai19@gmail.com
+- Lucas Messenger lmesseng@cisco.com
+- Manish Kumar krmanish260@gmail.com
+- x0b x0bdev@gmail.com
+- CERN through the CS3MESH4EOSC Project
Forum
diff --git a/MANUAL.md b/MANUAL.md
index 847363041..255dfc3af 100644
--- a/MANUAL.md
+++ b/MANUAL.md
@@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
-% Feb 02, 2021
+% Mar 31, 2021
# Rclone syncs your files to cloud storage
@@ -179,6 +179,9 @@ See below for some expanded Linux / macOS instructions.
See the [Usage section](https://rclone.org/docs/#usage) of the docs for how to use rclone, or
run `rclone -h`.
+Already installed rclone can be easily updated to the latest version
+using the [rclone selfupdate](https://rclone.org/commands/rclone_selfupdate/) command.
+
## Script installation ##
To install rclone on Linux/macOS/BSD systems, run:
@@ -342,7 +345,7 @@ kill %1
## Install from source ##
-Make sure you have at least [Go](https://golang.org/) 1.12
+Make sure you have at least [Go](https://golang.org/) go1.13
installed. [Download go](https://golang.org/dl/) if necessary. The
latest release is recommended. Then
@@ -390,8 +393,8 @@ Configure
First, you'll need to configure rclone. As the object storage systems
have quite complicated authentication these are kept in a config file.
-(See the `--config` entry for how to find the config file and choose
-its location.)
+(See the [`--config`](#config-config-file) entry for how to find the config
+file and choose its location.)
The easiest way to make the config is to run rclone with the config
option:
@@ -596,7 +599,8 @@ Make source and dest identical, modifying destination only.
Sync the source to the destination, changing the destination
only. Doesn't transfer unchanged files, testing by size and
modification time or MD5SUM. Destination is updated to match
-source, including deleting files if necessary.
+source, including deleting files if necessary (except duplicate
+objects, see below).
**Important**: Since this can cause data loss, test first with the
`--dry-run` or the `--interactive`/`-i` flag.
@@ -604,7 +608,8 @@ source, including deleting files if necessary.
rclone sync -i SOURCE remote:DESTINATION
Note that files in the destination won't be deleted if there were any
-errors at any point.
+errors at any point. Duplicate objects (files with the same name, on
+those providers that support it) are also not yet handled.
It is always the contents of the directory that is synced, not the
directory so when source:path is a directory, it's the contents of
@@ -616,6 +621,9 @@ go there.
**Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics
+**Note**: Use the `rclone dedupe` command to deal with "Duplicate object/directory found in source/destination - ignoring" errors.
+See [this forum post](https://forum.rclone.org/t/sync-not-clearing-duplicates/14372) for more info.
+
```
rclone sync source:path dest:path [flags]
@@ -1169,14 +1177,21 @@ Show the version number.
## Synopsis
-Show the version number, the go version and the architecture.
+Show the rclone version number, the go version, the build target OS and
+architecture, build tags and the type of executable (static or dynamic).
-Eg
+For example:
$ rclone version
- rclone v1.41
- - os/arch: linux/amd64
- - go version: go1.10
+ rclone v1.54
+ - os/type: linux
+ - os/arch: amd64
+ - go/version: go1.16
+ - go/linking: static
+ - go/tags: none
+
+Note: before rclone version 1.55 the os/type and os/arch lines were merged,
+ and the "go/version" line was tagged as "go version".
If you supply the --check flag, then it will do an online check to
compare your version with the latest release and the latest beta.
@@ -1253,8 +1268,8 @@ By default `dedupe` interactively finds files with duplicate
names and offers to delete all but one or rename them to be
different. This is known as deduping by name.
-Deduping by name is only useful with backends like Google Drive which
-can have duplicate file names. It can be run on wrapping backends
+Deduping by name is only useful with a small group of backends (e.g. Google Drive,
+Opendrive) that can have duplicate file names. It can be run on wrapping backends
(e.g. crypt) if they wrap a backend which supports duplicate file
names.
@@ -1994,15 +2009,16 @@ Copy url content to dest.
Download a URL's content and copy it to the destination without saving
it in temporary storage.
-Setting --auto-filename will cause the file name to be retrieved from
+Setting `--auto-filename`will cause the file name to be retrieved from
the from URL (after any redirections) and used in the destination
-path.
+path. With `--print-filename` in addition, the resuling file name will
+be printed.
-Setting --no-clobber will prevent overwriting file on the
+Setting `--no-clobber` will prevent overwriting file on the
destination if there is one with the same name.
-Setting --stdout or making the output file name "-" will cause the
-output to be written to standard output.
+Setting `--stdout` or making the output file name `-`
+will cause the output to be written to standard output.
```
@@ -2012,10 +2028,11 @@ rclone copyurl https://example.com dest:path [flags]
## Options
```
- -a, --auto-filename Get the file name from the URL and use it for destination file path
- -h, --help help for copyurl
- --no-clobber Prevent overwriting file with same name
- --stdout Write the output to stdout rather than a file
+ -a, --auto-filename Get the file name from the URL and use it for destination file path
+ -h, --help help for copyurl
+ --no-clobber Prevent overwriting file with same name
+ -p, --print-filename Print the resulting name from --auto-filename
+ --stdout Write the output to stdout rather than a file
```
See the [global flags page](https://rclone.org/flags/) for global options not listed here.
@@ -2759,15 +2776,15 @@ is an **empty** **existing** directory:
On Windows you can start a mount in different ways. See [below](#mounting-modes-on-windows)
for details. The following examples will mount to an automatically assigned drive,
-to specific drive letter `X:`, to path `C:\path\to\nonexistent\directory`
-(which must be **non-existent** subdirectory of an **existing** parent directory or drive,
+to specific drive letter `X:`, to path `C:\path\parent\mount`
+(where parent directory or drive must exist, and mount must **not** exist,
and is not supported when [mounting as a network drive](#mounting-modes-on-windows)), and
the last example will mount as network share `\\cloud\remote` and map it to an
automatically assigned drive:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
- rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+ rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files \\cloud\remote
When the program ends while in foreground mode, either via Ctrl+C or receiving
@@ -2821,14 +2838,14 @@ and experience unexpected program errors, freezes or other issues, consider moun
as a network drive instead.
When mounting as a fixed disk drive you can either mount to an unused drive letter,
-or to a path - which must be **non-existent** subdirectory of an **existing** parent
+or to a path representing a **non-existent** subdirectory of an **existing** parent
directory or drive. Using the special value `*` will tell rclone to
automatically assign the next available drive letter, starting with Z: and moving backward.
Examples:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
- rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+ rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files X:
Option `--volname` can be used to set a custom volume name for the mounted
@@ -2901,10 +2918,24 @@ Note that the mapping of permissions is not always trivial, and the result
you see in Windows Explorer may not be exactly like you expected.
For example, when setting a value that includes write access, this will be
mapped to individual permissions "write attributes", "write data" and "append data",
-but not "write extended attributes" (WinFsp does not support extended attributes,
-see [this](https://github.com/billziss-gh/winfsp/wiki/NTFS-Compatibility)).
-Windows will then show this as basic permission "Special" instead of "Write",
-because "Write" includes the "write extended attributes" permission.
+but not "write extended attributes". Windows will then show this as basic
+permission "Special" instead of "Write", because "Write" includes the
+"write extended attributes" permission.
+
+If you set POSIX permissions for only allowing access to the owner, using
+`--file-perms 0600 --dir-perms 0700`, the user group and the built-in "Everyone"
+group will still be given some special permissions, such as "read attributes"
+and "read permissions", in Windows. This is done for compatibility reasons,
+e.g. to allow users without additional permissions to be able to read basic
+metadata about files like in UNIX. One case that may arise is that other programs
+(incorrectly) interprets this as the file being accessible by everyone. For example
+an SSH client may warn about "unprotected private key file".
+
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity",
+that allows the complete specification of file security descriptors using
+[SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
+With this you can work around issues such as the mentioned "unprotected private key file"
+by specifying `-o FileSecurity="D:P(A;;FA;;;OW)"`, for file all access (FA) to the owner (OW).
### Windows caveats
@@ -3108,6 +3139,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -3251,6 +3289,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
```
rclone mount remote:path /path/to/mountpoint [flags]
@@ -3295,6 +3346,7 @@ rclone mount remote:path /path/to/mountpoint [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
--volname string Set the volume name. Supported on Windows and OSX only.
@@ -3667,6 +3719,83 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+# rclone selfupdate
+
+Update the rclone binary.
+
+## Synopsis
+
+
+This command downloads the latest release of rclone and replaces
+the currently running binary. The download is verified with a hashsum
+and cryptographically signed signature.
+
+If used without flags (or with implied `--stable` flag), this command
+will install the latest stable release. However, some issues may be fixed
+(or features added) only in the latest beta release. In such cases you should
+run the command with the `--beta` flag, i.e. `rclone selfupdate --beta`.
+You can check in advance what version would be installed by adding the
+`--check` flag, then repeat the command without it when you are satisfied.
+
+Sometimes the rclone team may recommend you a concrete beta or stable
+rclone release to troubleshoot your issue or add a bleeding edge feature.
+The `--version VER` flag, if given, will update to the concrete version
+instead of the latest one. If you omit micro version from `VER` (for
+example `1.53`), the latest matching micro version will be used.
+
+Upon successful update rclone will print a message that contains a previous
+version number. You will need it if you later decide to revert your update
+for some reason. Then you'll have to note the previous version and run the
+following command: `rclone selfupdate [--beta] OLDVER`.
+If the old version contains only dots and digits (for example `v1.54.0`)
+then it's a stable release so you won't need the `--beta` flag. Beta releases
+have an additional information similar to `v1.54.0-beta.5111.06f1c0c61`.
+(if you are a developer and use a locally built rclone, the version number
+will end with `-DEV`, you will have to rebuild it as it obvisously can't
+be distributed).
+
+If you previously installed rclone via a package manager, the package may
+include local documentation or configure services. You may wish to update
+with the flag `--package deb` or `--package rpm` (whichever is correct for
+your OS) to update these too. This command with the default `--package zip`
+will update only the rclone executable so the local manual may become
+inaccurate after it.
+
+The `rclone mount` command (https://rclone.org/commands/rclone_mount/) may
+or may not support extended FUSE options depending on the build and OS.
+`selfupdate` will refuse to update if the capability would be discarded.
+
+Note: Windows forbids deletion of a currently running executable so this
+command will rename the old executable to 'rclone.old.exe' upon success.
+
+Please note that this command was not available before rclone version 1.55.
+If it fails for you with the message `unknown command "selfupdate"` then
+you will need to update manually following the install instructions located
+at https://rclone.org/install/
+
+
+```
+rclone selfupdate [flags]
+```
+
+## Options
+
+```
+ --beta Install beta release.
+ --check Check for latest release, do not download.
+ -h, --help help for selfupdate
+ --output string Save the downloaded binary at a given path (default: replace running binary)
+ --package string Package format: zip|deb|rpm (default: zip)
+ --stable Install stable release (this is the default)
+ --version string Install the given rclone version (default: latest)
+```
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+
# rclone serve
Serve a remote over a protocol.
@@ -3832,6 +3961,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -3975,6 +4111,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
```
rclone serve dlna remote:path [flags]
@@ -4007,6 +4156,7 @@ rclone serve dlna remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
@@ -4145,6 +4295,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -4288,6 +4445,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
@@ -4406,6 +4576,7 @@ rclone serve ftp remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
@@ -4616,6 +4787,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -4759,6 +4937,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
```
rclone serve http remote:path [flags]
@@ -4801,6 +4992,7 @@ rclone serve http remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
@@ -5153,6 +5345,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -5296,6 +5495,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
@@ -5413,6 +5625,7 @@ rclone serve sftp remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
@@ -5631,6 +5844,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -5774,6 +5994,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
@@ -5900,6 +6133,7 @@ rclone serve webdav remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
@@ -5956,6 +6190,152 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+# rclone test
+
+Run a test command
+
+## Synopsis
+
+Rclone test is used to run test commands.
+
+Select which test comand you want with the subcommand, eg
+
+ rclone test memory remote:
+
+Each subcommand has its own options which you can see in their help.
+
+**NB** Be careful running these commands, they may do strange things
+so reading their documentation first is recommended.
+
+
+## Options
+
+```
+ -h, --help help for test
+```
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+* [rclone test histogram](https://rclone.org/commands/rclone_test_histogram/) - Makes a histogram of file name characters.
+* [rclone test info](https://rclone.org/commands/rclone_test_info/) - Discovers file name or other limitations for paths.
+* [rclone test makefiles](https://rclone.org/commands/rclone_test_makefiles/) - Make a random file hierarchy in
+* [rclone test memory](https://rclone.org/commands/rclone_test_memory/) - Load all the objects at remote:path into memory and report memory stats.
+
+# rclone test histogram
+
+Makes a histogram of file name characters.
+
+## Synopsis
+
+This command outputs JSON which shows the histogram of characters used
+in filenames in the remote:path specified.
+
+The data doesn't contain any identifying information but is useful for
+the rclone developers when developing filename compression.
+
+
+```
+rclone test histogram [remote:path] [flags]
+```
+
+## Options
+
+```
+ -h, --help help for histogram
+```
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+
+# rclone test info
+
+Discovers file name or other limitations for paths.
+
+## Synopsis
+
+rclone info discovers what filenames and upload methods are possible
+to write to the paths passed in and how long they can be. It can take some
+time. It will write test files into the remote:path passed in. It outputs
+a bit of go code for each one.
+
+**NB** this can create undeletable files and other hazards - use with care
+
+
+```
+rclone test info [remote:path]+ [flags]
+```
+
+## Options
+
+```
+ --all Run all tests.
+ --check-control Check control characters.
+ --check-length Check max filename length.
+ --check-normalization Check UTF-8 Normalization.
+ --check-streaming Check uploads with indeterminate file size.
+ -h, --help help for info
+ --upload-wait duration Wait after writing a file.
+ --write-json string Write results to file.
+```
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+
+# rclone test makefiles
+
+Make a random file hierarchy in
+
+```
+rclone test makefiles [flags]
+```
+
+## Options
+
+```
+ --files int Number of files to create (default 1000)
+ --files-per-directory int Average number of files per directory (default 10)
+ -h, --help help for makefiles
+ --max-file-size SizeSuffix Maximum size of files to create (default 100)
+ --max-name-length int Maximum size of file names (default 12)
+ --min-file-size SizeSuffix Minimum size of file to create
+ --min-name-length int Minimum size of file names (default 4)
+```
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+
+# rclone test memory
+
+Load all the objects at remote:path into memory and report memory stats.
+
+```
+rclone test memory remote:path [flags]
+```
+
+## Options
+
+```
+ -h, --help help for memory
+```
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+
# rclone touch
Create new file or change file modification time.
@@ -6104,8 +6484,9 @@ The syntax of the paths passed to the rclone command are as follows.
This refers to the local file system.
-On Windows only `\` may be used instead of `/` in local paths
-**only**, non local paths must use `/`.
+On Windows `\` may be used instead of `/` in local paths **only**,
+non local paths must use `/`. See [local filesystem](https://rclone.org/local/#windows-paths)
+documentation for more about Windows-specific paths.
These paths needn't start with a leading `/` - if they don't then they
will be relative to the current directory.
@@ -6150,6 +6531,108 @@ To copy files and directories in `https://example.com/path/to/dir` to `/tmp/dir`
To copy files and directories from `example.com` in the relative
directory `path/to/dir` to `/tmp/dir` using sftp.
+### Connection strings {#connection-strings}
+
+The above examples can also be written using a connection string
+syntax, so instead of providing the arguments as command line
+parameters `--http-url https://pub.rclone.org` they are provided as
+part of the remote specification as a kind of connection string.
+
+ rclone lsd ":http,url='https://pub.rclone.org':"
+ rclone lsf ":http,url='https://example.com':path/to/dir"
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+ rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
+
+These can apply to modify existing remotes as well as create new
+remotes with the on the fly syntax. This example is equivalent to
+adding the `--drive-shared-with-me` parameter to the remote `gdrive:`.
+
+ rclone lsf "gdrive,shared_with_me:path/to/dir"
+
+The major advantage to using the connection string style syntax is
+that it only applies the the remote, not to all the remotes of that
+type of the command line. A common confusion is this attempt to copy a
+file shared on google drive to the normal drive which **does not
+work** because the `--drive-shared-with-me` flag applies to both the
+source and the destination.
+
+ rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
+
+However using the connection string syntax, this does work.
+
+ rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
+
+The connection strings have the following syntax
+
+ remote,parameter=value,parameter2=value2:path/to/dir
+ :backend,parameter=value,parameter2=value2:path/to/dir
+
+If the `parameter` has a `:` or `,` then it must be placed in quotes `"` or
+`'`, so
+
+ remote,parameter="colon:value",parameter2="comma,value":path/to/dir
+ :backend,parameter='colon:value',parameter2='comma,value':path/to/dir
+
+If a quoted value needs to include that quote, then it should be
+doubled, so
+
+ remote,parameter="with""quote",parameter2='with''quote':path/to/dir
+
+This will make `parameter` be `with"quote` and `parameter2` be
+`with'quote`.
+
+If you leave off the `=parameter` then rclone will substitute `=true`
+which works very well with flags. For example to use s3 configured in
+the environment you could use:
+
+ rclone lsd :s3,env_auth:
+
+Which is equivalent to
+
+ rclone lsd :s3,env_auth=true:
+
+Note that on the command line you might need to surround these
+connection strings with `"` or `'` to stop the shell interpreting any
+special characters within them.
+
+If you are a shell master then you'll know which strings are OK and
+which aren't, but if you aren't sure then enclose them in `"` and use
+`'` as the inside quote. This syntax works on all OSes.
+
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+
+On Linux/macOS some characters are still interpreted inside `"`
+strings in the shell (notably `\` and `$` and `"`) so if your strings
+contain those you can swap the roles of `"` and `'` thus. (This syntax
+does not work on Windows.)
+
+ rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
+
+#### Connection strings, config and logging
+
+If you supply extra configuration to a backend by command line flag,
+environment variable or connection string then rclone will add a
+suffix based on the hash of the config to the name of the remote, eg
+
+ rclone -vv lsf --s3-chunk-size 20M s3:
+
+Has the log message
+
+ DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
+
+This is so rclone can tell the modified remote apart from the
+unmodified remote when caching the backends.
+
+This should only be noticeable in the logs.
+
+This means that on the fly backends such as
+
+ rclone -vv lsf :s3,env_auth:
+
+Will get their own names
+
+ DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
+
### Valid remote names
- Remote names may only contain 0-9, A-Z ,a-z ,_ , - and space.
@@ -6306,7 +6789,7 @@ This option controls the bandwidth limit. For example
--bwlimit 10M
-would mean limit the upload and download bandwidth to 10 Mbyte/s.
+would mean limit the upload and download bandwidth to 10 MByte/s.
**NB** this is **bytes** per second not **bits** per second. To use a
single limit, specify the desired bandwidth in kBytes/s, or use a
suffix b|k|M|G. The default is `0` which means to not limit bandwidth.
@@ -6316,7 +6799,7 @@ The upload and download bandwidth can be specified seperately, as
--bwlimit 10M:100k
-would mean limit the upload bandwidth to 10 Mbyte/s and the download
+would mean limit the upload bandwidth to 10 MByte/s and the download
bandwidth to 100 kByte/s. Either limit can be "off" meaning no limit, so
to just limit the upload bandwidth you would use
@@ -6345,9 +6828,9 @@ working hours could be:
`--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"`
In this example, the transfer bandwidth will be set to 512kBytes/sec
-at 8am every day. At noon, it will rise to 10Mbytes/s, and drop back
+at 8am every day. At noon, it will rise to 10MByte/s, and drop back
to 512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to
-30MBytes/s, and at 11pm it will be completely disabled (full speed).
+30MByte/s, and at 11pm it will be completely disabled (full speed).
Anything between 11pm and 8am will remain unlimited.
An example of timetable with `WEEKDAY` could be:
@@ -6355,8 +6838,8 @@ An example of timetable with `WEEKDAY` could be:
`--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"`
It means that, the transfer bandwidth will be set to 512kBytes/sec on
-Monday. It will rise to 10Mbytes/s before the end of Friday. At 10:00
-on Saturday it will be set to 1Mbyte/s. From 20:00 on Sunday it will
+Monday. It will rise to 10MByte/s before the end of Friday. At 10:00
+on Saturday it will be set to 1MByte/s. From 20:00 on Sunday it will
be unlimited.
Timeslots without `WEEKDAY` are extended to the whole week. So this
@@ -6479,7 +6962,7 @@ See `--copy-dest` and `--backup-dir`.
### --config=CONFIG_FILE ###
-Specify the location of the rclone config file.
+Specify the location of the rclone configuration file.
Normally the config file is in your home directory as a file called
`.config/rclone/rclone.conf` (or `.rclone.conf` if created with an
@@ -6496,6 +6979,45 @@ location is for you.
Use this flag to override the config location, e.g. `rclone
--config=".myconfig" .config`.
+If the location is set to empty string `""` or the special value
+`/notfound`, or the os null device represented by value `NUL` on
+Windows and `/dev/null` on Unix systems, then rclone will keep the
+config file in memory only.
+
+The file format is basic [INI](https://en.wikipedia.org/wiki/INI_file#Format):
+Sections of text, led by a `[section]` header and followed by
+`key=value` entries on separate lines. In rclone each remote is
+represented by its own section, where the section name defines the
+name of the remote. Options are specified as the `key=value` entries,
+where the key is the option name without the `--backend-` prefix,
+in lowercase and with `_` instead of `-`. E.g. option `--mega-hard-delete`
+corresponds to key `hard_delete`. Only backend options can be specified.
+A special, and required, key `type` identifies the [storage system](https://rclone.org/overview/),
+where the value is the internal lowercase name as returned by command
+`rclone help backends`. Comments are indicated by `;` or `#` at the
+beginning of a line.
+
+Example:
+
+ [megaremote]
+ type = mega
+ user = you@example.com
+ pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
+
+Note that passwords are in [obscured](https://rclone.org/commands/rclone_obscure/)
+form. Also, many storage systems uses token-based authentication instead
+of passwords, and this requires additional steps. It is easier, and safer,
+to use the interactive command `rclone config` instead of manually
+editing the configuration file.
+
+The configuration file will typically contain login information, and
+should therefore have restricted permissions so that only the current user
+can read it. Rclone tries to ensure this when it writes the file.
+You may also choose to [encrypt](#configuration-encryption) the file.
+
+When token-based authentication are used, the configuration file
+must be writable, because rclone needs to update the tokens inside it.
+
### --contimeout=TIME ###
Set the connection timeout. This should be in go time format which
@@ -6543,6 +7065,21 @@ This flag can be useful for debugging and in exceptional circumstances
(e.g. Google Drive limiting the total volume of Server Side Copies to
100GB/day).
+### --dscp VALUE ###
+
+Specify a DSCP value or name to use in connections. This could help QoS
+system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are allowed.
+
+See the description of [differentiated services](https://en.wikipedia.org/wiki/Differentiated_services) to get an idea of
+this field. Setting this to 1 (LE) to identify the flow to SCAVENGER class
+can avoid occupying too much bandwidth in a network with DiffServ support ([RFC 8622](https://tools.ietf.org/html/rfc8622)).
+
+For example, if you configured QoS on router to handle LE properly. Running:
+```
+rclone copy --dscp LE from:/from to:/to
+```
+would make the priority lower than usual internet flows.
+
### -n, --dry-run ###
Do a trial run with no permanent changes. Use this to see what rclone
@@ -6573,6 +7110,27 @@ triggering follow-on actions if data was copied, or skipping if not.
NB: Enabling this option turns a usually non-fatal error into a potentially
fatal one - please check and adjust your scripts accordingly!
+### --fs-cache-expire-duration=TIME
+
+When using rclone via the API rclone caches created remotes for 5
+minutes by default in the "fs cache". This means that if you do
+repeated actions on the same remote then rclone won't have to build it
+again from scratch, which makes it more efficient.
+
+This flag sets the time that the remotes are cached for. If you set it
+to `0` (or negative) then rclone won't cache the remotes at all.
+
+Note that if you use some flags, eg `--backup-dir` and if this is set
+to `0` rclone may build two remotes (one for the source or destination
+and one for the `--backup-dir` where it may have only built one
+before.
+
+### --fs-cache-expire-interval=TIME
+
+This controls how often rclone checks for cached remotes to expire.
+See the `--fs-cache-expire-duration` documentation above for more
+info. The default is 60s, set to 0 to disable expiry.
+
### --header ###
Add an HTTP header for all transactions. The flag can be repeated to
@@ -7544,7 +8102,7 @@ Configuration Encryption
------------------------
Your configuration file contains information for logging in to
your cloud services. This means that you should keep your
-`.rclone.conf` file in a secure location.
+`rclone.conf` file in a secure location.
If you are in an environment where that isn't possible, you can
add a password to your configuration. This means that you will
@@ -7650,6 +8208,16 @@ password prompts. To do that, pass the parameter
of asking for a password if `RCLONE_CONFIG_PASS` doesn't contain
a valid password, and `--password-command` has not been supplied.
+Some rclone commands, such as `genautocomplete`, do not require configuration.
+Nevertheless, rclone will read any configuration file found
+according to the rules described [above](https://rclone.org/docs/#config-config-file).
+If an encrypted configuration file is found, this means you will be prompted for
+password (unless using `--password-command`). To avoid this, you can bypass
+the loading of the configuration file by overriding the location with an empty
+string `""` or the special value `/notfound`, or the os null device represented
+by value `NUL` on Windows and `/dev/null` on Unix systems (before rclone
+version 1.55 only this null device alternative was supported).
+E.g. `rclone --config="" genautocomplete bash`.
Developer options
-----------------
@@ -7851,11 +8419,8 @@ so they take exactly the same form.
### Config file ###
You can set defaults for values in the config file on an individual
-remote basis. If you want to use this feature, you will need to
-discover the name of the config items that you want. The easiest way
-is to run through `rclone config` by hand, then look in the config
-file to see what the values are (the config file can be found by
-looking at the help for `--config` in `rclone help`).
+remote basis. The names of the config items are documented in the page
+for each backend.
To find the name of the environment variable, you need to set, take
`RCLONE_CONFIG_` + name of remote + `_` + name of config file option
@@ -7877,6 +8442,11 @@ mys3:
Note that if you want to create a remote using environment variables
you must create the `..._TYPE` variable as above.
+Note also that now rclone has [connectionstrings](#connection-strings),
+it is probably easier to use those instead which makes the above example
+
+ rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
+
### Precedence
The various different methods of backend configuration are read in
@@ -8071,6 +8641,24 @@ separator or the beginning of the path/file.
- doesn't match "afile.jpg"
- doesn't match "directory/file.jpg"
+The top level of the remote may not be the top level of the drive.
+
+E.g. for a Microsoft Windows local directory structure
+
+ F:
+ ├── bkp
+ ├── data
+ │ ├── excl
+ │ │ ├── 123.jpg
+ │ │ └── 456.jpg
+ │ ├── incl
+ │ │ └── document.pdf
+
+To copy the contents of folder `data` into folder `bkp` excluding the contents of subfolder
+`excl`the following command treats `F:\data` and `F:\bkp` as top level for filtering.
+
+`rclone copy F:\data\ F:\bkp\ --exclude=/excl/**`
+
**Important** Use `/` in path/file name patterns and not `\` even if
running on Microsoft Windows.
@@ -8088,7 +8676,7 @@ With `--ignore-case`
## How filter rules are applied to files
-Rclone path / file name filters are made up of one or more of the following flags:
+Rclone path/file name filters are made up of one or more of the following flags:
* `--include`
* `--include-from`
@@ -8114,7 +8702,7 @@ To mix up the order of processing includes and excludes use `--filter...`
flags.
Within `--include-from`, `--exclude-from` and `--filter-from` flags
-rules are processed from top to bottom of the referenced file..
+rules are processed from top to bottom of the referenced file.
If there is an `--include` or `--include-from` flag specified, rclone
implies a `- **` rule which it adds to the bottom of the internal rule
@@ -8146,7 +8734,7 @@ classes. [Go regular expression reference](https://golang.org/pkg/regexp/syntax/
### How filter rules are applied to directories
-Rclone commands filter, and are applied to, path/file names not
+Rclone commands are applied to path/file names not
directories. The entire contents of a directory can be matched
to a filter by the pattern `directory/*` or recursively by
`directory/**`.
@@ -8160,15 +8748,15 @@ recurse into subdirectories. This potentially optimises access to a remote
by avoiding listing unnecessary directories. Whether optimisation is
desirable depends on the specific filter rules and source remote content.
-Optimisation occurs if either:
+Directory recursion optimisation occurs if either:
-* A source remote does not support the rclone `ListR` primitive. `local`,
-`sftp`, `Microsoft OneDrive` and `WebDav` do not support `ListR`. Google
+* A source remote does not support the rclone `ListR` primitive. local,
+sftp, Microsoft OneDrive and WebDav do not support `ListR`. Google
Drive and most bucket type storage do. [Full list](https://rclone.org/overview/#optional-features)
-* On other remotes, if the rclone command is not naturally recursive,
+* On other remotes (those that support `ListR`), if the rclone command is not naturally recursive, and
provided it is not run with the `--fast-list` flag. `ls`, `lsf -R` and
-`size` are recursive but `sync`, `copy` and `move` are not.
+`size` are naturally recursive but `sync`, `copy` and `move` are not.
* Whenever the `--disable ListR` flag is applied to an rclone command.
@@ -8190,7 +8778,7 @@ to be specified.
E.g. `rclone ls remote: --include /directory/` will not match any
files. Because it is an `--include` option the `--exclude **` rule
-is implied, and the `\directory\` pattern serves only to optimise
+is implied, and the `/directory/` pattern serves only to optimise
access to the remote by ignoring everything outside of that directory.
E.g. `rclone ls remote: --filter-from filter-list.txt` with a file
@@ -8203,7 +8791,7 @@ E.g. `rclone ls remote: --filter-from filter-list.txt` with a file
All files in directories `dir1` or `dir2` or their subdirectories
are completely excluded from the listing. Only files of suffix
-`'pdf` in the root of `remote:` or its subdirectories are listed.
+`pdf` in the root of `remote:` or its subdirectories are listed.
The `- **` rule prevents listing of any path/files not previously
matched by the rules above.
@@ -8211,6 +8799,28 @@ Option `exclude-if-present` creates a directory exclude rule based
on the presence of a file in a directory and takes precedence over
other rclone directory filter rules.
+When using pattern list syntax, if a pattern item contains either
+`/` or `**`, then rclone will not able to imply a directory filter rule
+from this pattern list.
+
+E.g. for an include rule
+
+ {dir1/**,dir2/**}
+
+Rclone will match files below directories `dir1` or `dir2` only,
+but will not be able to use this filter to exclude a directory `dir3`
+from being traversed.
+
+Directory recursion optimisation may affect performance, but normally
+not the result. One exception to this is sync operations with option
+`--create-empty-src-dirs`, where any traversed empty directories will
+be created. With the pattern list example `{dir1/**,dir2/**}` above,
+this would create an empty directory `dir3` on destination (when it exists
+on source). Changing the filter to `{dir1,dir2}/**`, or splitting it into
+two include rules `--include dir1/** --include dir2/**`, will match the
+same files while also filtering directories, with the result that an empty
+directory `dir3` will no longer be created.
+
### `--exclude` - Exclude files matching pattern
Excludes path/file names from an rclone command based on a single exclude
@@ -8234,8 +8844,8 @@ directories.
E.g. on Microsoft Windows `rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"`
lists the files in `remote:` with `[JP]` or `[KR]` or `[HK]` in
-their name. The single quotes prevent the shell from interpreting the `\`
-characters. The `\` characters escape the `[` and `]` so ran clone filter
+their name. Quotes prevent the shell from interpreting the `\`
+characters.`\` characters escape the `[` and `]` so an rclone filter
treats them literally rather than as a character-range. The `{` and `}`
define an rclone pattern list. For other operating systems single quotes are
required ie `rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'`
@@ -8371,7 +8981,7 @@ processed in.
Arrange the order of filter rules with the most restrictive first and
work down.
-E.g. For `filter-file.txt`:
+E.g. for `filter-file.txt`:
# a sample filter rule file
- secret*.jpg
@@ -8482,13 +9092,13 @@ The three files are transferred as follows:
/home/user1/dir/ford → remote:backup/user1/dir/file
/home/user2/prefect → remote:backup/user2/stuff
-Alternatively if `/` is chosen as root `files-from.txt` would be:
+Alternatively if `/` is chosen as root `files-from.txt` will be:
/home/user1/42
/home/user1/dir/ford
/home/user2/prefect
-The copy command would be:
+The copy command will be:
rclone copy --files-from files-from.txt / remote:backup
@@ -8569,10 +9179,10 @@ Default units are seconds or the following abbreviations are valid:
`--max-age` can also be specified as an absolute time in the following
formats:
-- RFC3339 - e.g. "2006-01-02T15:04:05Z07:00"
-- ISO8601 Date and time, local timezone - "2006-01-02T15:04:05"
-- ISO8601 Date and time, local timezone - "2006-01-02 15:04:05"
-- ISO8601 Date - "2006-01-02" (YYYY-MM-DD)
+- RFC3339 - e.g. `2006-01-02T15:04:05Z` or `2006-01-02T15:04:05+07:00`
+- ISO8601 Date and time, local timezone - `2006-01-02T15:04:05`
+- ISO8601 Date and time, local timezone - `2006-01-02 15:04:05`
+- ISO8601 Date - `2006-01-02` (YYYY-MM-DD)
`--max-age` applies only to files and not to directories.
@@ -8596,7 +9206,7 @@ old or more.
**Important** this flag is dangerous to your data - use with `--dry-run`
and `-v` first.
-In conjunction with `rclone sync` the `--delete-excluded deletes any files
+In conjunction with `rclone sync`, `--delete-excluded` deletes any files
on the destination which are excluded from the command.
E.g. the scope of `rclone sync -i A: B:` can be restricted:
@@ -8636,7 +9246,7 @@ not list `dir3`, `file3` or `.ignore`.
## Common pitfalls
The most frequent filter support issues on
-the [rclone forum](https://https://forum.rclone.org/) are:
+the [rclone forum](https://forum.rclone.org/) are:
* Not using paths relative to the root of the remote
* Not using `/` to match from the root of a remote
@@ -8946,8 +9556,6 @@ Rather than
rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
```
-
-
## Special parameters
The rc interface supports some special parameters which apply to
@@ -9018,6 +9626,69 @@ $ rclone rc job/list
}
```
+### Setting config flags with _config
+
+If you wish to set config (the equivalent of the global flags) for the
+duration of an rc call only then pass in the `_config` parameter.
+
+This should be in the same format as the `config` key returned by
+[options/get](#options-get).
+
+For example, if you wished to run a sync with the `--checksum`
+parameter, you would pass this parameter in your JSON blob.
+
+ "_config":{"CheckSum": true}
+
+If using `rclone rc` this could be passed as
+
+ rclone rc operations/sync ... _config='{"CheckSum": true}'
+
+Any config parameters you don't set will inherit the global defaults
+which were set with command line flags or environment variables.
+
+Note that it is possible to set some values as strings or integers -
+see [data types](/#data-types) for more info. Here is an example
+setting the equivalent of `--buffer-size` in string or integer format.
+
+ "_config":{"BufferSize": "42M"}
+ "_config":{"BufferSize": 44040192}
+
+If you wish to check the `_config` assignment has worked properly then
+calling `options/local` will show what the value got set to.
+
+### Setting filter flags with _filter
+
+If you wish to set filters for the duration of an rc call only then
+pass in the `_filter` parameter.
+
+This should be in the same format as the `filter` key returned by
+[options/get](#options-get).
+
+For example, if you wished to run a sync with these flags
+
+ --max-size 1M --max-age 42s --include "a" --include "b"
+
+you would pass this parameter in your JSON blob.
+
+ "_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}
+
+If using `rclone rc` this could be passed as
+
+ rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}'
+
+Any filter parameters you don't set will inherit the global defaults
+which were set with command line flags or environment variables.
+
+Note that it is possible to set some values as strings or integers -
+see [data types](/#data-types) for more info. Here is an example
+setting the equivalent of `--buffer-size` in string or integer format.
+
+ "_filter":{"MinSize": "42M"}
+ "_filter":{"MinSize": 44040192}
+
+If you wish to check the `_filter` assignment has worked properly then
+calling `options/local` will show what the value got set to.
+
### Assigning operations to groups with _group = value
Each rc call has its own stats group for tracking its metrics. By default
@@ -9037,6 +9708,78 @@ $ rclone rc --json '{ "group": "job/1" }' core/stats
}
```
+## Data types {#data-types}
+
+When the API returns types, these will mostly be straight forward
+integer, string or boolean types.
+
+However some of the types returned by the [options/get](#options-get)
+call and taken by the [options/set](#options-set) calls as well as the
+`vfsOpt`, `mountOpt` and the `_config` parameters.
+
+- `Duration` - these are returned as an integer duration in
+ nanoseconds. They may be set as an integer, or they may be set with
+ time string, eg "5s". See the [options section](https://rclone.org/docs/#options) for
+ more info.
+- `Size` - these are returned as an integer number of bytes. They may
+ be set as an integer or they may be set with a size suffix string,
+ eg "10M". See the [options section](https://rclone.org/docs/#options) for more info.
+- Enumerated type (such as `CutoffMode`, `DumpFlags`, `LogLevel`,
+ `VfsCacheMode` - these will be returned as an integer and may be set
+ as an integer but more conveniently they can be set as a string, eg
+ "HARD" for `CutoffMode` or `DEBUG` for `LogLevel`.
+- `BandwidthSpec` - this will be set and returned as a string, eg
+ "1M".
+
+## Specifying remotes to work on
+
+Remotes are specified with the `fs=`, `srcFs=`, `dstFs=`
+parameters depending on the command being used.
+
+The parameters can be a string as per the rest of rclone, eg
+`s3:bucket/path` or `:sftp:/my/dir`. They can also be specified as
+JSON blobs.
+
+If specifyng a JSON blob it should be a object mapping strings to
+strings. These values will be used to configure the remote. There are
+3 special values which may be set:
+
+- `type` - set to `type` to specify a remote called `:type:`
+- `_name` - set to `name` to specify a remote called `name:`
+- `_root` - sets the root of the remote - may be empty
+
+One of `_name` or `type` should normally be set. If the `local`
+backend is desired then `type` should be set to `local`. If `_root`
+isn't specified then it defaults to the root of the remote.
+
+For example this JSON is equivalent to `remote:/tmp`
+
+```
+{
+ "_name": "remote",
+ "_path": "/tmp"
+}
+```
+
+And this is equivalent to `:sftp,host='example.com':/tmp`
+
+```
+{
+ "type": "sftp",
+ "host": "example.com",
+ "_path": "/tmp"
+}
+```
+
+And this is equivalent to `/tmp/dir`
+
+```
+{
+ type = "local",
+ _ path = "/tmp/dir"
+}
+```
+
## Supported commands
### backend/command: Runs a backend command. {#backend-command}
@@ -9375,18 +10118,22 @@ Returns the following values:
```
{
- "speed": average speed in bytes/sec since start of the process,
- "bytes": total transferred bytes since the start of the process,
+ "bytes": total transferred bytes since the start of the group,
+ "checks": number of files checked,
+ "deletes" : number of files deleted,
+ "elapsedTime": time in floating point seconds since rclone was started,
"errors": number of errors,
- "fatalError": whether there has been at least one FatalError,
- "retryError": whether there has been at least one non-NoRetryError,
- "checks": number of checked files,
- "transfers": number of transferred files,
- "deletes" : number of deleted files,
- "renames" : number of renamed files,
+ "eta": estimated time in seconds until the group completes,
+ "fatalError": boolean whether there has been at least one fatal error,
+ "lastError": last error string,
+ "renames" : number of files renamed,
+ "retryError": boolean showing whether there has been at least one non-NoRetryError,
+ "speed": average speed in bytes/sec since start of the group,
+ "totalBytes": total number of bytes in the group,
+ "totalChecks": total number of checks in the group,
+ "totalTransfers": total number of transfers in the group,
"transferTime" : total time spent on running jobs,
- "elapsedTime": time in seconds since the start of the process,
- "lastError": last occurred error,
+ "transfers": number of transferred files,
"transferring": an array of currently active file transfers:
[
{
@@ -9467,6 +10214,8 @@ This shows the current version of go and the go runtime
- os - OS in use as according to Go
- arch - cpu architecture in use according to Go
- goVersion - version of Go runtime in use
+- linking - type of rclone executable (static or dynamic)
+- goTags - space separated build tags or "none"
### debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. {#debug-set-block-profile-rate}
@@ -9506,6 +10255,26 @@ Results
- previousRate - int
+### fscache/clear: Clear the Fs cache. {#fscache-clear}
+
+This clears the fs cache. This is where remotes created from backends
+are cached for a short while to make repeated rc calls more efficient.
+
+If you change the parameters of a backend then you may want to call
+this to clear an existing remote out of the cache before re-creating
+it.
+
+**Authentication is required for this call.**
+
+### fscache/entries: Returns the number of entries in the fs cache. {#fscache-entries}
+
+This returns the number of entries in the fs cache.
+
+Returns
+- entries - number of items in the cache
+
+**Authentication is required for this call.**
+
### job/list: Lists the IDs of the running jobs {#job-list}
Parameters - None
@@ -9875,11 +10644,31 @@ See the [uploadfile command](https://rclone.org/commands/rclone_uploadfile/) com
Returns
- options - a list of the options block names
-### options/get: Get all the options {#options-get}
+### options/get: Get all the global options {#options-get}
Returns an object where keys are option block names and values are an
object with the current option values in.
+Note that these are the global options which are unaffected by use of
+the _config and _filter parameters. If you wish to read the parameters
+set in _config then use options/config and for _filter use options/filter.
+
+This shows the internal names of the option within rclone which should
+map to the external options very easily with a few exceptions.
+
+### options/local: Get the currently active config for this call {#options-local}
+
+Returns an object with the keys "config" and "filter".
+The "config" key contains the local config and the "filter" key contains
+the local filters.
+
+Note that these are the local options specific to this rc call. If
+_config was not supplied then they will be the global options.
+Likewise with "_filter".
+
+This call is mostly useful for seeing if _config and _filter passing
+is working.
+
This shows the internal names of the option within rclone which should
map to the external options very easily with a few exceptions.
@@ -9898,17 +10687,18 @@ changed like this.
For example:
-This sets DEBUG level logs (-vv)
+This sets DEBUG level logs (-vv) (these can be set by number or string)
+ rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'
rclone rc options/set --json '{"main": {"LogLevel": 8}}'
And this sets INFO level logs (-v)
- rclone rc options/set --json '{"main": {"LogLevel": 7}}'
+ rclone rc options/set --json '{"main": {"LogLevel": "INFO"}}'
And this sets NOTICE level logs (normal without -v)
- rclone rc options/set --json '{"main": {"LogLevel": 6}}'
+ rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}'
### pluginsctl/addPlugin: Add a plugin using url {#pluginsctl-addPlugin}
@@ -10031,6 +10821,7 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
See the [copy command](https://rclone.org/commands/rclone_copy/) command for more information on the above.
@@ -10043,6 +10834,7 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
- deleteEmptySrcDirs - delete empty src directories if set
@@ -10056,6 +10848,7 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
See the [sync command](https://rclone.org/commands/rclone_sync/) command for more information on the above.
@@ -10685,43 +11478,43 @@ remote itself may assign the MIME type.
All rclone remotes support a base command set. Other features depend
upon backend specific capabilities.
-| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir |
-| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:| :------: |
-| 1Fichier | No | No | No | No | No | No | No | No | No | Yes |
-| Amazon Drive | Yes | No | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes |
-| Amazon S3 | No | Yes | No | No | Yes | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No |
-| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
-| Box | Yes | Yes | Yes | Yes | Yes ‡‡ | No | Yes | Yes | No | Yes |
-| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes |
-| Dropbox | Yes | Yes | Yes | Yes | No [#575](https://github.com/rclone/rclone/issues/575) | No | Yes | Yes | Yes | Yes |
-| Enterprise File Fabric | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
-| FTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes |
-| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No |
-| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
-| Google Photos | No | No | No | No | No | No | No | No | No | No |
-| HDFS | Yes | No | No | No | No | No | Yes | No | Yes | Yes |
-| HTTP | No | No | No | No | No | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | Yes |
-| Hubic | Yes †| Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No |
-| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
-| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
-| Mega | Yes | No | Yes | Yes | Yes | No | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes |
-| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No |
-| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No |
-| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
-| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
-| OpenStack Swift | Yes †| Yes | No | No | No | Yes | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | No |
-| pCloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
-| premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes |
-| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes |
-| QingStor | No | Yes | No | No | Yes | Yes | No | No [#2178](https://github.com/rclone/rclone/issues/2178) | No | No |
-| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
-| SFTP | No | No | Yes | Yes | No | No | Yes | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes |
-| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes |
-| Tardigrade | Yes †| No | No | No | No | Yes | Yes | No | No | No |
-| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No [#2178](https://github.com/rclone/rclone/issues/2178) | Yes | Yes |
-| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |
-| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | Yes | Yes |
-| The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
+| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir |
+| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:|:--------:|
+| 1Fichier | No | Yes | Yes | No | No | No | No | Yes | No | Yes |
+| Amazon Drive | Yes | No | Yes | Yes | No | No | No | No | No | Yes |
+| Amazon S3 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
+| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
+| Box | Yes | Yes | Yes | Yes | Yes ‡‡ | No | Yes | Yes | Yes | Yes |
+| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes |
+| Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes |
+| Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | Yes |
+| FTP | No | No | Yes | Yes | No | No | Yes | No | No | Yes |
+| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No |
+| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
+| Google Photos | No | No | No | No | No | No | No | No | No | No |
+| HDFS | Yes | No | No | No | No | No | Yes | No | Yes | Yes |
+| HTTP | No | No | No | No | No | No | No | No | No | Yes |
+| Hubic | Yes †| Yes | No | No | No | Yes | Yes | No | Yes | No |
+| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
+| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
+| Mega | Yes | No | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
+| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No |
+| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No |
+| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
+| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
+| OpenStack Swift | Yes †| Yes | No | No | No | Yes | Yes | No | Yes | No |
+| pCloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
+| premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes |
+| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | Yes | Yes |
+| QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No |
+| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
+| SFTP | No | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
+| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes |
+| Tardigrade | Yes †| No | No | No | No | Yes | Yes | No | No | No |
+| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No | Yes | Yes |
+| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |
+| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | Yes | Yes |
+| The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
### Purge ###
@@ -10831,10 +11624,10 @@ These flags are available for every command.
-c, --checksum Skip based on checksum (if available) & size, not mod-time & size
--client-cert string Client SSL certificate (PEM) for mutual TLS auth
--client-key string Client SSL private key (PEM) for mutual TLS auth
- --compare-dest string Include additional server-side path during comparison.
+ --compare-dest stringArray Include additional comma separated server-side paths during comparison.
--config string Config file. (default "$HOME/.config/rclone/rclone.conf")
--contimeout duration Connect timeout (default 1m0s)
- --copy-dest string Implies --compare-dest but also copies files from path into destination.
+ --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination.
--cpuprofile string Write cpu profile to file
--cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
--delete-after When synchronizing, delete files on destination after transferring (default)
@@ -10843,6 +11636,7 @@ These flags are available for every command.
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
-n, --dry-run Do a trial run with no permanent changes
+ --dscp string Set DSCP value to connections. Can be value or names, eg. CS1, LE, DF, AF21.
--dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
--dump-headers Dump HTTP headers - may contain sensitive info
@@ -10856,6 +11650,8 @@ These flags are available for every command.
--files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
-f, --filter stringArray Add a file-filtering rule
--filter-from stringArray Read filtering patterns from a file (use - to read from stdin)
+ --fs-cache-expire-duration duration cache remotes for this long (0 to disable caching) (default 5m0s)
+ --fs-cache-expire-interval duration interval to check for expired remotes (default 1m0s)
--header stringArray Set HTTP header for all transactions
--header-download stringArray Set HTTP header for download transactions
--header-upload stringArray Set HTTP header for upload transactions
@@ -10954,7 +11750,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
- --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.54.0")
+ --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
-v, --verbose count Print lots more stuff (repeat for more)
```
@@ -10987,6 +11783,7 @@ and may be set in the config file.
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_object_id or msi_mi_res_id specified.
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_object_id specified.
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_mi_res_id specified.
+ --azureblob-public-access string Public access level of a container: blob, container.
--azureblob-sas-url string SAS URL for container level access only
--azureblob-service-principal-file string Path to file containing credentials for use with a service principal.
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256MB). (Deprecated)
@@ -11050,6 +11847,7 @@ and may be set in the config file.
-L, --copy-links Follow symlinks and copy the pointed to item.
--crypt-directory-name-encryption Option to either encrypt directory names or leave them intact. (default true)
--crypt-filename-encryption string How to encrypt the filenames. (default "standard")
+ --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted.
--crypt-password string Password or pass phrase for encryption. (obscured)
--crypt-password2 string Password or pass phrase for salt. Optional but recommended. (obscured)
--crypt-remote string Remote to encrypt/decrypt.
@@ -11085,7 +11883,7 @@ and may be set in the config file.
--drive-starred-only Only show files that are starred.
--drive-stop-on-download-limit Make download limit errors be fatal
--drive-stop-on-upload-limit Make upload limit errors be fatal
- --drive-team-drive string ID of the Team Drive
+ --drive-team-drive string ID of the Shared Drive (Team Drive)
--drive-token string OAuth Access Token as a JSON blob.
--drive-token-url string Token server url.
--drive-trashed-only Only show files that are in the trash.
@@ -11114,12 +11912,14 @@ and may be set in the config file.
--filefabric-token-expiry string Token expiry time
--filefabric-url string URL of the Enterprise File Fabric to connect to
--filefabric-version string Version read from the file fabric
+ --ftp-close-timeout Duration Maximum time to wait for a response to close. (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
--ftp-disable-epsv Disable using EPSV even if server advertises support
--ftp-disable-mlsd Disable using MLSD even if server advertises support
--ftp-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot)
--ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
--ftp-host string FTP host to connect to
+ --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--ftp-no-check-certificate Do not verify the TLS certificate of the server
--ftp-pass string FTP password (obscured)
--ftp-port string FTP port, leave blank to use default (21)
@@ -11181,6 +11981,7 @@ and may be set in the config file.
--local-case-sensitive Force the filesystem to report itself as case sensitive.
--local-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot)
--local-no-check-updated Don't check to see if the files change during upload
+ --local-no-preallocate Disable preallocation of disk space for transferred files
--local-no-set-modtime Disable setting modtime
--local-no-sparse Disable sparse files for multi-thread downloads
--local-no-unicode-normalization Don't apply unicode normalization to paths and filenames (Deprecated)
@@ -11285,8 +12086,10 @@ and may be set in the config file.
--seafile-url string URL of seafile host to connect to
--seafile-user string User name (usually email address)
--sftp-ask-password Allow asking for SFTP password when needed.
+ --sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available.
--sftp-host string SSH host to connect to
+ --sftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--sftp-key-file string Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.
--sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file. (obscured)
--sftp-key-pem string Raw PEM-encoded private key, If specified, will override key_file parameter.
@@ -11356,9 +12159,10 @@ and may be set in the config file.
--union-upstreams string List of space separated upstreams.
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
--webdav-bearer-token-command string Command to run to get a bearer token
+ --webdav-encoding string This sets the encoding for the backend.
--webdav-pass string Password. (obscured)
--webdav-url string URL of http host to connect to
- --webdav-user string User name
+ --webdav-user string User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-auth-url string Auth server URL.
--yandex-client-id string OAuth Client Id
@@ -11366,8 +12170,13 @@ and may be set in the config file.
--yandex-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot)
--yandex-token string OAuth Access Token as a JSON blob.
--yandex-token-url string Token server url.
+ --zoho-auth-url string Auth server URL.
+ --zoho-client-id string OAuth Client Id
+ --zoho-client-secret string OAuth Client Secret
--zoho-encoding MultiEncoder This sets the encoding for the backend. (default Del,Ctl,InvalidUtf8)
--zoho-region string Zoho region to connect to. You'll have to use the region you organization is registered in.
+ --zoho-token string OAuth Access Token as a JSON blob.
+ --zoho-token-url string Token server url.
```
1Fichier
@@ -12281,6 +13090,16 @@ You'd then do a full `rclone sync` less often.
Note that `--fast-list` isn't required in the top-up sync.
+#### Avoiding HEAD requests after PUT
+
+By default rclone will HEAD every object it uploads. It does this to
+check the object got uploaded correctly.
+
+You can disable this with the [--s3-no-head](#s3-no-head) option - see
+there for more details.
+
+Setting this flag increases the chance for undetected upload failures.
+
### Hashes ###
For small objects which weren't uploaded as multipart uploads (objects
@@ -15706,7 +16525,7 @@ These URLs are used by Plex internally to connect to the Plex server securely.
The format for these URLs is the following:
-https://ip-with-dots-replaced.server-hash.plex.direct:32400/
+`https://ip-with-dots-replaced.server-hash.plex.direct:32400/`
The `ip-with-dots-replaced` part can be any IPv4 address, where the dots
have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`.
@@ -16291,6 +17110,9 @@ Note that `list` assembles composite directory entries only when chunk names
match the configured format and treats non-conforming file names as normal
non-chunked files.
+When using `norename` transactions, chunk names will additionally have a unique
+file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
+
### Metadata
@@ -16310,6 +17132,7 @@ for composite files. Meta objects carry the following fields:
- `nchunks` - number of data chunks in file
- `md5` - MD5 hashsum of composite file (if present)
- `sha1` - SHA1 hashsum (if present)
+- `txn` - identifies current version of the file
There is no field for composite file name as it's simply equal to the name
of meta object on the wrapped remote. Please refer to respective sections
@@ -16382,8 +17205,8 @@ use modification time of the first data chunk.
### Migrations
-The idiomatic way to migrate to a different chunk size, hash type or
-chunk naming scheme is to:
+The idiomatic way to migrate to a different chunk size, hash type, transaction
+style or chunk naming scheme is to:
- Collect all your chunked files under a directory and have your
chunker remote point to it.
@@ -16443,6 +17266,8 @@ Chunker included in rclone releases up to `v1.54` can sometimes fail to
detect metadata produced by recent versions of rclone. We recommend users
to keep rclone up-to-date to avoid data corruption.
+Changing `transactions` is dangerous and requires explicit migration.
+
### Standard Options
@@ -16550,6 +17375,29 @@ Choose how chunker should handle files with missing or invalid chunks.
- "false"
- Warn user, skip incomplete file and proceed.
+#### --chunker-transactions
+
+Choose how chunker should handle temporary files during transactions.
+
+- Config: transactions
+- Env Var: RCLONE_CHUNKER_TRANSACTIONS
+- Type: string
+- Default: "rename"
+- Examples:
+ - "rename"
+ - Rename temporary files after a successful transaction.
+ - "norename"
+ - Leave temporary file names and write transaction ID to metadata file.
+ - Metadata is required for no rename transactions (meta format cannot be "none").
+ - If you are using norename transactions you should be careful not to downgrade Rclone
+ - as older versions of Rclone don't support this transaction style and will misinterpret
+ - files manipulated by norename transactions.
+ - This method is EXPERIMENTAL, don't use on production systems.
+ - "auto"
+ - Rename or norename will be used depending on capabilities of the backend.
+ - If meta format is set to "none", rename transactions will always be used.
+ - This method is EXPERIMENTAL, don't use on production systems.
+
## Citrix ShareFile
@@ -16870,7 +17718,7 @@ as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
and rclone will encrypt and decrypt as needed on the fly.
If you access the wrapped remote `remote:path` directly you will bypass
the encryption, and anything you read will be in encrypted form, and
-anything you write will be undencrypted. To avoid issues it is best to
+anything you write will be unencrypted. To avoid issues it is best to
configure a dedicated path for encrypted content, and access it
exclusively through a crypt remote.
@@ -17305,6 +18153,20 @@ names, or for debugging purposes.
- Type: bool
- Default: false
+#### --crypt-no-data-encryption
+
+Option to either encrypt file data or leave it unencrypted.
+
+- Config: no_data_encryption
+- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Type: bool
+- Default: false
+- Examples:
+ - "true"
+ - Don't encrypt file data, leave it unencrypted.
+ - "false"
+ - Encrypt file data.
+
### Backend commands
Here are the commands specific to the crypt backend.
@@ -17821,6 +18683,21 @@ memory. It can be set smaller if you are tight on memory.
Impersonate this user when using a business account.
+Note that if you want to use impersonate, you should make sure this
+flag is set when running "rclone config" as this will cause rclone to
+request the "members.read" scope which it won't normally. This is
+needed to lookup a members email address into the internal ID that
+dropbox uses in the API.
+
+Using the "members.read" scope will require a Dropbox Team Admin
+to approve during the OAuth flow.
+
+You will have to use your own App (setting your own client_id and
+client_secret) to use this option as currently rclone's default set of
+permissions doesn't include "members.read". This can be added once
+v1.55 or later is in use everywhere.
+
+
- Config: impersonate
- Env Var: RCLONE_DROPBOX_IMPERSONATE
- Type: string
@@ -18277,8 +19154,8 @@ excess files in the directory.
Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to
be enabled in the FTP backend config for the remote, or with
-`[--ftp-tls]{#ftp-tls}`. The default FTPS port is `990`, not `21` and
-can be set with `[--ftp-port]{#ftp-port}`.
+[`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and
+can be set with [`--ftp-port`](#ftp-port).
### Standard Options
@@ -18391,6 +19268,30 @@ Disable using MLSD even if server advertises support
- Type: bool
- Default: false
+#### --ftp-idle-timeout
+
+Max time before closing idle connections
+
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+
+Set to 0 to keep connections indefinitely.
+
+
+- Config: idle_timeout
+- Env Var: RCLONE_FTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
+#### --ftp-close-timeout
+
+Maximum time to wait for a response to close.
+
+- Config: close_timeout
+- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
#### --ftp-encoding
This sets the encoding for the backend.
@@ -19042,7 +19943,7 @@ If your browser doesn't open automatically go to the following link: http://127.
Log in and authorize rclone for access
Waiting for code...
Got code
-Configure this as a team drive?
+Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> n
@@ -19249,23 +20150,24 @@ Note: in case you configured a specific root folder on gdrive and rclone is unab
`rclone -v foo@example.com lsf gdrive:backup`
-### Team drives ###
+### Shared drives (team drives) ###
-If you want to configure the remote to point to a Google Team Drive
-then answer `y` to the question `Configure this as a team drive?`.
+If you want to configure the remote to point to a Google Shared Drive
+(previously known as Team Drives) then answer `y` to the question
+`Configure this as a Shared Drive (Team Drive)?`.
-This will fetch the list of Team Drives from google and allow you to
-configure which one you want to use. You can also type in a team
-drive ID if you prefer.
+This will fetch the list of Shared Drives from google and allow you to
+configure which one you want to use. You can also type in a Shared
+Drive ID if you prefer.
For example:
```
-Configure this as a team drive?
+Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> y
-Fetching team drive list...
+Fetching Shared Drive list...
Choose a number from below, or type in your own value
1 / Rclone Test
\ "xxxxxxxxxxxxxxxxxxxx"
@@ -19273,7 +20175,7 @@ Choose a number from below, or type in your own value
\ "yyyyyyyyyyyyyyyyyyyy"
3 / Rclone Test 3
\ "zzzzzzzzzzzzzzzzzzzz"
-Enter a Team Drive ID> 1
+Enter a Shared Drive ID> 1
--------------------
[remote]
client_id =
@@ -19644,7 +20546,7 @@ Needed only if you want use SA instead of interactive login.
#### --drive-team-drive
-ID of the Team Drive
+ID of the Shared Drive (Team Drive)
- Config: team_drive
- Env Var: RCLONE_DRIVE_TEAM_DRIVE
@@ -20107,11 +21009,11 @@ Options:
#### drives
-List the shared drives available to this account
+List the Shared Drives available to this account
rclone backend drives remote: [options] [+]
-This command lists the shared drives (teamdrives) available to this
+This command lists the Shared Drives (Team Drives) available to this
account.
Usage:
@@ -21815,7 +22717,7 @@ Note that Koofr is case insensitive so you can't have a file called
Mail.ru Cloud
----------------------------------------
-[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/), available only on Windows. (Please note that official sites are in Russian)
+[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS.
Currently it is recommended to disable 2FA on Mail.ru accounts intended for rclone until it gets eventually implemented.
@@ -22799,6 +23701,22 @@ See: the [encoding section in the overview](https://rclone.org/overview/#encodin
- Type: MultiEncoder
- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
+#### --azureblob-public-access
+
+Public access level of a container: blob, container.
+
+- Config: public_access
+- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
+- Type: string
+- Default: ""
+- Examples:
+ - ""
+ - The container and its blobs can be accessed only with an authorized request. It's a default value
+ - "blob"
+ - Blob data within this container can be read via anonymous request.
+ - "container"
+ - Allow full public read access for container and blob data.
+
### Limitations ###
@@ -25717,6 +26635,44 @@ any given time.
- Type: bool
- Default: false
+#### --sftp-disable-concurrent-reads
+
+If set don't use concurrent reads
+
+Normally concurrent reads are safe to use and not using them will
+degrade performance, so this option is disabled by default.
+
+Some servers limit the amount number of times a file can be
+downloaded. Using concurrent reads can trigger this limit, so if you
+have a server which returns
+
+ Failed to copy: file does not exist
+
+Then you may need to enable this flag.
+
+If concurrent reads are disabled, the use_fstat option is ignored.
+
+
+- Config: disable_concurrent_reads
+- Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
+- Type: bool
+- Default: false
+
+#### --sftp-idle-timeout
+
+Max time before closing idle connections
+
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+
+Set to 0 to keep connections indefinitely.
+
+
+- Config: idle_timeout
+- Env Var: RCLONE_SFTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
### Limitations ###
@@ -25747,8 +26703,8 @@ The Go SSH library disables the use of the aes128-cbc cipher by
default, due to security concerns. This can be re-enabled on a
per-connection basis by setting the `use_insecure_cipher` setting in
the configuration file to `true`. Further details on the insecurity of
-this cipher can be found [in this paper]
-(http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
+this cipher can be found
+[in this paper](http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
SFTP isn't supported under plan9 until [this
issue](https://github.com/pkg/sftp/issues/156) is fixed.
@@ -26599,9 +27555,11 @@ Choose a number from below, or type in your own value
\ "nextcloud"
2 / Owncloud
\ "owncloud"
- 3 / Sharepoint
+ 3 / Sharepoint Online, authenticated by Microsoft account.
\ "sharepoint"
- 4 / Other site/service or software
+ 4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
+ \ "sharepoint-ntlm"
+ 5 / Other site/service or software
\ "other"
vendor> 1
User name
@@ -26689,13 +27647,15 @@ Name of the Webdav site/service/software you are using
- "owncloud"
- Owncloud
- "sharepoint"
- - Sharepoint
+ - Sharepoint Online, authenticated by Microsoft account.
+ - "sharepoint-ntlm"
+ - Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
- "other"
- Other site/service or software
#### --webdav-user
-User name
+User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
- Config: user
- Env Var: RCLONE_WEBDAV_USER
@@ -26735,6 +27695,19 @@ Command to run to get a bearer token
- Type: string
- Default: ""
+#### --webdav-encoding
+
+This sets the encoding for the backend.
+
+See: the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+
+Default encoding is Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8 for sharepoint-ntlm or identity otherwise.
+
+- Config: encoding
+- Env Var: RCLONE_WEBDAV_ENCODING
+- Type: string
+- Default: ""
+
## Provider notes ##
@@ -26755,7 +27728,7 @@ This is configured in an identical way to Owncloud. Note that
Nextcloud initially did not support streaming of files (`rcat`) whereas
Owncloud did, but [this](https://github.com/nextcloud/nextcloud-snap/issues/365) seems to be fixed as of 2020-11-27 (tested with rclone v1.53.1 and Nextcloud Server v19).
-### Sharepoint ###
+### Sharepoint Online ###
Rclone can be used with Sharepoint provided by OneDrive for Business
or Office365 Education Accounts.
@@ -26791,11 +27764,40 @@ Your config file should look like this:
[sharepoint]
type = webdav
url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents
-vendor = other
+vendor = sharepoint
user = YourEmailAddress
pass = encryptedpassword
```
+### Sharepoint with NTLM Authentication ###
+
+Use this option in case your (hosted) Sharepoint is not tied to OneDrive accounts and uses NTLM authentication.
+
+To get the `url` configuration, similarly to the above, first navigate to the desired directory in your browser to get the URL,
+then strip everything after the name of the opened directory.
+
+Example:
+If the URL is:
+https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx
+
+The configuration to use would be:
+https://example.sharepoint.com/sites/12345/Documents
+
+Set the `vendor` to `sharepoint-ntlm`.
+
+NTLM uses domain and user name combination for authentication,
+set `user` to `DOMAIN\username`.
+
+Your config file should look like this:
+
+```
+[sharepoint]
+type = webdav
+url = https://[YOUR-DOMAIN]/some-path-to/Documents
+vendor = sharepoint-ntlm
+user = DOMAIN\user
+pass = encryptedpassword
+```
#### Required Flags for SharePoint ####
As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer.
@@ -27148,11 +28150,12 @@ y/e/d>
See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
machine with no Internet browser available.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Zoho Workdrive. This only runs from the moment it
-opens your browser to the moment you get back the verification code.
-This is on `http://127.0.0.1:53682/` and this it may require you to
-unblock it temporarily if you are running a host firewall.
+Rclone runs a webserver on your local computer to collect the
+authorization token from Zoho Workdrive. This is only from the moment
+your browser is opened until the token is returned.
+The webserver runs on `http://127.0.0.1:53682/`.
+If local port `53682` is protected by a firewall you may need to temporarily
+unblock the firewall to complete authorization.
Once configured you can then use `rclone` like this,
@@ -27199,6 +28202,26 @@ from filenames during upload.
Here are the standard options specific to zoho (Zoho).
+#### --zoho-client-id
+
+OAuth Client Id
+Leave blank normally.
+
+- Config: client_id
+- Env Var: RCLONE_ZOHO_CLIENT_ID
+- Type: string
+- Default: ""
+
+#### --zoho-client-secret
+
+OAuth Client Secret
+Leave blank normally.
+
+- Config: client_secret
+- Env Var: RCLONE_ZOHO_CLIENT_SECRET
+- Type: string
+- Default: ""
+
#### --zoho-region
Zoho region to connect to. You'll have to use the region you organization is registered in.
@@ -27221,6 +28244,35 @@ Zoho region to connect to. You'll have to use the region you organization is reg
Here are the advanced options specific to zoho (Zoho).
+#### --zoho-token
+
+OAuth Access Token as a JSON blob.
+
+- Config: token
+- Env Var: RCLONE_ZOHO_TOKEN
+- Type: string
+- Default: ""
+
+#### --zoho-auth-url
+
+Auth server URL.
+Leave blank to use the provider defaults.
+
+- Config: auth_url
+- Env Var: RCLONE_ZOHO_AUTH_URL
+- Type: string
+- Default: ""
+
+#### --zoho-token-url
+
+Token server url.
+Leave blank to use the provider defaults.
+
+- Config: token_url
+- Env Var: RCLONE_ZOHO_TOKEN_URL
+- Type: string
+- Default: ""
+
#### --zoho-encoding
This sets the encoding for the backend.
@@ -27241,15 +28293,17 @@ Local paths are specified as normal filesystem paths, e.g. `/path/to/wherever`,
rclone sync -i /home/source /tmp/destination
-Will sync `/home/source` to `/tmp/destination`
+Will sync `/home/source` to `/tmp/destination`.
-These can be configured into the config file for consistencies sake,
-but it is probably easier not to.
+For consistencies sake one can also configure a remote of type
+`local` in the config file, and access the local filesystem using
+rclone remote paths, e.g. `remote:path/to/wherever`, but it is probably
+easier not to.
### Modified time ###
Rclone reads and writes the modified time using an accuracy determined by
-the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second
+the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second
on OS X.
### Filenames ###
@@ -27340,15 +28394,35 @@ These only get replaced if they are the last character in the name:
Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
as they can't be converted to UTF-16.
-### Long paths on Windows ###
+### Paths on Windows ###
-Rclone handles long paths automatically, by converting all paths to long
-[UNC paths](https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath)
-which allows paths up to 32,767 characters.
+On Windows there are many ways of specifying a path to a file system resource.
+Both absolute paths like `C:\path\to\wherever`, and relative paths like
+`..\wherever` can be used, and path separator can be either
+`\` (as in `C:\path\to\wherever`) or `/` (as in `C:/path/to/wherever`).
+Length of these paths are limited to 259 characters for files and 247
+characters for directories, but there is an alternative extended-length
+path format increasing the limit to (approximately) 32,767 characters.
+This format requires absolute paths and the use of prefix `\\?\`,
+e.g. `\\?\D:\some\very\long\path`. For convenience rclone will automatically
+convert regular paths into the corresponding extended-length paths,
+so in most cases you do not have to worry about this (read more [below](#long-paths)).
-This is why you will see that your paths, for instance `c:\files` is
-converted to the UNC path `\\?\c:\files` in the output,
-and `\\server\share` is converted to `\\?\UNC\server\share`.
+Note that Windows supports using the same prefix `\\?\` to
+specify path to volumes identified by their GUID, e.g.
+`\\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path`.
+This is *not* supported in rclone, due to an [issue](https://github.com/golang/go/issues/39785)
+in go.
+
+#### Long paths ####
+
+Rclone handles long paths automatically, by converting all paths to
+[extended-length path format](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation), which allows paths up to 32,767 characters.
+
+This conversion will ensure paths are absolute and prefix them with
+the `\\?\`. This is why you will see that your paths, for instance
+`.\files` is shown as path `\\?\C:\files` in the output, and `\\server\share`
+as `\\?\UNC\server\share`.
However, in rare cases this may cause problems with buggy file
system drivers like [EncFS](https://github.com/rclone/rclone/issues/261).
@@ -27372,7 +28446,7 @@ And use rclone like this:
This will use UNC paths on `c:\src` but not on `z:\dst`.
Of course this will cause problems if the absolute path length of a
-file exceeds 258 characters on z, so only use this option if you have to.
+file exceeds 259 characters on z, so only use this option if you have to.
### Symlinks / Junction points
@@ -27529,9 +28603,9 @@ filesystem.
where it isn't supported (e.g. Windows) it will be ignored.
-### Standard Options
+### Advanced Options
-Here are the standard options specific to local (Local Disk).
+Here are the advanced options specific to local (Local Disk).
#### --local-nounc
@@ -27545,10 +28619,6 @@ Disable UNC (long path names) conversion on Windows
- "true"
- Disables long file names
-### Advanced Options
-
-Here are the advanced options specific to local (Local Disk).
-
#### --copy-links / -L
Follow symlinks and copy the pointed to item.
@@ -27678,6 +28748,21 @@ to override the default choice.
- Type: bool
- Default: false
+#### --local-no-preallocate
+
+Disable preallocation of disk space for transferred files
+
+Preallocation of disk space helps prevent filesystem fragmentation.
+However, some virtual filesystem layers (such as Google Drive File
+Stream) may incorrectly set the actual file size equal to the
+preallocated space, causing checksum and file size checks to fail.
+Use this flag to disable preallocation.
+
+- Config: no_preallocate
+- Env Var: RCLONE_LOCAL_NO_PREALLOCATE
+- Type: bool
+- Default: false
+
#### --local-no-sparse
Disable sparse files for multi-thread downloads
@@ -27752,12 +28837,203 @@ Options:
# Changelog
+## v1.55.0 - 2021-03-31
+
+[See commits](https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)
+
+* New commands
+ * [selfupdate](https://rclone.org/commands/rclone_selfupdate/) (Ivan Andreev)
+ * Allows rclone to update itself in-place or via a package (using `--package` flag)
+ * Reads cryptographically signed signatures for non beta releases
+ * Works on all OSes.
+ * [test](https://rclone.org/commands/rclone_test/) - these are test commands - use with care!
+ * `histogram` - Makes a histogram of file name characters.
+ * `info` - Discovers file name or other limitations for paths.
+ * `makefiles` - Make a random file hierarchy for testing.
+ * `memory` - Load all the objects at remote:path into memory and report memory stats.
+* New Features
+ * [Connection strings](https://rclone.org/docs/#connection-strings)
+ * Config parameters can now be passed as part of the remote name as a connection string.
+ * For example to do the equivalent of `--drive-shared-with-me` use `drive,shared_with_me:`
+ * Make sure we don't save on the fly remote config to the config file (Nick Craig-Wood)
+ * Make sure backends with additional config have a different name for caching (Nick Craig-Wood)
+ * This work was sponsored by CERN, through the [CS3MESH4EOSC Project](https://cs3mesh4eosc.eu/).
+ * CS3MESH4EOSC has received funding from the European Union’s Horizon 2020
+ * research and innovation programme under Grant Agreement no. 863353.
+ * build
+ * Update go build version to go1.16 and raise minimum go version to go1.13 (Nick Craig-Wood)
+ * Make a macOS ARM64 build to support Apple Silicon (Nick Craig-Wood)
+ * Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood)
+ * Use `GO386=softfloat` instead of deprecated `GO386=387` for 386 builds (Nick Craig-Wood)
+ * Disable IOS builds for the time being (Nick Craig-Wood)
+ * Androids builds made with up to date NDK (x0b)
+ * Add an rclone user to the Docker image but don't use it by default (cynthia kwok)
+ * dedupe: Make largest directory primary to minimize data moved (Saksham Khanna)
+ * config
+ * Wrap config library in an interface (Fionera)
+ * Make config file system pluggable (Nick Craig-Wood)
+ * `--config ""` or `"/notfound"` for in memory config only (Nick Craig-Wood)
+ * Clear fs cache of stale entries when altering config (Nick Craig-Wood)
+ * copyurl: Add option to print resulting auto-filename (albertony)
+ * delete: Make `--rmdirs` obey the filters (Nick Craig-Wood)
+ * docs - many fixes and reworks from edwardxml, albertony, pvalls, Ivan Andreev, Evan Harris, buengese, Alexey Tabakman
+ * encoder/filename - add SCSU as tables (Klaus Post)
+ * Add multiple paths support to `--compare-dest` and `--copy-dest` flag (K265)
+ * filter: Make `--exclude "dir/"` equivalent to `--exclude "dir/**"` (Nick Craig-Wood)
+ * fshttp: Add DSCP support with `--dscp` for QoS with differentiated services (Max Sum)
+ * lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood)
+ * lib/file
+ * Make pre-allocate detect disk full errors and return them (Nick Craig-Wood)
+ * Don't run preallocate concurrently (Nick Craig-Wood)
+ * Retry preallocate on EINTR (Nick Craig-Wood)
+ * operations: Made copy and sync operations obey a RetryAfterError (Ankur Gupta)
+ * rc
+ * Add string alternatives for setting options over the rc (Nick Craig-Wood)
+ * Add `options/local` to see the options configured in the context (Nick Craig-Wood)
+ * Add `_config` parameter to set global config for just this rc call (Nick Craig-Wood)
+ * Implement passing filter config with `_filter` parameter (Nick Craig-Wood)
+ * Add `fscache/clear` and `fscache/entries` to control the fs cache (Nick Craig-Wood)
+ * Avoid +Inf value for speed in `core/stats` (albertony)
+ * Add a full set of stats to `core/stats` (Nick Craig-Wood)
+ * Allow `fs=` params to be a JSON blob (Nick Craig-Wood)
+ * rcd: Added systemd notification during the `rclone rcd` command. (Naveen Honest Raj)
+ * rmdirs: Make `--rmdirs` obey the filters (Nick Craig-Wood)
+ * version: Show build tags and type of executable (Ivan Andreev)
+* Bug Fixes
+ * install.sh: make it fail on download errors (Ivan Andreev)
+ * Fix excessive retries missing `--max-duration` timeout (Nick Craig-Wood)
+ * Fix crash when `--low-level-retries=0` (Nick Craig-Wood)
+ * Fix failed token refresh on mounts created via the rc (Nick Craig-Wood)
+ * fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood)
+ * lib/atexit
+ * Unregister interrupt handler once it has fired so users can interrupt again (Nick Craig-Wood)
+ * Fix occasional failure to unmount with CTRL-C (Nick Craig-Wood)
+ * Fix deadlock calling Finalise while Run is running (Nick Craig-Wood)
+ * lib/rest: Fix multipart uploads not stopping on context cancel (Nick Craig-Wood)
+* Mount
+ * Allow mounting to root directory on windows (albertony)
+ * Improved handling of relative paths on windows (albertony)
+ * Fix unicode issues with accented characters on macOS (Nick Craig-Wood)
+ * Docs: document the new FileSecurity option in WinFsp 2021 (albertony)
+ * Docs: add note about volume path syntax on windows (albertony)
+ * Fix caching of old directories after renaming them (Nick Craig-Wood)
+ * Update cgofuse to the latest version to bring in macfuse 4 fix (Nick Craig-Wood)
+* VFS
+ * `--vfs-used-is-size` to report used space using recursive scan (tYYGH)
+ * Don't set modification time if it was already correct (Nick Craig-Wood)
+ * Fix Create causing windows explorer to truncate files on CTRL-C CTRL-V (Nick Craig-Wood)
+ * Fix modtimes not updating when writing via cache (Nick Craig-Wood)
+ * Fix modtimes changing by fractional seconds after upload (Nick Craig-Wood)
+ * Fix modtime set if `--vfs-cache-mode writes`/`full` and no write (Nick Craig-Wood)
+ * Rename files in cache and cancel uploads on directory rename (Nick Craig-Wood)
+ * Fix directory renaming by renaming dirs cached in memory (Nick Craig-Wood)
+* Local
+ * Add flag `--local-no-preallocate` (David Sze)
+ * Make `nounc` an advanced option except on Windows (albertony)
+ * Don't ignore preallocate disk full errors (Nick Craig-Wood)
+* Cache
+ * Add `--fs-cache-expire-duration` to control the fs cache (Nick Craig-Wood)
+* Crypt
+ * Add option to not encrypt data (Vesnyx)
+ * Log hash ok on upload (albertony)
+* Azure Blob
+ * Add container public access level support. (Manish Kumar)
+* B2
+ * Fix HTML files downloaded via cloudflare (Nick Craig-Wood)
+* Box
+ * Fix transfers getting stuck on token expiry after API change (Nick Craig-Wood)
+* Chunker
+ * Partially implement no-rename transactions (Maxwell Calman)
+* Drive
+ * Don't stop server side copy if couldn't read description (Nick Craig-Wood)
+ * Pass context on to drive SDK - to help with cancellation (Nick Craig-Wood)
+* Dropbox
+ * Add polling for changes support (Robert Thomas)
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+ * Raise priority of rate limited message to INFO to make it more noticeable (Nick Craig-Wood)
+* Fichier
+ * Implement copy & move (buengese)
+ * Implement public link (buengese)
+* FTP
+ * Implement Shutdown method (Nick Craig-Wood)
+ * Close idle connections after `--ftp-idle-timeout` (1m by default) (Nick Craig-Wood)
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+ * Add `--ftp-close-timeout` flag for use with awkward ftp servers (Nick Craig-Wood)
+ * Retry connections and logins on 421 errors (Nick Craig-Wood)
+* Hdfs
+ * Fix permissions for when directory is created (Lucas Messenger)
+* Onedrive
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+* S3
+ * Fix `--s3-profile` which wasn't working (Nick Craig-Wood)
+* SFTP
+ * Close idle connections after `--sftp-idle-timeout` (1m by default) (Nick Craig-Wood)
+ * Fix "file not found" errors for read once servers (Nick Craig-Wood)
+ * Fix SetModTime stat failed: object not found with `--sftp-set-modtime=false` (Nick Craig-Wood)
+* Swift
+ * Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood)
+ * Implement copying large objects (nguyenhuuluan434)
+* Union
+ * Fix crash when using epff policy (Nick Craig-Wood)
+ * Fix union attempting to update files on a read only file system (Nick Craig-Wood)
+ * Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick Craig-Wood)
+ * Fix initialisation broken in refactor (Nick Craig-Wood)
+* WebDAV
+ * Add support for sharepoint with NTLM authentication (Rauno Ots)
+ * Make sharepoint-ntlm docs more consistent (Alex Chen)
+ * Improve terminology in sharepoint-ntlm docs (Ivan Andreev)
+ * Disable HTTP/2 for NTLM authentication (georne)
+ * Fix sharepoint-ntlm error 401 for parallel actions (Ivan Andreev)
+ * Check that purged directory really exists (Ivan Andreev)
+* Yandex
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+* Zoho
+ * Replace client id - you will need to `rclone config reconnect` after this (buengese)
+ * Add forgotten setupRegion() to NewFs - this finally fixes regions other than EU (buengese)
+
+## v1.54.1 - 2021-03-08
+
+[See commits](https://github.com/rclone/rclone/compare/v1.54.0...v1.54.1)
+
+* Bug Fixes
+ * accounting: Fix --bwlimit when up or down is off (Nick Craig-Wood)
+ * docs
+ * Fix nesting of brackets and backticks in ftp docs (edwardxml)
+ * Fix broken link in sftp page (edwardxml)
+ * Fix typo in crypt.md (Romeo Kienzler)
+ * Changelog: Correct link to digitalis.io (Alex JOST)
+ * Replace #file-caching with #vfs-file-caching (Miron Veryanskiy)
+ * Convert bogus example link to code (edwardxml)
+ * Remove dead link from rc.md (edwardxml)
+ * rc: Sync,copy,move: document createEmptySrcDirs parameter (Nick Craig-Wood)
+ * lsjson: Fix unterminated JSON in the presence of errors (Nick Craig-Wood)
+* Mount
+ * Fix mount dropping on macOS by setting --daemon-timeout 10m (Nick Craig-Wood)
+* VFS
+ * Document simultaneous usage with the same cache shouldn't be used (Nick Craig-Wood)
+* B2
+ * Automatically raise upload cutoff to avoid spurious error (Nick Craig-Wood)
+ * Fix failed to create file system with application key limited to a prefix (Nick Craig-Wood)
+* Drive
+ * Refer to Shared Drives instead of Team Drives (Nick Craig-Wood)
+* Dropbox
+ * Add scopes to oauth request and optionally "members.read" (Nick Craig-Wood)
+* S3
+ * Fix failed to create file system with folder level permissions policy (Nick Craig-Wood)
+ * Fix Wasabi HEAD requests returning stale data by using only 1 transport (Nick Craig-Wood)
+ * Fix shared_credentials_file auth (Dmitry Chepurovskiy)
+ * Add --s3-no-head to reducing costs docs (Nick Craig-Wood)
+* Union
+ * Fix mkdir at root with remote:/ (Nick Craig-Wood)
+* Zoho
+ * Fix custom client id's (buengese)
+
## v1.54.0 - 2021-02-02
[See commits](https://github.com/rclone/rclone/compare/v1.53.0...v1.54.0)
* New backends
- * Compression remote (experimental)(buengese)
+ * Compression remote (experimental) (buengese)
* Enterprise File Fabric (Nick Craig-Wood)
* This work was sponsored by [Storage Made Easy](https://storagemadeeasy.com/)
* HDFS (Hadoop Distributed File System) (Yury Stankevich)
@@ -27765,41 +29041,35 @@ Options:
* New Features
* Deglobalise the config (Nick Craig-Wood)
* Global config now read from the context
- * Global config can be passed into the rc
- * This work was sponsored by [Digitalis](digitalis.io)
+ * This will enable passing of global config via the rc
+ * This work was sponsored by [Digitalis](https://digitalis.io/)
* Add `--bwlimit` for upload and download (Nick Craig-Wood)
* Obey bwlimit in http Transport for better limiting
* Enhance systemd integration (Hekmon)
- * log level identification
- * manual activation with flag
- * automatic systemd launch detection
+ * log level identification, manual activation with flag, automatic systemd launch detection
* Don't compile systemd log integration for non unix systems (Benjamin Gustin)
- * Add a download flag to hashsum and related commands to force rclone to download and hash files locally (lostheli)
+ * Add a `--download` flag to md5sum/sha1sum/hashsum to force rclone to download and hash files locally (lostheli)
+ * Add `--progress-terminal-title` to print ETA to terminal title (LaSombra)
+ * Make backend env vars show in help as the defaults for backend flags (Nick Craig-Wood)
* build
* Raise minimum go version to go1.12 (Nick Craig-Wood)
- * check
- * Make the error count match up in the log message (Nick Craig-Wood)
- * cmd
- * Add --progress-terminal-title to print ETA to terminal title (LaSombra)
- * Make backend env vars show in help as the defaults for backend flags (Nick Craig-Wood)
* dedupe
- * Add --by-hash to dedupe on hash not file name (Nick Craig-Wood)
- * Add --dedupe-mode list to just list dupes, changing nothing (Nick Craig-Wood)
+ * Add `--by-hash` to dedupe on content hash not file name (Nick Craig-Wood)
+ * Add `--dedupe-mode list` to just list dupes, changing nothing (Nick Craig-Wood)
* Add warning if used on a remote which can't have duplicate names (Nick Craig-Wood)
- * flags: Improve error message when reading environment vars (Nick Craig-Wood)
* fs
* Add Shutdown optional method for backends (Nick Craig-Wood)
- * When using --files-from check files concurrently (zhucan)
- * Accumulate stats when using --dry-run (Ingo Weiss)
- * Always show stats when using --dry-run or --interactive (Nick Craig-Wood)
- * Add support for flag --no-console on windows to hide the console window (albertony)
+ * When using `--files-from` check files concurrently (zhucan)
+ * Accumulate stats when using `--dry-run` (Ingo Weiss)
+ * Always show stats when using `--dry-run` or `--interactive` (Nick Craig-Wood)
+ * Add support for flag `--no-console` on windows to hide the console window (albertony)
* genautocomplete: Add support to output to stdout (Ingo)
* ncdu
* Highlight read errors instead of aborting (Claudio Bantaloukas)
* Add sort by average size in directory (Adam Plánský)
- * Add toggle option for average size in directory - key 'a' (Adam Plánský)
+ * Add toggle option for average s3ize in directory - key 'a' (Adam Plánský)
* Add empty folder flag into ncdu browser (Adam Plánský)
- * Add ! (errror) and . (unreadable) file flags to go with e (empty) (Nick Craig-Wood)
+ * Add `!` (errror) and `.` (unreadable) file flags to go with `e` (empty) (Nick Craig-Wood)
* obscure: Make `rclone osbcure -` ignore newline at end of line (Nick Craig-Wood)
* operations
* Add logs when need to upload files to set mod times (Nick Craig-Wood)
@@ -27817,26 +29087,22 @@ Options:
* Prompt user for updating webui if an update is available (Chaitanya Bankanhal)
* Fix plugins initialization (negative0)
* Bug Fixes
- * build
- * Explicitly set ARM version to fix build (Nick Craig-Wood)
- * Don't explicitly set ARM version to fix ARMv5 build (Nick Craig-Wood)
- * Fix nfpm install (Nick Craig-Wood)
- * Fix docker build by upgrading ilteoood/docker_buildx (Nick Craig-Wood)
- * Temporary fix for Windows build errors (Ivan Andreev)
* fs
* Fix nil pointer on copy & move operations directly to remote (Anagh Kumar Baranwal)
* Fix parsing of .. when joining remotes (Nick Craig-Wood)
- * log: Fix enabling systemd logging when using --log-file (Nick Craig-Wood)
- * move: Fix data loss when moving the same object (Nick Craig-Wood)
+ * log: Fix enabling systemd logging when using `--log-file` (Nick Craig-Wood)
+ * check
+ * Make the error count match up in the log message (Nick Craig-Wood)
+ * move: Fix data loss when source and destination are the same object (Nick Craig-Wood)
* operations
- * Fix --cutof-mode hard not cutting off immediately (Nick Craig-Wood)
- * Fix --immutable error message (Nick Craig-Wood)
+ * Fix `--cutof-mode` hard not cutting off immediately (Nick Craig-Wood)
+ * Fix `--immutable` error message (Nick Craig-Wood)
* sync
- * Fix --cutoff-mode soft & cautious so it doesn't end the transfer early (Nick Craig-Wood)
- * Fix --immutable errors retrying many times (Nick Craig-Wood)
+ * Fix `--cutoff-mode` soft & cautious so it doesn't end the transfer early (Nick Craig-Wood)
+ * Fix `--immutable` errors retrying many times (Nick Craig-Wood)
* Docs
* Many fixes and a rewrite of the filtering docs (edwardxml)
- * Many spelling and grammar problems (Josh Soref)
+ * Many spelling and grammar fixes (Josh Soref)
* Doc fixes for commands delete, purge, rmdir, rmdirs and mount (albertony)
* And thanks to these people for many doc fixes too numerous to list
* Ameer Dawood, Antoine GIRARD, Bob Bagwill, Christopher Stewart
@@ -27846,46 +29112,44 @@ Options:
* Mount
* Update systemd status with cache stats (Hekmon)
* Disable bazil/fuse based mount on macOS (Nick Craig-Wood)
- * Make mount be cmount under macOS (Nick Craig-Wood)
+ * Make `rclone mount` actually run `rclone cmount` under macOS (Nick Craig-Wood)
* Implement mknod to make NFS file creation work (Nick Craig-Wood)
* Make sure we don't call umount more than once (Nick Craig-Wood)
- * Don't call host.Umount if a signal has been received (Nick Craig-Wood)
* More user friendly mounting as network drive on windows (albertony)
- * Cleanup OS specific option handling and documentation (albertony)
* Detect if uid or gid are set in same option string: -o uid=123,gid=456 (albertony)
* Don't attempt to unmount if fs has been destroyed already (Nick Craig-Wood)
* VFS
* Fix virtual entries causing deleted files to still appear (Nick Craig-Wood)
* Fix "file already exists" error for stale cache files (Nick Craig-Wood)
- * Fix file leaks with --vfs-cache-mode full and --buffer-size 0 (Nick Craig-Wood)
+ * Fix file leaks with `--vfs-cache-mode` full and `--buffer-size 0` (Nick Craig-Wood)
* Fix invalid cache path on windows when using :backend: as remote (albertony)
* Local
* Continue listing files/folders when a circular symlink is detected (Manish Gupta)
- * New flag --local-zero-size-links to fix sync on some virtual filesystems (Riccardo Iaconelli)
+ * New flag `--local-zero-size-links` to fix sync on some virtual filesystems (Riccardo Iaconelli)
* Azure Blob
* Add support for service principals (James Lim)
- * Utilize streaming capabilities (Denis Neuling)
- * Update SDK to v0.13.0 and fix API breakage (Nick Craig-Wood, Mitsuo Heijo)
+ * Add support for managed identities (Brad Ackerman)
+ * Add examples for access tier (Bob Pusateri)
+ * Utilize the streaming capabilities from the SDK for multipart uploads (Denis Neuling)
* Fix setting of mime types (Nick Craig-Wood)
* Fix crash when listing outside a SAS URL's root (Nick Craig-Wood)
- * Delete archive tier blobs before update if --azureblob-archive-tier-delete (Nick Craig-Wood)
- * Add support for managed identities (Brad Ackerman)
+ * Delete archive tier blobs before update if `--azureblob-archive-tier-delete` (Nick Craig-Wood)
* Fix crash on startup (Nick Craig-Wood)
- * Add examples for access tier (Bob Pusateri)
- * Fix memory usage by upgrading the SDK and implementing a TransferManager (Nick Craig-Wood)
+ * Fix memory usage by upgrading the SDK to v0.13.0 and implementing a TransferManager (Nick Craig-Wood)
* Require go1.14+ to compile due to SDK changes (Nick Craig-Wood)
* B2
* Make NewObject use less expensive API calls (Nick Craig-Wood)
- * Fixed possible crash when accessing Backblaze b2 remote (lluuaapp)
+ * This will improve `--files-from` and `restic serve` in particular
+ * Fixed crash on an empty file name (lluuaapp)
* Box
* Fix NewObject for files that differ in case (Nick Craig-Wood)
* Fix finding directories in a case insentive way (Nick Craig-Wood)
* Chunker
* Skip long local hashing, hash in-transit (fixes) (Ivan Andreev)
- * Set Features.ReadMimeType=false as Object.MimeType not supported (Nick Craig-Wood)
+ * Set Features ReadMimeType to false as Object.MimeType not supported (Nick Craig-Wood)
* Fix case-insensitive NewObject, test metadata detection (Ivan Andreev)
* Drive
- * Implement "rclone backend copyid" command for copying files by ID (Nick Craig-Wood)
+ * Implement `rclone backend copyid` command for copying files by ID (Nick Craig-Wood)
* Added flag `--drive-stop-on-download-limit` to stop transfers when the download limit is exceeded (Anagh Kumar Baranwal)
* Implement CleanUp workaround for team drives (buengese)
* Allow shortcut resolution and creation to be retried (Nick Craig-Wood)
@@ -27893,44 +29157,43 @@ Options:
* Add xdg office icons to xdg desktop files (Pau Rodriguez-Estivill)
* Dropbox
* Add support for viewing shared files and folders (buengese)
- * Implement IDer (buengese)
- * Set Features.ReadMimeType=false as Object.MimeType not supported (Nick Craig-Wood)
- * Tidy repeated error message (Nick Craig-Wood)
+ * Enable short lived access tokens (Nick Craig-Wood)
+ * Implement IDer on Objects so `rclone lsf` etc can read the IDs (buengese)
+ * Set Features ReadMimeType to false as Object.MimeType not supported (Nick Craig-Wood)
* Make malformed_path errors from too long files not retriable (Nick Craig-Wood)
* Test file name length before upload to fix upload loop (Nick Craig-Wood)
- * Enable short lived access tokens (Nick Craig-Wood)
* Fichier
- * Set Features.ReadMimeType=true as Object.MimeType is supported (Nick Craig-Wood)
+ * Set Features ReadMimeType to true as Object.MimeType is supported (Nick Craig-Wood)
* FTP
- * Add --ftp-disable-msld option to ignore MLSD for really old servers (Nick Craig-Wood)
- * Make --tpslimit apply (Nick Craig-Wood)
+ * Add `--ftp-disable-msld` option to ignore MLSD for really old servers (Nick Craig-Wood)
+ * Make `--tpslimit apply` (Nick Craig-Wood)
* Google Cloud Storage
* Storage class object header support (Laurens Janssen)
* Fix anonymous client to use rclone's HTTP client (Nick Craig-Wood)
- * Fix Entry doesn't belong in directory "" (same as directory) - ignoring (Nick Craig-Wood)
+ * Fix `Entry doesn't belong in directory "" (same as directory) - ignoring` (Nick Craig-Wood)
* Googlephotos
- * New flag --gphotos-include-archived (Nicolas Rueff)
+ * New flag `--gphotos-include-archived` to show archived photos as well (Nicolas Rueff)
* Jottacloud
- * Don't erroniously report support for writing mime types (buengese)
- * Add support for Telia Cloud (#4930) (Patrik Nordlén)
+ * Don't erroneously report support for writing mime types (buengese)
+ * Add support for Telia Cloud (Patrik Nordlén)
* Mailru
+ * Accept special folders eg camera-upload (Ivan Andreev)
+ * Avoid prehashing of large local files (Ivan Andreev)
* Fix uploads after recent changes on server (Ivan Andreev)
* Fix range requests after June 2020 changes on server (Ivan Andreev)
* Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
* Remove deprecated protocol quirks (Ivan Andreev)
- * Accept special folders eg camera-upload (Ivan Andreev)
- * Avoid prehashing of large local files (Ivan Andreev)
* Memory
* Fix setting of mime types (Nick Craig-Wood)
* Onedrive
- * Add support for china region operated by 21vianet and other regional suppliers (#4963) (NyaMisty)
+ * Add support for China region operated by 21vianet and other regional suppliers (NyaMisty)
* Warn on gateway timeout errors (Nick Craig-Wood)
- * Fall back to normal copy if server-side copy unavailable (#4903) (Alex Chen)
+ * Fall back to normal copy if server-side copy unavailable (Alex Chen)
* Fix server-side copy completely disabled on OneDrive for Business (Cnly)
- * (business only) workaround to replace existing file on server-side copy (#4904) (Alex Chen)
+ * (business only) workaround to replace existing file on server-side copy (Alex Chen)
* Enhance link creation with expiry, scope, type and password (Nick Craig-Wood)
- * Remove % and # from the set of encoded characters (#4909) (Alex Chen)
- * Support addressing site by server-relative URL (#4761) (kice)
+ * Remove % and # from the set of encoded characters (Alex Chen)
+ * Support addressing site by server-relative URL (kice)
* Opendrive
* Fix finding directories in a case insensitive way (Nick Craig-Wood)
* Pcloud
@@ -27943,13 +29206,13 @@ Options:
* S3
* Added `--s3-disable-http2` to disable http/2 (Anagh Kumar Baranwal)
* Complete SSE-C implementation (Nick Craig-Wood)
- * Fix hashes on small files with AWS:KMS and SSE-C (Nick Craig-Wood)
- * Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C (Nick Craig-Wood)
+ * Fix hashes on small files with AWS:KMS and SSE-C (Nick Craig-Wood)
+ * Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C (Nick Craig-Wood)
+ * Add `--s3-no-head parameter` to minimise transactions on upload (Nick Craig-Wood)
* Update docs with a Reducing Costs section (Nick Craig-Wood)
* Added error handling for error code 429 indicating too many requests (Anagh Kumar Baranwal)
* Add requester pays option (kelv)
* Fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch' (Louis Koo)
- * Add --s3-no-head parameter to minimise transactions on upload (Nick Craig-Wood)
* SFTP
* Allow cert based auth via optional pubkey (Stephen Harris)
* Allow user to optionally check server hosts key to add security (Stephen Harris)
@@ -27957,20 +29220,20 @@ Options:
* Remember entered password in AskPass mode (Stephen Harris)
* Implement Shutdown method (Nick Craig-Wood)
* Implement keyboard interactive authentication (Nick Craig-Wood)
- * Make --tpslimit apply (Nick Craig-Wood)
- * Implement --sftp-use-fstat (Nick Craig-Wood)
+ * Make `--tpslimit` apply (Nick Craig-Wood)
+ * Implement `--sftp-use-fstat` for unusual SFTP servers (Nick Craig-Wood)
* Sugarsync
* Fix NewObject for files that differ in case (Nick Craig-Wood)
* Fix finding directories in a case insentive way (Nick Craig-Wood)
* Swift
* Fix deletion of parts of Static Large Object (SLO) (Nguyễn Hữu Luân)
- * Ensure partially uploaded large files are uploaded unless --swift-leave-parts-on-error (Nguyễn Hữu Luân)
+ * Ensure partially uploaded large files are uploaded unless `--swift-leave-parts-on-error` (Nguyễn Hữu Luân)
* Tardigrade
* Upgrade to uplink v1.4.1 (Caleb Case)
* WebDAV
* Updated docs to show streaming to nextcloud is working (Durval Menezes)
* Yandex
- * Set Features.WriteMimeType=false as Yandex ignores mime types (Nick Craig-Wood)
+ * Set Features WriteMimeType to false as Yandex ignores mime types (Nick Craig-Wood)
## v1.53.4 - 2021-01-20
@@ -31652,6 +32915,28 @@ put them back in again.` >}}
* Nicolas Rueff
* Pau Rodriguez-Estivill
* Bob Pusateri
+ * Alex JOST <25005220+dimejo@users.noreply.github.com>
+ * Alexey Tabakman
+ * David Sze
+ * cynthia kwok
+ * Ankur Gupta
+ * Miron Veryanskiy
+ * K265
+ * Vesnyx
+ * Dmitry Chepurovskiy
+ * Rauno Ots
+ * Georg Neugschwandtner
+ * pvalls
+ * Robert Thomas <31854736+wolveix@users.noreply.github.com>
+ * Romeo Kienzler
+ * tYYGH
+ * georne <77802995+georne@users.noreply.github.com>
+ * Maxwell Calman
+ * Naveen Honest Raj
+ * Lucas Messenger
+ * Manish Kumar
+ * x0b
+ * CERN through the CS3MESH4EOSC Project
# Contact the rclone project #
diff --git a/MANUAL.txt b/MANUAL.txt
index 1d3dbbbb3..849ac5763 100644
--- a/MANUAL.txt
+++ b/MANUAL.txt
@@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
-Feb 02, 2021
+Mar 31, 2021
@@ -178,6 +178,9 @@ See below for some expanded Linux / macOS instructions.
See the Usage section of the docs for how to use rclone, or run
rclone -h.
+Already installed rclone can be easily updated to the latest version
+using the rclone selfupdate command.
+
Script installation
@@ -348,8 +351,8 @@ Here are some commands tested on an Ubuntu 18.04.3 host:
Install from source
-Make sure you have at least Go 1.12 installed. Download go if necessary.
-The latest release is recommended. Then
+Make sure you have at least Go go1.13 installed. Download go if
+necessary. The latest release is recommended. Then
git clone https://github.com/rclone/rclone.git
cd rclone
@@ -606,7 +609,7 @@ Synopsis
Sync the source to the destination, changing the destination only.
Doesn't transfer unchanged files, testing by size and modification time
or MD5SUM. Destination is updated to match source, including deleting
-files if necessary.
+files if necessary (except duplicate objects, see below).
IMPORTANT: Since this can cause data loss, test first with the --dry-run
or the --interactive/-i flag.
@@ -614,7 +617,8 @@ or the --interactive/-i flag.
rclone sync -i SOURCE remote:DESTINATION
Note that files in the destination won't be deleted if there were any
-errors at any point.
+errors at any point. Duplicate objects (files with the same name, on
+those providers that support it) are also not yet handled.
It is always the contents of the directory that is synced, not the
directory so when source:path is a directory, it's the contents of
@@ -626,6 +630,10 @@ go there.
NOTE: Use the -P/--progress flag to view real-time transfer statistics
+NOTE: Use the rclone dedupe command to deal with "Duplicate
+object/directory found in source/destination - ignoring" errors. See
+this forum post for more info.
+
rclone sync source:path dest:path [flags]
@@ -1184,14 +1192,21 @@ Show the version number.
Synopsis
-Show the version number, the go version and the architecture.
+Show the rclone version number, the go version, the build target OS and
+architecture, build tags and the type of executable (static or dynamic).
-Eg
+For example:
$ rclone version
- rclone v1.41
- - os/arch: linux/amd64
- - go version: go1.10
+ rclone v1.54
+ - os/type: linux
+ - os/arch: amd64
+ - go/version: go1.16
+ - go/linking: static
+ - go/tags: none
+
+Note: before rclone version 1.55 the os/type and os/arch lines were
+merged, and the "go/version" line was tagged as "go version".
If you supply the --check flag, then it will do an online check to
compare your version with the latest release and the latest beta.
@@ -1266,9 +1281,10 @@ By default dedupe interactively finds files with duplicate names and
offers to delete all but one or rename them to be different. This is
known as deduping by name.
-Deduping by name is only useful with backends like Google Drive which
-can have duplicate file names. It can be run on wrapping backends (e.g.
-crypt) if they wrap a backend which supports duplicate file names.
+Deduping by name is only useful with a small group of backends (e.g.
+Google Drive, Opendrive) that can have duplicate file names. It can be
+run on wrapping backends (e.g. crypt) if they wrap a backend which
+supports duplicate file names.
However if --by-hash is passed in then dedupe will find files with
duplicate hashes instead which will work on any backend which supports
@@ -2020,24 +2036,26 @@ Synopsis
Download a URL's content and copy it to the destination without saving
it in temporary storage.
-Setting --auto-filename will cause the file name to be retrieved from
-the from URL (after any redirections) and used in the destination path.
+Setting --auto-filenamewill cause the file name to be retrieved from the
+from URL (after any redirections) and used in the destination path. With
+--print-filename in addition, the resuling file name will be printed.
Setting --no-clobber will prevent overwriting file on the destination if
there is one with the same name.
-Setting --stdout or making the output file name "-" will cause the
-output to be written to standard output.
+Setting --stdout or making the output file name - will cause the output
+to be written to standard output.
rclone copyurl https://example.com dest:path [flags]
Options
- -a, --auto-filename Get the file name from the URL and use it for destination file path
- -h, --help help for copyurl
- --no-clobber Prevent overwriting file with same name
- --stdout Write the output to stdout rather than a file
+ -a, --auto-filename Get the file name from the URL and use it for destination file path
+ -h, --help help for copyurl
+ --no-clobber Prevent overwriting file with same name
+ -p, --print-filename Print the resulting name from --auto-filename
+ --stdout Write the output to stdout rather than a file
See the global flags page for global options not listed here.
@@ -2785,16 +2803,15 @@ On Linux/macOS/FreeBSD start the mount like this, where
On Windows you can start a mount in different ways. See below for
details. The following examples will mount to an automatically assigned
-drive, to specific drive letter X:, to path
-C:\path\to\nonexistent\directory (which must be NON-EXISTENT
-subdirectory of an EXISTING parent directory or drive, and is not
-supported when mounting as a network drive), and the last example will
-mount as network share \\cloud\remote and map it to an automatically
-assigned drive:
+drive, to specific drive letter X:, to path C:\path\parent\mount (where
+parent directory or drive must exist, and mount must NOT exist, and is
+not supported when mounting as a network drive), and the last example
+will mount as network share \\cloud\remote and map it to an
+automatically assigned drive:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
- rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+ rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files \\cloud\remote
When the program ends while in foreground mode, either via Ctrl+C or
@@ -2853,14 +2870,14 @@ unexpected program errors, freezes or other issues, consider mounting as
a network drive instead.
When mounting as a fixed disk drive you can either mount to an unused
-drive letter, or to a path - which must be NON-EXISTENT subdirectory of
+drive letter, or to a path representing a NON-EXISTENT subdirectory of
an EXISTING parent directory or drive. Using the special value * will
tell rclone to automatically assign the next available drive letter,
starting with Z: and moving backward. Examples:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
- rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+ rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files X:
Option --volname can be used to set a custom volume name for the mounted
@@ -2937,10 +2954,26 @@ Note that the mapping of permissions is not always trivial, and the
result you see in Windows Explorer may not be exactly like you expected.
For example, when setting a value that includes write access, this will
be mapped to individual permissions "write attributes", "write data" and
-"append data", but not "write extended attributes" (WinFsp does not
-support extended attributes, see this). Windows will then show this as
-basic permission "Special" instead of "Write", because "Write" includes
-the "write extended attributes" permission.
+"append data", but not "write extended attributes". Windows will then
+show this as basic permission "Special" instead of "Write", because
+"Write" includes the "write extended attributes" permission.
+
+If you set POSIX permissions for only allowing access to the owner,
+using --file-perms 0600 --dir-perms 0700, the user group and the
+built-in "Everyone" group will still be given some special permissions,
+such as "read attributes" and "read permissions", in Windows. This is
+done for compatibility reasons, e.g. to allow users without additional
+permissions to be able to read basic metadata about files like in UNIX.
+One case that may arise is that other programs (incorrectly) interprets
+this as the file being accessible by everyone. For example an SSH client
+may warn about "unprotected private key file".
+
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option
+"FileSecurity", that allows the complete specification of file security
+descriptors using SDDL. With this you can work around issues such as the
+mentioned "unprotected private key file" by specifying
+-o FileSecurity="D:P(A;;FA;;;OW)", for file all access (FA) to the owner
+(OW).
Windows caveats
@@ -2962,8 +2995,8 @@ Limitations
Without the use of --vfs-cache-mode this can only write files
sequentially, it can only seek when reading. This means that many
applications won't work with their files on an rclone mount without
---vfs-cache-mode writes or --vfs-cache-mode full. See the File Caching
-section for more info.
+--vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File
+Caching section for more info.
The bucket based remotes (e.g. Swift, S3, Google Compute Storage, B2,
Hubic) do not support the concept of empty directories, so empty
@@ -2979,7 +3012,7 @@ File systems expect things to be 100% reliable, whereas cloud storage
systems are a long way from 100% reliable. The rclone sync/copy commands
cope with this with lots of retries. However rclone mount can't use
retries in the same way without making local copies of the uploads. Look
-at the file caching for solutions to make mount more reliable.
+at the VFS File Caching for solutions to make mount more reliable.
Attribute caching
@@ -3150,6 +3183,12 @@ for two reasons. Firstly because it is only checked every
--vfs-cache-poll-interval. Secondly because open files cannot be evicted
from the cache.
+You SHOULD NOT run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
+
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote
@@ -3298,6 +3337,21 @@ depends on the operating system where rclone runs: "true" on Windows and
macOS, "false" otherwise. If the flag is provided without a value, then
it is "true".
+
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+_WARNING._ Contrary to rclone size, this flag ignores filters so that
+the result is accurate. However, this is very inefficient and may cost
+lots of API calls resulting in extra charges. Use it as a last resort
+and only with caching.
+
rclone mount remote:path /path/to/mountpoint [flags]
@@ -3339,6 +3393,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
--volname string Set the volume name. Supported on Windows and OSX only.
@@ -3707,6 +3762,84 @@ SEE ALSO
+RCLONE SELFUPDATE
+
+
+Update the rclone binary.
+
+
+Synopsis
+
+This command downloads the latest release of rclone and replaces the
+currently running binary. The download is verified with a hashsum and
+cryptographically signed signature.
+
+If used without flags (or with implied --stable flag), this command will
+install the latest stable release. However, some issues may be fixed (or
+features added) only in the latest beta release. In such cases you
+should run the command with the --beta flag, i.e.
+rclone selfupdate --beta. You can check in advance what version would be
+installed by adding the --check flag, then repeat the command without it
+when you are satisfied.
+
+Sometimes the rclone team may recommend you a concrete beta or stable
+rclone release to troubleshoot your issue or add a bleeding edge
+feature. The --version VER flag, if given, will update to the concrete
+version instead of the latest one. If you omit micro version from VER
+(for example 1.53), the latest matching micro version will be used.
+
+Upon successful update rclone will print a message that contains a
+previous version number. You will need it if you later decide to revert
+your update for some reason. Then you'll have to note the previous
+version and run the following command:
+rclone selfupdate [--beta] OLDVER. If the old version contains only dots
+and digits (for example v1.54.0) then it's a stable release so you won't
+need the --beta flag. Beta releases have an additional information
+similar to v1.54.0-beta.5111.06f1c0c61. (if you are a developer and use
+a locally built rclone, the version number will end with -DEV, you will
+have to rebuild it as it obvisously can't be distributed).
+
+If you previously installed rclone via a package manager, the package
+may include local documentation or configure services. You may wish to
+update with the flag --package deb or --package rpm (whichever is
+correct for your OS) to update these too. This command with the default
+--package zip will update only the rclone executable so the local manual
+may become inaccurate after it.
+
+The rclone mount command (https://rclone.org/commands/rclone_mount/) may
+or may not support extended FUSE options depending on the build and OS.
+selfupdate will refuse to update if the capability would be discarded.
+
+Note: Windows forbids deletion of a currently running executable so this
+command will rename the old executable to 'rclone.old.exe' upon success.
+
+Please note that this command was not available before rclone version
+1.55. If it fails for you with the message unknown command "selfupdate"
+then you will need to update manually following the install instructions
+located at https://rclone.org/install/
+
+ rclone selfupdate [flags]
+
+
+Options
+
+ --beta Install beta release.
+ --check Check for latest release, do not download.
+ -h, --help help for selfupdate
+ --output string Save the downloaded binary at a given path (default: replace running binary)
+ --package string Package format: zip|deb|rpm (default: zip)
+ --stable Install stable release (this is the default)
+ --version string Install the given rclone version (default: latest)
+
+See the global flags page for global options not listed here.
+
+
+SEE ALSO
+
+- rclone - Show help for rclone commands, flags and backends.
+
+
+
RCLONE SERVE
@@ -3878,6 +4011,12 @@ for two reasons. Firstly because it is only checked every
--vfs-cache-poll-interval. Secondly because open files cannot be evicted
from the cache.
+You SHOULD NOT run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
+
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote
@@ -4026,6 +4165,21 @@ depends on the operating system where rclone runs: "true" on Windows and
macOS, "false" otherwise. If the flag is provided without a value, then
it is "true".
+
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+_WARNING._ Contrary to rclone size, this flag ignores filters so that
+the result is accurate. However, this is very inefficient and may cost
+lots of API calls resulting in extra charges. Use it as a last resort
+and only with caching.
+
rclone serve dlna remote:path [flags]
@@ -4055,6 +4209,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
@@ -4200,6 +4355,12 @@ for two reasons. Firstly because it is only checked every
--vfs-cache-poll-interval. Secondly because open files cannot be evicted
from the cache.
+You SHOULD NOT run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
+
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote
@@ -4349,6 +4510,21 @@ macOS, "false" otherwise. If the flag is provided without a value, then
it is "true".
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+_WARNING._ Contrary to rclone size, this flag ignores filters so that
+the result is accurate. However, this is very inefficient and may cost
+lots of API calls resulting in extra charges. Use it as a last resort
+and only with caching.
+
+
Auth Proxy
If you supply the parameter --auth-proxy /path/to/program then rclone
@@ -4454,6 +4630,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
@@ -4697,6 +4874,12 @@ for two reasons. Firstly because it is only checked every
--vfs-cache-poll-interval. Secondly because open files cannot be evicted
from the cache.
+You SHOULD NOT run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
+
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote
@@ -4845,6 +5028,21 @@ depends on the operating system where rclone runs: "true" on Windows and
macOS, "false" otherwise. If the flag is provided without a value, then
it is "true".
+
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+_WARNING._ Contrary to rclone size, this flag ignores filters so that
+the result is accurate. However, this is very inefficient and may cost
+lots of API calls resulting in extra charges. Use it as a last resort
+and only with caching.
+
rclone serve http remote:path [flags]
@@ -4884,6 +5082,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
@@ -5268,6 +5467,12 @@ for two reasons. Firstly because it is only checked every
--vfs-cache-poll-interval. Secondly because open files cannot be evicted
from the cache.
+You SHOULD NOT run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
+
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote
@@ -5417,6 +5622,21 @@ macOS, "false" otherwise. If the flag is provided without a value, then
it is "true".
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+_WARNING._ Contrary to rclone size, this flag ignores filters so that
+the result is accurate. However, this is very inefficient and may cost
+lots of API calls resulting in extra charges. Use it as a last resort
+and only with caching.
+
+
Auth Proxy
If you supply the parameter --auth-proxy /path/to/program then rclone
@@ -5521,6 +5741,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
@@ -5771,6 +5992,12 @@ for two reasons. Firstly because it is only checked every
--vfs-cache-poll-interval. Secondly because open files cannot be evicted
from the cache.
+You SHOULD NOT run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
+
--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote
@@ -5920,6 +6147,21 @@ macOS, "false" otherwise. If the flag is provided without a value, then
it is "true".
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+_WARNING._ Contrary to rclone size, this flag ignores filters so that
+the result is accurate. However, this is very inefficient and may cost
+lots of API calls resulting in extra charges. Use it as a last resort
+and only with caching.
+
+
Auth Proxy
If you supply the parameter --auth-proxy /path/to/program then rclone
@@ -6033,6 +6275,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
@@ -6092,6 +6335,160 @@ SEE ALSO
+RCLONE TEST
+
+
+Run a test command
+
+
+Synopsis
+
+Rclone test is used to run test commands.
+
+Select which test comand you want with the subcommand, eg
+
+ rclone test memory remote:
+
+Each subcommand has its own options which you can see in their help.
+
+NB Be careful running these commands, they may do strange things so
+reading their documentation first is recommended.
+
+
+Options
+
+ -h, --help help for test
+
+See the global flags page for global options not listed here.
+
+
+SEE ALSO
+
+- rclone - Show help for rclone commands, flags and backends.
+- rclone test histogram - Makes a histogram of file name characters.
+- rclone test info - Discovers file name or other limitations for
+ paths.
+- rclone test makefiles - Make a random file hierarchy in
+- rclone test memory - Load all the objects at remote:path into memory
+ and report memory stats.
+
+
+
+RCLONE TEST HISTOGRAM
+
+
+Makes a histogram of file name characters.
+
+
+Synopsis
+
+This command outputs JSON which shows the histogram of characters used
+in filenames in the remote:path specified.
+
+The data doesn't contain any identifying information but is useful for
+the rclone developers when developing filename compression.
+
+ rclone test histogram [remote:path] [flags]
+
+
+Options
+
+ -h, --help help for histogram
+
+See the global flags page for global options not listed here.
+
+
+SEE ALSO
+
+- rclone test - Run a test command
+
+
+
+RCLONE TEST INFO
+
+
+Discovers file name or other limitations for paths.
+
+
+Synopsis
+
+rclone info discovers what filenames and upload methods are possible to
+write to the paths passed in and how long they can be. It can take some
+time. It will write test files into the remote:path passed in. It
+outputs a bit of go code for each one.
+
+NB this can create undeletable files and other hazards - use with care
+
+ rclone test info [remote:path]+ [flags]
+
+
+Options
+
+ --all Run all tests.
+ --check-control Check control characters.
+ --check-length Check max filename length.
+ --check-normalization Check UTF-8 Normalization.
+ --check-streaming Check uploads with indeterminate file size.
+ -h, --help help for info
+ --upload-wait duration Wait after writing a file.
+ --write-json string Write results to file.
+
+See the global flags page for global options not listed here.
+
+
+SEE ALSO
+
+- rclone test - Run a test command
+
+
+
+RCLONE TEST MAKEFILES
+
+
+Make a random file hierarchy in
+ rclone test makefiles [flags]
+
+
+Options
+
+ --files int Number of files to create (default 1000)
+ --files-per-directory int Average number of files per directory (default 10)
+ -h, --help help for makefiles
+ --max-file-size SizeSuffix Maximum size of files to create (default 100)
+ --max-name-length int Maximum size of file names (default 12)
+ --min-file-size SizeSuffix Minimum size of file to create
+ --min-name-length int Minimum size of file names (default 4)
+
+See the global flags page for global options not listed here.
+
+
+SEE ALSO
+
+- rclone test - Run a test command
+
+
+
+RCLONE TEST MEMORY
+
+
+Load all the objects at remote:path into memory and report memory stats.
+
+ rclone test memory remote:path [flags]
+
+
+Options
+
+ -h, --help help for memory
+
+See the global flags page for global options not listed here.
+
+
+SEE ALSO
+
+- rclone test - Run a test command
+
+
+
RCLONE TOUCH
@@ -6238,8 +6635,9 @@ The syntax of the paths passed to the rclone command are as follows.
This refers to the local file system.
-On Windows only \ may be used instead of / in local paths ONLY, non
-local paths must use /.
+On Windows \ may be used instead of / in local paths ONLY, non local
+paths must use /. See local filesystem documentation for more about
+Windows-specific paths.
These paths needn't start with a leading / - if they don't then they
will be relative to the current directory.
@@ -6284,6 +6682,107 @@ To copy files and directories in https://example.com/path/to/dir to
To copy files and directories from example.com in the relative directory
path/to/dir to /tmp/dir using sftp.
+Connection strings
+
+The above examples can also be written using a connection string syntax,
+so instead of providing the arguments as command line parameters
+--http-url https://pub.rclone.org they are provided as part of the
+remote specification as a kind of connection string.
+
+ rclone lsd ":http,url='https://pub.rclone.org':"
+ rclone lsf ":http,url='https://example.com':path/to/dir"
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+ rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
+
+These can apply to modify existing remotes as well as create new remotes
+with the on the fly syntax. This example is equivalent to adding the
+--drive-shared-with-me parameter to the remote gdrive:.
+
+ rclone lsf "gdrive,shared_with_me:path/to/dir"
+
+The major advantage to using the connection string style syntax is that
+it only applies the the remote, not to all the remotes of that type of
+the command line. A common confusion is this attempt to copy a file
+shared on google drive to the normal drive which DOES NOT WORK because
+the --drive-shared-with-me flag applies to both the source and the
+destination.
+
+ rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
+
+However using the connection string syntax, this does work.
+
+ rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
+
+The connection strings have the following syntax
+
+ remote,parameter=value,parameter2=value2:path/to/dir
+ :backend,parameter=value,parameter2=value2:path/to/dir
+
+If the parameter has a : or , then it must be placed in quotes " or ',
+so
+
+ remote,parameter="colon:value",parameter2="comma,value":path/to/dir
+ :backend,parameter='colon:value',parameter2='comma,value':path/to/dir
+
+If a quoted value needs to include that quote, then it should be
+doubled, so
+
+ remote,parameter="with""quote",parameter2='with''quote':path/to/dir
+
+This will make parameter be with"quote and parameter2 be with'quote.
+
+If you leave off the =parameter then rclone will substitute =true which
+works very well with flags. For example to use s3 configured in the
+environment you could use:
+
+ rclone lsd :s3,env_auth:
+
+Which is equivalent to
+
+ rclone lsd :s3,env_auth=true:
+
+Note that on the command line you might need to surround these
+connection strings with " or ' to stop the shell interpreting any
+special characters within them.
+
+If you are a shell master then you'll know which strings are OK and
+which aren't, but if you aren't sure then enclose them in " and use ' as
+the inside quote. This syntax works on all OSes.
+
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+
+On Linux/macOS some characters are still interpreted inside " strings in
+the shell (notably \ and $ and ") so if your strings contain those you
+can swap the roles of " and ' thus. (This syntax does not work on
+Windows.)
+
+ rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
+
+Connection strings, config and logging
+
+If you supply extra configuration to a backend by command line flag,
+environment variable or connection string then rclone will add a suffix
+based on the hash of the config to the name of the remote, eg
+
+ rclone -vv lsf --s3-chunk-size 20M s3:
+
+Has the log message
+
+ DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
+
+This is so rclone can tell the modified remote apart from the unmodified
+remote when caching the backends.
+
+This should only be noticeable in the logs.
+
+This means that on the fly backends such as
+
+ rclone -vv lsf :s3,env_auth:
+
+Will get their own names
+
+ DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
+
Valid remote names
- Remote names may only contain 0-9, A-Z ,a-z ,_ , - and space.
@@ -6437,7 +6936,7 @@ This option controls the bandwidth limit. For example
--bwlimit 10M
-would mean limit the upload and download bandwidth to 10 Mbyte/s. NB
+would mean limit the upload and download bandwidth to 10 MByte/s. NB
this is BYTES per second not BITS per second. To use a single limit,
specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The
default is 0 which means to not limit bandwidth.
@@ -6447,7 +6946,7 @@ The upload and download bandwidth can be specified seperately, as
--bwlimit 10M:100k
-would mean limit the upload bandwidth to 10 Mbyte/s and the download
+would mean limit the upload bandwidth to 10 MByte/s and the download
bandwidth to 100 kByte/s. Either limit can be "off" meaning no limit, so
to just limit the upload bandwidth you would use
@@ -6477,9 +6976,9 @@ daytime working hours could be:
--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"
In this example, the transfer bandwidth will be set to 512kBytes/sec at
-8am every day. At noon, it will rise to 10Mbytes/s, and drop back to
+8am every day. At noon, it will rise to 10MByte/s, and drop back to
512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to
-30MBytes/s, and at 11pm it will be completely disabled (full speed).
+30MByte/s, and at 11pm it will be completely disabled (full speed).
Anything between 11pm and 8am will remain unlimited.
An example of timetable with WEEKDAY could be:
@@ -6487,8 +6986,8 @@ An example of timetable with WEEKDAY could be:
--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"
It means that, the transfer bandwidth will be set to 512kBytes/sec on
-Monday. It will rise to 10Mbytes/s before the end of Friday. At 10:00 on
-Saturday it will be set to 1Mbyte/s. From 20:00 on Sunday it will be
+Monday. It will rise to 10MByte/s before the end of Friday. At 10:00 on
+Saturday it will be set to 1MByte/s. From 20:00 on Sunday it will be
unlimited.
Timeslots without WEEKDAY are extended to the whole week. So this
@@ -6609,7 +7108,7 @@ See --copy-dest and --backup-dir.
--config=CONFIG_FILE
-Specify the location of the rclone config file.
+Specify the location of the rclone configuration file.
Normally the config file is in your home directory as a file called
.config/rclone/rclone.conf (or .rclone.conf if created with an older
@@ -6626,6 +7125,44 @@ for you.
Use this flag to override the config location, e.g.
rclone --config=".myconfig" .config.
+If the location is set to empty string "" or the special value
+/notfound, or the os null device represented by value NUL on Windows and
+/dev/null on Unix systems, then rclone will keep the config file in
+memory only.
+
+The file format is basic INI: Sections of text, led by a [section]
+header and followed by key=value entries on separate lines. In rclone
+each remote is represented by its own section, where the section name
+defines the name of the remote. Options are specified as the key=value
+entries, where the key is the option name without the --backend- prefix,
+in lowercase and with _ instead of -. E.g. option --mega-hard-delete
+corresponds to key hard_delete. Only backend options can be specified. A
+special, and required, key type identifies the storage system, where the
+value is the internal lowercase name as returned by command
+rclone help backends. Comments are indicated by ; or # at the beginning
+of a line.
+
+Example:
+
+ [megaremote]
+ type = mega
+ user = you@example.com
+ pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
+
+Note that passwords are in obscured form. Also, many storage systems
+uses token-based authentication instead of passwords, and this requires
+additional steps. It is easier, and safer, to use the interactive
+command rclone config instead of manually editing the configuration
+file.
+
+The configuration file will typically contain login information, and
+should therefore have restricted permissions so that only the current
+user can read it. Rclone tries to ensure this when it writes the file.
+You may also choose to encrypt the file.
+
+When token-based authentication are used, the configuration file must be
+writable, because rclone needs to update the tokens inside it.
+
--contimeout=TIME
Set the connection timeout. This should be in go time format which looks
@@ -6674,6 +7211,24 @@ This flag can be useful for debugging and in exceptional circumstances
(e.g. Google Drive limiting the total volume of Server Side Copies to
100GB/day).
+--dscp VALUE
+
+Specify a DSCP value or name to use in connections. This could help QoS
+system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are
+allowed.
+
+See the description of differentiated services to get an idea of this
+field. Setting this to 1 (LE) to identify the flow to SCAVENGER class
+can avoid occupying too much bandwidth in a network with DiffServ
+support (RFC 8622).
+
+For example, if you configured QoS on router to handle LE properly.
+Running:
+
+ rclone copy --dscp LE from:/from to:/to
+
+would make the priority lower than usual internet flows.
+
-n, --dry-run
Do a trial run with no permanent changes. Use this to see what rclone
@@ -6705,6 +7260,26 @@ NB: Enabling this option turns a usually non-fatal error into a
potentially fatal one - please check and adjust your scripts
accordingly!
+--fs-cache-expire-duration=TIME
+
+When using rclone via the API rclone caches created remotes for 5
+minutes by default in the "fs cache". This means that if you do repeated
+actions on the same remote then rclone won't have to build it again from
+scratch, which makes it more efficient.
+
+This flag sets the time that the remotes are cached for. If you set it
+to 0 (or negative) then rclone won't cache the remotes at all.
+
+Note that if you use some flags, eg --backup-dir and if this is set to 0
+rclone may build two remotes (one for the source or destination and one
+for the --backup-dir where it may have only built one before.
+
+--fs-cache-expire-interval=TIME
+
+This controls how often rclone checks for cached remotes to expire. See
+the --fs-cache-expire-duration documentation above for more info. The
+default is 60s, set to 0 to disable expiry.
+
--header
Add an HTTP header for all transactions. The flag can be repeated to add
@@ -7661,8 +8236,8 @@ THIS SHOULD BE USED ONLY FOR TESTING.
Configuration Encryption
Your configuration file contains information for logging in to your
-cloud services. This means that you should keep your .rclone.conf file
-in a secure location.
+cloud services. This means that you should keep your rclone.conf file in
+a secure location.
If you are in an environment where that isn't possible, you can add a
password to your configuration. This means that you will have to supply
@@ -7759,6 +8334,17 @@ make rclone fail instead of asking for a password if RCLONE_CONFIG_PASS
doesn't contain a valid password, and --password-command has not been
supplied.
+Some rclone commands, such as genautocomplete, do not require
+configuration. Nevertheless, rclone will read any configuration file
+found according to the rules described above. If an encrypted
+configuration file is found, this means you will be prompted for
+password (unless using --password-command). To avoid this, you can
+bypass the loading of the configuration file by overriding the location
+with an empty string "" or the special value /notfound, or the os null
+device represented by value NUL on Windows and /dev/null on Unix systems
+(before rclone version 1.55 only this null device alternative was
+supported). E.g. rclone --config="" genautocomplete bash.
+
Developer options
@@ -7959,11 +8545,8 @@ they take exactly the same form.
Config file
You can set defaults for values in the config file on an individual
-remote basis. If you want to use this feature, you will need to discover
-the name of the config items that you want. The easiest way is to run
-through rclone config by hand, then look in the config file to see what
-the values are (the config file can be found by looking at the help for
---config in rclone help).
+remote basis. The names of the config items are documented in the page
+for each backend.
To find the name of the environment variable, you need to set, take
RCLONE_CONFIG_ + name of remote + _ + name of config file option and
@@ -7983,6 +8566,11 @@ For example, to configure an S3 remote named mys3: without a config file
Note that if you want to create a remote using environment variables you
must create the ..._TYPE variable as above.
+Note also that now rclone has connectionstrings, it is probably easier
+to use those instead which makes the above example
+
+ rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
+
Precedence
The various different methods of backend configuration are read in this
@@ -8182,6 +8770,25 @@ beginning of the path/file.
- doesn't match "afile.jpg"
- doesn't match "directory/file.jpg"
+The top level of the remote may not be the top level of the drive.
+
+E.g. for a Microsoft Windows local directory structure
+
+ F:
+ ├── bkp
+ ├── data
+ │ ├── excl
+ │ │ ├── 123.jpg
+ │ │ └── 456.jpg
+ │ ├── incl
+ │ │ └── document.pdf
+
+To copy the contents of folder data into folder bkp excluding the
+contents of subfolder exclthe following command treats F:\data and
+F:\bkp as top level for filtering.
+
+rclone copy F:\data\ F:\bkp\ --exclude=/excl/**
+
IMPORTANT Use / in path/file name patterns and not \ even if running on
Microsoft Windows.
@@ -8201,7 +8808,7 @@ With --ignore-case
How filter rules are applied to files
-Rclone path / file name filters are made up of one or more of the
+Rclone path/file name filters are made up of one or more of the
following flags:
- --include
@@ -8228,7 +8835,7 @@ To mix up the order of processing includes and excludes use --filter...
flags.
Within --include-from, --exclude-from and --filter-from flags rules are
-processed from top to bottom of the referenced file..
+processed from top to bottom of the referenced file.
If there is an --include or --include-from flag specified, rclone
implies a - ** rule which it adds to the bottom of the internal rule
@@ -8259,9 +8866,9 @@ character classes. Go regular expression reference
How filter rules are applied to directories
-Rclone commands filter, and are applied to, path/file names not
-directories. The entire contents of a directory can be matched to a
-filter by the pattern directory/* or recursively by directory/**.
+Rclone commands are applied to path/file names not directories. The
+entire contents of a directory can be matched to a filter by the pattern
+directory/* or recursively by directory/**.
Directory filter rules are defined with a closing / separator.
@@ -8273,15 +8880,16 @@ remote by avoiding listing unnecessary directories. Whether optimisation
is desirable depends on the specific filter rules and source remote
content.
-Optimisation occurs if either:
+Directory recursion optimisation occurs if either:
- A source remote does not support the rclone ListR primitive. local,
sftp, Microsoft OneDrive and WebDav do not support ListR. Google
Drive and most bucket type storage do. Full list
-- On other remotes, if the rclone command is not naturally recursive,
- provided it is not run with the --fast-list flag. ls, lsf -R and
- size are recursive but sync, copy and move are not.
+- On other remotes (those that support ListR), if the rclone command
+ is not naturally recursive, and provided it is not run with the
+ --fast-list flag. ls, lsf -R and size are naturally recursive but
+ sync, copy and move are not.
- Whenever the --disable ListR flag is applied to an rclone command.
@@ -8303,7 +8911,7 @@ specified.
E.g. rclone ls remote: --include /directory/ will not match any files.
Because it is an --include option the --exclude ** rule is implied, and
-the \directory\ pattern serves only to optimise access to the remote by
+the /directory/ pattern serves only to optimise access to the remote by
ignoring everything outside of that directory.
E.g. rclone ls remote: --filter-from filter-list.txt with a file
@@ -8315,7 +8923,7 @@ filter-list.txt:
- **
All files in directories dir1 or dir2 or their subdirectories are
-completely excluded from the listing. Only files of suffix 'pdf in the
+completely excluded from the listing. Only files of suffix pdf in the
root of remote: or its subdirectories are listed. The - ** rule prevents
listing of any path/files not previously matched by the rules above.
@@ -8323,6 +8931,28 @@ Option exclude-if-present creates a directory exclude rule based on the
presence of a file in a directory and takes precedence over other rclone
directory filter rules.
+When using pattern list syntax, if a pattern item contains either / or
+**, then rclone will not able to imply a directory filter rule from this
+pattern list.
+
+E.g. for an include rule
+
+ {dir1/**,dir2/**}
+
+Rclone will match files below directories dir1 or dir2 only, but will
+not be able to use this filter to exclude a directory dir3 from being
+traversed.
+
+Directory recursion optimisation may affect performance, but normally
+not the result. One exception to this is sync operations with option
+--create-empty-src-dirs, where any traversed empty directories will be
+created. With the pattern list example {dir1/**,dir2/**} above, this
+would create an empty directory dir3 on destination (when it exists on
+source). Changing the filter to {dir1,dir2}/**, or splitting it into two
+include rules --include dir1/** --include dir2/**, will match the same
+files while also filtering directories, with the result that an empty
+directory dir3 will no longer be created.
+
--exclude - Exclude files matching pattern
Excludes path/file names from an rclone command based on a single
@@ -8345,11 +8975,11 @@ all files on remote: excluding those in root directory dir and sub
directories.
E.g. on Microsoft Windows rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"
-lists the files in remote: with [JP] or [KR] or [HK] in their name. The
-single quotes prevent the shell from interpreting the \ characters. The
-\ characters escape the [ and ] so ran clone filter treats them
-literally rather than as a character-range. The { and } define an rclone
-pattern list. For other operating systems single quotes are required ie
+lists the files in remote: with [JP] or [KR] or [HK] in their name.
+Quotes prevent the shell from interpreting the \ characters.\ characters
+escape the [ and ] so an rclone filter treats them literally rather than
+as a character-range. The { and } define an rclone pattern list. For
+other operating systems single quotes are required ie
rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'
--exclude-from - Read exclude patterns from file
@@ -8481,7 +9111,7 @@ processed in.
Arrange the order of filter rules with the most restrictive first and
work down.
-E.g. For filter-file.txt:
+E.g. for filter-file.txt:
# a sample filter rule file
- secret*.jpg
@@ -8589,13 +9219,13 @@ The three files are transferred as follows:
/home/user1/dir/ford → remote:backup/user1/dir/file
/home/user2/prefect → remote:backup/user2/stuff
-Alternatively if / is chosen as root files-from.txt would be:
+Alternatively if / is chosen as root files-from.txt will be:
/home/user1/42
/home/user1/dir/ford
/home/user2/prefect
-The copy command would be:
+The copy command will be:
rclone copy --files-from files-from.txt / remote:backup
@@ -8678,10 +9308,10 @@ Default units are seconds or the following abbreviations are valid:
--max-age can also be specified as an absolute time in the following
formats:
-- RFC3339 - e.g. "2006-01-02T15:04:05Z07:00"
-- ISO8601 Date and time, local timezone - "2006-01-02T15:04:05"
-- ISO8601 Date and time, local timezone - "2006-01-02 15:04:05"
-- ISO8601 Date - "2006-01-02" (YYYY-MM-DD)
+- RFC3339 - e.g. 2006-01-02T15:04:05Z or 2006-01-02T15:04:05+07:00
+- ISO8601 Date and time, local timezone - 2006-01-02T15:04:05
+- ISO8601 Date and time, local timezone - 2006-01-02 15:04:05
+- ISO8601 Date - 2006-01-02 (YYYY-MM-DD)
--max-age applies only to files and not to directories.
@@ -8706,8 +9336,8 @@ Other flags
IMPORTANT this flag is dangerous to your data - use with --dry-run and
-v first.
-In conjunction with rclone sync the `--delete-excluded deletes any files
-on the destination which are excluded from the command.
+In conjunction with rclone sync, --delete-excluded deletes any files on
+the destination which are excluded from the command.
E.g. the scope of rclone sync -i A: B: can be restricted:
@@ -9139,6 +9769,69 @@ job/list can be used to show the running or recently completed jobs
]
}
+Setting config flags with _config
+
+If you wish to set config (the equivalent of the global flags) for the
+duration of an rc call only then pass in the _config parameter.
+
+This should be in the same format as the config key returned by
+options/get.
+
+For example, if you wished to run a sync with the --checksum parameter,
+you would pass this parameter in your JSON blob.
+
+ "_config":{"CheckSum": true}
+
+If using rclone rc this could be passed as
+
+ rclone rc operations/sync ... _config='{"CheckSum": true}'
+
+Any config parameters you don't set will inherit the global defaults
+which were set with command line flags or environment variables.
+
+Note that it is possible to set some values as strings or integers - see
+data types for more info. Here is an example setting the equivalent of
+--buffer-size in string or integer format.
+
+ "_config":{"BufferSize": "42M"}
+ "_config":{"BufferSize": 44040192}
+
+If you wish to check the _config assignment has worked properly then
+calling options/local will show what the value got set to.
+
+Setting filter flags with _filter
+
+If you wish to set filters for the duration of an rc call only then pass
+in the _filter parameter.
+
+This should be in the same format as the filter key returned by
+options/get.
+
+For example, if you wished to run a sync with these flags
+
+ --max-size 1M --max-age 42s --include "a" --include "b"
+
+you would pass this parameter in your JSON blob.
+
+ "_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}
+
+If using rclone rc this could be passed as
+
+ rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}'
+
+Any filter parameters you don't set will inherit the global defaults
+which were set with command line flags or environment variables.
+
+Note that it is possible to set some values as strings or integers - see
+data types for more info. Here is an example setting the equivalent of
+--buffer-size in string or integer format.
+
+ "_filter":{"MinSize": "42M"}
+ "_filter":{"MinSize": 44040192}
+
+If you wish to check the _filter assignment has worked properly then
+calling options/local will show what the value got set to.
+
Assigning operations to groups with _group = value
Each rc call has its own stats group for tracking its metrics. By
@@ -9157,6 +9850,72 @@ Stats for specific group can be accessed by passing group to core/stats:
}
+Data types
+
+When the API returns types, these will mostly be straight forward
+integer, string or boolean types.
+
+However some of the types returned by the options/get call and taken by
+the options/set calls as well as the vfsOpt, mountOpt and the _config
+parameters.
+
+- Duration - these are returned as an integer duration in nanoseconds.
+ They may be set as an integer, or they may be set with time string,
+ eg "5s". See the options section for more info.
+- Size - these are returned as an integer number of bytes. They may be
+ set as an integer or they may be set with a size suffix string, eg
+ "10M". See the options section for more info.
+- Enumerated type (such as CutoffMode, DumpFlags, LogLevel,
+ VfsCacheMode - these will be returned as an integer and may be set
+ as an integer but more conveniently they can be set as a string, eg
+ "HARD" for CutoffMode or DEBUG for LogLevel.
+- BandwidthSpec - this will be set and returned as a string, eg "1M".
+
+
+Specifying remotes to work on
+
+Remotes are specified with the fs=, srcFs=, dstFs= parameters depending
+on the command being used.
+
+The parameters can be a string as per the rest of rclone, eg
+s3:bucket/path or :sftp:/my/dir. They can also be specified as JSON
+blobs.
+
+If specifyng a JSON blob it should be a object mapping strings to
+strings. These values will be used to configure the remote. There are 3
+special values which may be set:
+
+- type - set to type to specify a remote called :type:
+- _name - set to name to specify a remote called name:
+- _root - sets the root of the remote - may be empty
+
+One of _name or type should normally be set. If the local backend is
+desired then type should be set to local. If _root isn't specified then
+it defaults to the root of the remote.
+
+For example this JSON is equivalent to remote:/tmp
+
+ {
+ "_name": "remote",
+ "_path": "/tmp"
+ }
+
+And this is equivalent to :sftp,host='example.com':/tmp
+
+ {
+ "type": "sftp",
+ "host": "example.com",
+ "_path": "/tmp"
+ }
+
+And this is equivalent to /tmp/dir
+
+ {
+ type = "local",
+ _ path = "/tmp/dir"
+ }
+
+
Supported commands
backend/command: Runs a backend command.
@@ -9484,18 +10243,22 @@ Parameters
Returns the following values:
{
- "speed": average speed in bytes/sec since start of the process,
- "bytes": total transferred bytes since the start of the process,
+ "bytes": total transferred bytes since the start of the group,
+ "checks": number of files checked,
+ "deletes" : number of files deleted,
+ "elapsedTime": time in floating point seconds since rclone was started,
"errors": number of errors,
- "fatalError": whether there has been at least one FatalError,
- "retryError": whether there has been at least one non-NoRetryError,
- "checks": number of checked files,
- "transfers": number of transferred files,
- "deletes" : number of deleted files,
- "renames" : number of renamed files,
+ "eta": estimated time in seconds until the group completes,
+ "fatalError": boolean whether there has been at least one fatal error,
+ "lastError": last error string,
+ "renames" : number of files renamed,
+ "retryError": boolean showing whether there has been at least one non-NoRetryError,
+ "speed": average speed in bytes/sec since start of the group,
+ "totalBytes": total number of bytes in the group,
+ "totalChecks": total number of checks in the group,
+ "totalTransfers": total number of transfers in the group,
"transferTime" : total time spent on running jobs,
- "elapsedTime": time in seconds since the start of the process,
- "lastError": last occurred error,
+ "transfers": number of transferred files,
"transferring": an array of currently active file transfers:
[
{
@@ -9576,6 +10339,8 @@ This shows the current version of go and the go runtime
- os - OS in use as according to Go
- arch - cpu architecture in use according to Go
- goVersion - version of Go runtime in use
+- linking - type of rclone executable (static or dynamic)
+- goTags - space separated build tags or "none"
debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling.
@@ -9615,6 +10380,24 @@ Results
- previousRate - int
+fscache/clear: Clear the Fs cache.
+
+This clears the fs cache. This is where remotes created from backends
+are cached for a short while to make repeated rc calls more efficient.
+
+If you change the parameters of a backend then you may want to call this
+to clear an existing remote out of the cache before re-creating it.
+
+AUTHENTICATION IS REQUIRED FOR THIS CALL.
+
+fscache/entries: Returns the number of entries in the fs cache.
+
+This returns the number of entries in the fs cache.
+
+Returns - entries - number of items in the cache
+
+AUTHENTICATION IS REQUIRED FOR THIS CALL.
+
job/list: Lists the IDs of the running jobs
Parameters - None
@@ -9994,11 +10777,32 @@ options/blocks: List all the option blocks
Returns - options - a list of the options block names
-options/get: Get all the options
+options/get: Get all the global options
Returns an object where keys are option block names and values are an
object with the current option values in.
+Note that these are the global options which are unaffected by use of
+the _config and _filter parameters. If you wish to read the parameters
+set in _config then use options/config and for _filter use
+options/filter.
+
+This shows the internal names of the option within rclone which should
+map to the external options very easily with a few exceptions.
+
+options/local: Get the currently active config for this call
+
+Returns an object with the keys "config" and "filter". The "config" key
+contains the local config and the "filter" key contains the local
+filters.
+
+Note that these are the local options specific to this rc call. If
+_config was not supplied then they will be the global options. Likewise
+with "_filter".
+
+This call is mostly useful for seeing if _config and _filter passing is
+working.
+
This shows the internal names of the option within rclone which should
map to the external options very easily with a few exceptions.
@@ -10017,17 +10821,18 @@ changed like this.
For example:
-This sets DEBUG level logs (-vv)
+This sets DEBUG level logs (-vv) (these can be set by number or string)
+ rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'
rclone rc options/set --json '{"main": {"LogLevel": 8}}'
And this sets INFO level logs (-v)
- rclone rc options/set --json '{"main": {"LogLevel": 7}}'
+ rclone rc options/set --json '{"main": {"LogLevel": "INFO"}}'
And this sets NOTICE level logs (normal without -v)
- rclone rc options/set --json '{"main": {"LogLevel": 6}}'
+ rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}'
pluginsctl/addPlugin: Add a plugin using url
@@ -10156,6 +10961,8 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if
+ set
See the copy command command for more information on the above.
@@ -10167,6 +10974,8 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if
+ set
- deleteEmptySrcDirs - delete empty src directories if set
See the move command command for more information on the above.
@@ -10179,6 +10988,8 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if
+ set
See the sync command command for more information on the above.
@@ -10780,38 +11591,38 @@ upon backend specific capabilities.
Name Purge Copy Move DirMove CleanUp ListR StreamUpload LinkSharing About EmptyDir
------------------------------ ------- ------ ------ --------- --------- ------- -------------- ------------- ------- ----------
- 1Fichier No No No No No No No No No Yes
- Amazon Drive Yes No Yes Yes No #575 No No No #2178 No Yes
- Amazon S3 No Yes No No Yes Yes Yes No #2178 No No
+ 1Fichier No Yes Yes No No No No Yes No Yes
+ Amazon Drive Yes No Yes Yes No No No No No Yes
+ Amazon S3 No Yes No No Yes Yes Yes Yes No No
Backblaze B2 No Yes No No Yes Yes Yes Yes No No
- Box Yes Yes Yes Yes Yes ‡‡ No Yes Yes No Yes
+ Box Yes Yes Yes Yes Yes ‡‡ No Yes Yes Yes Yes
Citrix ShareFile Yes Yes Yes Yes No No Yes No No Yes
- Dropbox Yes Yes Yes Yes No #575 No Yes Yes Yes Yes
- Enterprise File Fabric Yes Yes Yes Yes No No No No No Yes
- FTP No No Yes Yes No No Yes No #2178 No Yes
- Google Cloud Storage Yes Yes No No No Yes Yes No #2178 No No
+ Dropbox Yes Yes Yes Yes No No Yes Yes Yes Yes
+ Enterprise File Fabric Yes Yes Yes Yes Yes No No No No Yes
+ FTP No No Yes Yes No No Yes No No Yes
+ Google Cloud Storage Yes Yes No No No Yes Yes No No No
Google Drive Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Google Photos No No No No No No No No No No
HDFS Yes No No No No No Yes No Yes Yes
- HTTP No No No No No No No No #2178 No Yes
- Hubic Yes †Yes No No No Yes Yes No #2178 Yes No
+ HTTP No No No No No No No No No Yes
+ Hubic Yes †Yes No No No Yes Yes No Yes No
Jottacloud Yes Yes Yes Yes Yes Yes No Yes Yes Yes
Mail.ru Cloud Yes Yes Yes Yes Yes No No Yes Yes Yes
- Mega Yes No Yes Yes Yes No No No #2178 Yes Yes
+ Mega Yes No Yes Yes Yes No No Yes Yes Yes
Memory No Yes No No No Yes Yes No No No
- Microsoft Azure Blob Storage Yes Yes No No No Yes Yes No #2178 No No
+ Microsoft Azure Blob Storage Yes Yes No No No Yes Yes No No No
Microsoft OneDrive Yes Yes Yes Yes Yes No No Yes Yes Yes
OpenDrive Yes Yes Yes Yes No No No No No Yes
- OpenStack Swift Yes †Yes No No No Yes Yes No #2178 Yes No
+ OpenStack Swift Yes †Yes No No No Yes Yes No Yes No
pCloud Yes Yes Yes Yes Yes No No Yes Yes Yes
premiumize.me Yes No Yes Yes No No No Yes Yes Yes
- put.io Yes No Yes Yes Yes No Yes No #2178 Yes Yes
- QingStor No Yes No No Yes Yes No No #2178 No No
+ put.io Yes No Yes Yes Yes No Yes No Yes Yes
+ QingStor No Yes No No Yes Yes No No No No
Seafile Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
- SFTP No No Yes Yes No No Yes No #2178 Yes Yes
+ SFTP No No Yes Yes No No Yes No Yes Yes
SugarSync Yes Yes Yes Yes No No Yes Yes No Yes
Tardigrade Yes †No No No No Yes Yes No No No
- WebDAV Yes Yes Yes Yes No No Yes ‡ No #2178 Yes Yes
+ WebDAV Yes Yes Yes Yes No No Yes ‡ No Yes Yes
Yandex Disk Yes Yes Yes Yes Yes No Yes Yes Yes Yes
Zoho WorkDrive Yes Yes Yes Yes No No No No Yes Yes
The local filesystem Yes No Yes Yes No No Yes No Yes Yes
@@ -10925,10 +11736,10 @@ These flags are available for every command.
-c, --checksum Skip based on checksum (if available) & size, not mod-time & size
--client-cert string Client SSL certificate (PEM) for mutual TLS auth
--client-key string Client SSL private key (PEM) for mutual TLS auth
- --compare-dest string Include additional server-side path during comparison.
+ --compare-dest stringArray Include additional comma separated server-side paths during comparison.
--config string Config file. (default "$HOME/.config/rclone/rclone.conf")
--contimeout duration Connect timeout (default 1m0s)
- --copy-dest string Implies --compare-dest but also copies files from path into destination.
+ --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination.
--cpuprofile string Write cpu profile to file
--cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
--delete-after When synchronizing, delete files on destination after transferring (default)
@@ -10937,6 +11748,7 @@ These flags are available for every command.
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
-n, --dry-run Do a trial run with no permanent changes
+ --dscp string Set DSCP value to connections. Can be value or names, eg. CS1, LE, DF, AF21.
--dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
--dump-headers Dump HTTP headers - may contain sensitive info
@@ -10950,6 +11762,8 @@ These flags are available for every command.
--files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
-f, --filter stringArray Add a file-filtering rule
--filter-from stringArray Read filtering patterns from a file (use - to read from stdin)
+ --fs-cache-expire-duration duration cache remotes for this long (0 to disable caching) (default 5m0s)
+ --fs-cache-expire-interval duration interval to check for expired remotes (default 1m0s)
--header stringArray Set HTTP header for all transactions
--header-download stringArray Set HTTP header for download transactions
--header-upload stringArray Set HTTP header for upload transactions
@@ -11048,7 +11862,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
- --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.54.0")
+ --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
-v, --verbose count Print lots more stuff (repeat for more)
@@ -11080,6 +11894,7 @@ and may be set in the config file.
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_object_id or msi_mi_res_id specified.
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_object_id specified.
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_mi_res_id specified.
+ --azureblob-public-access string Public access level of a container: blob, container.
--azureblob-sas-url string SAS URL for container level access only
--azureblob-service-principal-file string Path to file containing credentials for use with a service principal.
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256MB). (Deprecated)
@@ -11143,6 +11958,7 @@ and may be set in the config file.
-L, --copy-links Follow symlinks and copy the pointed to item.
--crypt-directory-name-encryption Option to either encrypt directory names or leave them intact. (default true)
--crypt-filename-encryption string How to encrypt the filenames. (default "standard")
+ --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted.
--crypt-password string Password or pass phrase for encryption. (obscured)
--crypt-password2 string Password or pass phrase for salt. Optional but recommended. (obscured)
--crypt-remote string Remote to encrypt/decrypt.
@@ -11178,7 +11994,7 @@ and may be set in the config file.
--drive-starred-only Only show files that are starred.
--drive-stop-on-download-limit Make download limit errors be fatal
--drive-stop-on-upload-limit Make upload limit errors be fatal
- --drive-team-drive string ID of the Team Drive
+ --drive-team-drive string ID of the Shared Drive (Team Drive)
--drive-token string OAuth Access Token as a JSON blob.
--drive-token-url string Token server url.
--drive-trashed-only Only show files that are in the trash.
@@ -11207,12 +12023,14 @@ and may be set in the config file.
--filefabric-token-expiry string Token expiry time
--filefabric-url string URL of the Enterprise File Fabric to connect to
--filefabric-version string Version read from the file fabric
+ --ftp-close-timeout Duration Maximum time to wait for a response to close. (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
--ftp-disable-epsv Disable using EPSV even if server advertises support
--ftp-disable-mlsd Disable using MLSD even if server advertises support
--ftp-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot)
--ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
--ftp-host string FTP host to connect to
+ --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--ftp-no-check-certificate Do not verify the TLS certificate of the server
--ftp-pass string FTP password (obscured)
--ftp-port string FTP port, leave blank to use default (21)
@@ -11274,6 +12092,7 @@ and may be set in the config file.
--local-case-sensitive Force the filesystem to report itself as case sensitive.
--local-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot)
--local-no-check-updated Don't check to see if the files change during upload
+ --local-no-preallocate Disable preallocation of disk space for transferred files
--local-no-set-modtime Disable setting modtime
--local-no-sparse Disable sparse files for multi-thread downloads
--local-no-unicode-normalization Don't apply unicode normalization to paths and filenames (Deprecated)
@@ -11378,8 +12197,10 @@ and may be set in the config file.
--seafile-url string URL of seafile host to connect to
--seafile-user string User name (usually email address)
--sftp-ask-password Allow asking for SFTP password when needed.
+ --sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available.
--sftp-host string SSH host to connect to
+ --sftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--sftp-key-file string Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.
--sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file. (obscured)
--sftp-key-pem string Raw PEM-encoded private key, If specified, will override key_file parameter.
@@ -11449,9 +12270,10 @@ and may be set in the config file.
--union-upstreams string List of space separated upstreams.
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
--webdav-bearer-token-command string Command to run to get a bearer token
+ --webdav-encoding string This sets the encoding for the backend.
--webdav-pass string Password. (obscured)
--webdav-url string URL of http host to connect to
- --webdav-user string User name
+ --webdav-user string User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-auth-url string Auth server URL.
--yandex-client-id string OAuth Client Id
@@ -11459,8 +12281,13 @@ and may be set in the config file.
--yandex-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot)
--yandex-token string OAuth Access Token as a JSON blob.
--yandex-token-url string Token server url.
+ --zoho-auth-url string Auth server URL.
+ --zoho-client-id string OAuth Client Id
+ --zoho-client-secret string OAuth Client Secret
--zoho-encoding MultiEncoder This sets the encoding for the backend. (default Del,Ctl,InvalidUtf8)
--zoho-region string Zoho region to connect to. You'll have to use the region you organization is registered in.
+ --zoho-token string OAuth Access Token as a JSON blob.
+ --zoho-token-url string Token server url.
1Fichier
@@ -12350,6 +13177,16 @@ You'd then do a full rclone sync less often.
Note that --fast-list isn't required in the top-up sync.
+Avoiding HEAD requests after PUT
+
+By default rclone will HEAD every object it uploads. It does this to
+check the object got uploaded correctly.
+
+You can disable this with the --s3-no-head option - see there for more
+details.
+
+Setting this flag increases the chance for undetected upload failures.
+
Hashes
For small objects which weren't uploaded as multipart uploads (objects
@@ -16313,6 +17150,10 @@ Note that list assembles composite directory entries only when chunk
names match the configured format and treats non-conforming file names
as normal non-chunked files.
+When using norename transactions, chunk names will additionally have a
+unique file version suffix. For example,
+BIG_FILE_NAME.rclone_chunk.001_bp562k.
+
Metadata
Besides data chunks chunker will by default create metadata object for a
@@ -16331,6 +17172,7 @@ for composite files. Meta objects carry the following fields:
- nchunks - number of data chunks in file
- md5 - MD5 hashsum of composite file (if present)
- sha1 - SHA1 hashsum (if present)
+- txn - identifies current version of the file
There is no field for composite file name as it's simply equal to the
name of meta object on the wrapped remote. Please refer to respective
@@ -16401,8 +17243,8 @@ first data chunk.
Migrations
-The idiomatic way to migrate to a different chunk size, hash type or
-chunk naming scheme is to:
+The idiomatic way to migrate to a different chunk size, hash type,
+transaction style or chunk naming scheme is to:
- Collect all your chunked files under a directory and have your
chunker remote point to it.
@@ -16460,6 +17302,8 @@ Chunker included in rclone releases up to v1.54 can sometimes fail to
detect metadata produced by recent versions of rclone. We recommend
users to keep rclone up-to-date to avoid data corruption.
+Changing transactions is dangerous and requires explicit migration.
+
Standard Options
Here are the standard options specific to chunker (Transparently
@@ -16571,6 +17415,37 @@ Choose how chunker should handle files with missing or invalid chunks.
- "false"
- Warn user, skip incomplete file and proceed.
+--chunker-transactions
+
+Choose how chunker should handle temporary files during transactions.
+
+- Config: transactions
+- Env Var: RCLONE_CHUNKER_TRANSACTIONS
+- Type: string
+- Default: "rename"
+- Examples:
+ - "rename"
+ - Rename temporary files after a successful transaction.
+ - "norename"
+ - Leave temporary file names and write transaction ID to
+ metadata file.
+ - Metadata is required for no rename transactions (meta format
+ cannot be "none").
+ - If you are using norename transactions you should be careful
+ not to downgrade Rclone
+ - as older versions of Rclone don't support this transaction
+ style and will misinterpret
+ - files manipulated by norename transactions.
+ - This method is EXPERIMENTAL, don't use on production
+ systems.
+ - "auto"
+ - Rename or norename will be used depending on capabilities of
+ the backend.
+ - If meta format is set to "none", rename transactions will
+ always be used.
+ - This method is EXPERIMENTAL, don't use on production
+ systems.
+
Citrix ShareFile
@@ -16882,9 +17757,8 @@ would with any other remote, e.g. rclone copy D:\docs secret:\docs, and
rclone will encrypt and decrypt as needed on the fly. If you access the
wrapped remote remote:path directly you will bypass the encryption, and
anything you read will be in encrypted form, and anything you write will
-be undencrypted. To avoid issues it is best to configure a dedicated
-path for encrypted content, and access it exclusively through a crypt
-remote.
+be unencrypted. To avoid issues it is best to configure a dedicated path
+for encrypted content, and access it exclusively through a crypt remote.
No remotes found - make a new one
n) New remote
@@ -17304,6 +18178,20 @@ names, or for debugging purposes.
- Type: bool
- Default: false
+--crypt-no-data-encryption
+
+Option to either encrypt file data or leave it unencrypted.
+
+- Config: no_data_encryption
+- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Type: bool
+- Default: false
+- Examples:
+ - "true"
+ - Don't encrypt file data, leave it unencrypted.
+ - "false"
+ - Encrypt file data.
+
Backend commands
Here are the commands specific to the crypt backend.
@@ -17814,6 +18702,20 @@ can be set smaller if you are tight on memory.
Impersonate this user when using a business account.
+Note that if you want to use impersonate, you should make sure this flag
+is set when running "rclone config" as this will cause rclone to request
+the "members.read" scope which it won't normally. This is needed to
+lookup a members email address into the internal ID that dropbox uses in
+the API.
+
+Using the "members.read" scope will require a Dropbox Team Admin to
+approve during the OAuth flow.
+
+You will have to use your own App (setting your own client_id and
+client_secret) to use this option as currently rclone's default set of
+permissions doesn't include "members.read". This can be added once v1.55
+or later is in use everywhere.
+
- Config: impersonate
- Env Var: RCLONE_DROPBOX_IMPERSONATE
- Type: string
@@ -18257,9 +19159,8 @@ Example without a config file
Implicit TLS
Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to be
-enabled in the FTP backend config for the remote, or with
-[--ftp-tls]{#ftp-tls}. The default FTPS port is 990, not 21 and can be
-set with [--ftp-port]{#ftp-port}.
+enabled in the FTP backend config for the remote, or with --ftp-tls. The
+default FTPS port is 990, not 21 and can be set with --ftp-port.
Standard Options
@@ -18371,6 +19272,29 @@ Disable using MLSD even if server advertises support
- Type: bool
- Default: false
+--ftp-idle-timeout
+
+Max time before closing idle connections
+
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+
+Set to 0 to keep connections indefinitely.
+
+- Config: idle_timeout
+- Env Var: RCLONE_FTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
+--ftp-close-timeout
+
+Maximum time to wait for a response to close.
+
+- Config: close_timeout
+- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
--ftp-encoding
This sets the encoding for the backend.
@@ -19009,7 +19933,7 @@ This will guide you through an interactive setup process:
Log in and authorize rclone for access
Waiting for code...
Got code
- Configure this as a team drive?
+ Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> n
@@ -19212,22 +20136,23 @@ you created/selected at step #1 - use rclone without specifying the
--drive-impersonate option, like this:
rclone -v foo@example.com lsf gdrive:backup
-Team drives
+Shared drives (team drives)
-If you want to configure the remote to point to a Google Team Drive then
-answer y to the question Configure this as a team drive?.
+If you want to configure the remote to point to a Google Shared Drive
+(previously known as Team Drives) then answer y to the question
+Configure this as a Shared Drive (Team Drive)?.
-This will fetch the list of Team Drives from google and allow you to
-configure which one you want to use. You can also type in a team drive
+This will fetch the list of Shared Drives from google and allow you to
+configure which one you want to use. You can also type in a Shared Drive
ID if you prefer.
For example:
- Configure this as a team drive?
+ Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> y
- Fetching team drive list...
+ Fetching Shared Drive list...
Choose a number from below, or type in your own value
1 / Rclone Test
\ "xxxxxxxxxxxxxxxxxxxx"
@@ -19235,7 +20160,7 @@ For example:
\ "yyyyyyyyyyyyyyyyyyyy"
3 / Rclone Test 3
\ "zzzzzzzzzzzzzzzzzzzz"
- Enter a Team Drive ID> 1
+ Enter a Shared Drive ID> 1
--------------------
[remote]
client_id =
@@ -19635,7 +20560,7 @@ if you want use SA instead of interactive login.
--drive-team-drive
-ID of the Team Drive
+ID of the Shared Drive (Team Drive)
- Config: team_drive
- Env Var: RCLONE_DRIVE_TEAM_DRIVE
@@ -20093,11 +21018,11 @@ Options:
drives
-List the shared drives available to this account
+List the Shared Drives available to this account
rclone backend drives remote: [options] [+]
-This command lists the shared drives (teamdrives) available to this
+This command lists the Shared Drives (Team Drives) available to this
account.
Usage:
@@ -21781,8 +22706,8 @@ Note that Koofr is case insensitive so you can't have a file called
Mail.ru Cloud
Mail.ru Cloud is a cloud storage provided by a Russian internet company
-Mail.Ru Group. The official desktop client is Disk-O:, available only on
-Windows. (Please note that official sites are in Russian)
+Mail.Ru Group. The official desktop client is Disk-O:, available on
+Windows and Mac OS.
Currently it is recommended to disable 2FA on Mail.ru accounts intended
for rclone until it gets eventually implemented.
@@ -22762,6 +23687,24 @@ See: the encoding section in the overview for more info.
- Type: MultiEncoder
- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
+--azureblob-public-access
+
+Public access level of a container: blob, container.
+
+- Config: public_access
+- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
+- Type: string
+- Default: ""
+- Examples:
+ - ""
+ - The container and its blobs can be accessed only with an
+ authorized request. It's a default value
+ - "blob"
+ - Blob data within this container can be read via anonymous
+ request.
+ - "container"
+ - Allow full public read access for container and blob data.
+
Limitations
MD5 sums are only uploaded with chunked files if the source has an MD5
@@ -25666,6 +26609,42 @@ opened at any given time.
- Type: bool
- Default: false
+--sftp-disable-concurrent-reads
+
+If set don't use concurrent reads
+
+Normally concurrent reads are safe to use and not using them will
+degrade performance, so this option is disabled by default.
+
+Some servers limit the amount number of times a file can be downloaded.
+Using concurrent reads can trigger this limit, so if you have a server
+which returns
+
+ Failed to copy: file does not exist
+
+Then you may need to enable this flag.
+
+If concurrent reads are disabled, the use_fstat option is ignored.
+
+- Config: disable_concurrent_reads
+- Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
+- Type: bool
+- Default: false
+
+--sftp-idle-timeout
+
+Max time before closing idle connections
+
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+
+Set to 0 to keep connections indefinitely.
+
+- Config: idle_timeout
+- Env Var: RCLONE_SFTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
Limitations
SFTP supports checksums if the same login has shell access and md5sum or
@@ -25693,7 +26672,7 @@ The Go SSH library disables the use of the aes128-cbc cipher by default,
due to security concerns. This can be re-enabled on a per-connection
basis by setting the use_insecure_cipher setting in the configuration
file to true. Further details on the insecurity of this cipher can be
-found [in this paper] (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
+found in this paper.
SFTP isn't supported under plan9 until this issue is fixed.
@@ -26609,9 +27588,11 @@ This will guide you through an interactive setup process:
\ "nextcloud"
2 / Owncloud
\ "owncloud"
- 3 / Sharepoint
+ 3 / Sharepoint Online, authenticated by Microsoft account.
\ "sharepoint"
- 4 / Other site/service or software
+ 4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
+ \ "sharepoint-ntlm"
+ 5 / Other site/service or software
\ "other"
vendor> 1
User name
@@ -26696,13 +27677,17 @@ Name of the Webdav site/service/software you are using
- "owncloud"
- Owncloud
- "sharepoint"
- - Sharepoint
+ - Sharepoint Online, authenticated by Microsoft account.
+ - "sharepoint-ntlm"
+ - Sharepoint with NTLM authentication. Usually self-hosted or
+ on-premises.
- "other"
- Other site/service or software
--webdav-user
-User name
+User name. In case NTLM authentication is used, the username should be
+in the format 'Domain'.
- Config: user
- Env Var: RCLONE_WEBDAV_USER
@@ -26742,6 +27727,21 @@ Command to run to get a bearer token
- Type: string
- Default: ""
+--webdav-encoding
+
+This sets the encoding for the backend.
+
+See: the encoding section in the overview for more info.
+
+Default encoding is
+Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8
+for sharepoint-ntlm or identity otherwise.
+
+- Config: encoding
+- Env Var: RCLONE_WEBDAV_ENCODING
+- Type: string
+- Default: ""
+
Provider notes
@@ -26762,7 +27762,7 @@ initially did not support streaming of files (rcat) whereas Owncloud
did, but this seems to be fixed as of 2020-11-27 (tested with rclone
v1.53.1 and Nextcloud Server v19).
-Sharepoint
+Sharepoint Online
Rclone can be used with Sharepoint provided by OneDrive for Business or
Office365 Education Accounts. This feature is only needed for a few of
@@ -26795,10 +27795,39 @@ Your config file should look like this:
[sharepoint]
type = webdav
url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents
- vendor = other
+ vendor = sharepoint
user = YourEmailAddress
pass = encryptedpassword
+Sharepoint with NTLM Authentication
+
+Use this option in case your (hosted) Sharepoint is not tied to OneDrive
+accounts and uses NTLM authentication.
+
+To get the url configuration, similarly to the above, first navigate to
+the desired directory in your browser to get the URL, then strip
+everything after the name of the opened directory.
+
+Example: If the URL is:
+https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx
+
+The configuration to use would be:
+https://example.sharepoint.com/sites/12345/Documents
+
+Set the vendor to sharepoint-ntlm.
+
+NTLM uses domain and user name combination for authentication, set user
+to DOMAIN\username.
+
+Your config file should look like this:
+
+ [sharepoint]
+ type = webdav
+ url = https://[YOUR-DOMAIN]/some-path-to/Documents
+ vendor = sharepoint-ntlm
+ user = DOMAIN\user
+ pass = encryptedpassword
+
Required Flags for SharePoint
As SharePoint does some special things with uploaded documents, you
@@ -27133,11 +28162,12 @@ This will guide you through an interactive setup process:
See the remote setup docs for how to set it up on a machine with no
Internet browser available.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Zoho Workdrive. This only runs from the moment it
-opens your browser to the moment you get back the verification code.
-This is on http://127.0.0.1:53682/ and this it may require you to
-unblock it temporarily if you are running a host firewall.
+Rclone runs a webserver on your local computer to collect the
+authorization token from Zoho Workdrive. This is only from the moment
+your browser is opened until the token is returned. The webserver runs
+on http://127.0.0.1:53682/. If local port 53682 is protected by a
+firewall you may need to temporarily unblock the firewall to complete
+authorization.
Once configured you can then use rclone like this,
@@ -27183,6 +28213,24 @@ Standard Options
Here are the standard options specific to zoho (Zoho).
+--zoho-client-id
+
+OAuth Client Id Leave blank normally.
+
+- Config: client_id
+- Env Var: RCLONE_ZOHO_CLIENT_ID
+- Type: string
+- Default: ""
+
+--zoho-client-secret
+
+OAuth Client Secret Leave blank normally.
+
+- Config: client_secret
+- Env Var: RCLONE_ZOHO_CLIENT_SECRET
+- Type: string
+- Default: ""
+
--zoho-region
Zoho region to connect to. You'll have to use the region you
@@ -27206,6 +28254,33 @@ Advanced Options
Here are the advanced options specific to zoho (Zoho).
+--zoho-token
+
+OAuth Access Token as a JSON blob.
+
+- Config: token
+- Env Var: RCLONE_ZOHO_TOKEN
+- Type: string
+- Default: ""
+
+--zoho-auth-url
+
+Auth server URL. Leave blank to use the provider defaults.
+
+- Config: auth_url
+- Env Var: RCLONE_ZOHO_AUTH_URL
+- Type: string
+- Default: ""
+
+--zoho-token-url
+
+Token server url. Leave blank to use the provider defaults.
+
+- Config: token_url
+- Env Var: RCLONE_ZOHO_TOKEN_URL
+- Type: string
+- Default: ""
+
--zoho-encoding
This sets the encoding for the backend.
@@ -27225,10 +28300,11 @@ Local paths are specified as normal filesystem paths, e.g.
rclone sync -i /home/source /tmp/destination
-Will sync /home/source to /tmp/destination
+Will sync /home/source to /tmp/destination.
-These can be configured into the config file for consistencies sake, but
-it is probably easier not to.
+For consistencies sake one can also configure a remote of type local in
+the config file, and access the local filesystem using rclone remote
+paths, e.g. remote:path/to/wherever, but it is probably easier not to.
Modified time
@@ -27322,14 +28398,34 @@ These only get replaced if they are the last character in the name:
Invalid UTF-8 bytes will also be replaced, as they can't be converted to
UTF-16.
-Long paths on Windows
+Paths on Windows
-Rclone handles long paths automatically, by converting all paths to long
-UNC paths which allows paths up to 32,767 characters.
+On Windows there are many ways of specifying a path to a file system
+resource. Both absolute paths like C:\path\to\wherever, and relative
+paths like ..\wherever can be used, and path separator can be either \
+(as in C:\path\to\wherever) or / (as in C:/path/to/wherever). Length of
+these paths are limited to 259 characters for files and 247 characters
+for directories, but there is an alternative extended-length path format
+increasing the limit to (approximately) 32,767 characters. This format
+requires absolute paths and the use of prefix \\?\, e.g.
+\\?\D:\some\very\long\path. For convenience rclone will automatically
+convert regular paths into the corresponding extended-length paths, so
+in most cases you do not have to worry about this (read more below).
-This is why you will see that your paths, for instance c:\files is
-converted to the UNC path \\?\c:\files in the output, and \\server\share
-is converted to \\?\UNC\server\share.
+Note that Windows supports using the same prefix \\?\ to specify path to
+volumes identified by their GUID, e.g.
+\\?\Volume{b75e2c83-0000-0000-0000-602f00000000}\some\path. This is
+_not_ supported in rclone, due to an issue in go.
+
+Long paths
+
+Rclone handles long paths automatically, by converting all paths to
+extended-length path format, which allows paths up to 32,767 characters.
+
+This conversion will ensure paths are absolute and prefix them with the
+\\?\. This is why you will see that your paths, for instance .\files is
+shown as path \\?\C:\files in the output, and \\server\share as
+\\?\UNC\server\share.
However, in rare cases this may cause problems with buggy file system
drivers like EncFS. To disable UNC conversion globally, add this to your
@@ -27350,7 +28446,7 @@ And use rclone like this:
rclone copy c:\src nounc:z:\dst
This will use UNC paths on c:\src but not on z:\dst. Of course this will
-cause problems if the absolute path length of a file exceeds 258
+cause problems if the absolute path length of a file exceeds 259
characters on z, so only use this option if you have to.
Symlinks / Junction points
@@ -27485,9 +28581,9 @@ mount to the same device as being on the same filesystem.
NB This flag is only available on Unix based systems. On systems where
it isn't supported (e.g. Windows) it will be ignored.
-Standard Options
+Advanced Options
-Here are the standard options specific to local (Local Disk).
+Here are the advanced options specific to local (Local Disk).
--local-nounc
@@ -27501,10 +28597,6 @@ Disable UNC (long path names) conversion on Windows
- "true"
- Disables long file names
-Advanced Options
-
-Here are the advanced options specific to local (Local Disk).
-
--copy-links / -L
Follow symlinks and copy the pointed to item.
@@ -27632,6 +28724,21 @@ override the default choice.
- Type: bool
- Default: false
+--local-no-preallocate
+
+Disable preallocation of disk space for transferred files
+
+Preallocation of disk space helps prevent filesystem fragmentation.
+However, some virtual filesystem layers (such as Google Drive File
+Stream) may incorrectly set the actual file size equal to the
+preallocated space, causing checksum and file size checks to fail. Use
+this flag to disable preallocation.
+
+- Config: no_preallocate
+- Env Var: RCLONE_LOCAL_NO_PREALLOCATE
+- Type: bool
+- Default: false
+
--local-no-sparse
Disable sparse files for multi-thread downloads
@@ -27707,12 +28814,279 @@ Options:
CHANGELOG
+v1.55.0 - 2021-03-31
+
+See commits
+
+- New commands
+ - selfupdate (Ivan Andreev)
+ - Allows rclone to update itself in-place or via a package
+ (using --package flag)
+ - Reads cryptographically signed signatures for non beta
+ releases
+ - Works on all OSes.
+ - test - these are test commands - use with care!
+ - histogram - Makes a histogram of file name characters.
+ - info - Discovers file name or other limitations for paths.
+ - makefiles - Make a random file hierarchy for testing.
+ - memory - Load all the objects at remote:path into memory and
+ report memory stats.
+- New Features
+ - Connection strings
+ - Config parameters can now be passed as part of the remote
+ name as a connection string.
+ - For example to do the equivalent of --drive-shared-with-me
+ use drive,shared_with_me:
+ - Make sure we don't save on the fly remote config to the
+ config file (Nick Craig-Wood)
+ - Make sure backends with additional config have a different
+ name for caching (Nick Craig-Wood)
+ - This work was sponsored by CERN, through the CS3MESH4EOSC
+ Project.
+ - CS3MESH4EOSC has received funding from the European
+ Union’s Horizon 2020
+ - research and innovation programme under Grant Agreement
+ no. 863353.
+ - build
+ - Update go build version to go1.16 and raise minimum go
+ version to go1.13 (Nick Craig-Wood)
+ - Make a macOS ARM64 build to support Apple Silicon (Nick
+ Craig-Wood)
+ - Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood)
+ - Use GO386=softfloat instead of deprecated GO386=387 for 386
+ builds (Nick Craig-Wood)
+ - Disable IOS builds for the time being (Nick Craig-Wood)
+ - Androids builds made with up to date NDK (x0b)
+ - Add an rclone user to the Docker image but don't use it by
+ default (cynthia kwok)
+ - dedupe: Make largest directory primary to minimize data moved
+ (Saksham Khanna)
+ - config
+ - Wrap config library in an interface (Fionera)
+ - Make config file system pluggable (Nick Craig-Wood)
+ - --config "" or "/notfound" for in memory config only (Nick
+ Craig-Wood)
+ - Clear fs cache of stale entries when altering config (Nick
+ Craig-Wood)
+ - copyurl: Add option to print resulting auto-filename (albertony)
+ - delete: Make --rmdirs obey the filters (Nick Craig-Wood)
+ - docs - many fixes and reworks from edwardxml, albertony, pvalls,
+ Ivan Andreev, Evan Harris, buengese, Alexey Tabakman
+ - encoder/filename - add SCSU as tables (Klaus Post)
+ - Add multiple paths support to --compare-dest and --copy-dest
+ flag (K265)
+ - filter: Make --exclude "dir/" equivalent to --exclude "dir/**"
+ (Nick Craig-Wood)
+ - fshttp: Add DSCP support with --dscp for QoS with differentiated
+ services (Max Sum)
+ - lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood)
+ - lib/file
+ - Make pre-allocate detect disk full errors and return them
+ (Nick Craig-Wood)
+ - Don't run preallocate concurrently (Nick Craig-Wood)
+ - Retry preallocate on EINTR (Nick Craig-Wood)
+ - operations: Made copy and sync operations obey a RetryAfterError
+ (Ankur Gupta)
+ - rc
+ - Add string alternatives for setting options over the rc
+ (Nick Craig-Wood)
+ - Add options/local to see the options configured in the
+ context (Nick Craig-Wood)
+ - Add _config parameter to set global config for just this rc
+ call (Nick Craig-Wood)
+ - Implement passing filter config with _filter parameter (Nick
+ Craig-Wood)
+ - Add fscache/clear and fscache/entries to control the fs
+ cache (Nick Craig-Wood)
+ - Avoid +Inf value for speed in core/stats (albertony)
+ - Add a full set of stats to core/stats (Nick Craig-Wood)
+ - Allow fs= params to be a JSON blob (Nick Craig-Wood)
+ - rcd: Added systemd notification during the rclone rcd command.
+ (Naveen Honest Raj)
+ - rmdirs: Make --rmdirs obey the filters (Nick Craig-Wood)
+ - version: Show build tags and type of executable (Ivan Andreev)
+- Bug Fixes
+ - install.sh: make it fail on download errors (Ivan Andreev)
+ - Fix excessive retries missing --max-duration timeout (Nick
+ Craig-Wood)
+ - Fix crash when --low-level-retries=0 (Nick Craig-Wood)
+ - Fix failed token refresh on mounts created via the rc (Nick
+ Craig-Wood)
+ - fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood)
+ - lib/atexit
+ - Unregister interrupt handler once it has fired so users can
+ interrupt again (Nick Craig-Wood)
+ - Fix occasional failure to unmount with CTRL-C (Nick
+ Craig-Wood)
+ - Fix deadlock calling Finalise while Run is running (Nick
+ Craig-Wood)
+ - lib/rest: Fix multipart uploads not stopping on context cancel
+ (Nick Craig-Wood)
+- Mount
+ - Allow mounting to root directory on windows (albertony)
+ - Improved handling of relative paths on windows (albertony)
+ - Fix unicode issues with accented characters on macOS (Nick
+ Craig-Wood)
+ - Docs: document the new FileSecurity option in WinFsp 2021
+ (albertony)
+ - Docs: add note about volume path syntax on windows (albertony)
+ - Fix caching of old directories after renaming them (Nick
+ Craig-Wood)
+ - Update cgofuse to the latest version to bring in macfuse 4 fix
+ (Nick Craig-Wood)
+- VFS
+ - --vfs-used-is-size to report used space using recursive scan
+ (tYYGH)
+ - Don't set modification time if it was already correct (Nick
+ Craig-Wood)
+ - Fix Create causing windows explorer to truncate files on CTRL-C
+ CTRL-V (Nick Craig-Wood)
+ - Fix modtimes not updating when writing via cache (Nick
+ Craig-Wood)
+ - Fix modtimes changing by fractional seconds after upload (Nick
+ Craig-Wood)
+ - Fix modtime set if --vfs-cache-mode writes/full and no write
+ (Nick Craig-Wood)
+ - Rename files in cache and cancel uploads on directory rename
+ (Nick Craig-Wood)
+ - Fix directory renaming by renaming dirs cached in memory (Nick
+ Craig-Wood)
+- Local
+ - Add flag --local-no-preallocate (David Sze)
+ - Make nounc an advanced option except on Windows (albertony)
+ - Don't ignore preallocate disk full errors (Nick Craig-Wood)
+- Cache
+ - Add --fs-cache-expire-duration to control the fs cache (Nick
+ Craig-Wood)
+- Crypt
+ - Add option to not encrypt data (Vesnyx)
+ - Log hash ok on upload (albertony)
+- Azure Blob
+ - Add container public access level support. (Manish Kumar)
+- B2
+ - Fix HTML files downloaded via cloudflare (Nick Craig-Wood)
+- Box
+ - Fix transfers getting stuck on token expiry after API change
+ (Nick Craig-Wood)
+- Chunker
+ - Partially implement no-rename transactions (Maxwell Calman)
+- Drive
+ - Don't stop server side copy if couldn't read description (Nick
+ Craig-Wood)
+ - Pass context on to drive SDK - to help with cancellation (Nick
+ Craig-Wood)
+- Dropbox
+ - Add polling for changes support (Robert Thomas)
+ - Make --timeout 0 work properly (Nick Craig-Wood)
+ - Raise priority of rate limited message to INFO to make it more
+ noticeable (Nick Craig-Wood)
+- Fichier
+ - Implement copy & move (buengese)
+ - Implement public link (buengese)
+- FTP
+ - Implement Shutdown method (Nick Craig-Wood)
+ - Close idle connections after --ftp-idle-timeout (1m by default)
+ (Nick Craig-Wood)
+ - Make --timeout 0 work properly (Nick Craig-Wood)
+ - Add --ftp-close-timeout flag for use with awkward ftp servers
+ (Nick Craig-Wood)
+ - Retry connections and logins on 421 errors (Nick Craig-Wood)
+- Hdfs
+ - Fix permissions for when directory is created (Lucas Messenger)
+- Onedrive
+ - Make --timeout 0 work properly (Nick Craig-Wood)
+- S3
+ - Fix --s3-profile which wasn't working (Nick Craig-Wood)
+- SFTP
+ - Close idle connections after --sftp-idle-timeout (1m by default)
+ (Nick Craig-Wood)
+ - Fix "file not found" errors for read once servers (Nick
+ Craig-Wood)
+ - Fix SetModTime stat failed: object not found with
+ --sftp-set-modtime=false (Nick Craig-Wood)
+- Swift
+ - Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood)
+ - Implement copying large objects (nguyenhuuluan434)
+- Union
+ - Fix crash when using epff policy (Nick Craig-Wood)
+ - Fix union attempting to update files on a read only file system
+ (Nick Craig-Wood)
+ - Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick
+ Craig-Wood)
+ - Fix initialisation broken in refactor (Nick Craig-Wood)
+- WebDAV
+ - Add support for sharepoint with NTLM authentication (Rauno Ots)
+ - Make sharepoint-ntlm docs more consistent (Alex Chen)
+ - Improve terminology in sharepoint-ntlm docs (Ivan Andreev)
+ - Disable HTTP/2 for NTLM authentication (georne)
+ - Fix sharepoint-ntlm error 401 for parallel actions (Ivan
+ Andreev)
+ - Check that purged directory really exists (Ivan Andreev)
+- Yandex
+ - Make --timeout 0 work properly (Nick Craig-Wood)
+- Zoho
+ - Replace client id - you will need to rclone config reconnect
+ after this (buengese)
+ - Add forgotten setupRegion() to NewFs - this finally fixes
+ regions other than EU (buengese)
+
+
+v1.54.1 - 2021-03-08
+
+See commits
+
+- Bug Fixes
+ - accounting: Fix --bwlimit when up or down is off (Nick
+ Craig-Wood)
+ - docs
+ - Fix nesting of brackets and backticks in ftp docs
+ (edwardxml)
+ - Fix broken link in sftp page (edwardxml)
+ - Fix typo in crypt.md (Romeo Kienzler)
+ - Changelog: Correct link to digitalis.io (Alex JOST)
+ - Replace #file-caching with #vfs-file-caching (Miron
+ Veryanskiy)
+ - Convert bogus example link to code (edwardxml)
+ - Remove dead link from rc.md (edwardxml)
+ - rc: Sync,copy,move: document createEmptySrcDirs parameter (Nick
+ Craig-Wood)
+ - lsjson: Fix unterminated JSON in the presence of errors (Nick
+ Craig-Wood)
+- Mount
+ - Fix mount dropping on macOS by setting --daemon-timeout 10m
+ (Nick Craig-Wood)
+- VFS
+ - Document simultaneous usage with the same cache shouldn't be
+ used (Nick Craig-Wood)
+- B2
+ - Automatically raise upload cutoff to avoid spurious error (Nick
+ Craig-Wood)
+ - Fix failed to create file system with application key limited to
+ a prefix (Nick Craig-Wood)
+- Drive
+ - Refer to Shared Drives instead of Team Drives (Nick Craig-Wood)
+- Dropbox
+ - Add scopes to oauth request and optionally "members.read" (Nick
+ Craig-Wood)
+- S3
+ - Fix failed to create file system with folder level permissions
+ policy (Nick Craig-Wood)
+ - Fix Wasabi HEAD requests returning stale data by using only 1
+ transport (Nick Craig-Wood)
+ - Fix shared_credentials_file auth (Dmitry Chepurovskiy)
+ - Add --s3-no-head to reducing costs docs (Nick Craig-Wood)
+- Union
+ - Fix mkdir at root with remote:/ (Nick Craig-Wood)
+- Zoho
+ - Fix custom client id's (buengese)
+
+
v1.54.0 - 2021-02-02
See commits
- New backends
- - Compression remote (experimental)(buengese)
+ - Compression remote (experimental) (buengese)
- Enterprise File Fabric (Nick Craig-Wood)
- This work was sponsored by Storage Made Easy
- HDFS (Hadoop Distributed File System) (Yury Stankevich)
@@ -27720,37 +29094,30 @@ See commits
- New Features
- Deglobalise the config (Nick Craig-Wood)
- Global config now read from the context
- - Global config can be passed into the rc
+ - This will enable passing of global config via the rc
- This work was sponsored by Digitalis
- Add --bwlimit for upload and download (Nick Craig-Wood)
- Obey bwlimit in http Transport for better limiting
- Enhance systemd integration (Hekmon)
- - log level identification
- - manual activation with flag
- - automatic systemd launch detection
+ - log level identification, manual activation with flag,
+ automatic systemd launch detection
- Don't compile systemd log integration for non unix systems
(Benjamin Gustin)
- - Add a download flag to hashsum and related commands to force
- rclone to download and hash files locally (lostheli)
+ - Add a --download flag to md5sum/sha1sum/hashsum to force rclone
+ to download and hash files locally (lostheli)
+ - Add --progress-terminal-title to print ETA to terminal title
+ (LaSombra)
+ - Make backend env vars show in help as the defaults for backend
+ flags (Nick Craig-Wood)
- build
- Raise minimum go version to go1.12 (Nick Craig-Wood)
- - check
- - Make the error count match up in the log message (Nick
- Craig-Wood)
- - cmd
- - Add --progress-terminal-title to print ETA to terminal title
- (LaSombra)
- - Make backend env vars show in help as the defaults for
- backend flags (Nick Craig-Wood)
- dedupe
- - Add --by-hash to dedupe on hash not file name (Nick
+ - Add --by-hash to dedupe on content hash not file name (Nick
Craig-Wood)
- Add --dedupe-mode list to just list dupes, changing nothing
(Nick Craig-Wood)
- Add warning if used on a remote which can't have duplicate
names (Nick Craig-Wood)
- - flags: Improve error message when reading environment vars (Nick
- Craig-Wood)
- fs
- Add Shutdown optional method for backends (Nick Craig-Wood)
- When using --files-from check files concurrently (zhucan)
@@ -27764,7 +29131,7 @@ See commits
- Highlight read errors instead of aborting (Claudio
Bantaloukas)
- Add sort by average size in directory (Adam Plánský)
- - Add toggle option for average size in directory - key 'a'
+ - Add toggle option for average s3ize in directory - key 'a'
(Adam Plánský)
- Add empty folder flag into ncdu browser (Adam Plánský)
- Add ! (errror) and . (unreadable) file flags to go with e
@@ -27794,22 +29161,17 @@ See commits
(Chaitanya Bankanhal)
- Fix plugins initialization (negative0)
- Bug Fixes
- - build
- - Explicitly set ARM version to fix build (Nick Craig-Wood)
- - Don't explicitly set ARM version to fix ARMv5 build (Nick
- Craig-Wood)
- - Fix nfpm install (Nick Craig-Wood)
- - Fix docker build by upgrading ilteoood/docker_buildx (Nick
- Craig-Wood)
- - Temporary fix for Windows build errors (Ivan Andreev)
- fs
- Fix nil pointer on copy & move operations directly to remote
(Anagh Kumar Baranwal)
- Fix parsing of .. when joining remotes (Nick Craig-Wood)
- log: Fix enabling systemd logging when using --log-file (Nick
Craig-Wood)
- - move: Fix data loss when moving the same object (Nick
- Craig-Wood)
+ - check
+ - Make the error count match up in the log message (Nick
+ Craig-Wood)
+ - move: Fix data loss when source and destination are the same
+ object (Nick Craig-Wood)
- operations
- Fix --cutof-mode hard not cutting off immediately (Nick
Craig-Wood)
@@ -27820,7 +29182,7 @@ See commits
- Fix --immutable errors retrying many times (Nick Craig-Wood)
- Docs
- Many fixes and a rewrite of the filtering docs (edwardxml)
- - Many spelling and grammar problems (Josh Soref)
+ - Many spelling and grammar fixes (Josh Soref)
- Doc fixes for commands delete, purge, rmdir, rmdirs and mount
(albertony)
- And thanks to these people for many doc fixes too numerous to
@@ -27834,15 +29196,12 @@ See commits
- Mount
- Update systemd status with cache stats (Hekmon)
- Disable bazil/fuse based mount on macOS (Nick Craig-Wood)
- - Make mount be cmount under macOS (Nick Craig-Wood)
+ - Make rclone mount actually run rclone cmount under macOS
+ (Nick Craig-Wood)
- Implement mknod to make NFS file creation work (Nick Craig-Wood)
- Make sure we don't call umount more than once (Nick Craig-Wood)
- - Don't call host.Umount if a signal has been received (Nick
- Craig-Wood)
- More user friendly mounting as network drive on windows
(albertony)
- - Cleanup OS specific option handling and documentation
- (albertony)
- Detect if uid or gid are set in same option string: -o
uid=123,gid=456 (albertony)
- Don't attempt to unmount if fs has been destroyed already (Nick
@@ -27863,37 +29222,37 @@ See commits
filesystems (Riccardo Iaconelli)
- Azure Blob
- Add support for service principals (James Lim)
- - Utilize streaming capabilities (Denis Neuling)
- - Update SDK to v0.13.0 and fix API breakage (Nick Craig-Wood,
- Mitsuo Heijo)
+ - Add support for managed identities (Brad Ackerman)
+ - Add examples for access tier (Bob Pusateri)
+ - Utilize the streaming capabilities from the SDK for multipart
+ uploads (Denis Neuling)
- Fix setting of mime types (Nick Craig-Wood)
- Fix crash when listing outside a SAS URL's root (Nick
Craig-Wood)
- Delete archive tier blobs before update if
--azureblob-archive-tier-delete (Nick Craig-Wood)
- - Add support for managed identities (Brad Ackerman)
- Fix crash on startup (Nick Craig-Wood)
- - Add examples for access tier (Bob Pusateri)
- - Fix memory usage by upgrading the SDK and implementing a
- TransferManager (Nick Craig-Wood)
+ - Fix memory usage by upgrading the SDK to v0.13.0 and
+ implementing a TransferManager (Nick Craig-Wood)
- Require go1.14+ to compile due to SDK changes (Nick Craig-Wood)
- B2
- Make NewObject use less expensive API calls (Nick Craig-Wood)
- - Fixed possible crash when accessing Backblaze b2 remote
- (lluuaapp)
+ - This will improve --files-from and restic serve in
+ particular
+ - Fixed crash on an empty file name (lluuaapp)
- Box
- Fix NewObject for files that differ in case (Nick Craig-Wood)
- Fix finding directories in a case insentive way (Nick
Craig-Wood)
- Chunker
- Skip long local hashing, hash in-transit (fixes) (Ivan Andreev)
- - Set Features.ReadMimeType=false as Object.MimeType not supported
- (Nick Craig-Wood)
+ - Set Features ReadMimeType to false as Object.MimeType not
+ supported (Nick Craig-Wood)
- Fix case-insensitive NewObject, test metadata detection (Ivan
Andreev)
- Drive
- - Implement "rclone backend copyid" command for copying files by
- ID (Nick Craig-Wood)
+ - Implement rclone backend copyid command for copying files by ID
+ (Nick Craig-Wood)
- Added flag --drive-stop-on-download-limit to stop transfers when
the download limit is exceeded (Anagh Kumar Baranwal)
- Implement CleanUp workaround for team drives (buengese)
@@ -27904,18 +29263,18 @@ See commits
Rodriguez-Estivill)
- Dropbox
- Add support for viewing shared files and folders (buengese)
- - Implement IDer (buengese)
- - Set Features.ReadMimeType=false as Object.MimeType not supported
- (Nick Craig-Wood)
- - Tidy repeated error message (Nick Craig-Wood)
+ - Enable short lived access tokens (Nick Craig-Wood)
+ - Implement IDer on Objects so rclone lsf etc can read the IDs
+ (buengese)
+ - Set Features ReadMimeType to false as Object.MimeType not
+ supported (Nick Craig-Wood)
- Make malformed_path errors from too long files not retriable
(Nick Craig-Wood)
- Test file name length before upload to fix upload loop (Nick
Craig-Wood)
- - Enable short lived access tokens (Nick Craig-Wood)
- Fichier
- - Set Features.ReadMimeType=true as Object.MimeType is supported
- (Nick Craig-Wood)
+ - Set Features ReadMimeType to true as Object.MimeType is
+ supported (Nick Craig-Wood)
- FTP
- Add --ftp-disable-msld option to ignore MLSD for really old
servers (Nick Craig-Wood)
@@ -27924,39 +29283,40 @@ See commits
- Storage class object header support (Laurens Janssen)
- Fix anonymous client to use rclone's HTTP client (Nick
Craig-Wood)
- - Fix Entry doesn't belong in directory "" (same as directory) -
- ignoring (Nick Craig-Wood)
+ - Fix
+ Entry doesn't belong in directory "" (same as directory) - ignoring
+ (Nick Craig-Wood)
- Googlephotos
- - New flag --gphotos-include-archived (Nicolas Rueff)
+ - New flag --gphotos-include-archived to show archived photos as
+ well (Nicolas Rueff)
- Jottacloud
- - Don't erroniously report support for writing mime types
+ - Don't erroneously report support for writing mime types
(buengese)
- - Add support for Telia Cloud (#4930) (Patrik Nordlén)
+ - Add support for Telia Cloud (Patrik Nordlén)
- Mailru
+ - Accept special folders eg camera-upload (Ivan Andreev)
+ - Avoid prehashing of large local files (Ivan Andreev)
- Fix uploads after recent changes on server (Ivan Andreev)
- Fix range requests after June 2020 changes on server (Ivan
Andreev)
- Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
- Remove deprecated protocol quirks (Ivan Andreev)
- - Accept special folders eg camera-upload (Ivan Andreev)
- - Avoid prehashing of large local files (Ivan Andreev)
- Memory
- Fix setting of mime types (Nick Craig-Wood)
- Onedrive
- - Add support for china region operated by 21vianet and other
- regional suppliers (#4963) (NyaMisty)
+ - Add support for China region operated by 21vianet and other
+ regional suppliers (NyaMisty)
- Warn on gateway timeout errors (Nick Craig-Wood)
- - Fall back to normal copy if server-side copy unavailable (#4903)
- (Alex Chen)
+ - Fall back to normal copy if server-side copy unavailable (Alex
+ Chen)
- Fix server-side copy completely disabled on OneDrive for
Business (Cnly)
- (business only) workaround to replace existing file on
- server-side copy (#4904) (Alex Chen)
+ server-side copy (Alex Chen)
- Enhance link creation with expiry, scope, type and password
(Nick Craig-Wood)
- - Remove % and # from the set of encoded characters (#4909) (Alex
- Chen)
- - Support addressing site by server-relative URL (#4761) (kice)
+ - Remove % and # from the set of encoded characters (Alex Chen)
+ - Support addressing site by server-relative URL (kice)
- Opendrive
- Fix finding directories in a case insensitive way (Nick
Craig-Wood)
@@ -27972,18 +29332,18 @@ See commits
- Added --s3-disable-http2 to disable http/2 (Anagh Kumar
Baranwal)
- Complete SSE-C implementation (Nick Craig-Wood)
- - Fix hashes on small files with AWS:KMS and SSE-C (Nick
- Craig-Wood)
- - Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C (Nick
- Craig-Wood)
+ - Fix hashes on small files with AWS:KMS and SSE-C (Nick
+ Craig-Wood)
+ - Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C
+ (Nick Craig-Wood)
+ - Add --s3-no-head parameter to minimise transactions on upload
+ (Nick Craig-Wood)
- Update docs with a Reducing Costs section (Nick Craig-Wood)
- Added error handling for error code 429 indicating too many
requests (Anagh Kumar Baranwal)
- Add requester pays option (kelv)
- Fix copy multipart with v2 auth failing with
'SignatureDoesNotMatch' (Louis Koo)
- - Add --s3-no-head parameter to minimise transactions on upload
- (Nick Craig-Wood)
- SFTP
- Allow cert based auth via optional pubkey (Stephen Harris)
- Allow user to optionally check server hosts key to add security
@@ -27994,7 +29354,8 @@ See commits
- Implement Shutdown method (Nick Craig-Wood)
- Implement keyboard interactive authentication (Nick Craig-Wood)
- Make --tpslimit apply (Nick Craig-Wood)
- - Implement --sftp-use-fstat (Nick Craig-Wood)
+ - Implement --sftp-use-fstat for unusual SFTP servers (Nick
+ Craig-Wood)
- Sugarsync
- Fix NewObject for files that differ in case (Nick Craig-Wood)
- Fix finding directories in a case insentive way (Nick
@@ -28010,7 +29371,7 @@ See commits
- Updated docs to show streaming to nextcloud is working (Durval
Menezes)
- Yandex
- - Set Features.WriteMimeType=false as Yandex ignores mime types
+ - Set Features WriteMimeType to false as Yandex ignores mime types
(Nick Craig-Wood)
@@ -32593,6 +33954,28 @@ email addresses removed from here need to be addeed to bin/.ignore-emails to mak
- Nicolas Rueff nicolas@rueff.fr
- Pau Rodriguez-Estivill prodrigestivill@gmail.com
- Bob Pusateri BobPusateri@users.noreply.github.com
+- Alex JOST 25005220+dimejo@users.noreply.github.com
+- Alexey Tabakman samosad.ru@gmail.com
+- David Sze sze.david@gmail.com
+- cynthia kwok cynthia.m.kwok@gmail.com
+- Ankur Gupta agupta@egnyte.com
+- Miron Veryanskiy MironVeryanskiy@gmail.com
+- K265 k.265@qq.com
+- Vesnyx Vesnyx@users.noreply.github.com
+- Dmitry Chepurovskiy me@dm3ch.net
+- Rauno Ots rauno.ots@cgi.com
+- Georg Neugschwandtner georg.neugschwandtner@gmx.net
+- pvalls polvallsrue@gmail.com
+- Robert Thomas 31854736+wolveix@users.noreply.github.com
+- Romeo Kienzler romeo.kienzler@gmail.com
+- tYYGH tYYGH@users.noreply.github.com
+- georne 77802995+georne@users.noreply.github.com
+- Maxwell Calman mcalman@MacBook-Pro.local
+- Naveen Honest Raj naveendurai19@gmail.com
+- Lucas Messenger lmesseng@cisco.com
+- Manish Kumar krmanish260@gmail.com
+- x0b x0bdev@gmail.com
+- CERN through the CS3MESH4EOSC Project
diff --git a/docs/content/authors.md b/docs/content/authors.md
index 4710e188e..bdc0022f0 100644
--- a/docs/content/authors.md
+++ b/docs/content/authors.md
@@ -477,3 +477,4 @@ put them back in again.` >}}
* Lucas Messenger
* Manish Kumar
* x0b
+ * CERN through the CS3MESH4EOSC Project
diff --git a/docs/content/azureblob.md b/docs/content/azureblob.md
index 8158383ff..842e87596 100644
--- a/docs/content/azureblob.md
+++ b/docs/content/azureblob.md
@@ -392,6 +392,22 @@ See: the [encoding section in the overview](/overview/#encoding) for more info.
- Type: MultiEncoder
- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
+#### --azureblob-public-access
+
+Public access level of a container: blob, container.
+
+- Config: public_access
+- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
+- Type: string
+- Default: ""
+- Examples:
+ - ""
+ - The container and its blobs can be accessed only with an authorized request. It's a default value
+ - "blob"
+ - Blob data within this container can be read via anonymous request.
+ - "container"
+ - Allow full public read access for container and blob data.
+
{{< rem autogenerated options stop >}}
### Limitations ###
diff --git a/docs/content/changelog.md b/docs/content/changelog.md
index 20b09eb37..3f8f54b28 100644
--- a/docs/content/changelog.md
+++ b/docs/content/changelog.md
@@ -5,6 +5,160 @@ description: "Rclone Changelog"
# Changelog
+## v1.55.0 - 2021-03-31
+
+[See commits](https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)
+
+* New commands
+ * [selfupdate](/commands/rclone_selfupdate/) (Ivan Andreev)
+ * Allows rclone to update itself in-place or via a package (using `--package` flag)
+ * Reads cryptographically signed signatures for non beta releases
+ * Works on all OSes.
+ * [test](/commands/rclone_test/) - these are test commands - use with care!
+ * `histogram` - Makes a histogram of file name characters.
+ * `info` - Discovers file name or other limitations for paths.
+ * `makefiles` - Make a random file hierarchy for testing.
+ * `memory` - Load all the objects at remote:path into memory and report memory stats.
+* New Features
+ * [Connection strings](/docs/#connection-strings)
+ * Config parameters can now be passed as part of the remote name as a connection string.
+ * For example to do the equivalent of `--drive-shared-with-me` use `drive,shared_with_me:`
+ * Make sure we don't save on the fly remote config to the config file (Nick Craig-Wood)
+ * Make sure backends with additional config have a different name for caching (Nick Craig-Wood)
+ * This work was sponsored by CERN, through the [CS3MESH4EOSC Project](https://cs3mesh4eosc.eu/).
+ * CS3MESH4EOSC has received funding from the European Union’s Horizon 2020
+ * research and innovation programme under Grant Agreement no. 863353.
+ * build
+ * Update go build version to go1.16 and raise minimum go version to go1.13 (Nick Craig-Wood)
+ * Make a macOS ARM64 build to support Apple Silicon (Nick Craig-Wood)
+ * Install macfuse 4.x instead of osxfuse 3.x (Nick Craig-Wood)
+ * Use `GO386=softfloat` instead of deprecated `GO386=387` for 386 builds (Nick Craig-Wood)
+ * Disable IOS builds for the time being (Nick Craig-Wood)
+ * Androids builds made with up to date NDK (x0b)
+ * Add an rclone user to the Docker image but don't use it by default (cynthia kwok)
+ * dedupe: Make largest directory primary to minimize data moved (Saksham Khanna)
+ * config
+ * Wrap config library in an interface (Fionera)
+ * Make config file system pluggable (Nick Craig-Wood)
+ * `--config ""` or `"/notfound"` for in memory config only (Nick Craig-Wood)
+ * Clear fs cache of stale entries when altering config (Nick Craig-Wood)
+ * copyurl: Add option to print resulting auto-filename (albertony)
+ * delete: Make `--rmdirs` obey the filters (Nick Craig-Wood)
+ * docs - many fixes and reworks from edwardxml, albertony, pvalls, Ivan Andreev, Evan Harris, buengese, Alexey Tabakman
+ * encoder/filename - add SCSU as tables (Klaus Post)
+ * Add multiple paths support to `--compare-dest` and `--copy-dest` flag (K265)
+ * filter: Make `--exclude "dir/"` equivalent to `--exclude "dir/**"` (Nick Craig-Wood)
+ * fshttp: Add DSCP support with `--dscp` for QoS with differentiated services (Max Sum)
+ * lib/cache: Add Delete and DeletePrefix methods (Nick Craig-Wood)
+ * lib/file
+ * Make pre-allocate detect disk full errors and return them (Nick Craig-Wood)
+ * Don't run preallocate concurrently (Nick Craig-Wood)
+ * Retry preallocate on EINTR (Nick Craig-Wood)
+ * operations: Made copy and sync operations obey a RetryAfterError (Ankur Gupta)
+ * rc
+ * Add string alternatives for setting options over the rc (Nick Craig-Wood)
+ * Add `options/local` to see the options configured in the context (Nick Craig-Wood)
+ * Add `_config` parameter to set global config for just this rc call (Nick Craig-Wood)
+ * Implement passing filter config with `_filter` parameter (Nick Craig-Wood)
+ * Add `fscache/clear` and `fscache/entries` to control the fs cache (Nick Craig-Wood)
+ * Avoid +Inf value for speed in `core/stats` (albertony)
+ * Add a full set of stats to `core/stats` (Nick Craig-Wood)
+ * Allow `fs=` params to be a JSON blob (Nick Craig-Wood)
+ * rcd: Added systemd notification during the `rclone rcd` command. (Naveen Honest Raj)
+ * rmdirs: Make `--rmdirs` obey the filters (Nick Craig-Wood)
+ * version: Show build tags and type of executable (Ivan Andreev)
+* Bug Fixes
+ * install.sh: make it fail on download errors (Ivan Andreev)
+ * Fix excessive retries missing `--max-duration` timeout (Nick Craig-Wood)
+ * Fix crash when `--low-level-retries=0` (Nick Craig-Wood)
+ * Fix failed token refresh on mounts created via the rc (Nick Craig-Wood)
+ * fshttp: Fix bandwidth limiting after bad merge (Nick Craig-Wood)
+ * lib/atexit
+ * Unregister interrupt handler once it has fired so users can interrupt again (Nick Craig-Wood)
+ * Fix occasional failure to unmount with CTRL-C (Nick Craig-Wood)
+ * Fix deadlock calling Finalise while Run is running (Nick Craig-Wood)
+ * lib/rest: Fix multipart uploads not stopping on context cancel (Nick Craig-Wood)
+* Mount
+ * Allow mounting to root directory on windows (albertony)
+ * Improved handling of relative paths on windows (albertony)
+ * Fix unicode issues with accented characters on macOS (Nick Craig-Wood)
+ * Docs: document the new FileSecurity option in WinFsp 2021 (albertony)
+ * Docs: add note about volume path syntax on windows (albertony)
+ * Fix caching of old directories after renaming them (Nick Craig-Wood)
+ * Update cgofuse to the latest version to bring in macfuse 4 fix (Nick Craig-Wood)
+* VFS
+ * `--vfs-used-is-size` to report used space using recursive scan (tYYGH)
+ * Don't set modification time if it was already correct (Nick Craig-Wood)
+ * Fix Create causing windows explorer to truncate files on CTRL-C CTRL-V (Nick Craig-Wood)
+ * Fix modtimes not updating when writing via cache (Nick Craig-Wood)
+ * Fix modtimes changing by fractional seconds after upload (Nick Craig-Wood)
+ * Fix modtime set if `--vfs-cache-mode writes`/`full` and no write (Nick Craig-Wood)
+ * Rename files in cache and cancel uploads on directory rename (Nick Craig-Wood)
+ * Fix directory renaming by renaming dirs cached in memory (Nick Craig-Wood)
+* Local
+ * Add flag `--local-no-preallocate` (David Sze)
+ * Make `nounc` an advanced option except on Windows (albertony)
+ * Don't ignore preallocate disk full errors (Nick Craig-Wood)
+* Cache
+ * Add `--fs-cache-expire-duration` to control the fs cache (Nick Craig-Wood)
+* Crypt
+ * Add option to not encrypt data (Vesnyx)
+ * Log hash ok on upload (albertony)
+* Azure Blob
+ * Add container public access level support. (Manish Kumar)
+* B2
+ * Fix HTML files downloaded via cloudflare (Nick Craig-Wood)
+* Box
+ * Fix transfers getting stuck on token expiry after API change (Nick Craig-Wood)
+* Chunker
+ * Partially implement no-rename transactions (Maxwell Calman)
+* Drive
+ * Don't stop server side copy if couldn't read description (Nick Craig-Wood)
+ * Pass context on to drive SDK - to help with cancellation (Nick Craig-Wood)
+* Dropbox
+ * Add polling for changes support (Robert Thomas)
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+ * Raise priority of rate limited message to INFO to make it more noticeable (Nick Craig-Wood)
+* Fichier
+ * Implement copy & move (buengese)
+ * Implement public link (buengese)
+* FTP
+ * Implement Shutdown method (Nick Craig-Wood)
+ * Close idle connections after `--ftp-idle-timeout` (1m by default) (Nick Craig-Wood)
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+ * Add `--ftp-close-timeout` flag for use with awkward ftp servers (Nick Craig-Wood)
+ * Retry connections and logins on 421 errors (Nick Craig-Wood)
+* Hdfs
+ * Fix permissions for when directory is created (Lucas Messenger)
+* Onedrive
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+* S3
+ * Fix `--s3-profile` which wasn't working (Nick Craig-Wood)
+* SFTP
+ * Close idle connections after `--sftp-idle-timeout` (1m by default) (Nick Craig-Wood)
+ * Fix "file not found" errors for read once servers (Nick Craig-Wood)
+ * Fix SetModTime stat failed: object not found with `--sftp-set-modtime=false` (Nick Craig-Wood)
+* Swift
+ * Update github.com/ncw/swift to v2.0.0 (Nick Craig-Wood)
+ * Implement copying large objects (nguyenhuuluan434)
+* Union
+ * Fix crash when using epff policy (Nick Craig-Wood)
+ * Fix union attempting to update files on a read only file system (Nick Craig-Wood)
+ * Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick Craig-Wood)
+ * Fix initialisation broken in refactor (Nick Craig-Wood)
+* WebDAV
+ * Add support for sharepoint with NTLM authentication (Rauno Ots)
+ * Make sharepoint-ntlm docs more consistent (Alex Chen)
+ * Improve terminology in sharepoint-ntlm docs (Ivan Andreev)
+ * Disable HTTP/2 for NTLM authentication (georne)
+ * Fix sharepoint-ntlm error 401 for parallel actions (Ivan Andreev)
+ * Check that purged directory really exists (Ivan Andreev)
+* Yandex
+ * Make `--timeout 0` work properly (Nick Craig-Wood)
+* Zoho
+ * Replace client id - you will need to `rclone config reconnect` after this (buengese)
+ * Add forgotten setupRegion() to NewFs - this finally fixes regions other than EU (buengese)
+
## v1.54.1 - 2021-03-08
[See commits](https://github.com/rclone/rclone/compare/v1.54.0...v1.54.1)
diff --git a/docs/content/chunker.md b/docs/content/chunker.md
index a304e8a99..710df3a3a 100644
--- a/docs/content/chunker.md
+++ b/docs/content/chunker.md
@@ -416,4 +416,27 @@ Choose how chunker should handle files with missing or invalid chunks.
- "false"
- Warn user, skip incomplete file and proceed.
+#### --chunker-transactions
+
+Choose how chunker should handle temporary files during transactions.
+
+- Config: transactions
+- Env Var: RCLONE_CHUNKER_TRANSACTIONS
+- Type: string
+- Default: "rename"
+- Examples:
+ - "rename"
+ - Rename temporary files after a successful transaction.
+ - "norename"
+ - Leave temporary file names and write transaction ID to metadata file.
+ - Metadata is required for no rename transactions (meta format cannot be "none").
+ - If you are using norename transactions you should be careful not to downgrade Rclone
+ - as older versions of Rclone don't support this transaction style and will misinterpret
+ - files manipulated by norename transactions.
+ - This method is EXPERIMENTAL, don't use on production systems.
+ - "auto"
+ - Rename or norename will be used depending on capabilities of the backend.
+ - If meta format is set to "none", rename transactions will always be used.
+ - This method is EXPERIMENTAL, don't use on production systems.
+
{{< rem autogenerated options stop >}}
diff --git a/docs/content/commands/rclone.md b/docs/content/commands/rclone.md
index a59739f60..071139d0f 100644
--- a/docs/content/commands/rclone.md
+++ b/docs/content/commands/rclone.md
@@ -72,11 +72,13 @@ See the [global flags page](/flags/) for global options not listed here.
* [rclone rcd](/commands/rclone_rcd/) - Run rclone listening to remote control commands only.
* [rclone rmdir](/commands/rclone_rmdir/) - Remove the empty directory at path.
* [rclone rmdirs](/commands/rclone_rmdirs/) - Remove empty directories under the path.
+* [rclone selfupdate](/commands/rclone_selfupdate/) - Update the rclone binary.
* [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol.
* [rclone settier](/commands/rclone_settier/) - Changes storage class/tier of objects in remote.
* [rclone sha1sum](/commands/rclone_sha1sum/) - Produces an sha1sum file for all the objects in the path.
* [rclone size](/commands/rclone_size/) - Prints the total size and number of objects in remote:path.
* [rclone sync](/commands/rclone_sync/) - Make source and dest identical, modifying destination only.
+* [rclone test](/commands/rclone_test/) - Run a test command
* [rclone touch](/commands/rclone_touch/) - Create new file or change file modification time.
* [rclone tree](/commands/rclone_tree/) - List the contents of the remote in a tree like fashion.
* [rclone version](/commands/rclone_version/) - Show the version number.
diff --git a/docs/content/commands/rclone_copyurl.md b/docs/content/commands/rclone_copyurl.md
index 6f3c6e52c..9d26b1958 100644
--- a/docs/content/commands/rclone_copyurl.md
+++ b/docs/content/commands/rclone_copyurl.md
@@ -15,15 +15,16 @@ Copy url content to dest.
Download a URL's content and copy it to the destination without saving
it in temporary storage.
-Setting --auto-filename will cause the file name to be retrieved from
+Setting `--auto-filename`will cause the file name to be retrieved from
the from URL (after any redirections) and used in the destination
-path.
+path. With `--print-filename` in addition, the resuling file name will
+be printed.
-Setting --no-clobber will prevent overwriting file on the
+Setting `--no-clobber` will prevent overwriting file on the
destination if there is one with the same name.
-Setting --stdout or making the output file name "-" will cause the
-output to be written to standard output.
+Setting `--stdout` or making the output file name `-`
+will cause the output to be written to standard output.
```
@@ -33,10 +34,11 @@ rclone copyurl https://example.com dest:path [flags]
## Options
```
- -a, --auto-filename Get the file name from the URL and use it for destination file path
- -h, --help help for copyurl
- --no-clobber Prevent overwriting file with same name
- --stdout Write the output to stdout rather than a file
+ -a, --auto-filename Get the file name from the URL and use it for destination file path
+ -h, --help help for copyurl
+ --no-clobber Prevent overwriting file with same name
+ -p, --print-filename Print the resulting name from --auto-filename
+ --stdout Write the output to stdout rather than a file
```
See the [global flags page](/flags/) for global options not listed here.
diff --git a/docs/content/commands/rclone_dedupe.md b/docs/content/commands/rclone_dedupe.md
index 8b201dda5..8a997195f 100644
--- a/docs/content/commands/rclone_dedupe.md
+++ b/docs/content/commands/rclone_dedupe.md
@@ -17,8 +17,8 @@ By default `dedupe` interactively finds files with duplicate
names and offers to delete all but one or rename them to be
different. This is known as deduping by name.
-Deduping by name is only useful with backends like Google Drive which
-can have duplicate file names. It can be run on wrapping backends
+Deduping by name is only useful with a small group of backends (e.g. Google Drive,
+Opendrive) that can have duplicate file names. It can be run on wrapping backends
(e.g. crypt) if they wrap a backend which supports duplicate file
names.
diff --git a/docs/content/commands/rclone_mount.md b/docs/content/commands/rclone_mount.md
index 0b0b7bb39..511711d35 100644
--- a/docs/content/commands/rclone_mount.md
+++ b/docs/content/commands/rclone_mount.md
@@ -29,15 +29,15 @@ is an **empty** **existing** directory:
On Windows you can start a mount in different ways. See [below](#mounting-modes-on-windows)
for details. The following examples will mount to an automatically assigned drive,
-to specific drive letter `X:`, to path `C:\path\to\nonexistent\directory`
-(which must be **non-existent** subdirectory of an **existing** parent directory or drive,
+to specific drive letter `X:`, to path `C:\path\parent\mount`
+(where parent directory or drive must exist, and mount must **not** exist,
and is not supported when [mounting as a network drive](#mounting-modes-on-windows)), and
the last example will mount as network share `\\cloud\remote` and map it to an
automatically assigned drive:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
- rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+ rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files \\cloud\remote
When the program ends while in foreground mode, either via Ctrl+C or receiving
@@ -91,14 +91,14 @@ and experience unexpected program errors, freezes or other issues, consider moun
as a network drive instead.
When mounting as a fixed disk drive you can either mount to an unused drive letter,
-or to a path - which must be **non-existent** subdirectory of an **existing** parent
+or to a path representing a **non-existent** subdirectory of an **existing** parent
directory or drive. Using the special value `*` will tell rclone to
automatically assign the next available drive letter, starting with Z: and moving backward.
Examples:
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
- rclone mount remote:path/to/files C:\path\to\nonexistent\directory
+ rclone mount remote:path/to/files C:\path\parent\mount
rclone mount remote:path/to/files X:
Option `--volname` can be used to set a custom volume name for the mounted
@@ -171,10 +171,24 @@ Note that the mapping of permissions is not always trivial, and the result
you see in Windows Explorer may not be exactly like you expected.
For example, when setting a value that includes write access, this will be
mapped to individual permissions "write attributes", "write data" and "append data",
-but not "write extended attributes" (WinFsp does not support extended attributes,
-see [this](https://github.com/billziss-gh/winfsp/wiki/NTFS-Compatibility)).
-Windows will then show this as basic permission "Special" instead of "Write",
-because "Write" includes the "write extended attributes" permission.
+but not "write extended attributes". Windows will then show this as basic
+permission "Special" instead of "Write", because "Write" includes the
+"write extended attributes" permission.
+
+If you set POSIX permissions for only allowing access to the owner, using
+`--file-perms 0600 --dir-perms 0700`, the user group and the built-in "Everyone"
+group will still be given some special permissions, such as "read attributes"
+and "read permissions", in Windows. This is done for compatibility reasons,
+e.g. to allow users without additional permissions to be able to read basic
+metadata about files like in UNIX. One case that may arise is that other programs
+(incorrectly) interprets this as the file being accessible by everyone. For example
+an SSH client may warn about "unprotected private key file".
+
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity",
+that allows the complete specification of file security descriptors using
+[SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
+With this you can work around issues such as the mentioned "unprotected private key file"
+by specifying `-o FileSecurity="D:P(A;;FA;;;OW)"`, for file all access (FA) to the owner (OW).
### Windows caveats
@@ -378,6 +392,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -521,6 +542,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
```
rclone mount remote:path /path/to/mountpoint [flags]
@@ -565,6 +599,7 @@ rclone mount remote:path /path/to/mountpoint [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
--volname string Set the volume name. Supported on Windows and OSX only.
diff --git a/docs/content/commands/rclone_selfupdate.md b/docs/content/commands/rclone_selfupdate.md
new file mode 100644
index 000000000..ee8ac83f4
--- /dev/null
+++ b/docs/content/commands/rclone_selfupdate.md
@@ -0,0 +1,84 @@
+---
+title: "rclone selfupdate"
+description: "Update the rclone binary."
+slug: rclone_selfupdate
+url: /commands/rclone_selfupdate/
+# autogenerated - DO NOT EDIT, instead edit the source code in cmd/selfupdate/ and as part of making a release run "make commanddocs"
+---
+# rclone selfupdate
+
+Update the rclone binary.
+
+## Synopsis
+
+
+This command downloads the latest release of rclone and replaces
+the currently running binary. The download is verified with a hashsum
+and cryptographically signed signature.
+
+If used without flags (or with implied `--stable` flag), this command
+will install the latest stable release. However, some issues may be fixed
+(or features added) only in the latest beta release. In such cases you should
+run the command with the `--beta` flag, i.e. `rclone selfupdate --beta`.
+You can check in advance what version would be installed by adding the
+`--check` flag, then repeat the command without it when you are satisfied.
+
+Sometimes the rclone team may recommend you a concrete beta or stable
+rclone release to troubleshoot your issue or add a bleeding edge feature.
+The `--version VER` flag, if given, will update to the concrete version
+instead of the latest one. If you omit micro version from `VER` (for
+example `1.53`), the latest matching micro version will be used.
+
+Upon successful update rclone will print a message that contains a previous
+version number. You will need it if you later decide to revert your update
+for some reason. Then you'll have to note the previous version and run the
+following command: `rclone selfupdate [--beta] OLDVER`.
+If the old version contains only dots and digits (for example `v1.54.0`)
+then it's a stable release so you won't need the `--beta` flag. Beta releases
+have an additional information similar to `v1.54.0-beta.5111.06f1c0c61`.
+(if you are a developer and use a locally built rclone, the version number
+will end with `-DEV`, you will have to rebuild it as it obvisously can't
+be distributed).
+
+If you previously installed rclone via a package manager, the package may
+include local documentation or configure services. You may wish to update
+with the flag `--package deb` or `--package rpm` (whichever is correct for
+your OS) to update these too. This command with the default `--package zip`
+will update only the rclone executable so the local manual may become
+inaccurate after it.
+
+The `rclone mount` command (https://rclone.org/commands/rclone_mount/) may
+or may not support extended FUSE options depending on the build and OS.
+`selfupdate` will refuse to update if the capability would be discarded.
+
+Note: Windows forbids deletion of a currently running executable so this
+command will rename the old executable to 'rclone.old.exe' upon success.
+
+Please note that this command was not available before rclone version 1.55.
+If it fails for you with the message `unknown command "selfupdate"` then
+you will need to update manually following the install instructions located
+at https://rclone.org/install/
+
+
+```
+rclone selfupdate [flags]
+```
+
+## Options
+
+```
+ --beta Install beta release.
+ --check Check for latest release, do not download.
+ -h, --help help for selfupdate
+ --output string Save the downloaded binary at a given path (default: replace running binary)
+ --package string Package format: zip|deb|rpm (default: zip)
+ --stable Install stable release (this is the default)
+ --version string Install the given rclone version (default: latest)
+```
+
+See the [global flags page](/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
+
diff --git a/docs/content/commands/rclone_serve_dlna.md b/docs/content/commands/rclone_serve_dlna.md
index 16c6eb7c8..2f5213dbe 100644
--- a/docs/content/commands/rclone_serve_dlna.md
+++ b/docs/content/commands/rclone_serve_dlna.md
@@ -134,6 +134,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -277,6 +284,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
```
rclone serve dlna remote:path [flags]
@@ -309,6 +329,7 @@ rclone serve dlna remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
diff --git a/docs/content/commands/rclone_serve_ftp.md b/docs/content/commands/rclone_serve_ftp.md
index 8f665c55b..14e37fc14 100644
--- a/docs/content/commands/rclone_serve_ftp.md
+++ b/docs/content/commands/rclone_serve_ftp.md
@@ -133,6 +133,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -276,6 +283,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
@@ -394,6 +414,7 @@ rclone serve ftp remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
diff --git a/docs/content/commands/rclone_serve_http.md b/docs/content/commands/rclone_serve_http.md
index 5614405f5..c43c095eb 100644
--- a/docs/content/commands/rclone_serve_http.md
+++ b/docs/content/commands/rclone_serve_http.md
@@ -205,6 +205,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -348,6 +355,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
```
rclone serve http remote:path [flags]
@@ -390,6 +410,7 @@ rclone serve http remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
diff --git a/docs/content/commands/rclone_serve_sftp.md b/docs/content/commands/rclone_serve_sftp.md
index f55388d73..2a9332498 100644
--- a/docs/content/commands/rclone_serve_sftp.md
+++ b/docs/content/commands/rclone_serve_sftp.md
@@ -144,6 +144,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -287,6 +294,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
@@ -404,6 +424,7 @@ rclone serve sftp remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
diff --git a/docs/content/commands/rclone_serve_webdav.md b/docs/content/commands/rclone_serve_webdav.md
index b898e03b8..2fd9a6350 100644
--- a/docs/content/commands/rclone_serve_webdav.md
+++ b/docs/content/commands/rclone_serve_webdav.md
@@ -213,6 +213,13 @@ for two reasons. Firstly because it is only checked every
`--vfs-cache-poll-interval`. Secondly because open files cannot be
evicted from the cache.
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
+
### --vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write
@@ -356,6 +363,19 @@ If the flag is not provided on the command line, then its default value depends
on the operating system where rclone runs: "true" on Windows and macOS, "false"
otherwise. If the flag is provided without a value, then it is "true".
+## Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
+
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
+
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
@@ -482,6 +502,7 @@ rclone serve webdav remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size.
--vfs-write-back duration Time to writeback files after last use when using cache. (default 5s)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
```
diff --git a/docs/content/commands/rclone_sync.md b/docs/content/commands/rclone_sync.md
index 1438ebd53..0d3f65982 100644
--- a/docs/content/commands/rclone_sync.md
+++ b/docs/content/commands/rclone_sync.md
@@ -15,7 +15,8 @@ Make source and dest identical, modifying destination only.
Sync the source to the destination, changing the destination
only. Doesn't transfer unchanged files, testing by size and
modification time or MD5SUM. Destination is updated to match
-source, including deleting files if necessary.
+source, including deleting files if necessary (except duplicate
+objects, see below).
**Important**: Since this can cause data loss, test first with the
`--dry-run` or the `--interactive`/`-i` flag.
@@ -23,7 +24,8 @@ source, including deleting files if necessary.
rclone sync -i SOURCE remote:DESTINATION
Note that files in the destination won't be deleted if there were any
-errors at any point.
+errors at any point. Duplicate objects (files with the same name, on
+those providers that support it) are also not yet handled.
It is always the contents of the directory that is synced, not the
directory so when source:path is a directory, it's the contents of
@@ -35,6 +37,9 @@ go there.
**Note**: Use the `-P`/`--progress` flag to view real-time transfer statistics
+**Note**: Use the `rclone dedupe` command to deal with "Duplicate object/directory found in source/destination - ignoring" errors.
+See [this forum post](https://forum.rclone.org/t/sync-not-clearing-duplicates/14372) for more info.
+
```
rclone sync source:path dest:path [flags]
diff --git a/docs/content/commands/rclone_test.md b/docs/content/commands/rclone_test.md
new file mode 100644
index 000000000..3ef4b9ac2
--- /dev/null
+++ b/docs/content/commands/rclone_test.md
@@ -0,0 +1,41 @@
+---
+title: "rclone test"
+description: "Run a test command"
+slug: rclone_test
+url: /commands/rclone_test/
+# autogenerated - DO NOT EDIT, instead edit the source code in cmd/test/ and as part of making a release run "make commanddocs"
+---
+# rclone test
+
+Run a test command
+
+## Synopsis
+
+Rclone test is used to run test commands.
+
+Select which test comand you want with the subcommand, eg
+
+ rclone test memory remote:
+
+Each subcommand has its own options which you can see in their help.
+
+**NB** Be careful running these commands, they may do strange things
+so reading their documentation first is recommended.
+
+
+## Options
+
+```
+ -h, --help help for test
+```
+
+See the [global flags page](/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
+* [rclone test histogram](/commands/rclone_test_histogram/) - Makes a histogram of file name characters.
+* [rclone test info](/commands/rclone_test_info/) - Discovers file name or other limitations for paths.
+* [rclone test makefiles](/commands/rclone_test_makefiles/) - Make a random file hierarchy in
+* [rclone test memory](/commands/rclone_test_memory/) - Load all the objects at remote:path into memory and report memory stats.
+
diff --git a/docs/content/commands/rclone_test_histogram.md b/docs/content/commands/rclone_test_histogram.md
new file mode 100644
index 000000000..b87ac83a0
--- /dev/null
+++ b/docs/content/commands/rclone_test_histogram.md
@@ -0,0 +1,36 @@
+---
+title: "rclone test histogram"
+description: "Makes a histogram of file name characters."
+slug: rclone_test_histogram
+url: /commands/rclone_test_histogram/
+# autogenerated - DO NOT EDIT, instead edit the source code in cmd/test/histogram/ and as part of making a release run "make commanddocs"
+---
+# rclone test histogram
+
+Makes a histogram of file name characters.
+
+## Synopsis
+
+This command outputs JSON which shows the histogram of characters used
+in filenames in the remote:path specified.
+
+The data doesn't contain any identifying information but is useful for
+the rclone developers when developing filename compression.
+
+
+```
+rclone test histogram [remote:path] [flags]
+```
+
+## Options
+
+```
+ -h, --help help for histogram
+```
+
+See the [global flags page](/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](/commands/rclone_test/) - Run a test command
+
diff --git a/docs/content/commands/rclone_test_info.md b/docs/content/commands/rclone_test_info.md
new file mode 100644
index 000000000..bcde8eb9d
--- /dev/null
+++ b/docs/content/commands/rclone_test_info.md
@@ -0,0 +1,44 @@
+---
+title: "rclone test info"
+description: "Discovers file name or other limitations for paths."
+slug: rclone_test_info
+url: /commands/rclone_test_info/
+# autogenerated - DO NOT EDIT, instead edit the source code in cmd/test/info/ and as part of making a release run "make commanddocs"
+---
+# rclone test info
+
+Discovers file name or other limitations for paths.
+
+## Synopsis
+
+rclone info discovers what filenames and upload methods are possible
+to write to the paths passed in and how long they can be. It can take some
+time. It will write test files into the remote:path passed in. It outputs
+a bit of go code for each one.
+
+**NB** this can create undeletable files and other hazards - use with care
+
+
+```
+rclone test info [remote:path]+ [flags]
+```
+
+## Options
+
+```
+ --all Run all tests.
+ --check-control Check control characters.
+ --check-length Check max filename length.
+ --check-normalization Check UTF-8 Normalization.
+ --check-streaming Check uploads with indeterminate file size.
+ -h, --help help for info
+ --upload-wait duration Wait after writing a file.
+ --write-json string Write results to file.
+```
+
+See the [global flags page](/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](/commands/rclone_test/) - Run a test command
+
diff --git a/docs/content/commands/rclone_test_makefiles.md b/docs/content/commands/rclone_test_makefiles.md
new file mode 100644
index 000000000..87b0fa9b6
--- /dev/null
+++ b/docs/content/commands/rclone_test_makefiles.md
@@ -0,0 +1,33 @@
+---
+title: "rclone test makefiles"
+description: "Make a random file hierarchy in "
+slug: rclone_test_makefiles
+url: /commands/rclone_test_makefiles/
+# autogenerated - DO NOT EDIT, instead edit the source code in cmd/test/makefiles/ and as part of making a release run "make commanddocs"
+---
+# rclone test makefiles
+
+Make a random file hierarchy in
+
+```
+rclone test makefiles [flags]
+```
+
+## Options
+
+```
+ --files int Number of files to create (default 1000)
+ --files-per-directory int Average number of files per directory (default 10)
+ -h, --help help for makefiles
+ --max-file-size SizeSuffix Maximum size of files to create (default 100)
+ --max-name-length int Maximum size of file names (default 12)
+ --min-file-size SizeSuffix Minimum size of file to create
+ --min-name-length int Minimum size of file names (default 4)
+```
+
+See the [global flags page](/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](/commands/rclone_test/) - Run a test command
+
diff --git a/docs/content/commands/rclone_test_memory.md b/docs/content/commands/rclone_test_memory.md
new file mode 100644
index 000000000..104e9f24d
--- /dev/null
+++ b/docs/content/commands/rclone_test_memory.md
@@ -0,0 +1,27 @@
+---
+title: "rclone test memory"
+description: "Load all the objects at remote:path into memory and report memory stats."
+slug: rclone_test_memory
+url: /commands/rclone_test_memory/
+# autogenerated - DO NOT EDIT, instead edit the source code in cmd/test/memory/ and as part of making a release run "make commanddocs"
+---
+# rclone test memory
+
+Load all the objects at remote:path into memory and report memory stats.
+
+```
+rclone test memory remote:path [flags]
+```
+
+## Options
+
+```
+ -h, --help help for memory
+```
+
+See the [global flags page](/flags/) for global options not listed here.
+
+## SEE ALSO
+
+* [rclone test](/commands/rclone_test/) - Run a test command
+
diff --git a/docs/content/commands/rclone_version.md b/docs/content/commands/rclone_version.md
index c5599c142..6e7774024 100644
--- a/docs/content/commands/rclone_version.md
+++ b/docs/content/commands/rclone_version.md
@@ -12,14 +12,21 @@ Show the version number.
## Synopsis
-Show the version number, the go version and the architecture.
+Show the rclone version number, the go version, the build target OS and
+architecture, build tags and the type of executable (static or dynamic).
-Eg
+For example:
$ rclone version
- rclone v1.41
- - os/arch: linux/amd64
- - go version: go1.10
+ rclone v1.54
+ - os/type: linux
+ - os/arch: amd64
+ - go/version: go1.16
+ - go/linking: static
+ - go/tags: none
+
+Note: before rclone version 1.55 the os/type and os/arch lines were merged,
+ and the "go/version" line was tagged as "go version".
If you supply the --check flag, then it will do an online check to
compare your version with the latest release and the latest beta.
diff --git a/docs/content/crypt.md b/docs/content/crypt.md
index f96494ba5..3055f8ff9 100644
--- a/docs/content/crypt.md
+++ b/docs/content/crypt.md
@@ -517,6 +517,20 @@ names, or for debugging purposes.
- Type: bool
- Default: false
+#### --crypt-no-data-encryption
+
+Option to either encrypt file data or leave it unencrypted.
+
+- Config: no_data_encryption
+- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Type: bool
+- Default: false
+- Examples:
+ - "true"
+ - Don't encrypt file data, leave it unencrypted.
+ - "false"
+ - Encrypt file data.
+
### Backend commands
Here are the commands specific to the crypt backend.
diff --git a/docs/content/dropbox.md b/docs/content/dropbox.md
index 8b5f5e83d..6acb34b1d 100644
--- a/docs/content/dropbox.md
+++ b/docs/content/dropbox.md
@@ -197,6 +197,21 @@ memory. It can be set smaller if you are tight on memory.
Impersonate this user when using a business account.
+Note that if you want to use impersonate, you should make sure this
+flag is set when running "rclone config" as this will cause rclone to
+request the "members.read" scope which it won't normally. This is
+needed to lookup a members email address into the internal ID that
+dropbox uses in the API.
+
+Using the "members.read" scope will require a Dropbox Team Admin
+to approve during the OAuth flow.
+
+You will have to use your own App (setting your own client_id and
+client_secret) to use this option as currently rclone's default set of
+permissions doesn't include "members.read". This can be added once
+v1.55 or later is in use everywhere.
+
+
- Config: impersonate
- Env Var: RCLONE_DROPBOX_IMPERSONATE
- Type: string
diff --git a/docs/content/flags.md b/docs/content/flags.md
index b1caae010..58cfd3b61 100755
--- a/docs/content/flags.md
+++ b/docs/content/flags.md
@@ -27,10 +27,10 @@ These flags are available for every command.
-c, --checksum Skip based on checksum (if available) & size, not mod-time & size
--client-cert string Client SSL certificate (PEM) for mutual TLS auth
--client-key string Client SSL private key (PEM) for mutual TLS auth
- --compare-dest string Include additional server-side path during comparison.
+ --compare-dest stringArray Include additional comma separated server-side paths during comparison.
--config string Config file. (default "$HOME/.config/rclone/rclone.conf")
--contimeout duration Connect timeout (default 1m0s)
- --copy-dest string Implies --compare-dest but also copies files from path into destination.
+ --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination.
--cpuprofile string Write cpu profile to file
--cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
--delete-after When synchronizing, delete files on destination after transferring (default)
@@ -39,10 +39,10 @@ These flags are available for every command.
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
-n, --dry-run Do a trial run with no permanent changes
+ --dscp string Set DSCP value to connections. Can be value or names, eg. CS1, LE, DF, AF21.
--dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
--dump-headers Dump HTTP headers - may contain sensitive info
- --dscp DSCP Name or Value (default 0)
--error-on-no-transfer Sets exit code 9 if no files are transferred, useful in scripts
--exclude stringArray Exclude files matching pattern
--exclude-from stringArray Read exclude patterns from file (use - to read from stdin)
@@ -53,6 +53,8 @@ These flags are available for every command.
--files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
-f, --filter stringArray Add a file-filtering rule
--filter-from stringArray Read filtering patterns from a file (use - to read from stdin)
+ --fs-cache-expire-duration duration cache remotes for this long (0 to disable caching) (default 5m0s)
+ --fs-cache-expire-interval duration interval to check for expired remotes (default 1m0s)
--header stringArray Set HTTP header for all transactions
--header-download stringArray Set HTTP header for download transactions
--header-upload stringArray Set HTTP header for upload transactions
@@ -151,7 +153,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
- --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.54.0")
+ --user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
-v, --verbose count Print lots more stuff (repeat for more)
```
@@ -184,6 +186,7 @@ and may be set in the config file.
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_object_id or msi_mi_res_id specified.
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_object_id specified.
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any. Leave blank if msi_client_id or msi_mi_res_id specified.
+ --azureblob-public-access string Public access level of a container: blob, container.
--azureblob-sas-url string SAS URL for container level access only
--azureblob-service-principal-file string Path to file containing credentials for use with a service principal.
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256MB). (Deprecated)
@@ -247,6 +250,7 @@ and may be set in the config file.
-L, --copy-links Follow symlinks and copy the pointed to item.
--crypt-directory-name-encryption Option to either encrypt directory names or leave them intact. (default true)
--crypt-filename-encryption string How to encrypt the filenames. (default "standard")
+ --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted.
--crypt-password string Password or pass phrase for encryption. (obscured)
--crypt-password2 string Password or pass phrase for salt. Optional but recommended. (obscured)
--crypt-remote string Remote to encrypt/decrypt.
@@ -282,7 +286,7 @@ and may be set in the config file.
--drive-starred-only Only show files that are starred.
--drive-stop-on-download-limit Make download limit errors be fatal
--drive-stop-on-upload-limit Make upload limit errors be fatal
- --drive-team-drive string ID of the Team Drive
+ --drive-team-drive string ID of the Shared Drive (Team Drive)
--drive-token string OAuth Access Token as a JSON blob.
--drive-token-url string Token server url.
--drive-trashed-only Only show files that are in the trash.
@@ -311,12 +315,14 @@ and may be set in the config file.
--filefabric-token-expiry string Token expiry time
--filefabric-url string URL of the Enterprise File Fabric to connect to
--filefabric-version string Version read from the file fabric
+ --ftp-close-timeout Duration Maximum time to wait for a response to close. (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
--ftp-disable-epsv Disable using EPSV even if server advertises support
--ftp-disable-mlsd Disable using MLSD even if server advertises support
--ftp-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot)
--ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
--ftp-host string FTP host to connect to
+ --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--ftp-no-check-certificate Do not verify the TLS certificate of the server
--ftp-pass string FTP password (obscured)
--ftp-port string FTP port, leave blank to use default (21)
@@ -378,6 +384,7 @@ and may be set in the config file.
--local-case-sensitive Force the filesystem to report itself as case sensitive.
--local-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot)
--local-no-check-updated Don't check to see if the files change during upload
+ --local-no-preallocate Disable preallocation of disk space for transferred files
--local-no-set-modtime Disable setting modtime
--local-no-sparse Disable sparse files for multi-thread downloads
--local-no-unicode-normalization Don't apply unicode normalization to paths and filenames (Deprecated)
@@ -482,8 +489,10 @@ and may be set in the config file.
--seafile-url string URL of seafile host to connect to
--seafile-user string User name (usually email address)
--sftp-ask-password Allow asking for SFTP password when needed.
+ --sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available.
--sftp-host string SSH host to connect to
+ --sftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
--sftp-key-file string Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.
--sftp-key-file-pass string The passphrase to decrypt the PEM-encoded private key file. (obscured)
--sftp-key-pem string Raw PEM-encoded private key, If specified, will override key_file parameter.
@@ -553,9 +562,10 @@ and may be set in the config file.
--union-upstreams string List of space separated upstreams.
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
--webdav-bearer-token-command string Command to run to get a bearer token
+ --webdav-encoding string This sets the encoding for the backend.
--webdav-pass string Password. (obscured)
--webdav-url string URL of http host to connect to
- --webdav-user string User name
+ --webdav-user string User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-auth-url string Auth server URL.
--yandex-client-id string OAuth Client Id
@@ -563,6 +573,11 @@ and may be set in the config file.
--yandex-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot)
--yandex-token string OAuth Access Token as a JSON blob.
--yandex-token-url string Token server url.
+ --zoho-auth-url string Auth server URL.
+ --zoho-client-id string OAuth Client Id
+ --zoho-client-secret string OAuth Client Secret
--zoho-encoding MultiEncoder This sets the encoding for the backend. (default Del,Ctl,InvalidUtf8)
--zoho-region string Zoho region to connect to. You'll have to use the region you organization is registered in.
+ --zoho-token string OAuth Access Token as a JSON blob.
+ --zoho-token-url string Token server url.
```
diff --git a/docs/content/ftp.md b/docs/content/ftp.md
index b78c3f0ff..07e9b3f55 100644
--- a/docs/content/ftp.md
+++ b/docs/content/ftp.md
@@ -223,6 +223,30 @@ Disable using MLSD even if server advertises support
- Type: bool
- Default: false
+#### --ftp-idle-timeout
+
+Max time before closing idle connections
+
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+
+Set to 0 to keep connections indefinitely.
+
+
+- Config: idle_timeout
+- Env Var: RCLONE_FTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
+#### --ftp-close-timeout
+
+Maximum time to wait for a response to close.
+
+- Config: close_timeout
+- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
#### --ftp-encoding
This sets the encoding for the backend.
diff --git a/docs/content/local.md b/docs/content/local.md
index 5b3579c4b..4bec9b224 100644
--- a/docs/content/local.md
+++ b/docs/content/local.md
@@ -320,9 +320,9 @@ filesystem.
where it isn't supported (e.g. Windows) it will be ignored.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/local/local.go then run make backenddocs" >}}
-### Standard Options
+### Advanced Options
-Here are the standard options specific to local (Local Disk).
+Here are the advanced options specific to local (Local Disk).
#### --local-nounc
@@ -336,10 +336,6 @@ Disable UNC (long path names) conversion on Windows
- "true"
- Disables long file names
-### Advanced Options
-
-Here are the advanced options specific to local (Local Disk).
-
#### --copy-links / -L
Follow symlinks and copy the pointed to item.
diff --git a/docs/content/rc.md b/docs/content/rc.md
index 8ef29ad29..db65dc56a 100644
--- a/docs/content/rc.md
+++ b/docs/content/rc.md
@@ -765,18 +765,22 @@ Returns the following values:
```
{
- "speed": average speed in bytes/sec since start of the process,
- "bytes": total transferred bytes since the start of the process,
+ "bytes": total transferred bytes since the start of the group,
+ "checks": number of files checked,
+ "deletes" : number of files deleted,
+ "elapsedTime": time in floating point seconds since rclone was started,
"errors": number of errors,
- "fatalError": whether there has been at least one FatalError,
- "retryError": whether there has been at least one non-NoRetryError,
- "checks": number of checked files,
- "transfers": number of transferred files,
- "deletes" : number of deleted files,
- "renames" : number of renamed files,
+ "eta": estimated time in seconds until the group completes,
+ "fatalError": boolean whether there has been at least one fatal error,
+ "lastError": last error string,
+ "renames" : number of files renamed,
+ "retryError": boolean showing whether there has been at least one non-NoRetryError,
+ "speed": average speed in bytes/sec since start of the group,
+ "totalBytes": total number of bytes in the group,
+ "totalChecks": total number of checks in the group,
+ "totalTransfers": total number of transfers in the group,
"transferTime" : total time spent on running jobs,
- "elapsedTime": time in seconds since the start of the process,
- "lastError": last occurred error,
+ "transfers": number of transferred files,
"transferring": an array of currently active file transfers:
[
{
@@ -857,6 +861,8 @@ This shows the current version of go and the go runtime
- os - OS in use as according to Go
- arch - cpu architecture in use according to Go
- goVersion - version of Go runtime in use
+- linking - type of rclone executable (static or dynamic)
+- goTags - space separated build tags or "none"
### debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. {#debug-set-block-profile-rate}
@@ -896,6 +902,26 @@ Results
- previousRate - int
+### fscache/clear: Clear the Fs cache. {#fscache-clear}
+
+This clears the fs cache. This is where remotes created from backends
+are cached for a short while to make repeated rc calls more efficient.
+
+If you change the parameters of a backend then you may want to call
+this to clear an existing remote out of the cache before re-creating
+it.
+
+**Authentication is required for this call.**
+
+### fscache/entries: Returns the number of entries in the fs cache. {#fscache-entries}
+
+This returns the number of entries in the fs cache.
+
+Returns
+- entries - number of items in the cache
+
+**Authentication is required for this call.**
+
### job/list: Lists the IDs of the running jobs {#job-list}
Parameters - None
@@ -1256,6 +1282,7 @@ This takes the following parameters
- fs - a remote name string e.g. "drive:"
- remote - a path within that remote e.g. "dir"
- each part in body represents a file to be uploaded
+See the [uploadfile command](/commands/rclone_uploadfile/) command for more information on the above.
**Authentication is required for this call.**
@@ -1264,11 +1291,31 @@ This takes the following parameters
Returns
- options - a list of the options block names
-### options/get: Get all the options {#options-get}
+### options/get: Get all the global options {#options-get}
Returns an object where keys are option block names and values are an
object with the current option values in.
+Note that these are the global options which are unaffected by use of
+the _config and _filter parameters. If you wish to read the parameters
+set in _config then use options/config and for _filter use options/filter.
+
+This shows the internal names of the option within rclone which should
+map to the external options very easily with a few exceptions.
+
+### options/local: Get the currently active config for this call {#options-local}
+
+Returns an object with the keys "config" and "filter".
+The "config" key contains the local config and the "filter" key contains
+the local filters.
+
+Note that these are the local options specific to this rc call. If
+_config was not supplied then they will be the global options.
+Likewise with "_filter".
+
+This call is mostly useful for seeing if _config and _filter passing
+is working.
+
This shows the internal names of the option within rclone which should
map to the external options very easily with a few exceptions.
@@ -1421,6 +1468,7 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
See the [copy command](/commands/rclone_copy/) command for more information on the above.
@@ -1433,6 +1481,7 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
- deleteEmptySrcDirs - delete empty src directories if set
@@ -1446,6 +1495,7 @@ This takes the following parameters
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
See the [sync command](/commands/rclone_sync/) command for more information on the above.
diff --git a/docs/content/sftp.md b/docs/content/sftp.md
index eb1427362..11b875047 100644
--- a/docs/content/sftp.md
+++ b/docs/content/sftp.md
@@ -496,6 +496,44 @@ any given time.
- Type: bool
- Default: false
+#### --sftp-disable-concurrent-reads
+
+If set don't use concurrent reads
+
+Normally concurrent reads are safe to use and not using them will
+degrade performance, so this option is disabled by default.
+
+Some servers limit the amount number of times a file can be
+downloaded. Using concurrent reads can trigger this limit, so if you
+have a server which returns
+
+ Failed to copy: file does not exist
+
+Then you may need to enable this flag.
+
+If concurrent reads are disabled, the use_fstat option is ignored.
+
+
+- Config: disable_concurrent_reads
+- Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
+- Type: bool
+- Default: false
+
+#### --sftp-idle-timeout
+
+Max time before closing idle connections
+
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+
+Set to 0 to keep connections indefinitely.
+
+
+- Config: idle_timeout
+- Env Var: RCLONE_SFTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+
{{< rem autogenerated options stop >}}
### Limitations ###
diff --git a/docs/content/webdav.md b/docs/content/webdav.md
index 557e00aec..7a74123f1 100644
--- a/docs/content/webdav.md
+++ b/docs/content/webdav.md
@@ -137,23 +137,21 @@ Name of the Webdav site/service/software you are using
- "owncloud"
- Owncloud
- "sharepoint"
- - Sharepoint
+ - Sharepoint Online, authenticated by Microsoft account.
- "sharepoint-ntlm"
- - Sharepoint with NTLM authentication
+ - Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
- "other"
- Other site/service or software
#### --webdav-user
-User name
+User name. In case NTLM authentication is used, the username should be in the format 'Domain\User'.
- Config: user
- Env Var: RCLONE_WEBDAV_USER
- Type: string
- Default: ""
-In case vendor mode `sharepoint-ntlm` is used, the user name is in the form `DOMAIN\user`
-
#### --webdav-pass
Password.
@@ -187,6 +185,19 @@ Command to run to get a bearer token
- Type: string
- Default: ""
+#### --webdav-encoding
+
+This sets the encoding for the backend.
+
+See: the [encoding section in the overview](/overview/#encoding) for more info.
+
+Default encoding is Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8 for sharepoint-ntlm or identity otherwise.
+
+- Config: encoding
+- Env Var: RCLONE_WEBDAV_ENCODING
+- Type: string
+- Default: ""
+
{{< rem autogenerated options stop >}}
## Provider notes ##
diff --git a/docs/content/zoho.md b/docs/content/zoho.md
index 1c44cc3f2..6775987f8 100644
--- a/docs/content/zoho.md
+++ b/docs/content/zoho.md
@@ -128,6 +128,26 @@ from filenames during upload.
Here are the standard options specific to zoho (Zoho).
+#### --zoho-client-id
+
+OAuth Client Id
+Leave blank normally.
+
+- Config: client_id
+- Env Var: RCLONE_ZOHO_CLIENT_ID
+- Type: string
+- Default: ""
+
+#### --zoho-client-secret
+
+OAuth Client Secret
+Leave blank normally.
+
+- Config: client_secret
+- Env Var: RCLONE_ZOHO_CLIENT_SECRET
+- Type: string
+- Default: ""
+
#### --zoho-region
Zoho region to connect to. You'll have to use the region you organization is registered in.
@@ -150,6 +170,35 @@ Zoho region to connect to. You'll have to use the region you organization is reg
Here are the advanced options specific to zoho (Zoho).
+#### --zoho-token
+
+OAuth Access Token as a JSON blob.
+
+- Config: token
+- Env Var: RCLONE_ZOHO_TOKEN
+- Type: string
+- Default: ""
+
+#### --zoho-auth-url
+
+Auth server URL.
+Leave blank to use the provider defaults.
+
+- Config: auth_url
+- Env Var: RCLONE_ZOHO_AUTH_URL
+- Type: string
+- Default: ""
+
+#### --zoho-token-url
+
+Token server url.
+Leave blank to use the provider defaults.
+
+- Config: token_url
+- Env Var: RCLONE_ZOHO_TOKEN_URL
+- Type: string
+- Default: ""
+
#### --zoho-encoding
This sets the encoding for the backend.
diff --git a/go.sum b/go.sum
index 987d04357..af5d8682c 100644
--- a/go.sum
+++ b/go.sum
@@ -11,7 +11,6 @@ cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6T
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
-cloud.google.com/go v0.56.0 h1:WRz29PgAsVEyPSDHyk+0fpEkwEFyfhHn+JbksT6gIL4=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
@@ -45,7 +44,6 @@ github.com/Azure/azure-storage-blob-go v0.13.0 h1:lgWHvFh+UYBNVQLFHXkvul2f6yOPA9
github.com/Azure/azure-storage-blob-go v0.13.0/go.mod h1:pA9kNqtjUeQF2zOSu4s//nUdBD+e64lEuc4sVnuOfNs=
github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs=
github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
-github.com/Azure/go-autorest/autorest/adal v0.9.2 h1:Aze/GQeAN1RRbGmnUJvUj+tFGBzFdIg3293/A9rbxC4=
github.com/Azure/go-autorest/autorest/adal v0.9.2/go.mod h1:/3SMAM86bP6wC9Ev35peQDUeqFZBMH07vvUOmg4z/fE=
github.com/Azure/go-autorest/autorest/adal v0.9.10 h1:r6fZHMaHD8B6LDCn0o5vyBFHIHrM6Ywwx7mb49lPItI=
github.com/Azure/go-autorest/autorest/adal v0.9.10/go.mod h1:B7KF7jKIeC9Mct5spmyCB/A8CG/sEz1vwIRGv/bbw7A=
@@ -62,7 +60,6 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/Julusian/godocdown v0.0.0-20170816220326-6d19f8ff2df8/go.mod h1:INZr5t32rG59/5xeltqoCJoNY7e5x/3xoY9WSWVWg74=
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
-github.com/Microsoft/go-winio v0.4.14 h1:+hMXMk01us9KgxGb7ftKQt2Xpf5hH/yky+TDA+qxleU=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
github.com/Microsoft/go-winio v0.4.16 h1:FtSW/jqD+l4ba5iPBj9CODVtgfYAD8w2wS923g/cFDk=
github.com/Microsoft/go-winio v0.4.16/go.mod h1:XB6nPKklQyQ7GC9LdcBEcBl8PF76WugXOPRXwdLnMv0=
@@ -111,8 +108,6 @@ github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+Ce
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
-github.com/billziss-gh/cgofuse v1.4.0 h1:kju2jDmdNuDDCrxPob2ggmZr5Mj/odCjU1Y8kx0Th9E=
-github.com/billziss-gh/cgofuse v1.4.0/go.mod h1:LJjoaUojlVjgo5GQoEJTcJNqZJeRU0nCR84CyxKt2YM=
github.com/billziss-gh/cgofuse v1.5.0 h1:kH516I/s+Ab4diL/Y/ayFeUjjA8ey+JK12xDfBf4HEs=
github.com/billziss-gh/cgofuse v1.5.0/go.mod h1:LJjoaUojlVjgo5GQoEJTcJNqZJeRU0nCR84CyxKt2YM=
github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84=
@@ -121,7 +116,6 @@ github.com/bradfitz/iter v0.0.0-20190303215204-33e6a9893b0c/go.mod h1:PyRFw1Lt2w
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
-github.com/btcsuite/btcutil v1.0.3-0.20201124182144-4031bdc69ded h1:WcPFZzCIqGt/TdFJHsOiX5dIlB/MUzrftltMhpjzfA8=
github.com/btcsuite/btcutil v1.0.3-0.20201124182144-4031bdc69ded/go.mod h1:0DVlHczLPewLcPGEIeUEzfOJhqGPQ0mJJRDBtD307+o=
github.com/btcsuite/btcutil v1.0.3-0.20201208143702-a53e38424cce h1:YtWJF7RHm2pYCvA5t0RPmAaLUhREsKuKd+SLhxFbFeQ=
github.com/btcsuite/btcutil v1.0.3-0.20201208143702-a53e38424cce/go.mod h1:0DVlHczLPewLcPGEIeUEzfOJhqGPQ0mJJRDBtD307+o=
@@ -171,7 +165,6 @@ github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
@@ -182,7 +175,6 @@ github.com/dropbox/dropbox-sdk-go-unofficial v1.0.1-0.20210114204226-41fdcdae8a5
github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/dustin/go-humanize v0.0.0-20180421182945-02af3965c54e/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
-github.com/dvyukov/go-fuzz v0.0.0-20200318091601-be3528f3a813 h1:NgO45/5mBLRVfiXerEFzH6ikcZ7DNRPS639xFg3ENzU=
github.com/dvyukov/go-fuzz v0.0.0-20200318091601-be3528f3a813/go.mod h1:11Gm+ccJnvAhCNLlf5+cS9KjtbaD5I5zaZpFMsTHWTw=
github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
@@ -222,7 +214,6 @@ github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/me
github.com/gogo/googleapis v1.1.0/go.mod h1:gf4bu3Q80BeJ6H1S1vYPm8/ELATdvryBaNFGgqEef3s=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
-github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE=
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
@@ -239,7 +230,6 @@ github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFU
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
-github.com/golang/mock v1.4.4 h1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.1.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -254,7 +244,6 @@ github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrU
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
-github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
@@ -267,10 +256,8 @@ github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
-github.com/google/go-cmp v0.5.0 h1:/QaMHBdZ26BB3SSst0Iwl10Epc+xhTquomWX0oZEB6w=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
-github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=
@@ -293,9 +280,7 @@ github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLe
github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.2.0 h1:qJYtXnJRWmpe7m/3XlyhrsLrEURqHRM2kxzoxXqyUDs=
github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -337,7 +322,6 @@ github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa
github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU=
github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
-github.com/hashicorp/go-uuid v1.0.1 h1:fv1ep09latC32wFoVwnqcnKJGnMSdBanPczbHAYm1BE=
github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-uuid v1.0.2 h1:cfejS+Tpcp13yd5nYHWDI6qVCny6wyX2Mt5SGur2IGE=
github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
@@ -390,7 +374,6 @@ github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlT
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
-github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
@@ -410,7 +393,6 @@ github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6
github.com/klauspost/compress v1.11.7 h1:0hzRabrMN4tSTvMfnL3SCv1ZGeAP23ynzodBgaHeMeg=
github.com/klauspost/compress v1.11.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
-github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/koofr/go-httpclient v0.0.0-20200420163713-93aa7c75b348 h1:Lrn8srO9JDBCf2iPjqy62stl49UDwoOxZ9/NGVi+fnk=
github.com/koofr/go-httpclient v0.0.0-20200420163713-93aa7c75b348/go.mod h1:JBLy//Q5jzU3XSMxdONTD5EIj1LhTPktosxG2Bw1iho=
@@ -419,7 +401,6 @@ github.com/koofr/go-koofrclient v0.0.0-20190724113126-8e5366da203a/go.mod h1:MRA
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
-github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
@@ -440,16 +421,13 @@ github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNx
github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
-github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.10 h1:CoZ3S2P7pvtP45xOtBw+/mDL2z0RKI576gSkzRRpdGg=
github.com/mattn/go-runewidth v0.0.10/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
-github.com/minio/minio-go/v6 v6.0.46 h1:waExJtO53xrnsNX//7cSc1h3478wqTryDx4RVD7o26I=
github.com/minio/minio-go/v6 v6.0.46/go.mod h1:qD0lajrGW49lKZLtXKtCB4X/qkMf0a5tBvN2PaZg7Gg=
-github.com/minio/sha256-simd v0.1.1 h1:5QHSlgo3nt5yKOJrC7W8w7X+NFl8cMPZm96iu8kKUJU=
github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
@@ -461,10 +439,8 @@ github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0Qu
github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
-github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
-github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/mschoch/smat v0.0.0-20160514031455-90eadee771ae/go.mod h1:qAyveg+e4CE+eKJXWVjKXM4ck2QobLqTDytGJbLLhJg=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -524,8 +500,6 @@ github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/profile v1.2.1/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA=
-github.com/pkg/sftp v1.12.0 h1:/f3b24xrDhkhddlaobPe2JgBqfdt+gC/NYl0QY9IOuI=
-github.com/pkg/sftp v1.12.0/go.mod h1:fUqqXB5vEgVCZ131L+9say31RAri6aF6KDViawhxKK8=
github.com/pkg/sftp v1.13.0 h1:Riw6pgOKK41foc1I1Uu03CjvbLZDXeGpInycM4shXoI=
github.com/pkg/sftp v1.13.0/go.mod h1:41g+FIPlQUTDCveupEmEA65IoiQFrtgCeDopC4ajGIM=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
@@ -536,7 +510,6 @@ github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeDPbaTKGT+JTgUa3og=
-github.com/prometheus/client_golang v1.7.1 h1:NTGy1Ja9pByO+xAeH/qiWnLrKtr3hJPNjaVUwnjpdpA=
github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
github.com/prometheus/client_golang v1.9.0 h1:Rrch9mh17XcxvEu9D9DEpb4isxjGBtcevQjKvxPRQIU=
github.com/prometheus/client_golang v1.9.0/go.mod h1:FqZLKOZnGdFAhOK4nqGHa7D66IdsO+O441Eve7ptJDU=
@@ -552,7 +525,6 @@ github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y8
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.7.0/go.mod h1:DjGbpBbp5NYNiECxcL/VnbXCCaQpKd3tt26CguLLsqA=
-github.com/prometheus/common v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc=
github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo=
github.com/prometheus/common v0.15.0 h1:4fgOnadei3EZvgRwxJ7RMpG1k1pOZth5Pc13tyspaKM=
github.com/prometheus/common v0.15.0/go.mod h1:U+gB1OBLb1lF3O42bTCL+FK18tX9Oar16Clt/msog/s=
@@ -561,9 +533,7 @@ github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
-github.com/prometheus/procfs v0.1.3 h1:F0+tqvhOksq22sc6iCHF5WGlWjdwj92p0udFh1VFBS8=
github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
-github.com/prometheus/procfs v0.2.0 h1:wH4vA7pcjKuZzjF7lM8awk4fnuJO6idemZXoKnULUx4=
github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/prometheus/procfs v0.3.0 h1:Uehi/mxLK0eiUc0H0++5tpMGTexB8wZ598MIgU8VpDM=
github.com/prometheus/procfs v0.3.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
@@ -573,14 +543,12 @@ github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8/go.mod h1:
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/rfjakob/eme v1.1.1 h1:t+CgvcOn+eDvj2xdglxsSnkgg8LM8jwdxnV7OnsrTn0=
github.com/rfjakob/eme v1.1.1/go.mod h1:U2bmx0hDj8EyDdcxmD5t3XHDnBFnyNNc22n1R4008eM=
-github.com/rivo/uniseg v0.1.0 h1:+2KBaVoUmb9XzDsrx/Ct0W/EYOSFf/nWTauy++DprtY=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/robertkrimen/godocdown v0.0.0-20130622164427-0bfa04905481/go.mod h1:C9WhFzY47SzYBIvzFqSvHIR6ROgDo4TtdTuRaOMjF/s=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
-github.com/russross/blackfriday/v2 v2.0.1 h1:lPqVAte+HuHNfhJ/0LC98ESWRz8afy9tM/0RK8m9o+Q=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -590,12 +558,10 @@ github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/sevlyar/go-daemon v0.1.5 h1:Zy/6jLbM8CfqJ4x4RPr7MJlSKt90f00kNM1D401C+Qk=
github.com/sevlyar/go-daemon v0.1.5/go.mod h1:6dJpPatBT9eUwM5VCw9Bt6CdX9Tk6UWvhW3MebLDRKE=
-github.com/shurcooL/sanitized_anchor_name v1.0.0 h1:PdmoCO6wvbs+7yrJyMORt4/BmY5IYyJwS/kOiWx8mHo=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
-github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/sirupsen/logrus v1.7.0 h1:ShrD1U9pZB12TX0cVy0DtePoCH97K8EtX+mg7ZARUtM=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
@@ -604,7 +570,6 @@ github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966/go.mod h1:s
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s=
-github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a h1:pa8hGb/2YqsZKovtsgrwcDH1RZhVbTKCjLp47XpqCDs=
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
@@ -631,7 +596,6 @@ github.com/streadway/amqp v0.0.0-20190404075320-75d898a42a94/go.mod h1:AZpEONHx3
github.com/streadway/amqp v0.0.0-20190827072141-edfb9018d271/go.mod h1:AZpEONHx3DKn8O/DFsRAY58/XVQiIPMTMB1SddzLXVw=
github.com/streadway/handy v0.0.0-20190108123426-d5acb3125c2a/go.mod h1:qNTQ5P5JnDBl6z3cMAg/SywNDC5ABu5ApDIw6lUbRmI=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
-github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.1/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
@@ -639,7 +603,6 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.3.1-0.20190311161405-34c6fa2dc709/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
-github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -684,32 +647,23 @@ go.opencensus.io v0.20.2/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
-go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
-go.opencensus.io v0.22.4 h1:LYy1Hy3MJdrCdMwwzxA/dRok4ejH+RwNGbuoD9fCjto=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
-go.opencensus.io v0.22.5 h1:dntmOdLpSpHlVqbW5Eay97DelsZHe+55D+xC6i0dDS0=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
go.opencensus.io v0.22.6 h1:BdkrbWrzDlV9dnbzoP7sfN+dHheJ4J9JOaYxcUDL+ok=
go.opencensus.io v0.22.6/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E=
go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
-go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
-go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
-go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
-go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A=
go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4=
go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
-go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
-go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
@@ -730,15 +684,9 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20200115085410-6d4e4cb37c7d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200117160349-530e935923ad/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a h1:vclmkQCjlDX5OydZ9wv8rBCcS0QyQY66Mpf/7BZbInM=
-golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0 h1:hb9wdF1z5waM+dSIICn1l0DkLVDT3hqhhQsDNUmHPRE=
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9 h1:umElSU9WZirRdgu2yFHY0ayQkEnKiOC1TtM3fWXFnoU=
golang.org/x/crypto v0.0.0-20201112155050-0c6587e931a9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad h1:DN0cp81fZ3njFcrLCytUHRSUkqBjfTo4Tx9RJTWs0EY=
golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
@@ -763,7 +711,6 @@ golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHl
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
-golang.org/x/lint v0.0.0-20200302205851-738671d3881b h1:Wh+f8QHJXR411sJR8/vRBTZ7YapZaRvUcLFFJhusH0k=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5 h1:2M3HP5CCK1Si9FQhwnzYhXdG6DXeebvUHFpre8QvbyI=
golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
@@ -774,7 +721,6 @@ golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.1 h1:Kvvh58BN8Y9/lBi7hTekvtMpm07eUZ0ck5pRHpsMWrY=
@@ -815,7 +761,6 @@ golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
-golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
@@ -829,9 +774,7 @@ golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAG
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
-golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
-golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43 h1:ld7aEMNHoBnnDAX15v1T6z31v8HwR2A9FYOuAhWqkwc=
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
@@ -845,9 +788,7 @@ golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208 h1:qwRHBd0NqMbJxfbotnDhm2ByMI1Shq4Y6oRJo21SGJA=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9 h1:SQFwaSi55rU7vdNs9Yr0Z324VNlrF+0wMqRXT4St8ck=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a h1:DcqTD9SDLc+1P/r1EmRBwnVsrOwW+kk2vWf9n+1sGhs=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -906,12 +847,10 @@ golang.org/x/sys v0.0.0-20201214210602-f9fddec55a1e/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210112091331-59c308dcf3cc/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c h1:VwygUrnw9jn88c4u8GD3rZQbqrP/tgas88tPUbBxQrk=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210313110737-8e9fff1a3a18 h1:jxr7/dEo+rR29uEBoLSWJ1tRHCFAMwFbGUU9nRqzpds=
golang.org/x/sys v0.0.0-20210313110737-8e9fff1a3a18/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
-golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf h1:MZ2shdL+ZM/XzY3ZGOnh4Nlpnxz5GSOhOmtHo3iPU6M=
golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -919,9 +858,7 @@ golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fq
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
-golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.3.4 h1:0YWbFKbhXG/wIiuHDSKpS0Iy7FSA+u45VtBMfQcFTTc=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5 h1:i6eZZ+zk0SOf0xgBpEpPD18qWcJda6q1sxt3S0kzyUQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@@ -939,7 +876,6 @@ golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-golang.org/x/tools v0.0.0-20190328211700-ab21143f2384 h1:TFlARGu6Czu1z7q93HTxcP1P+/ZFC/IKythI5RzrnRg=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
@@ -991,7 +927,6 @@ golang.org/x/tools v0.1.0 h1:po9/4sTYwZU9lPhi1tOrb4hCv3qrhiQ77LZfGa2OjwY=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -1009,7 +944,6 @@ google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
-google.golang.org/api v0.28.0 h1:jMF5hhVfMkTZwHW1SDpKq5CkgWLXOb31Foaca9Zr3oM=
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
@@ -1022,9 +956,7 @@ google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
-google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
-google.golang.org/appengine v1.6.6 h1:lMO5rYAqUxkmaj76jAkRUvt5JZgFymx/+Q5Mzfivuhc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
@@ -1049,7 +981,6 @@ google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfG
google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
-google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940 h1:MRHtG0U6SnaUb+s+LhNE1qt1FQ1wlhqr5E4usBKC0uA=
google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
@@ -1080,10 +1011,8 @@ google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQ
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
-google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4=
google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
-google.golang.org/grpc v1.30.0 h1:M5a8xTlYTxwMn5ZFkwhRabsygDY5G8TYLyQDBxJNAxE=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
@@ -1097,7 +1026,6 @@ google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQ
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
-google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
@@ -1106,7 +1034,6 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -1115,7 +1042,6 @@ gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/gcfg.v1 v1.2.3/go.mod h1:yesOnuUOFQAhST5vPY4nbZsb/huCgGGXlipJsBn0b3o=
-gopkg.in/ini.v1 v1.42.0 h1:7N3gPTt50s8GuLortA00n8AqRTk75qOP98+mTPpgzRk=
gopkg.in/ini.v1 v1.42.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
@@ -1127,14 +1053,11 @@ gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.5 h1:ymVxjfMaHvXD8RqPRmzHHsB3VvucivSkIAvJFDI5O3c=
gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
-gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
diff --git a/rclone.1 b/rclone.1
index 6a09bdded..5b62a8474 100644
--- a/rclone.1
+++ b/rclone.1
@@ -1,7 +1,7 @@
.\"t
.\" Automatically generated by Pandoc 2.5
.\"
-.TH "rclone" "1" "Feb 02, 2021" "User Manual" ""
+.TH "rclone" "1" "Mar 31, 2021" "User Manual" ""
.hy
.SH Rclone syncs your files to cloud storage
.PP
@@ -276,6 +276,10 @@ See below for some expanded Linux / macOS instructions.
.PP
See the Usage section (https://rclone.org/docs/#usage) of the docs for
how to use rclone, or run \f[C]rclone \-h\f[R].
+.PP
+Already installed rclone can be easily updated to the latest version
+using the rclone
+selfupdate (https://rclone.org/commands/rclone_selfupdate/) command.
.SS Script installation
.PP
To install rclone on Linux/macOS/BSD systems, run:
@@ -515,7 +519,7 @@ kill %1
.fi
.SS Install from source
.PP
-Make sure you have at least Go (https://golang.org/) 1.12 installed.
+Make sure you have at least Go (https://golang.org/) go1.13 installed.
Download go (https://golang.org/dl/) if necessary.
The latest release is recommended.
Then
@@ -891,7 +895,7 @@ Sync the source to the destination, changing the destination only.
Doesn\[aq]t transfer unchanged files, testing by size and modification
time or MD5SUM.
Destination is updated to match source, including deleting files if
-necessary.
+necessary (except duplicate objects, see below).
.PP
\f[B]Important\f[R]: Since this can cause data loss, test first with the
\f[C]\-\-dry\-run\f[R] or the \f[C]\-\-interactive\f[R]/\f[C]\-i\f[R]
@@ -905,6 +909,8 @@ rclone sync \-i SOURCE remote:DESTINATION
.PP
Note that files in the destination won\[aq]t be deleted if there were
any errors at any point.
+Duplicate objects (files with the same name, on those providers that
+support it) are also not yet handled.
.PP
It is always the contents of the directory that is synced, not the
directory so when source:path is a directory, it\[aq]s the contents of
@@ -916,6 +922,13 @@ contents go there.
.PP
\f[B]Note\f[R]: Use the \f[C]\-P\f[R]/\f[C]\-\-progress\f[R] flag to
view real\-time transfer statistics
+.PP
+\f[B]Note\f[R]: Use the \f[C]rclone dedupe\f[R] command to deal with
+\[dq]Duplicate object/directory found in source/destination \-
+ignoring\[dq] errors.
+See this forum
+post (https://forum.rclone.org/t/sync-not-clearing-duplicates/14372) for
+more info.
.IP
.nf
\f[C]
@@ -1549,19 +1562,27 @@ commands, flags and backends.
Show the version number.
.SS Synopsis
.PP
-Show the version number, the go version and the architecture.
+Show the rclone version number, the go version, the build target OS and
+architecture, build tags and the type of executable (static or dynamic).
.PP
-Eg
+For example:
.IP
.nf
\f[C]
$ rclone version
-rclone v1.41
-\- os/arch: linux/amd64
-\- go version: go1.10
+rclone v1.54
+\- os/type: linux
+\- os/arch: amd64
+\- go/version: go1.16
+\- go/linking: static
+\- go/tags: none
\f[R]
.fi
.PP
+Note: before rclone version 1.55 the os/type and os/arch lines were
+merged, and the \[dq]go/version\[dq] line was tagged as \[dq]go
+version\[dq].
+.PP
If you supply the \-\-check flag, then it will do an online check to
compare your version with the latest release and the latest beta.
.IP
@@ -1644,8 +1665,8 @@ By default \f[C]dedupe\f[R] interactively finds files with duplicate
names and offers to delete all but one or rename them to be different.
This is known as deduping by name.
.PP
-Deduping by name is only useful with backends like Google Drive which
-can have duplicate file names.
+Deduping by name is only useful with a small group of backends (e.g.
+Google Drive, Opendrive) that can have duplicate file names.
It can be run on wrapping backends (e.g.
crypt) if they wrap a backend which supports duplicate file names.
.PP
@@ -2537,14 +2558,17 @@ Copy url content to dest.
Download a URL\[aq]s content and copy it to the destination without
saving it in temporary storage.
.PP
-Setting \-\-auto\-filename will cause the file name to be retrieved from
-the from URL (after any redirections) and used in the destination path.
+Setting \f[C]\-\-auto\-filename\f[R]will cause the file name to be
+retrieved from the from URL (after any redirections) and used in the
+destination path.
+With \f[C]\-\-print\-filename\f[R] in addition, the resuling file name
+will be printed.
.PP
-Setting \-\-no\-clobber will prevent overwriting file on the destination
-if there is one with the same name.
+Setting \f[C]\-\-no\-clobber\f[R] will prevent overwriting file on the
+destination if there is one with the same name.
.PP
-Setting \-\-stdout or making the output file name \[dq]\-\[dq] will
-cause the output to be written to standard output.
+Setting \f[C]\-\-stdout\f[R] or making the output file name \f[C]\-\f[R]
+will cause the output to be written to standard output.
.IP
.nf
\f[C]
@@ -2555,10 +2579,11 @@ rclone copyurl https://example.com dest:path [flags]
.IP
.nf
\f[C]
- \-a, \-\-auto\-filename Get the file name from the URL and use it for destination file path
- \-h, \-\-help help for copyurl
- \-\-no\-clobber Prevent overwriting file with same name
- \-\-stdout Write the output to stdout rather than a file
+ \-a, \-\-auto\-filename Get the file name from the URL and use it for destination file path
+ \-h, \-\-help help for copyurl
+ \-\-no\-clobber Prevent overwriting file with same name
+ \-p, \-\-print\-filename Print the resulting name from \-\-auto\-filename
+ \-\-stdout Write the output to stdout rather than a file
\f[R]
.fi
.PP
@@ -3438,18 +3463,17 @@ On Windows you can start a mount in different ways.
See below for details.
The following examples will mount to an automatically assigned drive, to
specific drive letter \f[C]X:\f[R], to path
-\f[C]C:\[rs]path\[rs]to\[rs]nonexistent\[rs]directory\f[R] (which must
-be \f[B]non\-existent\f[R] subdirectory of an \f[B]existing\f[R] parent
-directory or drive, and is not supported when mounting as a network
-drive), and the last example will mount as network share
-\f[C]\[rs]\[rs]cloud\[rs]remote\f[R] and map it to an automatically
-assigned drive:
+\f[C]C:\[rs]path\[rs]parent\[rs]mount\f[R] (where parent directory or
+drive must exist, and mount must \f[B]not\f[R] exist, and is not
+supported when mounting as a network drive), and the last example will
+mount as network share \f[C]\[rs]\[rs]cloud\[rs]remote\f[R] and map it
+to an automatically assigned drive:
.IP
.nf
\f[C]
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
-rclone mount remote:path/to/files C:\[rs]path\[rs]to\[rs]nonexistent\[rs]directory
+rclone mount remote:path/to/files C:\[rs]path\[rs]parent\[rs]mount
rclone mount remote:path/to/files \[rs]\[rs]cloud\[rs]remote
\f[R]
.fi
@@ -3518,7 +3542,7 @@ experience unexpected program errors, freezes or other issues, consider
mounting as a network drive instead.
.PP
When mounting as a fixed disk drive you can either mount to an unused
-drive letter, or to a path \- which must be \f[B]non\-existent\f[R]
+drive letter, or to a path representing a \f[B]non\-existent\f[R]
subdirectory of an \f[B]existing\f[R] parent directory or drive.
Using the special value \f[C]*\f[R] will tell rclone to automatically
assign the next available drive letter, starting with Z: and moving
@@ -3529,7 +3553,7 @@ Examples:
\f[C]
rclone mount remote:path/to/files *
rclone mount remote:path/to/files X:
-rclone mount remote:path/to/files C:\[rs]path\[rs]to\[rs]nonexistent\[rs]directory
+rclone mount remote:path/to/files C:\[rs]path\[rs]parent\[rs]mount
rclone mount remote:path/to/files X:
\f[R]
.fi
@@ -3635,12 +3659,32 @@ result you see in Windows Explorer may not be exactly like you expected.
For example, when setting a value that includes write access, this will
be mapped to individual permissions \[dq]write attributes\[dq],
\[dq]write data\[dq] and \[dq]append data\[dq], but not \[dq]write
-extended attributes\[dq] (WinFsp does not support extended attributes,
-see
-this (https://github.com/billziss-gh/winfsp/wiki/NTFS-Compatibility)).
+extended attributes\[dq].
Windows will then show this as basic permission \[dq]Special\[dq]
instead of \[dq]Write\[dq], because \[dq]Write\[dq] includes the
\[dq]write extended attributes\[dq] permission.
+.PP
+If you set POSIX permissions for only allowing access to the owner,
+using \f[C]\-\-file\-perms 0600 \-\-dir\-perms 0700\f[R], the user group
+and the built\-in \[dq]Everyone\[dq] group will still be given some
+special permissions, such as \[dq]read attributes\[dq] and \[dq]read
+permissions\[dq], in Windows.
+This is done for compatibility reasons, e.g.
+to allow users without additional permissions to be able to read basic
+metadata about files like in UNIX.
+One case that may arise is that other programs (incorrectly) interprets
+this as the file being accessible by everyone.
+For example an SSH client may warn about \[dq]unprotected private key
+file\[dq].
+.PP
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option
+\[dq]FileSecurity\[dq], that allows the complete specification of file
+security descriptors using
+SDDL (https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
+With this you can work around issues such as the mentioned
+\[dq]unprotected private key file\[dq] by specifying
+\f[C]\-o FileSecurity=\[dq]D:P(A;;FA;;;OW)\[dq]\f[R], for file all
+access (FA) to the owner (OW).
.SS Windows caveats
.PP
Note that drives created as Administrator are not visible by other
@@ -3663,7 +3707,7 @@ files sequentially, it can only seek when reading.
This means that many applications won\[aq]t work with their files on an
rclone mount without \f[C]\-\-vfs\-cache\-mode writes\f[R] or
\f[C]\-\-vfs\-cache\-mode full\f[R].
-See the File Caching section for more info.
+See the VFS File Caching section for more info.
.PP
The bucket based remotes (e.g.
Swift, S3, Google Compute Storage, B2, Hubic) do not support the concept
@@ -3678,7 +3722,7 @@ systems are a long way from 100% reliable.
The rclone sync/copy commands cope with this with lots of retries.
However rclone mount can\[aq]t use retries in the same way without
making local copies of the uploads.
-Look at the file caching for solutions to make mount more reliable.
+Look at the VFS File Caching for solutions to make mount more reliable.
.SS Attribute caching
.PP
You can use the flag \f[C]\-\-attr\-timeout\f[R] to set the time the
@@ -3875,6 +3919,15 @@ exceed this size for two reasons.
Firstly because it is only checked every
\f[C]\-\-vfs\-cache\-poll\-interval\f[R].
Secondly because open files cannot be evicted from the cache.
+.PP
+You \f[B]should not\f[R] run two copies of rclone using the same VFS
+cache with the same or overlapping remotes if using
+\f[C]\-\-vfs\-cache\-mode > off\f[R].
+This can potentially cause data corruption if you do.
+You can work around this by giving each rclone its own cache hierarchy
+with \f[C]\-\-cache\-dir\f[R].
+You don\[aq]t need to worry about this if the remotes in use don\[aq]t
+overlap.
.SS \-\-vfs\-cache\-mode off
.PP
In this mode (the default) the cache will read directly from the remote
@@ -4052,6 +4105,21 @@ If the flag is not provided on the command line, then its default value
depends on the operating system where rclone runs: \[dq]true\[dq] on
Windows and macOS, \[dq]false\[dq] otherwise.
If the flag is provided without a value, then it is \[dq]true\[dq].
+.SS Alternate report of used bytes
+.PP
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running \f[C]df\f[R]
+on the filesystem, then pass the flag \f[C]\-\-vfs\-used\-is\-size\f[R]
+to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to
+\f[C]rclone size\f[R] and compute the total used space itself.
+.PP
+\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
+filters so that the result is accurate.
+However, this is very inefficient and may cost lots of API calls
+resulting in extra charges.
+Use it as a last resort and only with caching.
.IP
.nf
\f[C]
@@ -4098,6 +4166,7 @@ rclone mount remote:path /path/to/mountpoint [flags]
\-\-vfs\-read\-chunk\-size SizeSuffix Read the source objects in chunks. (default 128M)
\-\-vfs\-read\-chunk\-size\-limit SizeSuffix If greater than \-\-vfs\-read\-chunk\-size, double the chunk size after each chunk read, until the limit is reached. \[aq]off\[aq] is unlimited. (default off)
\-\-vfs\-read\-wait duration Time to wait for in\-sequence read before seeking. (default 20ms)
+ \-\-vfs\-used\-is\-size rclone size Use the rclone size algorithm for Used size.
\-\-vfs\-write\-back duration Time to writeback files after last use when using cache. (default 5s)
\-\-vfs\-write\-wait duration Time to wait for in\-sequence write before giving error. (default 1s)
\-\-volname string Set the volume name. Supported on Windows and OSX only.
@@ -4521,6 +4590,101 @@ not listed here.
.IP \[bu] 2
rclone (https://rclone.org/commands/rclone/) \- Show help for rclone
commands, flags and backends.
+.SH rclone selfupdate
+.PP
+Update the rclone binary.
+.SS Synopsis
+.PP
+This command downloads the latest release of rclone and replaces the
+currently running binary.
+The download is verified with a hashsum and cryptographically signed
+signature.
+.PP
+If used without flags (or with implied \f[C]\-\-stable\f[R] flag), this
+command will install the latest stable release.
+However, some issues may be fixed (or features added) only in the latest
+beta release.
+In such cases you should run the command with the \f[C]\-\-beta\f[R]
+flag, i.e.
+\f[C]rclone selfupdate \-\-beta\f[R].
+You can check in advance what version would be installed by adding the
+\f[C]\-\-check\f[R] flag, then repeat the command without it when you
+are satisfied.
+.PP
+Sometimes the rclone team may recommend you a concrete beta or stable
+rclone release to troubleshoot your issue or add a bleeding edge
+feature.
+The \f[C]\-\-version VER\f[R] flag, if given, will update to the
+concrete version instead of the latest one.
+If you omit micro version from \f[C]VER\f[R] (for example
+\f[C]1.53\f[R]), the latest matching micro version will be used.
+.PP
+Upon successful update rclone will print a message that contains a
+previous version number.
+You will need it if you later decide to revert your update for some
+reason.
+Then you\[aq]ll have to note the previous version and run the following
+command: \f[C]rclone selfupdate [\-\-beta] OLDVER\f[R].
+If the old version contains only dots and digits (for example
+\f[C]v1.54.0\f[R]) then it\[aq]s a stable release so you won\[aq]t need
+the \f[C]\-\-beta\f[R] flag.
+Beta releases have an additional information similar to
+\f[C]v1.54.0\-beta.5111.06f1c0c61\f[R].
+(if you are a developer and use a locally built rclone, the version
+number will end with \f[C]\-DEV\f[R], you will have to rebuild it as it
+obvisously can\[aq]t be distributed).
+.PP
+If you previously installed rclone via a package manager, the package
+may include local documentation or configure services.
+You may wish to update with the flag \f[C]\-\-package deb\f[R] or
+\f[C]\-\-package rpm\f[R] (whichever is correct for your OS) to update
+these too.
+This command with the default \f[C]\-\-package zip\f[R] will update only
+the rclone executable so the local manual may become inaccurate after
+it.
+.PP
+The \f[C]rclone mount\f[R] command
+(https://rclone.org/commands/rclone_mount/) may or may not support
+extended FUSE options depending on the build and OS.
+\f[C]selfupdate\f[R] will refuse to update if the capability would be
+discarded.
+.PP
+Note: Windows forbids deletion of a currently running executable so this
+command will rename the old executable to \[aq]rclone.old.exe\[aq] upon
+success.
+.PP
+Please note that this command was not available before rclone version
+1.55.
+If it fails for you with the message
+\f[C]unknown command \[dq]selfupdate\[dq]\f[R] then you will need to
+update manually following the install instructions located at
+https://rclone.org/install/
+.IP
+.nf
+\f[C]
+rclone selfupdate [flags]
+\f[R]
+.fi
+.SS Options
+.IP
+.nf
+\f[C]
+ \-\-beta Install beta release.
+ \-\-check Check for latest release, do not download.
+ \-h, \-\-help help for selfupdate
+ \-\-output string Save the downloaded binary at a given path (default: replace running binary)
+ \-\-package string Package format: zip|deb|rpm (default: zip)
+ \-\-stable Install stable release (this is the default)
+ \-\-version string Install the given rclone version (default: latest)
+\f[R]
+.fi
+.PP
+See the global flags page (https://rclone.org/flags/) for global options
+not listed here.
+.SS SEE ALSO
+.IP \[bu] 2
+rclone (https://rclone.org/commands/rclone/) \- Show help for rclone
+commands, flags and backends.
.SH rclone serve
.PP
Serve a remote over a protocol.
@@ -4730,6 +4894,15 @@ exceed this size for two reasons.
Firstly because it is only checked every
\f[C]\-\-vfs\-cache\-poll\-interval\f[R].
Secondly because open files cannot be evicted from the cache.
+.PP
+You \f[B]should not\f[R] run two copies of rclone using the same VFS
+cache with the same or overlapping remotes if using
+\f[C]\-\-vfs\-cache\-mode > off\f[R].
+This can potentially cause data corruption if you do.
+You can work around this by giving each rclone its own cache hierarchy
+with \f[C]\-\-cache\-dir\f[R].
+You don\[aq]t need to worry about this if the remotes in use don\[aq]t
+overlap.
.SS \-\-vfs\-cache\-mode off
.PP
In this mode (the default) the cache will read directly from the remote
@@ -4907,6 +5080,21 @@ If the flag is not provided on the command line, then its default value
depends on the operating system where rclone runs: \[dq]true\[dq] on
Windows and macOS, \[dq]false\[dq] otherwise.
If the flag is provided without a value, then it is \[dq]true\[dq].
+.SS Alternate report of used bytes
+.PP
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running \f[C]df\f[R]
+on the filesystem, then pass the flag \f[C]\-\-vfs\-used\-is\-size\f[R]
+to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to
+\f[C]rclone size\f[R] and compute the total used space itself.
+.PP
+\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
+filters so that the result is accurate.
+However, this is very inefficient and may cost lots of API calls
+resulting in extra charges.
+Use it as a last resort and only with caching.
.IP
.nf
\f[C]
@@ -4941,6 +5129,7 @@ rclone serve dlna remote:path [flags]
\-\-vfs\-read\-chunk\-size SizeSuffix Read the source objects in chunks. (default 128M)
\-\-vfs\-read\-chunk\-size\-limit SizeSuffix If greater than \-\-vfs\-read\-chunk\-size, double the chunk size after each chunk read, until the limit is reached. \[aq]off\[aq] is unlimited. (default off)
\-\-vfs\-read\-wait duration Time to wait for in\-sequence read before seeking. (default 20ms)
+ \-\-vfs\-used\-is\-size rclone size Use the rclone size algorithm for Used size.
\-\-vfs\-write\-back duration Time to writeback files after last use when using cache. (default 5s)
\-\-vfs\-write\-wait duration Time to wait for in\-sequence write before giving error. (default 1s)
\f[R]
@@ -5102,6 +5291,15 @@ exceed this size for two reasons.
Firstly because it is only checked every
\f[C]\-\-vfs\-cache\-poll\-interval\f[R].
Secondly because open files cannot be evicted from the cache.
+.PP
+You \f[B]should not\f[R] run two copies of rclone using the same VFS
+cache with the same or overlapping remotes if using
+\f[C]\-\-vfs\-cache\-mode > off\f[R].
+This can potentially cause data corruption if you do.
+You can work around this by giving each rclone its own cache hierarchy
+with \f[C]\-\-cache\-dir\f[R].
+You don\[aq]t need to worry about this if the remotes in use don\[aq]t
+overlap.
.SS \-\-vfs\-cache\-mode off
.PP
In this mode (the default) the cache will read directly from the remote
@@ -5279,6 +5477,21 @@ If the flag is not provided on the command line, then its default value
depends on the operating system where rclone runs: \[dq]true\[dq] on
Windows and macOS, \[dq]false\[dq] otherwise.
If the flag is provided without a value, then it is \[dq]true\[dq].
+.SS Alternate report of used bytes
+.PP
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running \f[C]df\f[R]
+on the filesystem, then pass the flag \f[C]\-\-vfs\-used\-is\-size\f[R]
+to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to
+\f[C]rclone size\f[R] and compute the total used space itself.
+.PP
+\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
+filters so that the result is accurate.
+However, this is very inefficient and may cost lots of API calls
+resulting in extra charges.
+Use it as a last resort and only with caching.
.SS Auth Proxy
.PP
If you supply the parameter \f[C]\-\-auth\-proxy /path/to/program\f[R]
@@ -5411,6 +5624,7 @@ rclone serve ftp remote:path [flags]
\-\-vfs\-read\-chunk\-size SizeSuffix Read the source objects in chunks. (default 128M)
\-\-vfs\-read\-chunk\-size\-limit SizeSuffix If greater than \-\-vfs\-read\-chunk\-size, double the chunk size after each chunk read, until the limit is reached. \[aq]off\[aq] is unlimited. (default off)
\-\-vfs\-read\-wait duration Time to wait for in\-sequence read before seeking. (default 20ms)
+ \-\-vfs\-used\-is\-size rclone size Use the rclone size algorithm for Used size.
\-\-vfs\-write\-back duration Time to writeback files after last use when using cache. (default 5s)
\-\-vfs\-write\-wait duration Time to wait for in\-sequence write before giving error. (default 1s)
\f[R]
@@ -5726,6 +5940,15 @@ exceed this size for two reasons.
Firstly because it is only checked every
\f[C]\-\-vfs\-cache\-poll\-interval\f[R].
Secondly because open files cannot be evicted from the cache.
+.PP
+You \f[B]should not\f[R] run two copies of rclone using the same VFS
+cache with the same or overlapping remotes if using
+\f[C]\-\-vfs\-cache\-mode > off\f[R].
+This can potentially cause data corruption if you do.
+You can work around this by giving each rclone its own cache hierarchy
+with \f[C]\-\-cache\-dir\f[R].
+You don\[aq]t need to worry about this if the remotes in use don\[aq]t
+overlap.
.SS \-\-vfs\-cache\-mode off
.PP
In this mode (the default) the cache will read directly from the remote
@@ -5903,6 +6126,21 @@ If the flag is not provided on the command line, then its default value
depends on the operating system where rclone runs: \[dq]true\[dq] on
Windows and macOS, \[dq]false\[dq] otherwise.
If the flag is provided without a value, then it is \[dq]true\[dq].
+.SS Alternate report of used bytes
+.PP
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running \f[C]df\f[R]
+on the filesystem, then pass the flag \f[C]\-\-vfs\-used\-is\-size\f[R]
+to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to
+\f[C]rclone size\f[R] and compute the total used space itself.
+.PP
+\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
+filters so that the result is accurate.
+However, this is very inefficient and may cost lots of API calls
+resulting in extra charges.
+Use it as a last resort and only with caching.
.IP
.nf
\f[C]
@@ -5947,6 +6185,7 @@ rclone serve http remote:path [flags]
\-\-vfs\-read\-chunk\-size SizeSuffix Read the source objects in chunks. (default 128M)
\-\-vfs\-read\-chunk\-size\-limit SizeSuffix If greater than \-\-vfs\-read\-chunk\-size, double the chunk size after each chunk read, until the limit is reached. \[aq]off\[aq] is unlimited. (default off)
\-\-vfs\-read\-wait duration Time to wait for in\-sequence read before seeking. (default 20ms)
+ \-\-vfs\-used\-is\-size rclone size Use the rclone size algorithm for Used size.
\-\-vfs\-write\-back duration Time to writeback files after last use when using cache. (default 5s)
\-\-vfs\-write\-wait duration Time to wait for in\-sequence write before giving error. (default 1s)
\f[R]
@@ -6418,6 +6657,15 @@ exceed this size for two reasons.
Firstly because it is only checked every
\f[C]\-\-vfs\-cache\-poll\-interval\f[R].
Secondly because open files cannot be evicted from the cache.
+.PP
+You \f[B]should not\f[R] run two copies of rclone using the same VFS
+cache with the same or overlapping remotes if using
+\f[C]\-\-vfs\-cache\-mode > off\f[R].
+This can potentially cause data corruption if you do.
+You can work around this by giving each rclone its own cache hierarchy
+with \f[C]\-\-cache\-dir\f[R].
+You don\[aq]t need to worry about this if the remotes in use don\[aq]t
+overlap.
.SS \-\-vfs\-cache\-mode off
.PP
In this mode (the default) the cache will read directly from the remote
@@ -6595,6 +6843,21 @@ If the flag is not provided on the command line, then its default value
depends on the operating system where rclone runs: \[dq]true\[dq] on
Windows and macOS, \[dq]false\[dq] otherwise.
If the flag is provided without a value, then it is \[dq]true\[dq].
+.SS Alternate report of used bytes
+.PP
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running \f[C]df\f[R]
+on the filesystem, then pass the flag \f[C]\-\-vfs\-used\-is\-size\f[R]
+to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to
+\f[C]rclone size\f[R] and compute the total used space itself.
+.PP
+\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
+filters so that the result is accurate.
+However, this is very inefficient and may cost lots of API calls
+resulting in extra charges.
+Use it as a last resort and only with caching.
.SS Auth Proxy
.PP
If you supply the parameter \f[C]\-\-auth\-proxy /path/to/program\f[R]
@@ -6726,6 +6989,7 @@ rclone serve sftp remote:path [flags]
\-\-vfs\-read\-chunk\-size SizeSuffix Read the source objects in chunks. (default 128M)
\-\-vfs\-read\-chunk\-size\-limit SizeSuffix If greater than \-\-vfs\-read\-chunk\-size, double the chunk size after each chunk read, until the limit is reached. \[aq]off\[aq] is unlimited. (default off)
\-\-vfs\-read\-wait duration Time to wait for in\-sequence read before seeking. (default 20ms)
+ \-\-vfs\-used\-is\-size rclone size Use the rclone size algorithm for Used size.
\-\-vfs\-write\-back duration Time to writeback files after last use when using cache. (default 5s)
\-\-vfs\-write\-wait duration Time to wait for in\-sequence write before giving error. (default 1s)
\f[R]
@@ -7044,6 +7308,15 @@ exceed this size for two reasons.
Firstly because it is only checked every
\f[C]\-\-vfs\-cache\-poll\-interval\f[R].
Secondly because open files cannot be evicted from the cache.
+.PP
+You \f[B]should not\f[R] run two copies of rclone using the same VFS
+cache with the same or overlapping remotes if using
+\f[C]\-\-vfs\-cache\-mode > off\f[R].
+This can potentially cause data corruption if you do.
+You can work around this by giving each rclone its own cache hierarchy
+with \f[C]\-\-cache\-dir\f[R].
+You don\[aq]t need to worry about this if the remotes in use don\[aq]t
+overlap.
.SS \-\-vfs\-cache\-mode off
.PP
In this mode (the default) the cache will read directly from the remote
@@ -7221,6 +7494,21 @@ If the flag is not provided on the command line, then its default value
depends on the operating system where rclone runs: \[dq]true\[dq] on
Windows and macOS, \[dq]false\[dq] otherwise.
If the flag is provided without a value, then it is \[dq]true\[dq].
+.SS Alternate report of used bytes
+.PP
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running \f[C]df\f[R]
+on the filesystem, then pass the flag \f[C]\-\-vfs\-used\-is\-size\f[R]
+to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to
+\f[C]rclone size\f[R] and compute the total used space itself.
+.PP
+\f[I]WARNING.\f[R] Contrary to \f[C]rclone size\f[R], this flag ignores
+filters so that the result is accurate.
+However, this is very inefficient and may cost lots of API calls
+resulting in extra charges.
+Use it as a last resort and only with caching.
.SS Auth Proxy
.PP
If you supply the parameter \f[C]\-\-auth\-proxy /path/to/program\f[R]
@@ -7361,6 +7649,7 @@ rclone serve webdav remote:path [flags]
\-\-vfs\-read\-chunk\-size SizeSuffix Read the source objects in chunks. (default 128M)
\-\-vfs\-read\-chunk\-size\-limit SizeSuffix If greater than \-\-vfs\-read\-chunk\-size, double the chunk size after each chunk read, until the limit is reached. \[aq]off\[aq] is unlimited. (default off)
\-\-vfs\-read\-wait duration Time to wait for in\-sequence read before seeking. (default 20ms)
+ \-\-vfs\-used\-is\-size rclone size Use the rclone size algorithm for Used size.
\-\-vfs\-write\-back duration Time to writeback files after last use when using cache. (default 5s)
\-\-vfs\-write\-wait duration Time to wait for in\-sequence write before giving error. (default 1s)
\f[R]
@@ -7433,6 +7722,176 @@ not listed here.
.IP \[bu] 2
rclone (https://rclone.org/commands/rclone/) \- Show help for rclone
commands, flags and backends.
+.SH rclone test
+.PP
+Run a test command
+.SS Synopsis
+.PP
+Rclone test is used to run test commands.
+.PP
+Select which test comand you want with the subcommand, eg
+.IP
+.nf
+\f[C]
+rclone test memory remote:
+\f[R]
+.fi
+.PP
+Each subcommand has its own options which you can see in their help.
+.PP
+\f[B]NB\f[R] Be careful running these commands, they may do strange
+things so reading their documentation first is recommended.
+.SS Options
+.IP
+.nf
+\f[C]
+ \-h, \-\-help help for test
+\f[R]
+.fi
+.PP
+See the global flags page (https://rclone.org/flags/) for global options
+not listed here.
+.SS SEE ALSO
+.IP \[bu] 2
+rclone (https://rclone.org/commands/rclone/) \- Show help for rclone
+commands, flags and backends.
+.IP \[bu] 2
+rclone test
+histogram (https://rclone.org/commands/rclone_test_histogram/) \- Makes
+a histogram of file name characters.
+.IP \[bu] 2
+rclone test info (https://rclone.org/commands/rclone_test_info/) \-
+Discovers file name or other limitations for paths.
+.IP \[bu] 2
+rclone test
+makefiles (https://rclone.org/commands/rclone_test_makefiles/) \- Make a
+random file hierarchy in
+.RS 2
+.RE
+.IP \[bu] 2
+rclone test memory (https://rclone.org/commands/rclone_test_memory/) \-
+Load all the objects at remote:path into memory and report memory stats.
+.SH rclone test histogram
+.PP
+Makes a histogram of file name characters.
+.SS Synopsis
+.PP
+This command outputs JSON which shows the histogram of characters used
+in filenames in the remote:path specified.
+.PP
+The data doesn\[aq]t contain any identifying information but is useful
+for the rclone developers when developing filename compression.
+.IP
+.nf
+\f[C]
+rclone test histogram [remote:path] [flags]
+\f[R]
+.fi
+.SS Options
+.IP
+.nf
+\f[C]
+ \-h, \-\-help help for histogram
+\f[R]
+.fi
+.PP
+See the global flags page (https://rclone.org/flags/) for global options
+not listed here.
+.SS SEE ALSO
+.IP \[bu] 2
+rclone test (https://rclone.org/commands/rclone_test/) \- Run a test
+command
+.SH rclone test info
+.PP
+Discovers file name or other limitations for paths.
+.SS Synopsis
+.PP
+rclone info discovers what filenames and upload methods are possible to
+write to the paths passed in and how long they can be.
+It can take some time.
+It will write test files into the remote:path passed in.
+It outputs a bit of go code for each one.
+.PP
+\f[B]NB\f[R] this can create undeletable files and other hazards \- use
+with care
+.IP
+.nf
+\f[C]
+rclone test info [remote:path]+ [flags]
+\f[R]
+.fi
+.SS Options
+.IP
+.nf
+\f[C]
+ \-\-all Run all tests.
+ \-\-check\-control Check control characters.
+ \-\-check\-length Check max filename length.
+ \-\-check\-normalization Check UTF\-8 Normalization.
+ \-\-check\-streaming Check uploads with indeterminate file size.
+ \-h, \-\-help help for info
+ \-\-upload\-wait duration Wait after writing a file.
+ \-\-write\-json string Write results to file.
+\f[R]
+.fi
+.PP
+See the global flags page (https://rclone.org/flags/) for global options
+not listed here.
+.SS SEE ALSO
+.IP \[bu] 2
+rclone test (https://rclone.org/commands/rclone_test/) \- Run a test
+command
+.SH rclone test makefiles
+Make a random file hierarchy in
+.IP
+.nf
+\f[C]
+rclone test makefiles [flags]
+\f[R]
+.fi
+.SS Options
+.IP
+.nf
+\f[C]
+ \-\-files int Number of files to create (default 1000)
+ \-\-files\-per\-directory int Average number of files per directory (default 10)
+ \-h, \-\-help help for makefiles
+ \-\-max\-file\-size SizeSuffix Maximum size of files to create (default 100)
+ \-\-max\-name\-length int Maximum size of file names (default 12)
+ \-\-min\-file\-size SizeSuffix Minimum size of file to create
+ \-\-min\-name\-length int Minimum size of file names (default 4)
+\f[R]
+.fi
+.PP
+See the global flags page (https://rclone.org/flags/) for global options
+not listed here.
+.SS SEE ALSO
+.IP \[bu] 2
+rclone test (https://rclone.org/commands/rclone_test/) \- Run a test
+command
+.SH rclone test memory
+.PP
+Load all the objects at remote:path into memory and report memory stats.
+.IP
+.nf
+\f[C]
+rclone test memory remote:path [flags]
+\f[R]
+.fi
+.SS Options
+.IP
+.nf
+\f[C]
+ \-h, \-\-help help for memory
+\f[R]
+.fi
+.PP
+See the global flags page (https://rclone.org/flags/) for global options
+not listed here.
+.SS SEE ALSO
+.IP \[bu] 2
+rclone test (https://rclone.org/commands/rclone_test/) \- Run a test
+command
.SH rclone touch
.PP
Create new file or change file modification time.
@@ -7604,8 +8063,10 @@ The syntax of the paths passed to the rclone command are as follows.
.PP
This refers to the local file system.
.PP
-On Windows only \f[C]\[rs]\f[R] may be used instead of \f[C]/\f[R] in
-local paths \f[B]only\f[R], non local paths must use \f[C]/\f[R].
+On Windows \f[C]\[rs]\f[R] may be used instead of \f[C]/\f[R] in local
+paths \f[B]only\f[R], non local paths must use \f[C]/\f[R].
+See local filesystem (https://rclone.org/local/#windows-paths)
+documentation for more about Windows\-specific paths.
.PP
These paths needn\[aq]t start with a leading \f[C]/\f[R] \- if they
don\[aq]t then they will be relative to the current directory.
@@ -7667,6 +8128,171 @@ rclone copy \-\-sftp\-host example.com :sftp:path/to/dir /tmp/dir
.PP
To copy files and directories from \f[C]example.com\f[R] in the relative
directory \f[C]path/to/dir\f[R] to \f[C]/tmp/dir\f[R] using sftp.
+.SS Connection strings
+.PP
+The above examples can also be written using a connection string syntax,
+so instead of providing the arguments as command line parameters
+\f[C]\-\-http\-url https://pub.rclone.org\f[R] they are provided as part
+of the remote specification as a kind of connection string.
+.IP
+.nf
+\f[C]
+rclone lsd \[dq]:http,url=\[aq]https://pub.rclone.org\[aq]:\[dq]
+rclone lsf \[dq]:http,url=\[aq]https://example.com\[aq]:path/to/dir\[dq]
+rclone copy \[dq]:http,url=\[aq]https://example.com\[aq]:path/to/dir\[dq] /tmp/dir
+rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
+\f[R]
+.fi
+.PP
+These can apply to modify existing remotes as well as create new remotes
+with the on the fly syntax.
+This example is equivalent to adding the
+\f[C]\-\-drive\-shared\-with\-me\f[R] parameter to the remote
+\f[C]gdrive:\f[R].
+.IP
+.nf
+\f[C]
+rclone lsf \[dq]gdrive,shared_with_me:path/to/dir\[dq]
+\f[R]
+.fi
+.PP
+The major advantage to using the connection string style syntax is that
+it only applies the the remote, not to all the remotes of that type of
+the command line.
+A common confusion is this attempt to copy a file shared on google drive
+to the normal drive which \f[B]does not work\f[R] because the
+\f[C]\-\-drive\-shared\-with\-me\f[R] flag applies to both the source
+and the destination.
+.IP
+.nf
+\f[C]
+rclone copy \-\-drive\-shared\-with\-me gdrive:shared\-file.txt gdrive:
+\f[R]
+.fi
+.PP
+However using the connection string syntax, this does work.
+.IP
+.nf
+\f[C]
+rclone copy \[dq]gdrive,shared_with_me:shared\-file.txt\[dq] gdrive:
+\f[R]
+.fi
+.PP
+The connection strings have the following syntax
+.IP
+.nf
+\f[C]
+remote,parameter=value,parameter2=value2:path/to/dir
+:backend,parameter=value,parameter2=value2:path/to/dir
+\f[R]
+.fi
+.PP
+If the \f[C]parameter\f[R] has a \f[C]:\f[R] or \f[C],\f[R] then it must
+be placed in quotes \f[C]\[dq]\f[R] or \f[C]\[aq]\f[R], so
+.IP
+.nf
+\f[C]
+remote,parameter=\[dq]colon:value\[dq],parameter2=\[dq]comma,value\[dq]:path/to/dir
+:backend,parameter=\[aq]colon:value\[aq],parameter2=\[aq]comma,value\[aq]:path/to/dir
+\f[R]
+.fi
+.PP
+If a quoted value needs to include that quote, then it should be
+doubled, so
+.IP
+.nf
+\f[C]
+remote,parameter=\[dq]with\[dq]\[dq]quote\[dq],parameter2=\[aq]with\[aq]\[aq]quote\[aq]:path/to/dir
+\f[R]
+.fi
+.PP
+This will make \f[C]parameter\f[R] be \f[C]with\[dq]quote\f[R] and
+\f[C]parameter2\f[R] be \f[C]with\[aq]quote\f[R].
+.PP
+If you leave off the \f[C]=parameter\f[R] then rclone will substitute
+\f[C]=true\f[R] which works very well with flags.
+For example to use s3 configured in the environment you could use:
+.IP
+.nf
+\f[C]
+rclone lsd :s3,env_auth:
+\f[R]
+.fi
+.PP
+Which is equivalent to
+.IP
+.nf
+\f[C]
+rclone lsd :s3,env_auth=true:
+\f[R]
+.fi
+.PP
+Note that on the command line you might need to surround these
+connection strings with \f[C]\[dq]\f[R] or \f[C]\[aq]\f[R] to stop the
+shell interpreting any special characters within them.
+.PP
+If you are a shell master then you\[aq]ll know which strings are OK and
+which aren\[aq]t, but if you aren\[aq]t sure then enclose them in
+\f[C]\[dq]\f[R] and use \f[C]\[aq]\f[R] as the inside quote.
+This syntax works on all OSes.
+.IP
+.nf
+\f[C]
+rclone copy \[dq]:http,url=\[aq]https://example.com\[aq]:path/to/dir\[dq] /tmp/dir
+\f[R]
+.fi
+.PP
+On Linux/macOS some characters are still interpreted inside
+\f[C]\[dq]\f[R] strings in the shell (notably \f[C]\[rs]\f[R] and
+\f[C]$\f[R] and \f[C]\[dq]\f[R]) so if your strings contain those you
+can swap the roles of \f[C]\[dq]\f[R] and \f[C]\[aq]\f[R] thus.
+(This syntax does not work on Windows.)
+.IP
+.nf
+\f[C]
+rclone copy \[aq]:http,url=\[dq]https://example.com\[dq]:path/to/dir\[aq] /tmp/dir
+\f[R]
+.fi
+.SS Connection strings, config and logging
+.PP
+If you supply extra configuration to a backend by command line flag,
+environment variable or connection string then rclone will add a suffix
+based on the hash of the config to the name of the remote, eg
+.IP
+.nf
+\f[C]
+rclone \-vv lsf \-\-s3\-chunk\-size 20M s3:
+\f[R]
+.fi
+.PP
+Has the log message
+.IP
+.nf
+\f[C]
+DEBUG : s3: detected overridden config \- adding \[dq]{Srj1p}\[dq] suffix to name
+\f[R]
+.fi
+.PP
+This is so rclone can tell the modified remote apart from the unmodified
+remote when caching the backends.
+.PP
+This should only be noticeable in the logs.
+.PP
+This means that on the fly backends such as
+.IP
+.nf
+\f[C]
+rclone \-vv lsf :s3,env_auth:
+\f[R]
+.fi
+.PP
+Will get their own names
+.IP
+.nf
+\f[C]
+DEBUG : :s3: detected overridden config \- adding \[dq]{YTu53}\[dq] suffix to name
+\f[R]
+.fi
.SS Valid remote names
.IP \[bu] 2
Remote names may only contain 0\-9, A\-Z ,a\-z ,_ , \- and space.
@@ -7869,7 +8495,7 @@ For example
\f[R]
.fi
.PP
-would mean limit the upload and download bandwidth to 10 Mbyte/s.
+would mean limit the upload and download bandwidth to 10 MByte/s.
\f[B]NB\f[R] this is \f[B]bytes\f[R] per second not \f[B]bits\f[R] per
second.
To use a single limit, specify the desired bandwidth in kBytes/s, or use
@@ -7885,7 +8511,7 @@ The upload and download bandwidth can be specified seperately, as
\f[R]
.fi
.PP
-would mean limit the upload bandwidth to 10 Mbyte/s and the download
+would mean limit the upload bandwidth to 10 MByte/s and the download
bandwidth to 100 kByte/s.
Either limit can be \[dq]off\[dq] meaning no limit, so to just limit the
upload bandwidth you would use
@@ -7924,9 +8550,9 @@ daytime working hours could be:
.PP
In this example, the transfer bandwidth will be set to 512kBytes/sec at
8am every day.
-At noon, it will rise to 10Mbytes/s, and drop back to 512kBytes/sec at
+At noon, it will rise to 10MByte/s, and drop back to 512kBytes/sec at
1pm.
-At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it
+At 6pm, the bandwidth limit will be set to 30MByte/s, and at 11pm it
will be completely disabled (full speed).
Anything between 11pm and 8am will remain unlimited.
.PP
@@ -7936,8 +8562,8 @@ An example of timetable with \f[C]WEEKDAY\f[R] could be:
.PP
It means that, the transfer bandwidth will be set to 512kBytes/sec on
Monday.
-It will rise to 10Mbytes/s before the end of Friday.
-At 10:00 on Saturday it will be set to 1Mbyte/s.
+It will rise to 10MByte/s before the end of Friday.
+At 10:00 on Saturday it will be set to 1MByte/s.
From 20:00 on Sunday it will be unlimited.
.PP
Timeslots without \f[C]WEEKDAY\f[R] are extended to the whole week.
@@ -8077,7 +8703,7 @@ The compare directory must not overlap the destination directory.
See \f[C]\-\-copy\-dest\f[R] and \f[C]\-\-backup\-dir\f[R].
.SS \-\-config=CONFIG_FILE
.PP
-Specify the location of the rclone config file.
+Specify the location of the rclone configuration file.
.PP
Normally the config file is in your home directory as a file called
\f[C].config/rclone/rclone.conf\f[R] (or \f[C].rclone.conf\f[R] if
@@ -8095,6 +8721,58 @@ location is for you.
.PP
Use this flag to override the config location, e.g.
\f[C]rclone \-\-config=\[dq].myconfig\[dq] .config\f[R].
+.PP
+If the location is set to empty string \f[C]\[dq]\[dq]\f[R] or the
+special value \f[C]/notfound\f[R], or the os null device represented by
+value \f[C]NUL\f[R] on Windows and \f[C]/dev/null\f[R] on Unix systems,
+then rclone will keep the config file in memory only.
+.PP
+The file format is basic
+INI (https://en.wikipedia.org/wiki/INI_file#Format): Sections of text,
+led by a \f[C][section]\f[R] header and followed by \f[C]key=value\f[R]
+entries on separate lines.
+In rclone each remote is represented by its own section, where the
+section name defines the name of the remote.
+Options are specified as the \f[C]key=value\f[R] entries, where the key
+is the option name without the \f[C]\-\-backend\-\f[R] prefix, in
+lowercase and with \f[C]_\f[R] instead of \f[C]\-\f[R].
+E.g.
+option \f[C]\-\-mega\-hard\-delete\f[R] corresponds to key
+\f[C]hard_delete\f[R].
+Only backend options can be specified.
+A special, and required, key \f[C]type\f[R] identifies the storage
+system (https://rclone.org/overview/), where the value is the internal
+lowercase name as returned by command \f[C]rclone help backends\f[R].
+Comments are indicated by \f[C];\f[R] or \f[C]#\f[R] at the beginning of
+a line.
+.PP
+Example:
+.IP
+.nf
+\f[C]
+[megaremote]
+type = mega
+user = you\[at]example.com
+pass = PDPcQVVjVtzFY\-GTdDFozqBhTdsPg3qH
+\f[R]
+.fi
+.PP
+Note that passwords are in
+obscured (https://rclone.org/commands/rclone_obscure/) form.
+Also, many storage systems uses token\-based authentication instead of
+passwords, and this requires additional steps.
+It is easier, and safer, to use the interactive command
+\f[C]rclone config\f[R] instead of manually editing the configuration
+file.
+.PP
+The configuration file will typically contain login information, and
+should therefore have restricted permissions so that only the current
+user can read it.
+Rclone tries to ensure this when it writes the file.
+You may also choose to encrypt the file.
+.PP
+When token\-based authentication are used, the configuration file must
+be writable, because rclone needs to update the tokens inside it.
.SS \-\-contimeout=TIME
.PP
Set the connection timeout.
@@ -8154,6 +8832,29 @@ This flag can be useful for debugging and in exceptional circumstances
(e.g.
Google Drive limiting the total volume of Server Side Copies to
100GB/day).
+.SS \-\-dscp VALUE
+.PP
+Specify a DSCP value or name to use in connections.
+This could help QoS system to identify traffic class.
+BE, EF, DF, LE, CSx and AFxx are allowed.
+.PP
+See the description of differentiated
+services (https://en.wikipedia.org/wiki/Differentiated_services) to get
+an idea of this field.
+Setting this to 1 (LE) to identify the flow to SCAVENGER class can avoid
+occupying too much bandwidth in a network with DiffServ support (RFC
+8622 (https://tools.ietf.org/html/rfc8622)).
+.PP
+For example, if you configured QoS on router to handle LE properly.
+Running:
+.IP
+.nf
+\f[C]
+rclone copy \-\-dscp LE from:/from to:/to
+\f[R]
+.fi
+.PP
+would make the priority lower than usual internet flows.
.SS \-n, \-\-dry\-run
.PP
Do a trial run with no permanent changes.
@@ -8185,6 +8886,28 @@ if data was copied, or skipping if not.
NB: Enabling this option turns a usually non\-fatal error into a
potentially fatal one \- please check and adjust your scripts
accordingly!
+.SS \-\-fs\-cache\-expire\-duration=TIME
+.PP
+When using rclone via the API rclone caches created remotes for 5
+minutes by default in the \[dq]fs cache\[dq].
+This means that if you do repeated actions on the same remote then
+rclone won\[aq]t have to build it again from scratch, which makes it
+more efficient.
+.PP
+This flag sets the time that the remotes are cached for.
+If you set it to \f[C]0\f[R] (or negative) then rclone won\[aq]t cache
+the remotes at all.
+.PP
+Note that if you use some flags, eg \f[C]\-\-backup\-dir\f[R] and if
+this is set to \f[C]0\f[R] rclone may build two remotes (one for the
+source or destination and one for the \f[C]\-\-backup\-dir\f[R] where it
+may have only built one before.
+.SS \-\-fs\-cache\-expire\-interval=TIME
+.PP
+This controls how often rclone checks for cached remotes to expire.
+See the \f[C]\-\-fs\-cache\-expire\-duration\f[R] documentation above
+for more info.
+The default is 60s, set to 0 to disable expiry.
.SS \-\-header
.PP
Add an HTTP header for all transactions.
@@ -9248,7 +9971,7 @@ This option defaults to \f[C]false\f[R].
.PP
Your configuration file contains information for logging in to your
cloud services.
-This means that you should keep your \f[C].rclone.conf\f[R] file in a
+This means that you should keep your \f[C]rclone.conf\f[R] file in a
secure location.
.PP
If you are in an environment where that isn\[aq]t possible, you can add
@@ -9374,6 +10097,21 @@ rclone.
This will make rclone fail instead of asking for a password if
\f[C]RCLONE_CONFIG_PASS\f[R] doesn\[aq]t contain a valid password, and
\f[C]\-\-password\-command\f[R] has not been supplied.
+.PP
+Some rclone commands, such as \f[C]genautocomplete\f[R], do not require
+configuration.
+Nevertheless, rclone will read any configuration file found according to
+the rules described above (https://rclone.org/docs/#config-config-file).
+If an encrypted configuration file is found, this means you will be
+prompted for password (unless using \f[C]\-\-password\-command\f[R]).
+To avoid this, you can bypass the loading of the configuration file by
+overriding the location with an empty string \f[C]\[dq]\[dq]\f[R] or the
+special value \f[C]/notfound\f[R], or the os null device represented by
+value \f[C]NUL\f[R] on Windows and \f[C]/dev/null\f[R] on Unix systems
+(before rclone version 1.55 only this null device alternative was
+supported).
+E.g.
+\f[C]rclone \-\-config=\[dq]\[dq] genautocomplete bash\f[R].
.SS Developer options
.PP
These options are useful when developing or debugging rclone.
@@ -9600,12 +10338,8 @@ they take exactly the same form.
.PP
You can set defaults for values in the config file on an individual
remote basis.
-If you want to use this feature, you will need to discover the name of
-the config items that you want.
-The easiest way is to run through \f[C]rclone config\f[R] by hand, then
-look in the config file to see what the values are (the config file can
-be found by looking at the help for \f[C]\-\-config\f[R] in
-\f[C]rclone help\f[R]).
+The names of the config items are documented in the page for each
+backend.
.PP
To find the name of the environment variable, you need to set, take
\f[C]RCLONE_CONFIG_\f[R] + name of remote + \f[C]_\f[R] + name of config
@@ -9628,6 +10362,15 @@ mys3:
.PP
Note that if you want to create a remote using environment variables you
must create the \f[C]..._TYPE\f[R] variable as above.
+.PP
+Note also that now rclone has connectionstrings, it is probably easier
+to use those instead which makes the above example
+.IP
+.nf
+\f[C]
+rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
+\f[R]
+.fi
.SS Precedence
.PP
The various different methods of backend configuration are read in this
@@ -9884,6 +10627,31 @@ file.jpg \- matches \[dq]file.jpg\[dq]
\f[R]
.fi
.PP
+The top level of the remote may not be the top level of the drive.
+.PP
+E.g.
+for a Microsoft Windows local directory structure
+.IP
+.nf
+\f[C]
+F:
+\[u251C]\[u2500]\[u2500] bkp
+\[u251C]\[u2500]\[u2500] data
+\[br] \[u251C]\[u2500]\[u2500] excl
+\[br] \[br] \[u251C]\[u2500]\[u2500] 123.jpg
+\[br] \[br] \[u2514]\[u2500]\[u2500] 456.jpg
+\[br] \[u251C]\[u2500]\[u2500] incl
+\[br] \[br] \[u2514]\[u2500]\[u2500] document.pdf
+\f[R]
+.fi
+.PP
+To copy the contents of folder \f[C]data\f[R] into folder \f[C]bkp\f[R]
+excluding the contents of subfolder \f[C]excl\f[R]the following command
+treats \f[C]F:\[rs]data\f[R] and \f[C]F:\[rs]bkp\f[R] as top level for
+filtering.
+.PP
+\f[C]rclone copy F:\[rs]data\[rs] F:\[rs]bkp\[rs] \-\-exclude=/excl/**\f[R]
+.PP
\f[B]Important\f[R] Use \f[C]/\f[R] in path/file name patterns and not
\f[C]\[rs]\f[R] even if running on Microsoft Windows.
.PP
@@ -9909,7 +10677,7 @@ potato \- matches \[dq]potato\[dq]
.fi
.SS How filter rules are applied to files
.PP
-Rclone path / file name filters are made up of one or more of the
+Rclone path/file name filters are made up of one or more of the
following flags:
.IP \[bu] 2
\f[C]\-\-include\f[R]
@@ -9943,7 +10711,7 @@ To mix up the order of processing includes and excludes use
.PP
Within \f[C]\-\-include\-from\f[R], \f[C]\-\-exclude\-from\f[R] and
\f[C]\-\-filter\-from\f[R] flags rules are processed from top to bottom
-of the referenced file..
+of the referenced file.
.PP
If there is an \f[C]\-\-include\f[R] or \f[C]\-\-include\-from\f[R] flag
specified, rclone implies a \f[C]\- **\f[R] rule which it adds to the
@@ -9976,8 +10744,7 @@ contain character classes.
Go regular expression reference (https://golang.org/pkg/regexp/syntax/)
.SS How filter rules are applied to directories
.PP
-Rclone commands filter, and are applied to, path/file names not
-directories.
+Rclone commands are applied to path/file names not directories.
The entire contents of a directory can be matched to a filter by the
pattern \f[C]directory/*\f[R] or recursively by \f[C]directory/**\f[R].
.PP
@@ -9993,18 +10760,19 @@ unnecessary directories.
Whether optimisation is desirable depends on the specific filter rules
and source remote content.
.PP
-Optimisation occurs if either:
+Directory recursion optimisation occurs if either:
.IP \[bu] 2
A source remote does not support the rclone \f[C]ListR\f[R] primitive.
-\f[C]local\f[R], \f[C]sftp\f[R], \f[C]Microsoft OneDrive\f[R] and
-\f[C]WebDav\f[R] do not support \f[C]ListR\f[R].
+local, sftp, Microsoft OneDrive and WebDav do not support
+\f[C]ListR\f[R].
Google Drive and most bucket type storage do.
Full list (https://rclone.org/overview/#optional-features)
.IP \[bu] 2
-On other remotes, if the rclone command is not naturally recursive,
-provided it is not run with the \f[C]\-\-fast\-list\f[R] flag.
-\f[C]ls\f[R], \f[C]lsf \-R\f[R] and \f[C]size\f[R] are recursive but
-\f[C]sync\f[R], \f[C]copy\f[R] and \f[C]move\f[R] are not.
+On other remotes (those that support \f[C]ListR\f[R]), if the rclone
+command is not naturally recursive, and provided it is not run with the
+\f[C]\-\-fast\-list\f[R] flag.
+\f[C]ls\f[R], \f[C]lsf \-R\f[R] and \f[C]size\f[R] are naturally
+recursive but \f[C]sync\f[R], \f[C]copy\f[R] and \f[C]move\f[R] are not.
.IP \[bu] 2
Whenever the \f[C]\-\-disable ListR\f[R] flag is applied to an rclone
command.
@@ -10039,9 +10807,9 @@ E.g.
\f[C]rclone ls remote: \-\-include /directory/\f[R] will not match any
files.
Because it is an \f[C]\-\-include\f[R] option the
-\f[C]\-\-exclude **\f[R] rule is implied, and the
-\f[C]\[rs]directory\[rs]\f[R] pattern serves only to optimise access to
-the remote by ignoring everything outside of that directory.
+\f[C]\-\-exclude **\f[R] rule is implied, and the \f[C]/directory/\f[R]
+pattern serves only to optimise access to the remote by ignoring
+everything outside of that directory.
.PP
E.g.
\f[C]rclone ls remote: \-\-filter\-from filter\-list.txt\f[R] with a
@@ -10058,14 +10826,44 @@ file \f[C]filter\-list.txt\f[R]:
.PP
All files in directories \f[C]dir1\f[R] or \f[C]dir2\f[R] or their
subdirectories are completely excluded from the listing.
-Only files of suffix \f[C]\[aq]pdf\f[R] in the root of \f[C]remote:\f[R]
-or its subdirectories are listed.
+Only files of suffix \f[C]pdf\f[R] in the root of \f[C]remote:\f[R] or
+its subdirectories are listed.
The \f[C]\- **\f[R] rule prevents listing of any path/files not
previously matched by the rules above.
.PP
Option \f[C]exclude\-if\-present\f[R] creates a directory exclude rule
based on the presence of a file in a directory and takes precedence over
other rclone directory filter rules.
+.PP
+When using pattern list syntax, if a pattern item contains either
+\f[C]/\f[R] or \f[C]**\f[R], then rclone will not able to imply a
+directory filter rule from this pattern list.
+.PP
+E.g.
+for an include rule
+.IP
+.nf
+\f[C]
+{dir1/**,dir2/**}
+\f[R]
+.fi
+.PP
+Rclone will match files below directories \f[C]dir1\f[R] or
+\f[C]dir2\f[R] only, but will not be able to use this filter to exclude
+a directory \f[C]dir3\f[R] from being traversed.
+.PP
+Directory recursion optimisation may affect performance, but normally
+not the result.
+One exception to this is sync operations with option
+\f[C]\-\-create\-empty\-src\-dirs\f[R], where any traversed empty
+directories will be created.
+With the pattern list example \f[C]{dir1/**,dir2/**}\f[R] above, this
+would create an empty directory \f[C]dir3\f[R] on destination (when it
+exists on source).
+Changing the filter to \f[C]{dir1,dir2}/**\f[R], or splitting it into
+two include rules \f[C]\-\-include dir1/** \-\-include dir2/**\f[R],
+will match the same files while also filtering directories, with the
+result that an empty directory \f[C]dir3\f[R] will no longer be created.
.SS \f[C]\-\-exclude\f[R] \- Exclude files matching pattern
.PP
Excludes path/file names from an rclone command based on a single
@@ -10095,10 +10893,9 @@ on Microsoft Windows
\f[C]rclone ls remote: \-\-exclude \[dq]*\[rs][{JP,KR,HK}\[rs]]*\[dq]\f[R]
lists the files in \f[C]remote:\f[R] with \f[C][JP]\f[R] or
\f[C][KR]\f[R] or \f[C][HK]\f[R] in their name.
-The single quotes prevent the shell from interpreting the
-\f[C]\[rs]\f[R] characters.
-The \f[C]\[rs]\f[R] characters escape the \f[C][\f[R] and \f[C]]\f[R] so
-ran clone filter treats them literally rather than as a
+Quotes prevent the shell from interpreting the \f[C]\[rs]\f[R]
+characters.\f[C]\[rs]\f[R] characters escape the \f[C][\f[R] and
+\f[C]]\f[R] so an rclone filter treats them literally rather than as a
character\-range.
The \f[C]{\f[R] and \f[C]}\f[R] define an rclone pattern list.
For other operating systems single quotes are required ie
@@ -10274,7 +11071,7 @@ Arrange the order of filter rules with the most restrictive first and
work down.
.PP
E.g.
-For \f[C]filter\-file.txt\f[R]:
+for \f[C]filter\-file.txt\f[R]:
.IP
.nf
\f[C]
@@ -10433,7 +11230,7 @@ The three files are transferred as follows:
.fi
.PP
Alternatively if \f[C]/\f[R] is chosen as root \f[C]files\-from.txt\f[R]
-would be:
+will be:
.IP
.nf
\f[C]
@@ -10443,7 +11240,7 @@ would be:
\f[R]
.fi
.PP
-The copy command would be:
+The copy command will be:
.IP
.nf
\f[C]
@@ -10545,13 +11342,14 @@ Default units are seconds or the following abbreviations are valid:
following formats:
.IP \[bu] 2
RFC3339 \- e.g.
-\[dq]2006\-01\-02T15:04:05Z07:00\[dq]
+\f[C]2006\-01\-02T15:04:05Z\f[R] or
+\f[C]2006\-01\-02T15:04:05+07:00\f[R]
.IP \[bu] 2
-ISO8601 Date and time, local timezone \- \[dq]2006\-01\-02T15:04:05\[dq]
+ISO8601 Date and time, local timezone \- \f[C]2006\-01\-02T15:04:05\f[R]
.IP \[bu] 2
-ISO8601 Date and time, local timezone \- \[dq]2006\-01\-02 15:04:05\[dq]
+ISO8601 Date and time, local timezone \- \f[C]2006\-01\-02 15:04:05\f[R]
.IP \[bu] 2
-ISO8601 Date \- \[dq]2006\-01\-02\[dq] (YYYY\-MM\-DD)
+ISO8601 Date \- \f[C]2006\-01\-02\f[R] (YYYY\-MM\-DD)
.PP
\f[C]\-\-max\-age\f[R] applies only to files and not to directories.
.PP
@@ -10574,9 +11372,9 @@ E.g.
\f[B]Important\f[R] this flag is dangerous to your data \- use with
\f[C]\-\-dry\-run\f[R] and \f[C]\-v\f[R] first.
.PP
-In conjunction with \f[C]rclone sync\f[R] the \[ga]\-\-delete\-excluded
-deletes any files on the destination which are excluded from the
-command.
+In conjunction with \f[C]rclone sync\f[R],
+\f[C]\-\-delete\-excluded\f[R] deletes any files on the destination
+which are excluded from the command.
.PP
E.g.
the scope of \f[C]rclone sync \-i A: B:\f[R] can be restricted:
@@ -10623,7 +11421,7 @@ command.
.SS Common pitfalls
.PP
The most frequent filter support issues on the rclone
-forum (https://https://forum.rclone.org/) are:
+forum (https://forum.rclone.org/) are:
.IP \[bu] 2
Not using paths relative to the root of the remote
.IP \[bu] 2
@@ -11038,6 +11836,100 @@ $ rclone rc job/list
}
\f[R]
.fi
+.SS Setting config flags with _config
+.PP
+If you wish to set config (the equivalent of the global flags) for the
+duration of an rc call only then pass in the \f[C]_config\f[R]
+parameter.
+.PP
+This should be in the same format as the \f[C]config\f[R] key returned
+by options/get.
+.PP
+For example, if you wished to run a sync with the \f[C]\-\-checksum\f[R]
+parameter, you would pass this parameter in your JSON blob.
+.IP
+.nf
+\f[C]
+\[dq]_config\[dq]:{\[dq]CheckSum\[dq]: true}
+\f[R]
+.fi
+.PP
+If using \f[C]rclone rc\f[R] this could be passed as
+.IP
+.nf
+\f[C]
+rclone rc operations/sync ... _config=\[aq]{\[dq]CheckSum\[dq]: true}\[aq]
+\f[R]
+.fi
+.PP
+Any config parameters you don\[aq]t set will inherit the global defaults
+which were set with command line flags or environment variables.
+.PP
+Note that it is possible to set some values as strings or integers \-
+see data types (/#data-types) for more info.
+Here is an example setting the equivalent of \f[C]\-\-buffer\-size\f[R]
+in string or integer format.
+.IP
+.nf
+\f[C]
+\[dq]_config\[dq]:{\[dq]BufferSize\[dq]: \[dq]42M\[dq]}
+\[dq]_config\[dq]:{\[dq]BufferSize\[dq]: 44040192}
+\f[R]
+.fi
+.PP
+If you wish to check the \f[C]_config\f[R] assignment has worked
+properly then calling \f[C]options/local\f[R] will show what the value
+got set to.
+.SS Setting filter flags with _filter
+.PP
+If you wish to set filters for the duration of an rc call only then pass
+in the \f[C]_filter\f[R] parameter.
+.PP
+This should be in the same format as the \f[C]filter\f[R] key returned
+by options/get.
+.PP
+For example, if you wished to run a sync with these flags
+.IP
+.nf
+\f[C]
+\-\-max\-size 1M \-\-max\-age 42s \-\-include \[dq]a\[dq] \-\-include \[dq]b\[dq]
+\f[R]
+.fi
+.PP
+you would pass this parameter in your JSON blob.
+.IP
+.nf
+\f[C]
+\[dq]_filter\[dq]:{\[dq]MaxSize\[dq]:\[dq]1M\[dq], \[dq]IncludeRule\[dq]:[\[dq]a\[dq],\[dq]b\[dq]], \[dq]MaxAge\[dq]:\[dq]42s\[dq]}
+\f[R]
+.fi
+.PP
+If using \f[C]rclone rc\f[R] this could be passed as
+.IP
+.nf
+\f[C]
+rclone rc ... _filter=\[aq]{\[dq]MaxSize\[dq]:\[dq]1M\[dq], \[dq]IncludeRule\[dq]:[\[dq]a\[dq],\[dq]b\[dq]], \[dq]MaxAge\[dq]:\[dq]42s\[dq]}\[aq]
+\f[R]
+.fi
+.PP
+Any filter parameters you don\[aq]t set will inherit the global defaults
+which were set with command line flags or environment variables.
+.PP
+Note that it is possible to set some values as strings or integers \-
+see data types (/#data-types) for more info.
+Here is an example setting the equivalent of \f[C]\-\-buffer\-size\f[R]
+in string or integer format.
+.IP
+.nf
+\f[C]
+\[dq]_filter\[dq]:{\[dq]MinSize\[dq]: \[dq]42M\[dq]}
+\[dq]_filter\[dq]:{\[dq]MinSize\[dq]: 44040192}
+\f[R]
+.fi
+.PP
+If you wish to check the \f[C]_filter\f[R] assignment has worked
+properly then calling \f[C]options/local\f[R] will show what the value
+got set to.
.SS Assigning operations to groups with _group = value
.PP
Each rc call has its own stats group for tracking its metrics.
@@ -11060,6 +11952,98 @@ $ rclone rc \-\-json \[aq]{ \[dq]group\[dq]: \[dq]job/1\[dq] }\[aq] core/stats
}
\f[R]
.fi
+.SS Data types
+.PP
+When the API returns types, these will mostly be straight forward
+integer, string or boolean types.
+.PP
+However some of the types returned by the options/get call and taken by
+the options/set calls as well as the \f[C]vfsOpt\f[R],
+\f[C]mountOpt\f[R] and the \f[C]_config\f[R] parameters.
+.IP \[bu] 2
+\f[C]Duration\f[R] \- these are returned as an integer duration in
+nanoseconds.
+They may be set as an integer, or they may be set with time string, eg
+\[dq]5s\[dq].
+See the options section (https://rclone.org/docs/#options) for more
+info.
+.IP \[bu] 2
+\f[C]Size\f[R] \- these are returned as an integer number of bytes.
+They may be set as an integer or they may be set with a size suffix
+string, eg \[dq]10M\[dq].
+See the options section (https://rclone.org/docs/#options) for more
+info.
+.IP \[bu] 2
+Enumerated type (such as \f[C]CutoffMode\f[R], \f[C]DumpFlags\f[R],
+\f[C]LogLevel\f[R], \f[C]VfsCacheMode\f[R] \- these will be returned as
+an integer and may be set as an integer but more conveniently they can
+be set as a string, eg \[dq]HARD\[dq] for \f[C]CutoffMode\f[R] or
+\f[C]DEBUG\f[R] for \f[C]LogLevel\f[R].
+.IP \[bu] 2
+\f[C]BandwidthSpec\f[R] \- this will be set and returned as a string, eg
+\[dq]1M\[dq].
+.SS Specifying remotes to work on
+.PP
+Remotes are specified with the \f[C]fs=\f[R], \f[C]srcFs=\f[R],
+\f[C]dstFs=\f[R] parameters depending on the command being used.
+.PP
+The parameters can be a string as per the rest of rclone, eg
+\f[C]s3:bucket/path\f[R] or \f[C]:sftp:/my/dir\f[R].
+They can also be specified as JSON blobs.
+.PP
+If specifyng a JSON blob it should be a object mapping strings to
+strings.
+These values will be used to configure the remote.
+There are 3 special values which may be set:
+.IP \[bu] 2
+\f[C]type\f[R] \- set to \f[C]type\f[R] to specify a remote called
+\f[C]:type:\f[R]
+.IP \[bu] 2
+\f[C]_name\f[R] \- set to \f[C]name\f[R] to specify a remote called
+\f[C]name:\f[R]
+.IP \[bu] 2
+\f[C]_root\f[R] \- sets the root of the remote \- may be empty
+.PP
+One of \f[C]_name\f[R] or \f[C]type\f[R] should normally be set.
+If the \f[C]local\f[R] backend is desired then \f[C]type\f[R] should be
+set to \f[C]local\f[R].
+If \f[C]_root\f[R] isn\[aq]t specified then it defaults to the root of
+the remote.
+.PP
+For example this JSON is equivalent to \f[C]remote:/tmp\f[R]
+.IP
+.nf
+\f[C]
+{
+ \[dq]_name\[dq]: \[dq]remote\[dq],
+ \[dq]_path\[dq]: \[dq]/tmp\[dq]
+}
+\f[R]
+.fi
+.PP
+And this is equivalent to
+\f[C]:sftp,host=\[aq]example.com\[aq]:/tmp\f[R]
+.IP
+.nf
+\f[C]
+{
+ \[dq]type\[dq]: \[dq]sftp\[dq],
+ \[dq]host\[dq]: \[dq]example.com\[dq],
+ \[dq]_path\[dq]: \[dq]/tmp\[dq]
+}
+\f[R]
+.fi
+.PP
+And this is equivalent to \f[C]/tmp/dir\f[R]
+.IP
+.nf
+\f[C]
+{
+ type = \[dq]local\[dq],
+ _ path = \[dq]/tmp/dir\[dq]
+}
+\f[R]
+.fi
.SS Supported commands
.SS backend/command: Runs a backend command.
.PP
@@ -11462,18 +12446,22 @@ Returns the following values:
.nf
\f[C]
{
- \[dq]speed\[dq]: average speed in bytes/sec since start of the process,
- \[dq]bytes\[dq]: total transferred bytes since the start of the process,
+ \[dq]bytes\[dq]: total transferred bytes since the start of the group,
+ \[dq]checks\[dq]: number of files checked,
+ \[dq]deletes\[dq] : number of files deleted,
+ \[dq]elapsedTime\[dq]: time in floating point seconds since rclone was started,
\[dq]errors\[dq]: number of errors,
- \[dq]fatalError\[dq]: whether there has been at least one FatalError,
- \[dq]retryError\[dq]: whether there has been at least one non\-NoRetryError,
- \[dq]checks\[dq]: number of checked files,
- \[dq]transfers\[dq]: number of transferred files,
- \[dq]deletes\[dq] : number of deleted files,
- \[dq]renames\[dq] : number of renamed files,
+ \[dq]eta\[dq]: estimated time in seconds until the group completes,
+ \[dq]fatalError\[dq]: boolean whether there has been at least one fatal error,
+ \[dq]lastError\[dq]: last error string,
+ \[dq]renames\[dq] : number of files renamed,
+ \[dq]retryError\[dq]: boolean showing whether there has been at least one non\-NoRetryError,
+ \[dq]speed\[dq]: average speed in bytes/sec since start of the group,
+ \[dq]totalBytes\[dq]: total number of bytes in the group,
+ \[dq]totalChecks\[dq]: total number of checks in the group,
+ \[dq]totalTransfers\[dq]: total number of transfers in the group,
\[dq]transferTime\[dq] : total time spent on running jobs,
- \[dq]elapsedTime\[dq]: time in seconds since the start of the process,
- \[dq]lastError\[dq]: last occurred error,
+ \[dq]transfers\[dq]: number of transferred files,
\[dq]transferring\[dq]: an array of currently active file transfers:
[
{
@@ -11567,6 +12555,10 @@ os \- OS in use as according to Go
arch \- cpu architecture in use according to Go
.IP \[bu] 2
goVersion \- version of Go runtime in use
+.IP \[bu] 2
+linking \- type of rclone executable (static or dynamic)
+.IP \[bu] 2
+goTags \- space separated build tags or \[dq]none\[dq]
.SS debug/set\-block\-profile\-rate: Set runtime.SetBlockProfileRate for blocking profiling.
.PP
SetBlockProfileRate controls the fraction of goroutine blocking events
@@ -11614,6 +12606,23 @@ rate \- int
Results
.IP \[bu] 2
previousRate \- int
+.SS fscache/clear: Clear the Fs cache.
+.PP
+This clears the fs cache.
+This is where remotes created from backends are cached for a short while
+to make repeated rc calls more efficient.
+.PP
+If you change the parameters of a backend then you may want to call this
+to clear an existing remote out of the cache before re\-creating it.
+.PP
+\f[B]Authentication is required for this call.\f[R]
+.SS fscache/entries: Returns the number of entries in the fs cache.
+.PP
+This returns the number of entries in the fs cache.
+.PP
+Returns \- entries \- number of items in the cache
+.PP
+\f[B]Authentication is required for this call.\f[R]
.SS job/list: Lists the IDs of the running jobs
.PP
Parameters \- None
@@ -12091,11 +13100,31 @@ more information on the above.
.SS options/blocks: List all the option blocks
.PP
Returns \- options \- a list of the options block names
-.SS options/get: Get all the options
+.SS options/get: Get all the global options
.PP
Returns an object where keys are option block names and values are an
object with the current option values in.
.PP
+Note that these are the global options which are unaffected by use of
+the _config and _filter parameters.
+If you wish to read the parameters set in _config then use
+options/config and for _filter use options/filter.
+.PP
+This shows the internal names of the option within rclone which should
+map to the external options very easily with a few exceptions.
+.SS options/local: Get the currently active config for this call
+.PP
+Returns an object with the keys \[dq]config\[dq] and \[dq]filter\[dq].
+The \[dq]config\[dq] key contains the local config and the
+\[dq]filter\[dq] key contains the local filters.
+.PP
+Note that these are the local options specific to this rc call.
+If _config was not supplied then they will be the global options.
+Likewise with \[dq]_filter\[dq].
+.PP
+This call is mostly useful for seeing if _config and _filter passing is
+working.
+.PP
This shows the internal names of the option within rclone which should
map to the external options very easily with a few exceptions.
.SS options/set: Set an option
@@ -12116,10 +13145,11 @@ Not all options will have an effect when changed like this.
.PP
For example:
.PP
-This sets DEBUG level logs (\-vv)
+This sets DEBUG level logs (\-vv) (these can be set by number or string)
.IP
.nf
\f[C]
+rclone rc options/set \-\-json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: \[dq]DEBUG\[dq]}}\[aq]
rclone rc options/set \-\-json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: 8}}\[aq]
\f[R]
.fi
@@ -12128,7 +13158,7 @@ And this sets INFO level logs (\-v)
.IP
.nf
\f[C]
-rclone rc options/set \-\-json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: 7}}\[aq]
+rclone rc options/set \-\-json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: \[dq]INFO\[dq]}}\[aq]
\f[R]
.fi
.PP
@@ -12136,7 +13166,7 @@ And this sets NOTICE level logs (normal without \-v)
.IP
.nf
\f[C]
-rclone rc options/set \-\-json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: 6}}\[aq]
+rclone rc options/set \-\-json \[aq]{\[dq]main\[dq]: {\[dq]LogLevel\[dq]: \[dq]NOTICE\[dq]}}\[aq]
\f[R]
.fi
.SS pluginsctl/addPlugin: Add a plugin using url
@@ -12274,6 +13304,8 @@ srcFs \- a remote name string e.g.
.IP \[bu] 2
dstFs \- a remote name string e.g.
\[dq]drive:dst\[dq] for the destination
+.IP \[bu] 2
+createEmptySrcDirs \- create empty src directories on destination if set
.PP
See the copy command (https://rclone.org/commands/rclone_copy/) command
for more information on the above.
@@ -12289,6 +13321,8 @@ srcFs \- a remote name string e.g.
dstFs \- a remote name string e.g.
\[dq]drive:dst\[dq] for the destination
.IP \[bu] 2
+createEmptySrcDirs \- create empty src directories on destination if set
+.IP \[bu] 2
deleteEmptySrcDirs \- delete empty src directories if set
.PP
See the move command (https://rclone.org/commands/rclone_move/) command
@@ -12304,6 +13338,8 @@ srcFs \- a remote name string e.g.
.IP \[bu] 2
dstFs \- a remote name string e.g.
\[dq]drive:dst\[dq] for the destination
+.IP \[bu] 2
+createEmptySrcDirs \- create empty src directories on destination if set
.PP
See the sync command (https://rclone.org/commands/rclone_sync/) command
for more information on the above.
@@ -13852,7 +14888,9 @@ T{
T}@T{
No
T}@T{
-No
+Yes
+T}@T{
+Yes
T}@T{
No
T}@T{
@@ -13862,9 +14900,7 @@ No
T}@T{
No
T}@T{
-No
-T}@T{
-No
+Yes
T}@T{
No
T}@T{
@@ -13881,13 +14917,13 @@ Yes
T}@T{
Yes
T}@T{
-No #575 (https://github.com/rclone/rclone/issues/575)
-T}@T{
No
T}@T{
No
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
+T}@T{
+No
T}@T{
No
T}@T{
@@ -13910,7 +14946,7 @@ Yes
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+Yes
T}@T{
No
T}@T{
@@ -13958,7 +14994,7 @@ Yes
T}@T{
Yes
T}@T{
-No
+Yes
T}@T{
Yes
T}
@@ -13996,7 +15032,7 @@ Yes
T}@T{
Yes
T}@T{
-No #575 (https://github.com/rclone/rclone/issues/575)
+No
T}@T{
No
T}@T{
@@ -14019,7 +15055,7 @@ Yes
T}@T{
Yes
T}@T{
-No
+Yes
T}@T{
No
T}@T{
@@ -14048,7 +15084,7 @@ No
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
No
T}@T{
@@ -14071,7 +15107,7 @@ Yes
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
No
T}@T{
@@ -14163,7 +15199,7 @@ No
T}@T{
No
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
No
T}@T{
@@ -14186,7 +15222,7 @@ Yes
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
Yes
T}@T{
@@ -14255,7 +15291,7 @@ No
T}@T{
No
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+Yes
T}@T{
Yes
T}@T{
@@ -14301,7 +15337,7 @@ Yes
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
No
T}@T{
@@ -14370,7 +15406,7 @@ Yes
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
Yes
T}@T{
@@ -14439,7 +15475,7 @@ No
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
Yes
T}@T{
@@ -14462,7 +15498,7 @@ Yes
T}@T{
No
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
No
T}@T{
@@ -14508,7 +15544,7 @@ No
T}@T{
Yes
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
Yes
T}@T{
@@ -14577,7 +15613,7 @@ No
T}@T{
Yes \[dd]
T}@T{
-No #2178 (https://github.com/rclone/rclone/issues/2178)
+No
T}@T{
Yes
T}@T{
@@ -14763,10 +15799,10 @@ These flags are available for every command.
\-c, \-\-checksum Skip based on checksum (if available) & size, not mod\-time & size
\-\-client\-cert string Client SSL certificate (PEM) for mutual TLS auth
\-\-client\-key string Client SSL private key (PEM) for mutual TLS auth
- \-\-compare\-dest string Include additional server\-side path during comparison.
+ \-\-compare\-dest stringArray Include additional comma separated server\-side paths during comparison.
\-\-config string Config file. (default \[dq]$HOME/.config/rclone/rclone.conf\[dq])
\-\-contimeout duration Connect timeout (default 1m0s)
- \-\-copy\-dest string Implies \-\-compare\-dest but also copies files from path into destination.
+ \-\-copy\-dest stringArray Implies \-\-compare\-dest but also copies files from paths into destination.
\-\-cpuprofile string Write cpu profile to file
\-\-cutoff\-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default \[dq]HARD\[dq])
\-\-delete\-after When synchronizing, delete files on destination after transferring (default)
@@ -14775,6 +15811,7 @@ These flags are available for every command.
\-\-delete\-excluded Delete files on dest excluded from sync
\-\-disable string Disable a comma separated list of features. Use help to see a list.
\-n, \-\-dry\-run Do a trial run with no permanent changes
+ \-\-dscp string Set DSCP value to connections. Can be value or names, eg. CS1, LE, DF, AF21.
\-\-dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
\-\-dump\-bodies Dump HTTP headers and bodies \- may contain sensitive info
\-\-dump\-headers Dump HTTP headers \- may contain sensitive info
@@ -14788,6 +15825,8 @@ These flags are available for every command.
\-\-files\-from\-raw stringArray Read list of source\-file names from file without any processing of lines (use \- to read from stdin)
\-f, \-\-filter stringArray Add a file\-filtering rule
\-\-filter\-from stringArray Read filtering patterns from a file (use \- to read from stdin)
+ \-\-fs\-cache\-expire\-duration duration cache remotes for this long (0 to disable caching) (default 5m0s)
+ \-\-fs\-cache\-expire\-interval duration interval to check for expired remotes (default 1m0s)
\-\-header stringArray Set HTTP header for all transactions
\-\-header\-download stringArray Set HTTP header for download transactions
\-\-header\-upload stringArray Set HTTP header for upload transactions
@@ -14886,7 +15925,7 @@ These flags are available for every command.
\-\-use\-json\-log Use json log format.
\-\-use\-mmap Use mmap allocator (see docs).
\-\-use\-server\-modtime Use server modified time instead of object metadata
- \-\-user\-agent string Set the user\-agent to a specified string. The default is rclone/ version (default \[dq]rclone/v1.54.0\[dq])
+ \-\-user\-agent string Set the user\-agent to a specified string. The default is rclone/ version (default \[dq]rclone/v1.55.0\[dq])
\-v, \-\-verbose count Print lots more stuff (repeat for more)
\f[R]
.fi
@@ -14920,6 +15959,7 @@ They control the backends and may be set in the config file.
\-\-azureblob\-msi\-client\-id string Object ID of the user\-assigned MSI to use, if any. Leave blank if msi_object_id or msi_mi_res_id specified.
\-\-azureblob\-msi\-mi\-res\-id string Azure resource ID of the user\-assigned MSI to use, if any. Leave blank if msi_client_id or msi_object_id specified.
\-\-azureblob\-msi\-object\-id string Object ID of the user\-assigned MSI to use, if any. Leave blank if msi_client_id or msi_mi_res_id specified.
+ \-\-azureblob\-public\-access string Public access level of a container: blob, container.
\-\-azureblob\-sas\-url string SAS URL for container level access only
\-\-azureblob\-service\-principal\-file string Path to file containing credentials for use with a service principal.
\-\-azureblob\-upload\-cutoff string Cutoff for switching to chunked upload (<= 256MB). (Deprecated)
@@ -14983,6 +16023,7 @@ They control the backends and may be set in the config file.
\-L, \-\-copy\-links Follow symlinks and copy the pointed to item.
\-\-crypt\-directory\-name\-encryption Option to either encrypt directory names or leave them intact. (default true)
\-\-crypt\-filename\-encryption string How to encrypt the filenames. (default \[dq]standard\[dq])
+ \-\-crypt\-no\-data\-encryption Option to either encrypt file data or leave it unencrypted.
\-\-crypt\-password string Password or pass phrase for encryption. (obscured)
\-\-crypt\-password2 string Password or pass phrase for salt. Optional but recommended. (obscured)
\-\-crypt\-remote string Remote to encrypt/decrypt.
@@ -15018,7 +16059,7 @@ They control the backends and may be set in the config file.
\-\-drive\-starred\-only Only show files that are starred.
\-\-drive\-stop\-on\-download\-limit Make download limit errors be fatal
\-\-drive\-stop\-on\-upload\-limit Make upload limit errors be fatal
- \-\-drive\-team\-drive string ID of the Team Drive
+ \-\-drive\-team\-drive string ID of the Shared Drive (Team Drive)
\-\-drive\-token string OAuth Access Token as a JSON blob.
\-\-drive\-token\-url string Token server url.
\-\-drive\-trashed\-only Only show files that are in the trash.
@@ -15047,12 +16088,14 @@ They control the backends and may be set in the config file.
\-\-filefabric\-token\-expiry string Token expiry time
\-\-filefabric\-url string URL of the Enterprise File Fabric to connect to
\-\-filefabric\-version string Version read from the file fabric
+ \-\-ftp\-close\-timeout Duration Maximum time to wait for a response to close. (default 1m0s)
\-\-ftp\-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
\-\-ftp\-disable\-epsv Disable using EPSV even if server advertises support
\-\-ftp\-disable\-mlsd Disable using MLSD even if server advertises support
\-\-ftp\-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,RightSpace,Dot)
\-\-ftp\-explicit\-tls Use Explicit FTPS (FTP over TLS)
\-\-ftp\-host string FTP host to connect to
+ \-\-ftp\-idle\-timeout Duration Max time before closing idle connections (default 1m0s)
\-\-ftp\-no\-check\-certificate Do not verify the TLS certificate of the server
\-\-ftp\-pass string FTP password (obscured)
\-\-ftp\-port string FTP port, leave blank to use default (21)
@@ -15114,6 +16157,7 @@ They control the backends and may be set in the config file.
\-\-local\-case\-sensitive Force the filesystem to report itself as case sensitive.
\-\-local\-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Dot)
\-\-local\-no\-check\-updated Don\[aq]t check to see if the files change during upload
+ \-\-local\-no\-preallocate Disable preallocation of disk space for transferred files
\-\-local\-no\-set\-modtime Disable setting modtime
\-\-local\-no\-sparse Disable sparse files for multi\-thread downloads
\-\-local\-no\-unicode\-normalization Don\[aq]t apply unicode normalization to paths and filenames (Deprecated)
@@ -15218,8 +16262,10 @@ They control the backends and may be set in the config file.
\-\-seafile\-url string URL of seafile host to connect to
\-\-seafile\-user string User name (usually email address)
\-\-sftp\-ask\-password Allow asking for SFTP password when needed.
+ \-\-sftp\-disable\-concurrent\-reads If set don\[aq]t use concurrent reads
\-\-sftp\-disable\-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available.
\-\-sftp\-host string SSH host to connect to
+ \-\-sftp\-idle\-timeout Duration Max time before closing idle connections (default 1m0s)
\-\-sftp\-key\-file string Path to PEM\-encoded private key file, leave blank or set key\-use\-agent to use ssh\-agent.
\-\-sftp\-key\-file\-pass string The passphrase to decrypt the PEM\-encoded private key file. (obscured)
\-\-sftp\-key\-pem string Raw PEM\-encoded private key, If specified, will override key_file parameter.
@@ -15289,9 +16335,10 @@ They control the backends and may be set in the config file.
\-\-union\-upstreams string List of space separated upstreams.
\-\-webdav\-bearer\-token string Bearer token instead of user/pass (e.g. a Macaroon)
\-\-webdav\-bearer\-token\-command string Command to run to get a bearer token
+ \-\-webdav\-encoding string This sets the encoding for the backend.
\-\-webdav\-pass string Password. (obscured)
\-\-webdav\-url string URL of http host to connect to
- \-\-webdav\-user string User name
+ \-\-webdav\-user string User name. In case NTLM authentication is used, the username should be in the format \[aq]Domain\[rs]User\[aq].
\-\-webdav\-vendor string Name of the Webdav site/service/software you are using
\-\-yandex\-auth\-url string Auth server URL.
\-\-yandex\-client\-id string OAuth Client Id
@@ -15299,8 +16346,13 @@ They control the backends and may be set in the config file.
\-\-yandex\-encoding MultiEncoder This sets the encoding for the backend. (default Slash,Del,Ctl,InvalidUtf8,Dot)
\-\-yandex\-token string OAuth Access Token as a JSON blob.
\-\-yandex\-token\-url string Token server url.
+ \-\-zoho\-auth\-url string Auth server URL.
+ \-\-zoho\-client\-id string OAuth Client Id
+ \-\-zoho\-client\-secret string OAuth Client Secret
\-\-zoho\-encoding MultiEncoder This sets the encoding for the backend. (default Del,Ctl,InvalidUtf8)
\-\-zoho\-region string Zoho region to connect to. You\[aq]ll have to use the region you organization is registered in.
+ \-\-zoho\-token string OAuth Access Token as a JSON blob.
+ \-\-zoho\-token\-url string Token server url.
\f[R]
.fi
.SS 1Fichier
@@ -16452,6 +17504,15 @@ You\[aq]d then do a full \f[C]rclone sync\f[R] less often.
.PP
Note that \f[C]\-\-fast\-list\f[R] isn\[aq]t required in the top\-up
sync.
+.SS Avoiding HEAD requests after PUT
+.PP
+By default rclone will HEAD every object it uploads.
+It does this to check the object got uploaded correctly.
+.PP
+You can disable this with the \-\-s3\-no\-head option \- see there for
+more details.
+.PP
+Setting this flag increases the chance for undetected upload failures.
.SS Hashes
.PP
For small objects which weren\[aq]t uploaded as multipart uploads
@@ -21620,7 +22681,7 @@ securely.
.PP
The format for these URLs is the following:
.PP
-https://ip\-with\-dots\-replaced.server\-hash.plex.direct:32400/
+\f[C]https://ip\-with\-dots\-replaced.server\-hash.plex.direct:32400/\f[R]
.PP
The \f[C]ip\-with\-dots\-replaced\f[R] part can be any IPv4 address,
where the dots have been replaced with dashes, e.g.
@@ -22337,6 +23398,10 @@ become \f[C]big_data.txt\-301.part\f[R].
Note that \f[C]list\f[R] assembles composite directory entries only when
chunk names match the configured format and treats non\-conforming file
names as normal non\-chunked files.
+.PP
+When using \f[C]norename\f[R] transactions, chunk names will
+additionally have a unique file version suffix.
+For example, \f[C]BIG_FILE_NAME.rclone_chunk.001_bp562k\f[R].
.SS Metadata
.PP
Besides data chunks chunker will by default create metadata object for a
@@ -22362,6 +23427,8 @@ Meta objects carry the following fields:
\f[C]md5\f[R] \- MD5 hashsum of composite file (if present)
.IP \[bu] 2
\f[C]sha1\f[R] \- SHA1 hashsum (if present)
+.IP \[bu] 2
+\f[C]txn\f[R] \- identifies current version of the file
.PP
There is no field for composite file name as it\[aq]s simply equal to
the name of meta object on the wrapped remote.
@@ -22439,8 +23506,8 @@ If file is chunked but metadata format is \f[C]none\f[R] then chunker
will use modification time of the first data chunk.
.SS Migrations
.PP
-The idiomatic way to migrate to a different chunk size, hash type or
-chunk naming scheme is to:
+The idiomatic way to migrate to a different chunk size, hash type,
+transaction style or chunk naming scheme is to:
.IP \[bu] 2
Collect all your chunked files under a directory and have your chunker
remote point to it.
@@ -22509,6 +23576,9 @@ and \[dq]hello.doc\[dq] in the same directory).
Chunker included in rclone releases up to \f[C]v1.54\f[R] can sometimes
fail to detect metadata produced by recent versions of rclone.
We recommend users to keep rclone up\-to\-date to avoid data corruption.
+.PP
+Changing \f[C]transactions\f[R] is dangerous and requires explicit
+migration.
.SS Standard Options
.PP
Here are the standard options specific to chunker (Transparently
@@ -22693,6 +23763,58 @@ Report errors and abort current command.
Warn user, skip incomplete file and proceed.
.RE
.RE
+.SS \-\-chunker\-transactions
+.PP
+Choose how chunker should handle temporary files during transactions.
+.IP \[bu] 2
+Config: transactions
+.IP \[bu] 2
+Env Var: RCLONE_CHUNKER_TRANSACTIONS
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]rename\[dq]
+.IP \[bu] 2
+Examples:
+.RS 2
+.IP \[bu] 2
+\[dq]rename\[dq]
+.RS 2
+.IP \[bu] 2
+Rename temporary files after a successful transaction.
+.RE
+.IP \[bu] 2
+\[dq]norename\[dq]
+.RS 2
+.IP \[bu] 2
+Leave temporary file names and write transaction ID to metadata file.
+.IP \[bu] 2
+Metadata is required for no rename transactions (meta format cannot be
+\[dq]none\[dq]).
+.IP \[bu] 2
+If you are using norename transactions you should be careful not to
+downgrade Rclone
+.IP \[bu] 2
+as older versions of Rclone don\[aq]t support this transaction style and
+will misinterpret
+.IP \[bu] 2
+files manipulated by norename transactions.
+.IP \[bu] 2
+This method is EXPERIMENTAL, don\[aq]t use on production systems.
+.RE
+.IP \[bu] 2
+\[dq]auto\[dq]
+.RS 2
+.IP \[bu] 2
+Rename or norename will be used depending on capabilities of the
+backend.
+.IP \[bu] 2
+If meta format is set to \[dq]none\[dq], rename transactions will always
+be used.
+.IP \[bu] 2
+This method is EXPERIMENTAL, don\[aq]t use on production systems.
+.RE
+.RE
.SS Citrix ShareFile
.PP
Citrix ShareFile (https://sharefile.com) is a secure file sharing and
@@ -23160,7 +24282,7 @@ just as you would with any other remote, e.g.
encrypt and decrypt as needed on the fly.
If you access the wrapped remote \f[C]remote:path\f[R] directly you will
bypass the encryption, and anything you read will be in encrypted form,
-and anything you write will be undencrypted.
+and anything you write will be unencrypted.
To avoid issues it is best to configure a dedicated path for encrypted
content, and access it exclusively through a crypt remote.
.IP
@@ -23674,6 +24796,33 @@ Env Var: RCLONE_CRYPT_SHOW_MAPPING
Type: bool
.IP \[bu] 2
Default: false
+.SS \-\-crypt\-no\-data\-encryption
+.PP
+Option to either encrypt file data or leave it unencrypted.
+.IP \[bu] 2
+Config: no_data_encryption
+.IP \[bu] 2
+Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+.IP \[bu] 2
+Type: bool
+.IP \[bu] 2
+Default: false
+.IP \[bu] 2
+Examples:
+.RS 2
+.IP \[bu] 2
+\[dq]true\[dq]
+.RS 2
+.IP \[bu] 2
+Don\[aq]t encrypt file data, leave it unencrypted.
+.RE
+.IP \[bu] 2
+\[dq]false\[dq]
+.RS 2
+.IP \[bu] 2
+Encrypt file data.
+.RE
+.RE
.SS Backend commands
.PP
Here are the commands specific to the crypt backend.
@@ -24321,6 +25470,20 @@ Default: 48M
.SS \-\-dropbox\-impersonate
.PP
Impersonate this user when using a business account.
+.PP
+Note that if you want to use impersonate, you should make sure this flag
+is set when running \[dq]rclone config\[dq] as this will cause rclone to
+request the \[dq]members.read\[dq] scope which it won\[aq]t normally.
+This is needed to lookup a members email address into the internal ID
+that dropbox uses in the API.
+.PP
+Using the \[dq]members.read\[dq] scope will require a Dropbox Team Admin
+to approve during the OAuth flow.
+.PP
+You will have to use your own App (setting your own client_id and
+client_secret) to use this option as currently rclone\[aq]s default set
+of permissions doesn\[aq]t include \[dq]members.read\[dq].
+This can be added once v1.55 or later is in use everywhere.
.IP \[bu] 2
Config: impersonate
.IP \[bu] 2
@@ -24866,9 +26029,9 @@ rclone lsf :ftp: \-\-ftp\-host=speedtest.tele2.net \-\-ftp\-user=anonymous \-\-f
.PP
Rlone FTP supports implicit FTP over TLS servers (FTPS).
This has to be enabled in the FTP backend config for the remote, or with
-\f[C][\-\-ftp\-tls]{#ftp\-tls}\f[R].
+\f[C]\-\-ftp\-tls\f[R].
The default FTPS port is \f[C]990\f[R], not \f[C]21\f[R] and can be set
-with \f[C][\-\-ftp\-port]{#ftp\-port}\f[R].
+with \f[C]\-\-ftp\-port\f[R].
.SS Standard Options
.PP
Here are the standard options specific to ftp (FTP Connection).
@@ -25005,6 +26168,33 @@ Env Var: RCLONE_FTP_DISABLE_MLSD
Type: bool
.IP \[bu] 2
Default: false
+.SS \-\-ftp\-idle\-timeout
+.PP
+Max time before closing idle connections
+.PP
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+.PP
+Set to 0 to keep connections indefinitely.
+.IP \[bu] 2
+Config: idle_timeout
+.IP \[bu] 2
+Env Var: RCLONE_FTP_IDLE_TIMEOUT
+.IP \[bu] 2
+Type: Duration
+.IP \[bu] 2
+Default: 1m0s
+.SS \-\-ftp\-close\-timeout
+.PP
+Maximum time to wait for a response to close.
+.IP \[bu] 2
+Config: close_timeout
+.IP \[bu] 2
+Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+.IP \[bu] 2
+Type: Duration
+.IP \[bu] 2
+Default: 1m0s
.SS \-\-ftp\-encoding
.PP
This sets the encoding for the backend.
@@ -25961,7 +27151,7 @@ If your browser doesn\[aq]t open automatically go to the following link: http://
Log in and authorize rclone for access
Waiting for code...
Got code
-Configure this as a team drive?
+Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> n
@@ -26200,25 +27390,25 @@ interface, share your root folder with the user/email of the new Service
Account you created/selected at step #1 \- use rclone without specifying
the \f[C]\-\-drive\-impersonate\f[R] option, like this:
\f[C]rclone \-v foo\[at]example.com lsf gdrive:backup\f[R]
-.SS Team drives
+.SS Shared drives (team drives)
.PP
-If you want to configure the remote to point to a Google Team Drive then
-answer \f[C]y\f[R] to the question
-\f[C]Configure this as a team drive?\f[R].
+If you want to configure the remote to point to a Google Shared Drive
+(previously known as Team Drives) then answer \f[C]y\f[R] to the
+question \f[C]Configure this as a Shared Drive (Team Drive)?\f[R].
.PP
-This will fetch the list of Team Drives from google and allow you to
+This will fetch the list of Shared Drives from google and allow you to
configure which one you want to use.
-You can also type in a team drive ID if you prefer.
+You can also type in a Shared Drive ID if you prefer.
.PP
For example:
.IP
.nf
\f[C]
-Configure this as a team drive?
+Configure this as a Shared Drive (Team Drive)?
y) Yes
n) No
y/n> y
-Fetching team drive list...
+Fetching Shared Drive list...
Choose a number from below, or type in your own value
1 / Rclone Test
\[rs] \[dq]xxxxxxxxxxxxxxxxxxxx\[dq]
@@ -26226,7 +27416,7 @@ Choose a number from below, or type in your own value
\[rs] \[dq]yyyyyyyyyyyyyyyyyyyy\[dq]
3 / Rclone Test 3
\[rs] \[dq]zzzzzzzzzzzzzzzzzzzz\[dq]
-Enter a Team Drive ID> 1
+Enter a Shared Drive ID> 1
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
[remote]
client_id =
@@ -26927,7 +28117,7 @@ Type: string
Default: \[dq]\[dq]
.SS \-\-drive\-team\-drive
.PP
-ID of the Team Drive
+ID of the Shared Drive (Team Drive)
.IP \[bu] 2
Config: team_drive
.IP \[bu] 2
@@ -27487,7 +28677,7 @@ Options:
\[dq]target\[dq]: optional target remote for the shortcut destination
.SS drives
.PP
-List the shared drives available to this account
+List the Shared Drives available to this account
.IP
.nf
\f[C]
@@ -27495,7 +28685,7 @@ rclone backend drives remote: [options] [+]
\f[R]
.fi
.PP
-This command lists the shared drives (teamdrives) available to this
+This command lists the Shared Drives (Team Drives) available to this
account.
.PP
Usage:
@@ -29608,9 +30798,8 @@ Note that Koofr is case insensitive so you can\[aq]t have a file called
.PP
Mail.ru Cloud (https://cloud.mail.ru/) is a cloud storage provided by a
Russian internet company Mail.Ru Group (https://mail.ru).
-The official desktop client is Disk\-O: (https://disk-o.cloud/),
-available only on Windows.
-(Please note that official sites are in Russian)
+The official desktop client is Disk\-O: (https://disk-o.cloud/en),
+available on Windows and Mac OS.
.PP
Currently it is recommended to disable 2FA on Mail.ru accounts intended
for rclone until it gets eventually implemented.
@@ -30970,6 +32159,41 @@ Env Var: RCLONE_AZUREBLOB_ENCODING
Type: MultiEncoder
.IP \[bu] 2
Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
+.SS \-\-azureblob\-public\-access
+.PP
+Public access level of a container: blob, container.
+.IP \[bu] 2
+Config: public_access
+.IP \[bu] 2
+Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
+.IP \[bu] 2
+Examples:
+.RS 2
+.IP \[bu] 2
+\[dq]\[dq]
+.RS 2
+.IP \[bu] 2
+The container and its blobs can be accessed only with an authorized
+request.
+It\[aq]s a default value
+.RE
+.IP \[bu] 2
+\[dq]blob\[dq]
+.RS 2
+.IP \[bu] 2
+Blob data within this container can be read via anonymous request.
+.RE
+.IP \[bu] 2
+\[dq]container\[dq]
+.RS 2
+.IP \[bu] 2
+Allow full public read access for container and blob data.
+.RE
+.RE
.SS Limitations
.PP
MD5 sums are only uploaded with chunked files if the source has an MD5
@@ -34967,6 +36191,50 @@ Env Var: RCLONE_SFTP_USE_FSTAT
Type: bool
.IP \[bu] 2
Default: false
+.SS \-\-sftp\-disable\-concurrent\-reads
+.PP
+If set don\[aq]t use concurrent reads
+.PP
+Normally concurrent reads are safe to use and not using them will
+degrade performance, so this option is disabled by default.
+.PP
+Some servers limit the amount number of times a file can be downloaded.
+Using concurrent reads can trigger this limit, so if you have a server
+which returns
+.IP
+.nf
+\f[C]
+Failed to copy: file does not exist
+\f[R]
+.fi
+.PP
+Then you may need to enable this flag.
+.PP
+If concurrent reads are disabled, the use_fstat option is ignored.
+.IP \[bu] 2
+Config: disable_concurrent_reads
+.IP \[bu] 2
+Env Var: RCLONE_SFTP_DISABLE_CONCURRENT_READS
+.IP \[bu] 2
+Type: bool
+.IP \[bu] 2
+Default: false
+.SS \-\-sftp\-idle\-timeout
+.PP
+Max time before closing idle connections
+.PP
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
+.PP
+Set to 0 to keep connections indefinitely.
+.IP \[bu] 2
+Config: idle_timeout
+.IP \[bu] 2
+Env Var: RCLONE_SFTP_IDLE_TIMEOUT
+.IP \[bu] 2
+Type: Duration
+.IP \[bu] 2
+Default: 1m0s
.SS Limitations
.PP
SFTP supports checksums if the same login has shell access and
@@ -35000,8 +36268,8 @@ default, due to security concerns.
This can be re\-enabled on a per\-connection basis by setting the
\f[C]use_insecure_cipher\f[R] setting in the configuration file to
\f[C]true\f[R].
-Further details on the insecurity of this cipher can be found [in this
-paper] (http://www.isg.rhul.ac.uk/\[ti]kp/SandPfinal.pdf).
+Further details on the insecurity of this cipher can be found in this
+paper (http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
.PP
SFTP isn\[aq]t supported under plan9 until this
issue (https://github.com/pkg/sftp/issues/156) is fixed.
@@ -36220,9 +37488,11 @@ Choose a number from below, or type in your own value
\[rs] \[dq]nextcloud\[dq]
2 / Owncloud
\[rs] \[dq]owncloud\[dq]
- 3 / Sharepoint
+ 3 / Sharepoint Online, authenticated by Microsoft account.
\[rs] \[dq]sharepoint\[dq]
- 4 / Other site/service or software
+ 4 / Sharepoint with NTLM authentication. Usually self\-hosted or on\-premises.
+ \[rs] \[dq]sharepoint\-ntlm\[dq]
+ 5 / Other site/service or software
\[rs] \[dq]other\[dq]
vendor> 1
User name
@@ -36345,7 +37615,14 @@ Owncloud
\[dq]sharepoint\[dq]
.RS 2
.IP \[bu] 2
-Sharepoint
+Sharepoint Online, authenticated by Microsoft account.
+.RE
+.IP \[bu] 2
+\[dq]sharepoint\-ntlm\[dq]
+.RS 2
+.IP \[bu] 2
+Sharepoint with NTLM authentication.
+Usually self\-hosted or on\-premises.
.RE
.IP \[bu] 2
\[dq]other\[dq]
@@ -36356,7 +37633,9 @@ Other site/service or software
.RE
.SS \-\-webdav\-user
.PP
-User name
+User name.
+In case NTLM authentication is used, the username should be in the
+format \[aq]Domain\[aq].
.IP \[bu] 2
Config: user
.IP \[bu] 2
@@ -36405,6 +37684,24 @@ Env Var: RCLONE_WEBDAV_BEARER_TOKEN_COMMAND
Type: string
.IP \[bu] 2
Default: \[dq]\[dq]
+.SS \-\-webdav\-encoding
+.PP
+This sets the encoding for the backend.
+.PP
+See: the encoding section in the
+overview (https://rclone.org/overview/#encoding) for more info.
+.PP
+Default encoding is
+Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Hash,Percent,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8
+for sharepoint\-ntlm or identity otherwise.
+.IP \[bu] 2
+Config: encoding
+.IP \[bu] 2
+Env Var: RCLONE_WEBDAV_ENCODING
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
.SS Provider notes
.PP
See below for notes on specific providers.
@@ -36425,7 +37722,7 @@ Note that Nextcloud initially did not support streaming of files
this (https://github.com/nextcloud/nextcloud-snap/issues/365) seems to
be fixed as of 2020\-11\-27 (tested with rclone v1.53.1 and Nextcloud
Server v19).
-.SS Sharepoint
+.SS Sharepoint Online
.PP
Rclone can be used with Sharepoint provided by OneDrive for Business or
Office365 Education Accounts.
@@ -36465,11 +37762,43 @@ Your config file should look like this:
[sharepoint]
type = webdav
url = https://[YOUR\-DOMAIN]\-my.sharepoint.com/personal/[YOUR\-EMAIL]/Documents
-vendor = other
+vendor = sharepoint
user = YourEmailAddress
pass = encryptedpassword
\f[R]
.fi
+.SS Sharepoint with NTLM Authentication
+.PP
+Use this option in case your (hosted) Sharepoint is not tied to OneDrive
+accounts and uses NTLM authentication.
+.PP
+To get the \f[C]url\f[R] configuration, similarly to the above, first
+navigate to the desired directory in your browser to get the URL, then
+strip everything after the name of the opened directory.
+.PP
+Example: If the URL is:
+https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx
+.PP
+The configuration to use would be:
+https://example.sharepoint.com/sites/12345/Documents
+.PP
+Set the \f[C]vendor\f[R] to \f[C]sharepoint\-ntlm\f[R].
+.PP
+NTLM uses domain and user name combination for authentication, set
+\f[C]user\f[R] to \f[C]DOMAIN\[rs]username\f[R].
+.PP
+Your config file should look like this:
+.IP
+.nf
+\f[C]
+[sharepoint]
+type = webdav
+url = https://[YOUR\-DOMAIN]/some\-path\-to/Documents
+vendor = sharepoint\-ntlm
+user = DOMAIN\[rs]user
+pass = encryptedpassword
+\f[R]
+.fi
.SS Required Flags for SharePoint
.PP
As SharePoint does some special things with uploaded documents, you
@@ -36878,12 +38207,13 @@ y/e/d>
See the remote setup docs (https://rclone.org/remote_setup/) for how to
set it up on a machine with no Internet browser available.
.PP
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Zoho Workdrive.
-This only runs from the moment it opens your browser to the moment you
-get back the verification code.
-This is on \f[C]http://127.0.0.1:53682/\f[R] and this it may require you
-to unblock it temporarily if you are running a host firewall.
+Rclone runs a webserver on your local computer to collect the
+authorization token from Zoho Workdrive.
+This is only from the moment your browser is opened until the token is
+returned.
+The webserver runs on \f[C]http://127.0.0.1:53682/\f[R].
+If local port \f[C]53682\f[R] is protected by a firewall you may need to
+temporarily unblock the firewall to complete authorization.
.PP
Once configured you can then use \f[C]rclone\f[R] like this,
.PP
@@ -36941,6 +38271,28 @@ and will be removed from filenames during upload.
.SS Standard Options
.PP
Here are the standard options specific to zoho (Zoho).
+.SS \-\-zoho\-client\-id
+.PP
+OAuth Client Id Leave blank normally.
+.IP \[bu] 2
+Config: client_id
+.IP \[bu] 2
+Env Var: RCLONE_ZOHO_CLIENT_ID
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
+.SS \-\-zoho\-client\-secret
+.PP
+OAuth Client Secret Leave blank normally.
+.IP \[bu] 2
+Config: client_secret
+.IP \[bu] 2
+Env Var: RCLONE_ZOHO_CLIENT_SECRET
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
.SS \-\-zoho\-region
.PP
Zoho region to connect to.
@@ -36984,6 +38336,41 @@ Australia
.SS Advanced Options
.PP
Here are the advanced options specific to zoho (Zoho).
+.SS \-\-zoho\-token
+.PP
+OAuth Access Token as a JSON blob.
+.IP \[bu] 2
+Config: token
+.IP \[bu] 2
+Env Var: RCLONE_ZOHO_TOKEN
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
+.SS \-\-zoho\-auth\-url
+.PP
+Auth server URL.
+Leave blank to use the provider defaults.
+.IP \[bu] 2
+Config: auth_url
+.IP \[bu] 2
+Env Var: RCLONE_ZOHO_AUTH_URL
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
+.SS \-\-zoho\-token\-url
+.PP
+Token server url.
+Leave blank to use the provider defaults.
+.IP \[bu] 2
+Config: token_url
+.IP \[bu] 2
+Env Var: RCLONE_ZOHO_TOKEN_URL
+.IP \[bu] 2
+Type: string
+.IP \[bu] 2
+Default: \[dq]\[dq]
.SS \-\-zoho\-encoding
.PP
This sets the encoding for the backend.
@@ -37009,10 +38396,12 @@ rclone sync \-i /home/source /tmp/destination
\f[R]
.fi
.PP
-Will sync \f[C]/home/source\f[R] to \f[C]/tmp/destination\f[R]
+Will sync \f[C]/home/source\f[R] to \f[C]/tmp/destination\f[R].
.PP
-These can be configured into the config file for consistencies sake, but
-it is probably easier not to.
+For consistencies sake one can also configure a remote of type
+\f[C]local\f[R] in the config file, and access the local filesystem
+using rclone remote paths, e.g.
+\f[C]remote:path/to/wherever\f[R], but it is probably easier not to.
.SS Modified time
.PP
Rclone reads and writes the modified time using an accuracy determined
@@ -37412,17 +38801,44 @@ T}
Invalid UTF\-8 bytes will also be
replaced (https://rclone.org/overview/#invalid-utf8), as they can\[aq]t
be converted to UTF\-16.
-.SS Long paths on Windows
+.SS Paths on Windows
.PP
-Rclone handles long paths automatically, by converting all paths to long
-UNC
-paths (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath)
+On Windows there are many ways of specifying a path to a file system
+resource.
+Both absolute paths like \f[C]C:\[rs]path\[rs]to\[rs]wherever\f[R], and
+relative paths like \f[C]..\[rs]wherever\f[R] can be used, and path
+separator can be either \f[C]\[rs]\f[R] (as in
+\f[C]C:\[rs]path\[rs]to\[rs]wherever\f[R]) or \f[C]/\f[R] (as in
+\f[C]C:/path/to/wherever\f[R]).
+Length of these paths are limited to 259 characters for files and 247
+characters for directories, but there is an alternative extended\-length
+path format increasing the limit to (approximately) 32,767 characters.
+This format requires absolute paths and the use of prefix
+\f[C]\[rs]\[rs]?\[rs]\f[R], e.g.
+\f[C]\[rs]\[rs]?\[rs]D:\[rs]some\[rs]very\[rs]long\[rs]path\f[R].
+For convenience rclone will automatically convert regular paths into the
+corresponding extended\-length paths, so in most cases you do not have
+to worry about this (read more below).
+.PP
+Note that Windows supports using the same prefix
+\f[C]\[rs]\[rs]?\[rs]\f[R] to specify path to volumes identified by
+their GUID, e.g.
+\f[C]\[rs]\[rs]?\[rs]Volume{b75e2c83\-0000\-0000\-0000\-602f00000000}\[rs]some\[rs]path\f[R].
+This is \f[I]not\f[R] supported in rclone, due to an
+issue (https://github.com/golang/go/issues/39785) in go.
+.SS Long paths
+.PP
+Rclone handles long paths automatically, by converting all paths to
+extended\-length path
+format (https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation),
which allows paths up to 32,767 characters.
.PP
+This conversion will ensure paths are absolute and prefix them with the
+\f[C]\[rs]\[rs]?\[rs]\f[R].
This is why you will see that your paths, for instance
-\f[C]c:\[rs]files\f[R] is converted to the UNC path
-\f[C]\[rs]\[rs]?\[rs]c:\[rs]files\f[R] in the output, and
-\f[C]\[rs]\[rs]server\[rs]share\f[R] is converted to
+\f[C].\[rs]files\f[R] is shown as path
+\f[C]\[rs]\[rs]?\[rs]C:\[rs]files\f[R] in the output, and
+\f[C]\[rs]\[rs]server\[rs]share\f[R] as
\f[C]\[rs]\[rs]?\[rs]UNC\[rs]server\[rs]share\f[R].
.PP
However, in rare cases this may cause problems with buggy file system
@@ -37455,7 +38871,7 @@ And use rclone like this:
This will use UNC paths on \f[C]c:\[rs]src\f[R] but not on
\f[C]z:\[rs]dst\f[R].
Of course this will cause problems if the absolute path length of a file
-exceeds 258 characters on z, so only use this option if you have to.
+exceeds 259 characters on z, so only use this option if you have to.
.SS Symlinks / Junction points
.PP
Normally rclone will ignore symlinks or junction points (which behave
@@ -37638,9 +39054,9 @@ device as being on the same filesystem.
\f[B]NB\f[R] This flag is only available on Unix based systems.
On systems where it isn\[aq]t supported (e.g.
Windows) it will be ignored.
-.SS Standard Options
+.SS Advanced Options
.PP
-Here are the standard options specific to local (Local Disk).
+Here are the advanced options specific to local (Local Disk).
.SS \-\-local\-nounc
.PP
Disable UNC (long path names) conversion on Windows
@@ -37662,9 +39078,6 @@ Examples:
Disables long file names
.RE
.RE
-.SS Advanced Options
-.PP
-Here are the advanced options specific to local (Local Disk).
.SS \-\-copy\-links / \-L
.PP
Follow symlinks and copy the pointed to item.
@@ -37821,6 +39234,23 @@ Env Var: RCLONE_LOCAL_CASE_INSENSITIVE
Type: bool
.IP \[bu] 2
Default: false
+.SS \-\-local\-no\-preallocate
+.PP
+Disable preallocation of disk space for transferred files
+.PP
+Preallocation of disk space helps prevent filesystem fragmentation.
+However, some virtual filesystem layers (such as Google Drive File
+Stream) may incorrectly set the actual file size equal to the
+preallocated space, causing checksum and file size checks to fail.
+Use this flag to disable preallocation.
+.IP \[bu] 2
+Config: no_preallocate
+.IP \[bu] 2
+Env Var: RCLONE_LOCAL_NO_PREALLOCATE
+.IP \[bu] 2
+Type: bool
+.IP \[bu] 2
+Default: false
.SS \-\-local\-no\-sparse
.PP
Disable sparse files for multi\-thread downloads
@@ -37910,6 +39340,527 @@ Options:
.IP \[bu] 2
\[dq]error\[dq]: return an error based on option value
.SH Changelog
+.SS v1.55.0 \- 2021\-03\-31
+.PP
+See commits (https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)
+.IP \[bu] 2
+New commands
+.RS 2
+.IP \[bu] 2
+selfupdate (https://rclone.org/commands/rclone_selfupdate/) (Ivan
+Andreev)
+.RS 2
+.IP \[bu] 2
+Allows rclone to update itself in\-place or via a package (using
+\f[C]\-\-package\f[R] flag)
+.IP \[bu] 2
+Reads cryptographically signed signatures for non beta releases
+.IP \[bu] 2
+Works on all OSes.
+.RE
+.IP \[bu] 2
+test (https://rclone.org/commands/rclone_test/) \- these are test
+commands \- use with care!
+.RS 2
+.IP \[bu] 2
+\f[C]histogram\f[R] \- Makes a histogram of file name characters.
+.IP \[bu] 2
+\f[C]info\f[R] \- Discovers file name or other limitations for paths.
+.IP \[bu] 2
+\f[C]makefiles\f[R] \- Make a random file hierarchy for testing.
+.IP \[bu] 2
+\f[C]memory\f[R] \- Load all the objects at remote:path into memory and
+report memory stats.
+.RE
+.RE
+.IP \[bu] 2
+New Features
+.RS 2
+.IP \[bu] 2
+Connection strings (https://rclone.org/docs/#connection-strings)
+.RS 2
+.IP \[bu] 2
+Config parameters can now be passed as part of the remote name as a
+connection string.
+.IP \[bu] 2
+For example to do the equivalent of
+\f[C]\-\-drive\-shared\-with\-me\f[R] use
+\f[C]drive,shared_with_me:\f[R]
+.IP \[bu] 2
+Make sure we don\[aq]t save on the fly remote config to the config file
+(Nick Craig\-Wood)
+.IP \[bu] 2
+Make sure backends with additional config have a different name for
+caching (Nick Craig\-Wood)
+.IP \[bu] 2
+This work was sponsored by CERN, through the CS3MESH4EOSC
+Project (https://cs3mesh4eosc.eu/).
+.RS 2
+.IP \[bu] 2
+CS3MESH4EOSC has received funding from the European Union\[cq]s Horizon
+2020
+.IP \[bu] 2
+research and innovation programme under Grant Agreement no.
+863353.
+.RE
+.RE
+.IP \[bu] 2
+build
+.RS 2
+.IP \[bu] 2
+Update go build version to go1.16 and raise minimum go version to go1.13
+(Nick Craig\-Wood)
+.IP \[bu] 2
+Make a macOS ARM64 build to support Apple Silicon (Nick Craig\-Wood)
+.IP \[bu] 2
+Install macfuse 4.x instead of osxfuse 3.x (Nick Craig\-Wood)
+.IP \[bu] 2
+Use \f[C]GO386=softfloat\f[R] instead of deprecated \f[C]GO386=387\f[R]
+for 386 builds (Nick Craig\-Wood)
+.IP \[bu] 2
+Disable IOS builds for the time being (Nick Craig\-Wood)
+.IP \[bu] 2
+Androids builds made with up to date NDK (x0b)
+.IP \[bu] 2
+Add an rclone user to the Docker image but don\[aq]t use it by default
+(cynthia kwok)
+.RE
+.IP \[bu] 2
+dedupe: Make largest directory primary to minimize data moved (Saksham
+Khanna)
+.IP \[bu] 2
+config
+.RS 2
+.IP \[bu] 2
+Wrap config library in an interface (Fionera)
+.IP \[bu] 2
+Make config file system pluggable (Nick Craig\-Wood)
+.IP \[bu] 2
+\f[C]\-\-config \[dq]\[dq]\f[R] or \f[C]\[dq]/notfound\[dq]\f[R] for in
+memory config only (Nick Craig\-Wood)
+.IP \[bu] 2
+Clear fs cache of stale entries when altering config (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+copyurl: Add option to print resulting auto\-filename (albertony)
+.IP \[bu] 2
+delete: Make \f[C]\-\-rmdirs\f[R] obey the filters (Nick Craig\-Wood)
+.IP \[bu] 2
+docs \- many fixes and reworks from edwardxml, albertony, pvalls, Ivan
+Andreev, Evan Harris, buengese, Alexey Tabakman
+.IP \[bu] 2
+encoder/filename \- add SCSU as tables (Klaus Post)
+.IP \[bu] 2
+Add multiple paths support to \f[C]\-\-compare\-dest\f[R] and
+\f[C]\-\-copy\-dest\f[R] flag (K265)
+.IP \[bu] 2
+filter: Make \f[C]\-\-exclude \[dq]dir/\[dq]\f[R] equivalent to
+\f[C]\-\-exclude \[dq]dir/**\[dq]\f[R] (Nick Craig\-Wood)
+.IP \[bu] 2
+fshttp: Add DSCP support with \f[C]\-\-dscp\f[R] for QoS with
+differentiated services (Max Sum)
+.IP \[bu] 2
+lib/cache: Add Delete and DeletePrefix methods (Nick Craig\-Wood)
+.IP \[bu] 2
+lib/file
+.RS 2
+.IP \[bu] 2
+Make pre\-allocate detect disk full errors and return them (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Don\[aq]t run preallocate concurrently (Nick Craig\-Wood)
+.IP \[bu] 2
+Retry preallocate on EINTR (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+operations: Made copy and sync operations obey a RetryAfterError (Ankur
+Gupta)
+.IP \[bu] 2
+rc
+.RS 2
+.IP \[bu] 2
+Add string alternatives for setting options over the rc (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Add \f[C]options/local\f[R] to see the options configured in the context
+(Nick Craig\-Wood)
+.IP \[bu] 2
+Add \f[C]_config\f[R] parameter to set global config for just this rc
+call (Nick Craig\-Wood)
+.IP \[bu] 2
+Implement passing filter config with \f[C]_filter\f[R] parameter (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Add \f[C]fscache/clear\f[R] and \f[C]fscache/entries\f[R] to control the
+fs cache (Nick Craig\-Wood)
+.IP \[bu] 2
+Avoid +Inf value for speed in \f[C]core/stats\f[R] (albertony)
+.IP \[bu] 2
+Add a full set of stats to \f[C]core/stats\f[R] (Nick Craig\-Wood)
+.IP \[bu] 2
+Allow \f[C]fs=\f[R] params to be a JSON blob (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+rcd: Added systemd notification during the \f[C]rclone rcd\f[R] command.
+(Naveen Honest Raj)
+.IP \[bu] 2
+rmdirs: Make \f[C]\-\-rmdirs\f[R] obey the filters (Nick Craig\-Wood)
+.IP \[bu] 2
+version: Show build tags and type of executable (Ivan Andreev)
+.RE
+.IP \[bu] 2
+Bug Fixes
+.RS 2
+.IP \[bu] 2
+install.sh: make it fail on download errors (Ivan Andreev)
+.IP \[bu] 2
+Fix excessive retries missing \f[C]\-\-max\-duration\f[R] timeout (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix crash when \f[C]\-\-low\-level\-retries=0\f[R] (Nick Craig\-Wood)
+.IP \[bu] 2
+Fix failed token refresh on mounts created via the rc (Nick Craig\-Wood)
+.IP \[bu] 2
+fshttp: Fix bandwidth limiting after bad merge (Nick Craig\-Wood)
+.IP \[bu] 2
+lib/atexit
+.RS 2
+.IP \[bu] 2
+Unregister interrupt handler once it has fired so users can interrupt
+again (Nick Craig\-Wood)
+.IP \[bu] 2
+Fix occasional failure to unmount with CTRL\-C (Nick Craig\-Wood)
+.IP \[bu] 2
+Fix deadlock calling Finalise while Run is running (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+lib/rest: Fix multipart uploads not stopping on context cancel (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+Mount
+.RS 2
+.IP \[bu] 2
+Allow mounting to root directory on windows (albertony)
+.IP \[bu] 2
+Improved handling of relative paths on windows (albertony)
+.IP \[bu] 2
+Fix unicode issues with accented characters on macOS (Nick Craig\-Wood)
+.IP \[bu] 2
+Docs: document the new FileSecurity option in WinFsp 2021 (albertony)
+.IP \[bu] 2
+Docs: add note about volume path syntax on windows (albertony)
+.IP \[bu] 2
+Fix caching of old directories after renaming them (Nick Craig\-Wood)
+.IP \[bu] 2
+Update cgofuse to the latest version to bring in macfuse 4 fix (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+VFS
+.RS 2
+.IP \[bu] 2
+\f[C]\-\-vfs\-used\-is\-size\f[R] to report used space using recursive
+scan (tYYGH)
+.IP \[bu] 2
+Don\[aq]t set modification time if it was already correct (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix Create causing windows explorer to truncate files on CTRL\-C CTRL\-V
+(Nick Craig\-Wood)
+.IP \[bu] 2
+Fix modtimes not updating when writing via cache (Nick Craig\-Wood)
+.IP \[bu] 2
+Fix modtimes changing by fractional seconds after upload (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix modtime set if \f[C]\-\-vfs\-cache\-mode writes\f[R]/\f[C]full\f[R]
+and no write (Nick Craig\-Wood)
+.IP \[bu] 2
+Rename files in cache and cancel uploads on directory rename (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix directory renaming by renaming dirs cached in memory (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+Local
+.RS 2
+.IP \[bu] 2
+Add flag \f[C]\-\-local\-no\-preallocate\f[R] (David Sze)
+.IP \[bu] 2
+Make \f[C]nounc\f[R] an advanced option except on Windows (albertony)
+.IP \[bu] 2
+Don\[aq]t ignore preallocate disk full errors (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Cache
+.RS 2
+.IP \[bu] 2
+Add \f[C]\-\-fs\-cache\-expire\-duration\f[R] to control the fs cache
+(Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Crypt
+.RS 2
+.IP \[bu] 2
+Add option to not encrypt data (Vesnyx)
+.IP \[bu] 2
+Log hash ok on upload (albertony)
+.RE
+.IP \[bu] 2
+Azure Blob
+.RS 2
+.IP \[bu] 2
+Add container public access level support.
+(Manish Kumar)
+.RE
+.IP \[bu] 2
+B2
+.RS 2
+.IP \[bu] 2
+Fix HTML files downloaded via cloudflare (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Box
+.RS 2
+.IP \[bu] 2
+Fix transfers getting stuck on token expiry after API change (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+Chunker
+.RS 2
+.IP \[bu] 2
+Partially implement no\-rename transactions (Maxwell Calman)
+.RE
+.IP \[bu] 2
+Drive
+.RS 2
+.IP \[bu] 2
+Don\[aq]t stop server side copy if couldn\[aq]t read description (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Pass context on to drive SDK \- to help with cancellation (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+Dropbox
+.RS 2
+.IP \[bu] 2
+Add polling for changes support (Robert Thomas)
+.IP \[bu] 2
+Make \f[C]\-\-timeout 0\f[R] work properly (Nick Craig\-Wood)
+.IP \[bu] 2
+Raise priority of rate limited message to INFO to make it more
+noticeable (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Fichier
+.RS 2
+.IP \[bu] 2
+Implement copy & move (buengese)
+.IP \[bu] 2
+Implement public link (buengese)
+.RE
+.IP \[bu] 2
+FTP
+.RS 2
+.IP \[bu] 2
+Implement Shutdown method (Nick Craig\-Wood)
+.IP \[bu] 2
+Close idle connections after \f[C]\-\-ftp\-idle\-timeout\f[R] (1m by
+default) (Nick Craig\-Wood)
+.IP \[bu] 2
+Make \f[C]\-\-timeout 0\f[R] work properly (Nick Craig\-Wood)
+.IP \[bu] 2
+Add \f[C]\-\-ftp\-close\-timeout\f[R] flag for use with awkward ftp
+servers (Nick Craig\-Wood)
+.IP \[bu] 2
+Retry connections and logins on 421 errors (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Hdfs
+.RS 2
+.IP \[bu] 2
+Fix permissions for when directory is created (Lucas Messenger)
+.RE
+.IP \[bu] 2
+Onedrive
+.RS 2
+.IP \[bu] 2
+Make \f[C]\-\-timeout 0\f[R] work properly (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+S3
+.RS 2
+.IP \[bu] 2
+Fix \f[C]\-\-s3\-profile\f[R] which wasn\[aq]t working (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+SFTP
+.RS 2
+.IP \[bu] 2
+Close idle connections after \f[C]\-\-sftp\-idle\-timeout\f[R] (1m by
+default) (Nick Craig\-Wood)
+.IP \[bu] 2
+Fix \[dq]file not found\[dq] errors for read once servers (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix SetModTime stat failed: object not found with
+\f[C]\-\-sftp\-set\-modtime=false\f[R] (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Swift
+.RS 2
+.IP \[bu] 2
+Update github.com/ncw/swift to v2.0.0 (Nick Craig\-Wood)
+.IP \[bu] 2
+Implement copying large objects (nguyenhuuluan434)
+.RE
+.IP \[bu] 2
+Union
+.RS 2
+.IP \[bu] 2
+Fix crash when using epff policy (Nick Craig\-Wood)
+.IP \[bu] 2
+Fix union attempting to update files on a read only file system (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Refactor to use fspath.SplitFs instead of fs.ParseRemote (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix initialisation broken in refactor (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+WebDAV
+.RS 2
+.IP \[bu] 2
+Add support for sharepoint with NTLM authentication (Rauno Ots)
+.IP \[bu] 2
+Make sharepoint\-ntlm docs more consistent (Alex Chen)
+.IP \[bu] 2
+Improve terminology in sharepoint\-ntlm docs (Ivan Andreev)
+.IP \[bu] 2
+Disable HTTP/2 for NTLM authentication (georne)
+.IP \[bu] 2
+Fix sharepoint\-ntlm error 401 for parallel actions (Ivan Andreev)
+.IP \[bu] 2
+Check that purged directory really exists (Ivan Andreev)
+.RE
+.IP \[bu] 2
+Yandex
+.RS 2
+.IP \[bu] 2
+Make \f[C]\-\-timeout 0\f[R] work properly (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Zoho
+.RS 2
+.IP \[bu] 2
+Replace client id \- you will need to \f[C]rclone config reconnect\f[R]
+after this (buengese)
+.IP \[bu] 2
+Add forgotten setupRegion() to NewFs \- this finally fixes regions other
+than EU (buengese)
+.RE
+.SS v1.54.1 \- 2021\-03\-08
+.PP
+See commits (https://github.com/rclone/rclone/compare/v1.54.0...v1.54.1)
+.IP \[bu] 2
+Bug Fixes
+.RS 2
+.IP \[bu] 2
+accounting: Fix \-\-bwlimit when up or down is off (Nick Craig\-Wood)
+.IP \[bu] 2
+docs
+.RS 2
+.IP \[bu] 2
+Fix nesting of brackets and backticks in ftp docs (edwardxml)
+.IP \[bu] 2
+Fix broken link in sftp page (edwardxml)
+.IP \[bu] 2
+Fix typo in crypt.md (Romeo Kienzler)
+.IP \[bu] 2
+Changelog: Correct link to digitalis.io (Alex JOST)
+.IP \[bu] 2
+Replace #file\-caching with #vfs\-file\-caching (Miron Veryanskiy)
+.IP \[bu] 2
+Convert bogus example link to code (edwardxml)
+.IP \[bu] 2
+Remove dead link from rc.md (edwardxml)
+.RE
+.IP \[bu] 2
+rc: Sync,copy,move: document createEmptySrcDirs parameter (Nick
+Craig\-Wood)
+.IP \[bu] 2
+lsjson: Fix unterminated JSON in the presence of errors (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+Mount
+.RS 2
+.IP \[bu] 2
+Fix mount dropping on macOS by setting \-\-daemon\-timeout 10m (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+VFS
+.RS 2
+.IP \[bu] 2
+Document simultaneous usage with the same cache shouldn\[aq]t be used
+(Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+B2
+.RS 2
+.IP \[bu] 2
+Automatically raise upload cutoff to avoid spurious error (Nick
+Craig\-Wood)
+.IP \[bu] 2
+Fix failed to create file system with application key limited to a
+prefix (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Drive
+.RS 2
+.IP \[bu] 2
+Refer to Shared Drives instead of Team Drives (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Dropbox
+.RS 2
+.IP \[bu] 2
+Add scopes to oauth request and optionally \[dq]members.read\[dq] (Nick
+Craig\-Wood)
+.RE
+.IP \[bu] 2
+S3
+.RS 2
+.IP \[bu] 2
+Fix failed to create file system with folder level permissions policy
+(Nick Craig\-Wood)
+.IP \[bu] 2
+Fix Wasabi HEAD requests returning stale data by using only 1 transport
+(Nick Craig\-Wood)
+.IP \[bu] 2
+Fix shared_credentials_file auth (Dmitry Chepurovskiy)
+.IP \[bu] 2
+Add \-\-s3\-no\-head to reducing costs docs (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Union
+.RS 2
+.IP \[bu] 2
+Fix mkdir at root with remote:/ (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+Zoho
+.RS 2
+.IP \[bu] 2
+Fix custom client id\[aq]s (buengese)
+.RE
.SS v1.54.0 \- 2021\-02\-02
.PP
See commits (https://github.com/rclone/rclone/compare/v1.53.0...v1.54.0)
@@ -37917,7 +39868,7 @@ See commits (https://github.com/rclone/rclone/compare/v1.53.0...v1.54.0)
New backends
.RS 2
.IP \[bu] 2
-Compression remote (experimental)(buengese)
+Compression remote (experimental) (buengese)
.IP \[bu] 2
Enterprise File Fabric (Nick Craig\-Wood)
.RS 2
@@ -37939,9 +39890,9 @@ Deglobalise the config (Nick Craig\-Wood)
.IP \[bu] 2
Global config now read from the context
.IP \[bu] 2
-Global config can be passed into the rc
+This will enable passing of global config via the rc
.IP \[bu] 2
-This work was sponsored by Digitalis (digitalis.io)
+This work was sponsored by Digitalis (https://digitalis.io/)
.RE
.IP \[bu] 2
Add \f[C]\-\-bwlimit\f[R] for upload and download (Nick Craig\-Wood)
@@ -37953,18 +39904,21 @@ Obey bwlimit in http Transport for better limiting
Enhance systemd integration (Hekmon)
.RS 2
.IP \[bu] 2
-log level identification
-.IP \[bu] 2
-manual activation with flag
-.IP \[bu] 2
-automatic systemd launch detection
+log level identification, manual activation with flag, automatic systemd
+launch detection
.IP \[bu] 2
Don\[aq]t compile systemd log integration for non unix systems (Benjamin
Gustin)
.RE
.IP \[bu] 2
-Add a download flag to hashsum and related commands to force rclone to
-download and hash files locally (lostheli)
+Add a \f[C]\-\-download\f[R] flag to md5sum/sha1sum/hashsum to force
+rclone to download and hash files locally (lostheli)
+.IP \[bu] 2
+Add \f[C]\-\-progress\-terminal\-title\f[R] to print ETA to terminal
+title (LaSombra)
+.IP \[bu] 2
+Make backend env vars show in help as the defaults for backend flags
+(Nick Craig\-Wood)
.IP \[bu] 2
build
.RS 2
@@ -37972,51 +39926,33 @@ build
Raise minimum go version to go1.12 (Nick Craig\-Wood)
.RE
.IP \[bu] 2
-check
-.RS 2
-.IP \[bu] 2
-Make the error count match up in the log message (Nick Craig\-Wood)
-.RE
-.IP \[bu] 2
-cmd
-.RS 2
-.IP \[bu] 2
-Add \-\-progress\-terminal\-title to print ETA to terminal title
-(LaSombra)
-.IP \[bu] 2
-Make backend env vars show in help as the defaults for backend flags
-(Nick Craig\-Wood)
-.RE
-.IP \[bu] 2
dedupe
.RS 2
.IP \[bu] 2
-Add \-\-by\-hash to dedupe on hash not file name (Nick Craig\-Wood)
-.IP \[bu] 2
-Add \-\-dedupe\-mode list to just list dupes, changing nothing (Nick
+Add \f[C]\-\-by\-hash\f[R] to dedupe on content hash not file name (Nick
Craig\-Wood)
.IP \[bu] 2
+Add \f[C]\-\-dedupe\-mode list\f[R] to just list dupes, changing nothing
+(Nick Craig\-Wood)
+.IP \[bu] 2
Add warning if used on a remote which can\[aq]t have duplicate names
(Nick Craig\-Wood)
.RE
.IP \[bu] 2
-flags: Improve error message when reading environment vars (Nick
-Craig\-Wood)
-.IP \[bu] 2
fs
.RS 2
.IP \[bu] 2
Add Shutdown optional method for backends (Nick Craig\-Wood)
.IP \[bu] 2
-When using \-\-files\-from check files concurrently (zhucan)
+When using \f[C]\-\-files\-from\f[R] check files concurrently (zhucan)
.IP \[bu] 2
-Accumulate stats when using \-\-dry\-run (Ingo Weiss)
+Accumulate stats when using \f[C]\-\-dry\-run\f[R] (Ingo Weiss)
.IP \[bu] 2
-Always show stats when using \-\-dry\-run or \-\-interactive (Nick
-Craig\-Wood)
+Always show stats when using \f[C]\-\-dry\-run\f[R] or
+\f[C]\-\-interactive\f[R] (Nick Craig\-Wood)
.IP \[bu] 2
-Add support for flag \-\-no\-console on windows to hide the console
-window (albertony)
+Add support for flag \f[C]\-\-no\-console\f[R] on windows to hide the
+console window (albertony)
.RE
.IP \[bu] 2
genautocomplete: Add support to output to stdout (Ingo)
@@ -38028,13 +39964,13 @@ Highlight read errors instead of aborting (Claudio Bantaloukas)
.IP \[bu] 2
Add sort by average size in directory (Adam Pl\['a]nsk\['y])
.IP \[bu] 2
-Add toggle option for average size in directory \- key \[aq]a\[aq] (Adam
-Pl\['a]nsk\['y])
+Add toggle option for average s3ize in directory \- key \[aq]a\[aq]
+(Adam Pl\['a]nsk\['y])
.IP \[bu] 2
Add empty folder flag into ncdu browser (Adam Pl\['a]nsk\['y])
.IP \[bu] 2
-Add ! (errror) and .
-(unreadable) file flags to go with e (empty) (Nick Craig\-Wood)
+Add \f[C]!\f[R] (errror) and \f[C].\f[R] (unreadable) file flags to go
+with \f[C]e\f[R] (empty) (Nick Craig\-Wood)
.RE
.IP \[bu] 2
obscure: Make \f[C]rclone osbcure \-\f[R] ignore newline at end of line
@@ -38085,21 +40021,6 @@ Fix plugins initialization (negative0)
Bug Fixes
.RS 2
.IP \[bu] 2
-build
-.RS 2
-.IP \[bu] 2
-Explicitly set ARM version to fix build (Nick Craig\-Wood)
-.IP \[bu] 2
-Don\[aq]t explicitly set ARM version to fix ARMv5 build (Nick
-Craig\-Wood)
-.IP \[bu] 2
-Fix nfpm install (Nick Craig\-Wood)
-.IP \[bu] 2
-Fix docker build by upgrading ilteoood/docker_buildx (Nick Craig\-Wood)
-.IP \[bu] 2
-Temporary fix for Windows build errors (Ivan Andreev)
-.RE
-.IP \[bu] 2
fs
.RS 2
.IP \[bu] 2
@@ -38110,26 +40031,35 @@ Fix parsing of ..
when joining remotes (Nick Craig\-Wood)
.RE
.IP \[bu] 2
-log: Fix enabling systemd logging when using \-\-log\-file (Nick
-Craig\-Wood)
+log: Fix enabling systemd logging when using \f[C]\-\-log\-file\f[R]
+(Nick Craig\-Wood)
.IP \[bu] 2
-move: Fix data loss when moving the same object (Nick Craig\-Wood)
+check
+.RS 2
+.IP \[bu] 2
+Make the error count match up in the log message (Nick Craig\-Wood)
+.RE
+.IP \[bu] 2
+move: Fix data loss when source and destination are the same object
+(Nick Craig\-Wood)
.IP \[bu] 2
operations
.RS 2
.IP \[bu] 2
-Fix \-\-cutof\-mode hard not cutting off immediately (Nick Craig\-Wood)
+Fix \f[C]\-\-cutof\-mode\f[R] hard not cutting off immediately (Nick
+Craig\-Wood)
.IP \[bu] 2
-Fix \-\-immutable error message (Nick Craig\-Wood)
+Fix \f[C]\-\-immutable\f[R] error message (Nick Craig\-Wood)
.RE
.IP \[bu] 2
sync
.RS 2
.IP \[bu] 2
-Fix \-\-cutoff\-mode soft & cautious so it doesn\[aq]t end the transfer
-early (Nick Craig\-Wood)
+Fix \f[C]\-\-cutoff\-mode\f[R] soft & cautious so it doesn\[aq]t end the
+transfer early (Nick Craig\-Wood)
.IP \[bu] 2
-Fix \-\-immutable errors retrying many times (Nick Craig\-Wood)
+Fix \f[C]\-\-immutable\f[R] errors retrying many times (Nick
+Craig\-Wood)
.RE
.RE
.IP \[bu] 2
@@ -38138,7 +40068,7 @@ Docs
.IP \[bu] 2
Many fixes and a rewrite of the filtering docs (edwardxml)
.IP \[bu] 2
-Many spelling and grammar problems (Josh Soref)
+Many spelling and grammar fixes (Josh Soref)
.IP \[bu] 2
Doc fixes for commands delete, purge, rmdir, rmdirs and mount
(albertony)
@@ -38164,20 +40094,16 @@ Update systemd status with cache stats (Hekmon)
Disable bazil/fuse based mount on macOS (Nick Craig\-Wood)
.RS 2
.IP \[bu] 2
-Make mount be cmount under macOS (Nick Craig\-Wood)
+Make \f[C]rclone mount\f[R] actually run \f[C]rclone cmount\f[R] under
+macOS (Nick Craig\-Wood)
.RE
.IP \[bu] 2
Implement mknod to make NFS file creation work (Nick Craig\-Wood)
.IP \[bu] 2
Make sure we don\[aq]t call umount more than once (Nick Craig\-Wood)
.IP \[bu] 2
-Don\[aq]t call host.Umount if a signal has been received (Nick
-Craig\-Wood)
-.IP \[bu] 2
More user friendly mounting as network drive on windows (albertony)
.IP \[bu] 2
-Cleanup OS specific option handling and documentation (albertony)
-.IP \[bu] 2
Detect if uid or gid are set in same option string: \-o uid=123,gid=456
(albertony)
.IP \[bu] 2
@@ -38194,8 +40120,8 @@ Craig\-Wood)
Fix \[dq]file already exists\[dq] error for stale cache files (Nick
Craig\-Wood)
.IP \[bu] 2
-Fix file leaks with \-\-vfs\-cache\-mode full and \-\-buffer\-size 0
-(Nick Craig\-Wood)
+Fix file leaks with \f[C]\-\-vfs\-cache\-mode\f[R] full and
+\f[C]\-\-buffer\-size 0\f[R] (Nick Craig\-Wood)
.IP \[bu] 2
Fix invalid cache path on windows when using :backend: as remote
(albertony)
@@ -38207,8 +40133,8 @@ Local
Continue listing files/folders when a circular symlink is detected
(Manish Gupta)
.IP \[bu] 2
-New flag \-\-local\-zero\-size\-links to fix sync on some virtual
-filesystems (Riccardo Iaconelli)
+New flag \f[C]\-\-local\-zero\-size\-links\f[R] to fix sync on some
+virtual filesystems (Riccardo Iaconelli)
.RE
.IP \[bu] 2
Azure Blob
@@ -38216,26 +40142,24 @@ Azure Blob
.IP \[bu] 2
Add support for service principals (James Lim)
.IP \[bu] 2
-Utilize streaming capabilities (Denis Neuling)
+Add support for managed identities (Brad Ackerman)
.IP \[bu] 2
-Update SDK to v0.13.0 and fix API breakage (Nick Craig\-Wood, Mitsuo
-Heijo)
+Add examples for access tier (Bob Pusateri)
+.IP \[bu] 2
+Utilize the streaming capabilities from the SDK for multipart uploads
+(Denis Neuling)
.IP \[bu] 2
Fix setting of mime types (Nick Craig\-Wood)
.IP \[bu] 2
Fix crash when listing outside a SAS URL\[aq]s root (Nick Craig\-Wood)
.IP \[bu] 2
Delete archive tier blobs before update if
-\-\-azureblob\-archive\-tier\-delete (Nick Craig\-Wood)
-.IP \[bu] 2
-Add support for managed identities (Brad Ackerman)
+\f[C]\-\-azureblob\-archive\-tier\-delete\f[R] (Nick Craig\-Wood)
.IP \[bu] 2
Fix crash on startup (Nick Craig\-Wood)
.IP \[bu] 2
-Add examples for access tier (Bob Pusateri)
-.IP \[bu] 2
-Fix memory usage by upgrading the SDK and implementing a TransferManager
-(Nick Craig\-Wood)
+Fix memory usage by upgrading the SDK to v0.13.0 and implementing a
+TransferManager (Nick Craig\-Wood)
.IP \[bu] 2
Require go1.14+ to compile due to SDK changes (Nick Craig\-Wood)
.RE
@@ -38244,8 +40168,13 @@ B2
.RS 2
.IP \[bu] 2
Make NewObject use less expensive API calls (Nick Craig\-Wood)
+.RS 2
.IP \[bu] 2
-Fixed possible crash when accessing Backblaze b2 remote (lluuaapp)
+This will improve \f[C]\-\-files\-from\f[R] and \f[C]restic serve\f[R]
+in particular
+.RE
+.IP \[bu] 2
+Fixed crash on an empty file name (lluuaapp)
.RE
.IP \[bu] 2
Box
@@ -38261,8 +40190,8 @@ Chunker
.IP \[bu] 2
Skip long local hashing, hash in\-transit (fixes) (Ivan Andreev)
.IP \[bu] 2
-Set Features.ReadMimeType=false as Object.MimeType not supported (Nick
-Craig\-Wood)
+Set Features ReadMimeType to false as Object.MimeType not supported
+(Nick Craig\-Wood)
.IP \[bu] 2
Fix case\-insensitive NewObject, test metadata detection (Ivan Andreev)
.RE
@@ -38270,7 +40199,7 @@ Fix case\-insensitive NewObject, test metadata detection (Ivan Andreev)
Drive
.RS 2
.IP \[bu] 2
-Implement \[dq]rclone backend copyid\[dq] command for copying files by
+Implement \f[C]rclone backend copyid\f[R] command for copying files by
ID (Nick Craig\-Wood)
.IP \[bu] 2
Added flag \f[C]\-\-drive\-stop\-on\-download\-limit\f[R] to stop
@@ -38290,36 +40219,35 @@ Dropbox
.IP \[bu] 2
Add support for viewing shared files and folders (buengese)
.IP \[bu] 2
-Implement IDer (buengese)
+Enable short lived access tokens (Nick Craig\-Wood)
.IP \[bu] 2
-Set Features.ReadMimeType=false as Object.MimeType not supported (Nick
-Craig\-Wood)
+Implement IDer on Objects so \f[C]rclone lsf\f[R] etc can read the IDs
+(buengese)
.IP \[bu] 2
-Tidy repeated error message (Nick Craig\-Wood)
+Set Features ReadMimeType to false as Object.MimeType not supported
+(Nick Craig\-Wood)
.IP \[bu] 2
Make malformed_path errors from too long files not retriable (Nick
Craig\-Wood)
.IP \[bu] 2
Test file name length before upload to fix upload loop (Nick
Craig\-Wood)
-.IP \[bu] 2
-Enable short lived access tokens (Nick Craig\-Wood)
.RE
.IP \[bu] 2
Fichier
.RS 2
.IP \[bu] 2
-Set Features.ReadMimeType=true as Object.MimeType is supported (Nick
+Set Features ReadMimeType to true as Object.MimeType is supported (Nick
Craig\-Wood)
.RE
.IP \[bu] 2
FTP
.RS 2
.IP \[bu] 2
-Add \-\-ftp\-disable\-msld option to ignore MLSD for really old servers
-(Nick Craig\-Wood)
+Add \f[C]\-\-ftp\-disable\-msld\f[R] option to ignore MLSD for really
+old servers (Nick Craig\-Wood)
.IP \[bu] 2
-Make \-\-tpslimit apply (Nick Craig\-Wood)
+Make \f[C]\-\-tpslimit apply\f[R] (Nick Craig\-Wood)
.RE
.IP \[bu] 2
Google Cloud Storage
@@ -38329,27 +40257,33 @@ Storage class object header support (Laurens Janssen)
.IP \[bu] 2
Fix anonymous client to use rclone\[aq]s HTTP client (Nick Craig\-Wood)
.IP \[bu] 2
-Fix Entry doesn\[aq]t belong in directory \[dq]\[dq] (same as directory)
-\- ignoring (Nick Craig\-Wood)
+Fix
+\f[C]Entry doesn\[aq]t belong in directory \[dq]\[dq] (same as directory) \- ignoring\f[R]
+(Nick Craig\-Wood)
.RE
.IP \[bu] 2
Googlephotos
.RS 2
.IP \[bu] 2
-New flag \-\-gphotos\-include\-archived (Nicolas Rueff)
+New flag \f[C]\-\-gphotos\-include\-archived\f[R] to show archived
+photos as well (Nicolas Rueff)
.RE
.IP \[bu] 2
Jottacloud
.RS 2
.IP \[bu] 2
-Don\[aq]t erroniously report support for writing mime types (buengese)
+Don\[aq]t erroneously report support for writing mime types (buengese)
.IP \[bu] 2
-Add support for Telia Cloud (#4930) (Patrik Nordl\['e]n)
+Add support for Telia Cloud (Patrik Nordl\['e]n)
.RE
.IP \[bu] 2
Mailru
.RS 2
.IP \[bu] 2
+Accept special folders eg camera\-upload (Ivan Andreev)
+.IP \[bu] 2
+Avoid prehashing of large local files (Ivan Andreev)
+.IP \[bu] 2
Fix uploads after recent changes on server (Ivan Andreev)
.IP \[bu] 2
Fix range requests after June 2020 changes on server (Ivan Andreev)
@@ -38357,10 +40291,6 @@ Fix range requests after June 2020 changes on server (Ivan Andreev)
Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
.IP \[bu] 2
Remove deprecated protocol quirks (Ivan Andreev)
-.IP \[bu] 2
-Accept special folders eg camera\-upload (Ivan Andreev)
-.IP \[bu] 2
-Avoid prehashing of large local files (Ivan Andreev)
.RE
.IP \[bu] 2
Memory
@@ -38372,26 +40302,25 @@ Fix setting of mime types (Nick Craig\-Wood)
Onedrive
.RS 2
.IP \[bu] 2
-Add support for china region operated by 21vianet and other regional
-suppliers (#4963) (NyaMisty)
+Add support for China region operated by 21vianet and other regional
+suppliers (NyaMisty)
.IP \[bu] 2
Warn on gateway timeout errors (Nick Craig\-Wood)
.IP \[bu] 2
-Fall back to normal copy if server\-side copy unavailable (#4903) (Alex
-Chen)
+Fall back to normal copy if server\-side copy unavailable (Alex Chen)
.IP \[bu] 2
Fix server\-side copy completely disabled on OneDrive for Business
(Cnly)
.IP \[bu] 2
(business only) workaround to replace existing file on server\-side copy
-(#4904) (Alex Chen)
+(Alex Chen)
.IP \[bu] 2
Enhance link creation with expiry, scope, type and password (Nick
Craig\-Wood)
.IP \[bu] 2
-Remove % and # from the set of encoded characters (#4909) (Alex Chen)
+Remove % and # from the set of encoded characters (Alex Chen)
.IP \[bu] 2
-Support addressing site by server\-relative URL (#4761) (kice)
+Support addressing site by server\-relative URL (kice)
.RE
.IP \[bu] 2
Opendrive
@@ -38427,11 +40356,16 @@ Added \f[C]\-\-s3\-disable\-http2\f[R] to disable http/2 (Anagh Kumar
Baranwal)
.IP \[bu] 2
Complete SSE\-C implementation (Nick Craig\-Wood)
+.RS 2
.IP \[bu] 2
Fix hashes on small files with AWS:KMS and SSE\-C (Nick Craig\-Wood)
.IP \[bu] 2
Add MD5 metadata to objects uploaded with SSE\-AWS/SSE\-C (Nick
Craig\-Wood)
+.RE
+.IP \[bu] 2
+Add \f[C]\-\-s3\-no\-head parameter\f[R] to minimise transactions on
+upload (Nick Craig\-Wood)
.IP \[bu] 2
Update docs with a Reducing Costs section (Nick Craig\-Wood)
.IP \[bu] 2
@@ -38442,9 +40376,6 @@ Add requester pays option (kelv)
.IP \[bu] 2
Fix copy multipart with v2 auth failing with
\[aq]SignatureDoesNotMatch\[aq] (Louis Koo)
-.IP \[bu] 2
-Add \-\-s3\-no\-head parameter to minimise transactions on upload (Nick
-Craig\-Wood)
.RE
.IP \[bu] 2
SFTP
@@ -38464,9 +40395,10 @@ Implement Shutdown method (Nick Craig\-Wood)
.IP \[bu] 2
Implement keyboard interactive authentication (Nick Craig\-Wood)
.IP \[bu] 2
-Make \-\-tpslimit apply (Nick Craig\-Wood)
+Make \f[C]\-\-tpslimit\f[R] apply (Nick Craig\-Wood)
.IP \[bu] 2
-Implement \-\-sftp\-use\-fstat (Nick Craig\-Wood)
+Implement \f[C]\-\-sftp\-use\-fstat\f[R] for unusual SFTP servers (Nick
+Craig\-Wood)
.RE
.IP \[bu] 2
Sugarsync
@@ -38484,7 +40416,8 @@ Fix deletion of parts of Static Large Object (SLO) (Nguy\[u1EC5]n
H\[u1EEF]u Lu\[^a]n)
.IP \[bu] 2
Ensure partially uploaded large files are uploaded unless
-\-\-swift\-leave\-parts\-on\-error (Nguy\[u1EC5]n H\[u1EEF]u Lu\[^a]n)
+\f[C]\-\-swift\-leave\-parts\-on\-error\f[R] (Nguy\[u1EC5]n H\[u1EEF]u
+Lu\[^a]n)
.RE
.IP \[bu] 2
Tardigrade
@@ -38502,7 +40435,7 @@ Updated docs to show streaming to nextcloud is working (Durval Menezes)
Yandex
.RS 2
.IP \[bu] 2
-Set Features.WriteMimeType=false as Yandex ignores mime types (Nick
+Set Features WriteMimeType to false as Yandex ignores mime types (Nick
Craig\-Wood)
.RE
.SS v1.53.4 \- 2021\-01\-20
@@ -47313,6 +49246,50 @@ Nicolas Rueff
Pau Rodriguez\-Estivill
.IP \[bu] 2
Bob Pusateri
+.IP \[bu] 2
+Alex JOST <25005220+dimejo@users.noreply.github.com>
+.IP \[bu] 2
+Alexey Tabakman
+.IP \[bu] 2
+David Sze
+.IP \[bu] 2
+cynthia kwok
+.IP \[bu] 2
+Ankur Gupta
+.IP \[bu] 2
+Miron Veryanskiy
+.IP \[bu] 2
+K265
+.IP \[bu] 2
+Vesnyx
+.IP \[bu] 2
+Dmitry Chepurovskiy
+.IP \[bu] 2
+Rauno Ots
+.IP \[bu] 2
+Georg Neugschwandtner
+.IP \[bu] 2
+pvalls
+.IP \[bu] 2
+Robert Thomas <31854736+wolveix@users.noreply.github.com>
+.IP \[bu] 2
+Romeo Kienzler
+.IP \[bu] 2
+tYYGH
+.IP \[bu] 2
+georne <77802995+georne@users.noreply.github.com>
+.IP \[bu] 2
+Maxwell Calman
+.IP \[bu] 2
+Naveen Honest Raj
+.IP \[bu] 2
+Lucas Messenger
+.IP \[bu] 2
+Manish Kumar
+.IP \[bu] 2
+x0b
+.IP \[bu] 2
+CERN through the CS3MESH4EOSC Project
.SH Contact the rclone project
.SS Forum
.PP