diff --git a/MANUAL.html b/MANUAL.html index 73fda2731..d16296fa0 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -81,7 +81,7 @@

rclone(1) User Manual

Nick Craig-Wood

-

Sep 11, 2023

+

Nov 26, 2023

Rclone syncs your files to cloud storage

rclone logo

@@ -170,11 +170,14 @@
  • Koofr
  • Leviia Object Storage
  • Liara Object Storage
    • +
  • Linkbox
    • +
  • Linode Object Storage
  • Mail.ru Cloud
  • Memset Memstore
  • Mega
  • Memory
  • Microsoft Azure Blob Storage
    • +
  • Microsoft Azure Files Storage
  • Microsoft OneDrive
  • Minio
  • Nextcloud
    • @@ -275,6 +278,12 @@ sudo mandb

    NOTE: This version of rclone will not support mount any more (see #5373). If mounting is wanted on macOS, either install a precompiled binary or enable the relevant option when installing from source.

    Note that this is a third party installer not controlled by the rclone developers so it may be out of date. Its current version is as below.

    Homebrew package

    +

    Installation with MacPorts (#macos-macports)

    +

    On macOS, rclone can also be installed via MacPorts:

    +
    sudo port install rclone
    +

    Note that this is a third party installer not controlled by the rclone developers so it may be out of date. Its current version is as below.

    +

    MacPorts port

    +

    More information here.

    Precompiled binary, using curl

    To avoid problems with macOS gatekeeper enforcing the binary to be signed and notarized it is enough to download with curl.

    Download the latest version of rclone.

    @@ -392,8 +401,8 @@ kill %1

    Snap installation

    Get it from the Snap Store

    Make sure you have Snapd installed

    -
    $ sudo snap install rclone
    -

    Due to the strict confinement of Snap, rclone snap cannot acess real /home/$USER/.config/rclone directory, default config path is as below.

    +
    $ sudo snap install rclone
    +

    Due to the strict confinement of Snap, rclone snap cannot access real /home/$USER/.config/rclone directory, default config path is as below.

    The default number of parallel checks is 8. See the --checkers=N option for more information.

    -
    rclone checksum <hash> sumfile src:path [flags]
    +
    rclone checksum <hash> sumfile dst:path [flags]

    Options

          --combined string         Make a combined report of changes to this file
       --differ string           Report all non-matching files to this file
@@ -2194,11 +2213,11 @@ if src is directory
   -c, --checksum                                    Check for changes with size & checksum (if available, or fallback to size only).
       --compare-dest stringArray                    Include additional comma separated server-side paths during comparison
       --copy-dest stringArray                       Implies --compare-dest but also copies files from paths into destination
-      --cutoff-mode string                          Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+      --cutoff-mode HARD|SOFT|CAUTIOUS              Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
       --ignore-case-sync                            Ignore case when synchronizing
       --ignore-checksum                             Skip post copy check of checksums
       --ignore-existing                             Skip all files that exist on destination
-      --ignore-size                                 Ignore size when skipping use mod-time or checksum
+      --ignore-size                                 Ignore size when skipping use modtime or checksum
   -I, --ignore-times                                Don't skip files that match size and time - transfer all files
       --immutable                                   Do not modify files, fail if existing files have been modified
       --inplace                                     Download directly to destination file instead of atomic download to temp/rename
@@ -2213,11 +2232,12 @@ if src is directory
       --multi-thread-write-buffer-size SizeSuffix   In memory buffer size for writing when in multi-thread mode (default 128Ki)
       --no-check-dest                               Don't check the destination, copy regardless
       --no-traverse                                 Don't traverse destination file system on copy
-      --no-update-modtime                           Don't update destination mod-time if files identical
+      --no-update-modtime                           Don't update destination modtime if files identical
       --order-by string                             Instructions on how to order the transfers, e.g. 'size,descending'
+      --partial-suffix string                       Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
       --refresh-times                               Refresh the modtime of remote files
       --server-side-across-configs                  Allow server-side operations (e.g. copy) to work across different configs
-      --size-only                                   Skip based on size only, not mod-time or checksum
+      --size-only                                   Skip based on size only, not modtime or checksum
       --streaming-upload-cutoff SizeSuffix          Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
   -u, --update                                      Skip files that are newer on the destination

    Important Options

    @@ -2481,15 +2501,11 @@ Supported hashes are: * sha1 * whirlpool * crc32 - * sha256 - * dropbox - * hidrive - * mailru - * quickxor + * sha256

    Then

    $ rclone hashsum MD5 remote:path

    Note that hash names are case insensitive and values are output in lower case.

    -
    rclone hashsum <hash> remote:path [flags]
    +
    rclone hashsum [<hash> remote:path] [flags]

    Options

          --base64               Output base64 encoded hashsum
   -C, --checkfile string     Validate hashes against a given SUM file instead of printing them
@@ -2840,7 +2856,9 @@ rclone mount remote:path/to/files * --volname \\cloud\remote

    It is also possible to make a drive mount available to everyone on the system, by running the process creating it as the built-in SYSTEM account. There are several ways to do this: One is to use the command-line utility PsExec, from Microsoft's Sysinternals suite, which has option -s to start processes as the SYSTEM account. Another alternative is to run the mount command from a Windows Scheduled Task, or a Windows Service, configured to run as the SYSTEM account. A third alternative is to use the WinFsp.Launcher infrastructure). Read more in the install documentation. Note that when running rclone as another user, it will not use the configuration file from your profile unless you tell it to with the --config option. Note also that it is now the SYSTEM account that will have the owner permissions, and other accounts will have permissions according to the group or others scopes. As mentioned above, these will then not get the "write extended attributes" permission, and this may prevent writing to files. You can work around this with the FileSecurity option, see example above.

    Note that mapping to a directory path, instead of a drive letter, does not suffer from the same limitations.

    Mounting on macOS

    -

    Mounting on macOS can be done either via macFUSE (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which "mounts" via an NFSv4 local server.

    +

    Mounting on macOS can be done either via built-in NFS server, macFUSE (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which "mounts" via an NFSv4 local server.

    +

    NFS mount

    +

    This method spins up an NFS server using serve nfs command and mounts it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to send SIGTERM signal to the rclone process using |kill| command to stop the mount.

    macFUSE Notes

    If installing macFUSE using dmg packages from the website, rclone will locate the macFUSE libraries without any further intervention. If however, macFUSE is installed using the macports package manager, the following addition steps are required.

    sudo mkdir /usr/local/lib
@@ -2860,7 +2878,7 @@ sudo ln -s /opt/local/lib/libfuse.2.dylib

    Read Only mounts

    When mounting with --read-only, attempts to write to files will fail silently as opposed to with a clear warning as in macFUSE.

    Limitations

    -

    Without the use of --vfs-cache-mode this can only write files sequentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount without --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File Caching section for more info.

    +

    Without the use of --vfs-cache-mode this can only write files sequentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount without --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File Caching section for more info. When using NFS mount on macOS, if you don't specify |--vfs-cache-mode| the mount point will be read-only.

    The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do not support the concept of empty directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.

    When rclone mount is invoked on Unix with --daemon flag, the main rclone program will wait for the background mount to become ready or until the timeout specified by the --daemon-wait flag. On Linux it can check mount status using ProcFS so the flag in fact sets maximum time to wait, while the real wait can be less. On macOS / BSD the time to wait is constant and the check is performed only at the end. We advise you to set wait time on macOS reasonably.

    Only supported on Linux, FreeBSD, OS X and Windows at the moment.

    @@ -2912,9 +2930,8 @@ WantedBy=multi-user.target
  • command=cmount can be used to run cmount or any other rclone command rather than the default mount.
  • args2env will pass mount options to the mount helper running in background via environment variables instead of command line arguments. This allows to hide secrets from such commands as ps or pgrep.
  • vv... will be transformed into appropriate --verbose=N
    • -
  • standard mount options like x-systemd.automount, _netdev, nosuid and alike are intended only for Automountd and ignored by rclone.
    • +
  • standard mount options like x-systemd.automount, _netdev, nosuid and alike are intended only for Automountd and ignored by rclone. ## VFS - Virtual File System
    • -

    VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -3075,6 +3092,7 @@ WantedBy=multi-user.target --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -3135,11 +3153,11 @@ if src is directory -c, --checksum Check for changes with size & checksum (if available, or fallback to size only). --compare-dest stringArray Include additional comma separated server-side paths during comparison --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination - --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD") + --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD) --ignore-case-sync Ignore case when synchronizing --ignore-checksum Skip post copy check of checksums --ignore-existing Skip all files that exist on destination - --ignore-size Ignore size when skipping use mod-time or checksum + --ignore-size Ignore size when skipping use modtime or checksum -I, --ignore-times Don't skip files that match size and time - transfer all files --immutable Do not modify files, fail if existing files have been modified --inplace Download directly to destination file instead of atomic download to temp/rename @@ -3154,11 +3172,12 @@ if src is directory --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki) --no-check-dest Don't check the destination, copy regardless --no-traverse Don't traverse destination file system on copy - --no-update-modtime Don't update destination mod-time if files identical + --no-update-modtime Don't update destination modtime if files identical --order-by string Instructions on how to order the transfers, e.g. 'size,descending' + --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial") --refresh-times Refresh the modtime of remote files --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs - --size-only Skip based on size only, not mod-time or checksum + --size-only Skip based on size only, not modtime or checksum --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki) -u, --update Skip files that are newer on the destination

    Important Options

    @@ -3451,6 +3470,37 @@ ffmpeg - | rclone rcat remote:path/to/file +

    The server also makes the following functions available so that they can be used within the template. These functions help extend the options for dynamic rendering of HTML. They can be used to render HTML based on specific conditions.

    + ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
    FunctionDescription
    afterEpochReturns the time since the epoch for the given time.
    containsChecks whether a given substring is present or not in a given string.
    hasPrefixChecks whether the given string begins with the specified prefix.
    hasSuffixChecks whether the given string end with the specified suffix.

    Authentication

    By default this will serve files without needing a login.

    You can either use an htpasswd file which can take lots of users, or set a single username and password with the --rc-user and --rc-pass flags.

    @@ -3565,7 +3615,9 @@ htpasswd -B htpasswd anotherUser
  • rclone serve docker - Serve any remote on docker's volume plugin API.
  • rclone serve ftp - Serve remote:path over FTP.
  • rclone serve http - Serve the remote over HTTP.
    • +
  • rclone serve nfs - Serve the remote as an NFS mount
  • rclone serve restic - Serve the remote for restic's REST API.
    • +
  • rclone serve s3 - Serve remote:path over s3.
  • rclone serve sftp - Serve the remote over SFTP.
  • rclone serve webdav - Serve remote:path over WebDAV.
    • @@ -3577,8 +3629,7 @@ htpasswd -B htpasswd anotherUser

    Server options

    Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs.

    Use --name to choose the friendly server name, which is by default "rclone (hostname)".

    -

    Use --log-trace in conjunction with -vv to enable additional debug logging of all UPNP traffic.

    -

    VFS - Virtual File System

    +

    Use --log-trace in conjunction with -vv to enable additional debug logging of all UPNP traffic. ## VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -3726,6 +3777,7 @@ htpasswd -B htpasswd anotherUser --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -3767,8 +3819,7 @@ htpasswd -B htpasswd anotherUser

    Running rclone serve docker will create the said socket, listening for commands from Docker to create the necessary Volumes. Normally you need not give the --socket-addr flag. The API will listen on the unix domain socket at /run/docker/plugins/rclone.sock. In the example above rclone will create a TCP socket and a small file /etc/docker/plugins/rclone.spec containing the socket address. We use sudo because both paths are writeable only by the root user.

    If you later decide to change listening socket, the docker daemon must be restarted to reconnect to /run/docker/plugins/rclone.sock or parse new /etc/docker/plugins/rclone.spec. Until you restart, any volume related docker commands will timeout trying to access the old socket. Running directly is supported on Linux only, not on Windows or MacOS. This is not a problem with managed plugin mode described in details in the full documentation.

    The command will create volume mounts under the path given by --base-dir (by default /var/lib/docker-volumes/rclone available only to root) and maintain the JSON formatted file docker-plugin.state in the rclone cache directory with book-keeping records of created and mounted volumes.

    -

    All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity.

    -

    VFS - Virtual File System

    +

    All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity. ## VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -3934,6 +3985,7 @@ htpasswd -B htpasswd anotherUser --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -3977,8 +4029,7 @@ htpasswd -B htpasswd anotherUser

    If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

    Authentication

    By default this will serve files without needing a login.

    -

    You can set a single username and password with the --user and --pass flags.

    -

    VFS - Virtual File System

    +

    You can set a single username and password with the --user and --pass flags. ## VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -4159,6 +4210,7 @@ htpasswd -B htpasswd anotherUser --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -4290,6 +4342,37 @@ htpasswd -B htpasswd anotherUser +

    The server also makes the following functions available so that they can be used within the template. These functions help extend the options for dynamic rendering of HTML. They can be used to render HTML based on specific conditions.

    + ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
    FunctionDescription
    afterEpochReturns the time since the epoch for the given time.
    containsChecks whether a given substring is present or not in a given string.
    hasPrefixChecks whether the given string begins with the specified prefix.
    hasSuffixChecks whether the given string end with the specified suffix.

    Authentication

    By default this will serve files without needing a login.

    You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

    @@ -4301,8 +4384,7 @@ htpasswd -B htpasswd user htpasswd -B htpasswd anotherUser

    The password file can be updated while rclone is running.

    Use --realm to set the authentication realm.

    -

    Use --salt to change the password hashing salt from the default.

    -

    VFS - Virtual File System

    +

    Use --salt to change the password hashing salt from the default. ## VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -4492,6 +4574,7 @@ htpasswd -B htpasswd anotherUser --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) @@ -4524,118 +4607,21 @@ htpasswd -B htpasswd anotherUser -

    rclone serve restic

    -

    Serve the remote for restic's REST API.

    +

    rclone serve nfs

    +

    Serve the remote as an NFS mount

    Synopsis

    -

    Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.

    -

    Restic is a command-line program for doing backups.

    -

    The server will log errors. Use -v to see access logs.

    -

    --bwlimit will be respected for file transfers. Use --stats to control the stats printing.

    -

    Setting up rclone for use by restic

    -

    First set up a remote for your chosen cloud provider.

    -

    Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.

    -

    Now start the rclone restic server

    -
    rclone serve restic -v remote:backup
    -

    Where you can replace "backup" in the above by whatever path in the remote you wish to use.

    -

    By default this will serve on "localhost:8080" you can change this with use of the --addr flag.

    -

    You might wish to start this server on boot.

    -

    Adding --cache-objects=false will cause rclone to stop caching objects returned from the List call. Caching is normally desirable as it speeds up downloading objects, saves transactions and uses very little memory.

    -

    Setting up restic to use rclone

    -

    Now you can follow the restic instructions on setting up restic.

    -

    Note that you will need restic 0.8.2 or later to interoperate with rclone.

    -

    For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.

    -

    For example:

    -
    $ export RESTIC_REPOSITORY=rest:http://localhost:8080/
-$ export RESTIC_PASSWORD=yourpassword
-$ restic init
-created restic backend 8b1a4b56ae at rest:http://localhost:8080/
-
-Please note that knowledge of your password is required to access
-the repository. Losing your password means that your data is
-irrecoverably lost.
-$ restic backup /path/to/files/to/backup
-scan [/path/to/files/to/backup]
-scanned 189 directories, 312 files in 0:00
-[0:00] 100.00%  38.128 MiB / 38.128 MiB  501 / 501 items  0 errors  ETA 0:00
-duration: 0:00
-snapshot 45c8fdd8 saved
    -

    Multiple repositories

    -

    Note that you can use the endpoint to host multiple repositories. Do this by adding a directory name or path after the URL. Note that these must end with /. Eg

    -
    $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
-# backup user1 stuff
-$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
-# backup user2 stuff
    -

    Private repositories

    -

    The--private-repos flag can be used to limit users to repositories starting with a path of /<username>/.

    -

    Server options

    -

    Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

    -

    If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

    -

    You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

    -

    --addr may be repeated to listen on multiple IPs/ports/sockets.

    -

    --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

    -

    --max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

    -

    --baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

    -

    TLS (SSL)

    -

    By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

    -

    --cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

    -

    --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").

    -

    Authentication

    -

    By default this will serve files without needing a login.

    -

    You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

    -

    If no static users are configured by either of the above methods, and client certificates are required by the --client-ca flag passed to the server, the client certificate common name will be considered as the username.

    -

    Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

    -

    To create an htpasswd file:

    -
    touch htpasswd
-htpasswd -B htpasswd user
-htpasswd -B htpasswd anotherUser
    -

    The password file can be updated while rclone is running.

    -

    Use --realm to set the authentication realm.

    -

    Use --salt to change the password hashing salt from the default.

    -
    rclone serve restic remote:path [flags]
    -

    Options

    -
          --addr stringArray                IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
-      --allow-origin string             Origin which cross-domain request (CORS) can be executed from
-      --append-only                     Disallow deletion of repository data
-      --baseurl string                  Prefix for URLs - leave blank for root
-      --cache-objects                   Cache listed objects (default true)
-      --cert string                     TLS PEM key (concatenation of certificate and CA certificate)
-      --client-ca string                Client certificate authority to verify clients with
-  -h, --help                            help for restic
-      --htpasswd string                 A htpasswd file - if not provided no authentication is done
-      --key string                      TLS PEM Private key
-      --max-header-bytes int            Maximum size of request header (default 4096)
-      --min-tls-version string          Minimum TLS version that is acceptable (default "tls1.0")
-      --pass string                     Password for authentication
-      --private-repos                   Users can only access their private repo
-      --realm string                    Realm for authentication
-      --salt string                     Password hashing salt (default "dlPL2MqE")
-      --server-read-timeout Duration    Timeout for server reading data (default 1h0m0s)
-      --server-write-timeout Duration   Timeout for server writing data (default 1h0m0s)
-      --stdio                           Run an HTTP2 server on stdin/stdout
-      --user string                     User name for authentication
    -

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    - -

    rclone serve sftp

    -

    Serve the remote over SFTP.

    -

    Synopsis

    -

    Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.

    -

    You can use the filter flags (e.g. --include, --exclude) to control what is served.

    -

    The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.

    -

    Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.

    -

    The server will log errors. Use -v to see access logs.

    -

    --bwlimit will be respected for file transfers. Use --stats to control the stats printing.

    -

    You must provide some means of authentication, either with --user/--pass, an authorized keys file (specify location with --authorized-keys - the default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no authentication when logging in.

    -

    If you don't supply a host --key then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir) in the "serve-sftp" directory.

    -

    By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example.

    -

    Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients.

    -

    If --stdio is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:

    -
    restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
    -

    On the client you need to set --transfers 1 when using --stdio. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.

    -

    The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.

    -

    VFS - Virtual File System

    +

    Create an NFS server that serves the given remote over the network.

    +

    The primary purpose for this command is to enable mount command on recent macOS versions where installing FUSE is very cumbersome.

    +

    Since this is running on NFSv3, no authentication method is available. Any client will be able to access the data. To limit access, you can use serve NFS on loopback address and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; meaning that it is only available to the local machine. If you want other machines to access the NFS mount over local network, you need to specify the listening address and port using --addr flag.

    +

    Modifying files through NFS protocol requires VFS caching. Usually you will need to specify --vfs-cache-mode in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, the mount will be read-only.

    +

    To serve NFS over the network use following command:

    +
    rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
    +

    We specify a specific port that we can use in the mount command:

    +

    To mount the server under Linux/macOS, use the following command:

    +
    mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
    +

    Where $PORT is the same port number we used in the serve nfs command.

    +

    This feature is only available on Unix platforms.

    +

    VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -4752,58 +4738,21 @@ htpasswd -B htpasswd anotherUser

    Alternate report of used bytes

    Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

    WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

    -

    Auth Proxy

    -

    If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

    -

    PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

    -

    There is an example program bin/test_proxy.py in the rclone source code.

    -

    The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

    -

    This config generated must have this extra parameter - _root - root to use for the backend

    -

    And it may have this parameter - _obscure - comma separated strings for parameters to obscure

    -

    If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

    -
    {
-    "user": "me",
-    "pass": "mypassword"
-}
    -

    If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

    -
    {
-    "user": "me",
-    "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
    -

    And as an example return this on STDOUT

    -
    {
-    "type": "sftp",
-    "_root": "",
-    "_obscure": "pass",
-    "user": "me",
-    "pass": "mypassword",
-    "host": "sftp.example.com"
-}
    -

    This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).

    -

    The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.

    -

    Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

    -

    This can be used to build general purpose proxies to any kind of backend that rclone supports.

    -
    rclone serve sftp remote:path [flags]
    -

    Options

    -
          --addr string                            IPaddress:Port or :Port to bind server to (default "localhost:2022")
-      --auth-proxy string                      A program to use to create the backend from the auth
-      --authorized-keys string                 Authorized keys file (default "~/.ssh/authorized_keys")
+
    rclone serve nfs remote:path [flags]
    
+
    Options
    
+
          --addr string                            IPaddress:Port or :Port to bind server to
       --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
       --dir-perms FileMode                     Directory permissions (default 0777)
       --file-perms FileMode                    File permissions (default 0666)
       --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
-  -h, --help                                   help for sftp
-      --key stringArray                        SSH private host key file (Can be multi-valued, leave blank to auto generate)
-      --no-auth                                Allow connections with no authentication if set
+  -h, --help                                   help for nfs
       --no-checksum                            Don't compare checksums on up/download
       --no-modtime                             Don't read/write the modification time (can speed things up)
       --no-seek                                Don't allow seeking in files
-      --pass string                            Password for authentication
       --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
       --read-only                              Only allow read-only access
-      --stdio                                  Run an sftp server on stdin/stdout
       --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
       --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
-      --user string                            User name for authentication
       --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
       --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
       --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
@@ -4816,6 +4765,7 @@ htpasswd -B htpasswd anotherUser
    
       --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
       --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
       --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
+      --vfs-refresh                            Refreshes the directory cache recursively on start
       --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
       --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
       --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
    @@ -4844,24 +4794,54 @@ htpasswd -B htpasswd anotherUser --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    SEE ALSO

    -

    rclone serve webdav

    -

    Serve remote:path over WebDAV.

    -

    Synopsis

    -

    Run a basic WebDAV server to serve a remote over HTTP via the WebDAV protocol. This can be viewed with a WebDAV client, through a web browser, or you can make a remote of type WebDAV to read and write it.

    -

    WebDAV options

    -

    --etag-hash

    -

    This controls the ETag header. Without this flag the ETag will be based on the ModTime and Size of the object.

    -

    If this flag is set to "auto" then rclone will choose the first supported hash on the backend or you can use a named hash such as "MD5" or "SHA-1". Use the hashsum command to see the full list.

    -

    Access WebDAV on Windows

    -

    WebDAV shared folder can be mapped as a drive on Windows, however the default settings prevent it. Windows will fail to connect to the server using insecure Basic authentication. It will not even display any login dialog. Windows requires SSL / HTTPS connection to be used with Basic. If you try to connect via Add Network Location Wizard you will get the following error: "The folder you entered does not appear to be valid. Please choose another". However, you still can connect if you set the following registry key on a client machine: HKEY_LOCAL_MACHINEto 2. The BasicAuthLevel can be set to the following values: 0 - Basic authentication disabled 1 - Basic authentication enabled for SSL connections only 2 - Basic authentication enabled for SSL connections and for non-SSL connections If required, increase the FileSizeLimitInBytes to a higher value. Navigate to the Services interface, then restart the WebClient service.

    -

    Access Office applications on WebDAV

    -

    Navigate to following registry HKEY_CURRENT_USER[14.0/15.0/16.0] Create a new DWORD BasicAuthLevel with value 2. 0 - Basic authentication disabled 1 - Basic authentication enabled for SSL connections only 2 - Basic authentication enabled for SSL and for non-SSL connections

    -

    https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint

    -

    Server options

    +

    rclone serve restic

    +

    Serve the remote for restic's REST API.

    +

    Synopsis

    +

    Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.

    +

    Restic is a command-line program for doing backups.

    +

    The server will log errors. Use -v to see access logs.

    +

    --bwlimit will be respected for file transfers. Use --stats to control the stats printing.

    +

    Setting up rclone for use by restic

    +

    First set up a remote for your chosen cloud provider.

    +

    Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.

    +

    Now start the rclone restic server

    +
    rclone serve restic -v remote:backup
    +

    Where you can replace "backup" in the above by whatever path in the remote you wish to use.

    +

    By default this will serve on "localhost:8080" you can change this with use of the --addr flag.

    +

    You might wish to start this server on boot.

    +

    Adding --cache-objects=false will cause rclone to stop caching objects returned from the List call. Caching is normally desirable as it speeds up downloading objects, saves transactions and uses very little memory.

    +

    Setting up restic to use rclone

    +

    Now you can follow the restic instructions on setting up restic.

    +

    Note that you will need restic 0.8.2 or later to interoperate with rclone.

    +

    For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.

    +

    For example:

    +
    $ export RESTIC_REPOSITORY=rest:http://localhost:8080/
+$ export RESTIC_PASSWORD=yourpassword
+$ restic init
+created restic backend 8b1a4b56ae at rest:http://localhost:8080/
+
+Please note that knowledge of your password is required to access
+the repository. Losing your password means that your data is
+irrecoverably lost.
+$ restic backup /path/to/files/to/backup
+scan [/path/to/files/to/backup]
+scanned 189 directories, 312 files in 0:00
+[0:00] 100.00%  38.128 MiB / 38.128 MiB  501 / 501 items  0 errors  ETA 0:00
+duration: 0:00
+snapshot 45c8fdd8 saved
    +

    Multiple repositories

    +

    Note that you can use the endpoint to host multiple repositories. Do this by adding a directory name or path after the URL. Note that these must end with /. Eg

    +
    $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
+# backup user1 stuff
+$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
+# backup user2 stuff
    +

    Private repositories

    +

    The--private-repos flag can be used to limit users to repositories starting with a path of /<username>/.

    +

    Server options

    Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

    If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

    You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

    @@ -4869,91 +4849,11 @@ htpasswd -B htpasswd anotherUser

    --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

    --max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

    --baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

    -

    TLS (SSL)

    +

    TLS (SSL)

    By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

    --cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

    --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").

    -

    Template

    -

    --template allows a user to specify a custom markup template for HTTP and WebDAV serve functions. The server exports the following markup to be used within the template to server pages:

    - ---- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    ParameterDescription
    .NameThe full path of a file/directory.
    .TitleDirectory listing of .Name
    .SortThe current sort used. This is changeable via ?sort= parameter
    Sort Options: namedirfirst,name,size,time (default namedirfirst)
    .OrderThe current ordering used. This is changeable via ?order= parameter
    Order Options: asc,desc (default asc)
    .QueryCurrently unused.
    .BreadcrumbAllows for creating a relative navigation
    -- .LinkThe relative to the root link of the Text.
    -- .TextThe Name of the directory.
    .EntriesInformation about a specific file/directory.
    -- .URLThe 'url' of an entry.
    -- .LeafCurrently same as 'URL' but intended to be 'just' the name.
    -- .IsDirBoolean for if an entry is a directory or not.
    -- .SizeSize in Bytes of the entry.
    -- .ModTimeThe UTC timestamp of an entry.
    -

    Authentication

    +

    Authentication

    By default this will serve files without needing a login.

    You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

    If no static users are configured by either of the above methods, and client certificates are required by the --client-ca flag passed to the server, the client certificate common name will be considered as the username.

    @@ -4965,7 +4865,103 @@ htpasswd -B htpasswd anotherUser

    The password file can be updated while rclone is running.

    Use --realm to set the authentication realm.

    Use --salt to change the password hashing salt from the default.

    -

    VFS - Virtual File System

    +
    rclone serve restic remote:path [flags]
    +

    Options

    +
          --addr stringArray                IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+      --allow-origin string             Origin which cross-domain request (CORS) can be executed from
+      --append-only                     Disallow deletion of repository data
+      --baseurl string                  Prefix for URLs - leave blank for root
+      --cache-objects                   Cache listed objects (default true)
+      --cert string                     TLS PEM key (concatenation of certificate and CA certificate)
+      --client-ca string                Client certificate authority to verify clients with
+  -h, --help                            help for restic
+      --htpasswd string                 A htpasswd file - if not provided no authentication is done
+      --key string                      TLS PEM Private key
+      --max-header-bytes int            Maximum size of request header (default 4096)
+      --min-tls-version string          Minimum TLS version that is acceptable (default "tls1.0")
+      --pass string                     Password for authentication
+      --private-repos                   Users can only access their private repo
+      --realm string                    Realm for authentication
+      --salt string                     Password hashing salt (default "dlPL2MqE")
+      --server-read-timeout Duration    Timeout for server reading data (default 1h0m0s)
+      --server-write-timeout Duration   Timeout for server writing data (default 1h0m0s)
+      --stdio                           Run an HTTP2 server on stdin/stdout
+      --user string                     User name for authentication
    +

    See the global flags page for global options not listed here.

    +

    SEE ALSO

    + +

    rclone serve s3

    +

    Serve remote:path over s3.

    +

    Synopsis

    +

    serve s3 implements a basic s3 server that serves a remote via s3. This can be viewed with an s3 client, or you can make an s3 type remote to read and write to it with rclone.

    +

    serve s3 is considered Experimental so use with care.

    +

    S3 server supports Signature Version 4 authentication. Just use --auth-key accessKey,secretKey and set the Authorization header correctly in the request. (See the AWS docs).

    +

    --auth-key can be repeated for multiple auth pairs. If --auth-key is not provided then serve s3 will allow anonymous access.

    +

    Please note that some clients may require HTTPS endpoints. See the SSL docs for more information.

    +

    This command uses the VFS directory cache. All the functionality will work with --vfs-cache-mode off. Using --vfs-cache-mode full (or writes) can be used to cache objects locally to improve performance.

    +

    Use --force-path-style=false if you want to use the bucket name as a part of the hostname (such as mybucket.local)

    +

    Use --etag-hash if you want to change the hash uses for the ETag. Note that using anything other than MD5 (the default) is likely to cause problems for S3 clients which rely on the Etag being the MD5.

    +

    Quickstart

    +

    For a simple set up, to serve remote:path over s3, run the server like this:

    +
    rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
    +

    This will be compatible with an rclone remote which is defined like this:

    +
    [serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = false
    +

    Note that setting disable_multipart_uploads = true is to work around a bug which will be fixed in due course.

    +

    Bugs

    +

    When uploading multipart files serve s3 holds all the parts in memory (see #7453). This is a limitaton of the library rclone uses for serving S3 and will hopefully be fixed at some point.

    +

    Multipart server side copies do not work (see #7454). These take a very long time and eventually fail. The default threshold for multipart server side copies is 5G which is the maximum it can be, so files above this side will fail to be server side copied.

    +

    For a current list of serve s3 bugs see the serve s3 bug category on GitHub.

    +

    Limitations

    +

    serve s3 will treat all directories in the root as buckets and ignore all files in the root. You can use CreateBucket to create folders under the root, but you can't create empty folders under other folders not in the root.

    +

    When using PutObject or DeleteObject, rclone will automatically create or clean up empty folders. If you don't want to clean up empty folders automatically, use --no-cleanup.

    +

    When using ListObjects, rclone will use / when the delimiter is empty. This reduces backend requests with no effect on most operations, but if the delimiter is something other than / and empty, rclone will do a full recursive search of the backend, which can take some time.

    +

    Versioning is not currently supported.

    +

    Metadata will only be saved in memory other than the rclone mtime metadata which will be set as the modification time of the file.

    +

    Supported operations

    +

    serve s3 currently supports the following operations.

    + +

    Other operations will return error Unimplemented.

    +

    Server options

    +

    Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

    +

    If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

    +

    You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

    +

    --addr may be repeated to listen on multiple IPs/ports/sockets.

    +

    --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

    +

    --max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

    +

    --baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

    +

    TLS (SSL)

    +

    By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

    +

    --cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

    +

    --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). ## VFS - Virtual File System

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    @@ -5082,6 +5078,574 @@ htpasswd -B htpasswd anotherUser

    Alternate report of used bytes

    Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

    WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

    +
    rclone serve s3 remote:path [flags]
    +

    Options

    +
          --addr stringArray                       IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+      --allow-origin string                    Origin which cross-domain request (CORS) can be executed from
+      --auth-key stringArray                   Set key pair for v4 authorization: access_key_id,secret_access_key
+      --baseurl string                         Prefix for URLs - leave blank for root
+      --cert string                            TLS PEM key (concatenation of certificate and CA certificate)
+      --client-ca string                       Client certificate authority to verify clients with
+      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
+      --dir-perms FileMode                     Directory permissions (default 0777)
+      --etag-hash string                       Which hash to use for the ETag, or auto or blank for off (default "MD5")
+      --file-perms FileMode                    File permissions (default 0666)
+      --force-path-style                       If true use path style access if false use virtual hosted style (default true) (default true)
+      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+  -h, --help                                   help for s3
+      --key string                             TLS PEM Private key
+      --max-header-bytes int                   Maximum size of request header (default 4096)
+      --min-tls-version string                 Minimum TLS version that is acceptable (default "tls1.0")
+      --no-checksum                            Don't compare checksums on up/download
+      --no-cleanup                             Not to cleanup empty folder after object is deleted
+      --no-modtime                             Don't read/write the modification time (can speed things up)
+      --no-seek                                Don't allow seeking in files
+      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+      --read-only                              Only allow read-only access
+      --server-read-timeout Duration           Timeout for server reading data (default 1h0m0s)
+      --server-write-timeout Duration          Timeout for server writing data (default 1h0m0s)
+      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
+      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
+      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
+      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
+      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
+      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
+      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
+      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
+      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
+      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
+      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
+      --vfs-refresh                            Refreshes the directory cache recursively on start
+      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
+      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
+      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
    +

    Filter Options

    +

    Flags for filtering directory listings.

    +
          --delete-excluded                     Delete files on dest excluded from sync
+      --exclude stringArray                 Exclude files matching pattern
+      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
+      --exclude-if-present stringArray      Exclude directories if filename is present
+      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
+      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
+  -f, --filter stringArray                  Add a file filtering rule
+      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
+      --ignore-case                         Ignore case in filters (case insensitive)
+      --include stringArray                 Include files matching pattern
+      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
+      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --max-depth int                       If set limits the recursion depth to this (default -1)
+      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+      --metadata-exclude stringArray        Exclude metadatas matching pattern
+      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
+      --metadata-filter stringArray         Add a metadata filtering rule
+      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
+      --metadata-include stringArray        Include metadatas matching pattern
+      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
+      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
    +

    See the global flags page for global options not listed here.

    +

    SEE ALSO

    + +

    rclone serve sftp

    +

    Serve the remote over SFTP.

    +

    Synopsis

    +

    Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.

    +

    You can use the filter flags (e.g. --include, --exclude) to control what is served.

    +

    The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.

    +

    Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.

    +

    The server will log errors. Use -v to see access logs.

    +

    --bwlimit will be respected for file transfers. Use --stats to control the stats printing.

    +

    You must provide some means of authentication, either with --user/--pass, an authorized keys file (specify location with --authorized-keys - the default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no authentication when logging in.

    +

    If you don't supply a host --key then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir) in the "serve-sftp" directory.

    +

    By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example.

    +

    Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients.

    +

    If --stdio is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:

    +
    restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
    +

    On the client you need to set --transfers 1 when using --stdio. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.

    +

    The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.

    +

    VFS - Virtual File System

    +

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    +

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    +

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    +

    VFS Directory Cache

    +

    Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

    +
    --dir-cache-time duration   Time to cache directory entries for (default 5m0s)
+--poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
    +

    However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

    +

    You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

    +
    kill -SIGHUP $(pidof rclone)
    +

    If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

    +
    rclone rc vfs/forget
    +

    Or individual files or directories:

    +
    rclone rc vfs/forget file=path/to/file dir=path/to/dir
    +

    VFS File Buffering

    +

    The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

    +

    Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

    +

    This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

    +

    The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

    +

    VFS File Caching

    +

    These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

    +

    For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

    +

    Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

    +
    --cache-dir string                     Directory rclone will use for caching.
+--vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
    +

    If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

    +

    The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

    +

    Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

    +

    If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

    +

    The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

    +

    You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

    +

    --vfs-cache-mode off

    +

    In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

    +

    This will mean some operations are not possible

    + +

    --vfs-cache-mode minimal

    +

    This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

    +

    These operations are not possible

    + +

    --vfs-cache-mode writes

    +

    In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

    +

    This mode should support all normal file system operations.

    +

    If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

    +

    --vfs-cache-mode full

    +

    In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

    +

    In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

    +

    So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

    +

    This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

    +

    When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

    +

    When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

    +

    IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

    +

    Fingerprinting

    +

    Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

    + +

    where available on an object.

    +

    On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

    +

    For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

    +

    If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

    +

    If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

    +

    Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

    +

    VFS Chunked Reading

    +

    When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

    +

    These flags control the chunking:

    +
    --vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
    +

    Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

    +

    With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

    +

    Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

    +

    VFS Performance

    +

    These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

    +

    In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

    +
    --no-checksum     Don't compare checksums on up/download.
+--no-modtime      Don't read/write the modification time (can speed things up).
+--no-seek         Don't allow seeking in files.
+--read-only       Only allow read-only access.
    +

    Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

    +
    --vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
    +

    When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

    +
    --transfers int  Number of file transfers to run in parallel (default 4)
    +

    VFS Case Sensitivity

    +

    Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

    +

    File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

    +

    Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

    +

    The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

    +

    The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

    +

    Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

    +

    If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

    +

    VFS Disk Options

    +

    This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

    +
    --vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
    +

    Alternate report of used bytes

    +

    Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

    +

    WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

    +

    Auth Proxy

    +

    If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

    +

    PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

    +

    There is an example program bin/test_proxy.py in the rclone source code.

    +

    The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.

    +

    This config generated must have this extra parameter - _root - root to use for the backend

    +

    And it may have this parameter - _obscure - comma separated strings for parameters to obscure

    +

    If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

    +
    {
+    "user": "me",
+    "pass": "mypassword"
+}
    +

    If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:

    +
    {
+    "user": "me",
+    "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
    +

    And as an example return this on STDOUT

    +
    {
+    "type": "sftp",
+    "_root": "",
+    "_obscure": "pass",
+    "user": "me",
+    "pass": "mypassword",
+    "host": "sftp.example.com"
+}
    +

    This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).

    +

    The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.

    +

    Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

    +

    This can be used to build general purpose proxies to any kind of backend that rclone supports.

    +
    rclone serve sftp remote:path [flags]
    +

    Options

    +
          --addr string                            IPaddress:Port or :Port to bind server to (default "localhost:2022")
+      --auth-proxy string                      A program to use to create the backend from the auth
+      --authorized-keys string                 Authorized keys file (default "~/.ssh/authorized_keys")
+      --dir-cache-time Duration                Time to cache directory entries for (default 5m0s)
+      --dir-perms FileMode                     Directory permissions (default 0777)
+      --file-perms FileMode                    File permissions (default 0666)
+      --gid uint32                             Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+  -h, --help                                   help for sftp
+      --key stringArray                        SSH private host key file (Can be multi-valued, leave blank to auto generate)
+      --no-auth                                Allow connections with no authentication if set
+      --no-checksum                            Don't compare checksums on up/download
+      --no-modtime                             Don't read/write the modification time (can speed things up)
+      --no-seek                                Don't allow seeking in files
+      --pass string                            Password for authentication
+      --poll-interval Duration                 Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+      --read-only                              Only allow read-only access
+      --stdio                                  Run an sftp server on stdin/stdout
+      --uid uint32                             Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+      --umask int                              Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+      --user string                            User name for authentication
+      --vfs-cache-max-age Duration             Max time since last access of objects in the cache (default 1h0m0s)
+      --vfs-cache-max-size SizeSuffix          Max total size of objects in the cache (default off)
+      --vfs-cache-min-free-space SizeSuffix    Target minimum free space on the disk containing the cache (default off)
+      --vfs-cache-mode CacheMode               Cache mode off|minimal|writes|full (default off)
+      --vfs-cache-poll-interval Duration       Interval to poll the cache for stale objects (default 1m0s)
+      --vfs-case-insensitive                   If a file name not found, find a case insensitive match
+      --vfs-disk-space-total-size SizeSuffix   Specify the total space of disk (default off)
+      --vfs-fast-fingerprint                   Use fast (less accurate) fingerprints for change detection
+      --vfs-read-ahead SizeSuffix              Extra read ahead over --buffer-size when using cache-mode full
+      --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks (default 128Mi)
+      --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+      --vfs-read-wait Duration                 Time to wait for in-sequence read before seeking (default 20ms)
+      --vfs-refresh                            Refreshes the directory cache recursively on start
+      --vfs-used-is-size rclone size           Use the rclone size algorithm for Used size
+      --vfs-write-back Duration                Time to writeback files after last use when using cache (default 5s)
+      --vfs-write-wait Duration                Time to wait for in-sequence write before giving error (default 1s)
    +

    Filter Options

    +

    Flags for filtering directory listings.

    +
          --delete-excluded                     Delete files on dest excluded from sync
+      --exclude stringArray                 Exclude files matching pattern
+      --exclude-from stringArray            Read file exclude patterns from file (use - to read from stdin)
+      --exclude-if-present stringArray      Exclude directories if filename is present
+      --files-from stringArray              Read list of source-file names from file (use - to read from stdin)
+      --files-from-raw stringArray          Read list of source-file names from file without any processing of lines (use - to read from stdin)
+  -f, --filter stringArray                  Add a file filtering rule
+      --filter-from stringArray             Read file filtering patterns from a file (use - to read from stdin)
+      --ignore-case                         Ignore case in filters (case insensitive)
+      --include stringArray                 Include files matching pattern
+      --include-from stringArray            Read file include patterns from file (use - to read from stdin)
+      --max-age Duration                    Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --max-depth int                       If set limits the recursion depth to this (default -1)
+      --max-size SizeSuffix                 Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+      --metadata-exclude stringArray        Exclude metadatas matching pattern
+      --metadata-exclude-from stringArray   Read metadata exclude patterns from file (use - to read from stdin)
+      --metadata-filter stringArray         Add a metadata filtering rule
+      --metadata-filter-from stringArray    Read metadata filtering patterns from a file (use - to read from stdin)
+      --metadata-include stringArray        Include metadatas matching pattern
+      --metadata-include-from stringArray   Read metadata include patterns from file (use - to read from stdin)
+      --min-age Duration                    Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+      --min-size SizeSuffix                 Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
    +

    See the global flags page for global options not listed here.

    +

    SEE ALSO

    + +

    rclone serve webdav

    +

    Serve remote:path over WebDAV.

    +

    Synopsis

    +

    Run a basic WebDAV server to serve a remote over HTTP via the WebDAV protocol. This can be viewed with a WebDAV client, through a web browser, or you can make a remote of type WebDAV to read and write it.

    +

    WebDAV options

    +

    --etag-hash

    +

    This controls the ETag header. Without this flag the ETag will be based on the ModTime and Size of the object.

    +

    If this flag is set to "auto" then rclone will choose the first supported hash on the backend or you can use a named hash such as "MD5" or "SHA-1". Use the hashsum command to see the full list.

    +

    Access WebDAV on Windows

    +

    WebDAV shared folder can be mapped as a drive on Windows, however the default settings prevent it. Windows will fail to connect to the server using insecure Basic authentication. It will not even display any login dialog. Windows requires SSL / HTTPS connection to be used with Basic. If you try to connect via Add Network Location Wizard you will get the following error: "The folder you entered does not appear to be valid. Please choose another". However, you still can connect if you set the following registry key on a client machine: HKEY_LOCAL_MACHINEto 2. The BasicAuthLevel can be set to the following values: 0 - Basic authentication disabled 1 - Basic authentication enabled for SSL connections only 2 - Basic authentication enabled for SSL connections and for non-SSL connections If required, increase the FileSizeLimitInBytes to a higher value. Navigate to the Services interface, then restart the WebClient service.

    +

    Access Office applications on WebDAV

    +

    Navigate to following registry HKEY_CURRENT_USER[14.0/15.0/16.0] Create a new DWORD BasicAuthLevel with value 2. 0 - Basic authentication disabled 1 - Basic authentication enabled for SSL connections only 2 - Basic authentication enabled for SSL and for non-SSL connections

    +

    https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint

    +

    Server options

    +

    Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.

    +

    If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

    +

    You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.

    +

    --addr may be repeated to listen on multiple IPs/ports/sockets.

    +

    --server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

    +

    --max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

    +

    --baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.

    +

    TLS (SSL)

    +

    By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.

    +

    --cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

    +

    --min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").

    +

    Template

    +

    --template allows a user to specify a custom markup template for HTTP and WebDAV serve functions. The server exports the following markup to be used within the template to server pages:

    + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    ParameterDescription
    .NameThe full path of a file/directory.
    .TitleDirectory listing of .Name
    .SortThe current sort used. This is changeable via ?sort= parameter
    Sort Options: namedirfirst,name,size,time (default namedirfirst)
    .OrderThe current ordering used. This is changeable via ?order= parameter
    Order Options: asc,desc (default asc)
    .QueryCurrently unused.
    .BreadcrumbAllows for creating a relative navigation
    -- .LinkThe relative to the root link of the Text.
    -- .TextThe Name of the directory.
    .EntriesInformation about a specific file/directory.
    -- .URLThe 'url' of an entry.
    -- .LeafCurrently same as 'URL' but intended to be 'just' the name.
    -- .IsDirBoolean for if an entry is a directory or not.
    -- .SizeSize in Bytes of the entry.
    -- .ModTimeThe UTC timestamp of an entry.
    +

    The server also makes the following functions available so that they can be used within the template. These functions help extend the options for dynamic rendering of HTML. They can be used to render HTML based on specific conditions.

    + ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
    FunctionDescription
    afterEpochReturns the time since the epoch for the given time.
    containsChecks whether a given substring is present or not in a given string.
    hasPrefixChecks whether the given string begins with the specified prefix.
    hasSuffixChecks whether the given string end with the specified suffix.
    +

    Authentication

    +

    By default this will serve files without needing a login.

    +

    You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

    +

    If no static users are configured by either of the above methods, and client certificates are required by the --client-ca flag passed to the server, the client certificate common name will be considered as the username.

    +

    Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

    +

    To create an htpasswd file:

    +
    touch htpasswd
+htpasswd -B htpasswd user
+htpasswd -B htpasswd anotherUser
    +

    The password file can be updated while rclone is running.

    +

    Use --realm to set the authentication realm.

    +

    Use --salt to change the password hashing salt from the default. ## VFS - Virtual File System

    +

    This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.

    +

    Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.

    +

    The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.

    +

    VFS Directory Cache

    +

    Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.

    +
    --dir-cache-time duration   Time to cache directory entries for (default 5m0s)
+--poll-interval duration    Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
    +

    However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.

    +

    You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

    +
    kill -SIGHUP $(pidof rclone)
    +

    If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:

    +
    rclone rc vfs/forget
    +

    Or individual files or directories:

    +
    rclone rc vfs/forget file=path/to/file dir=path/to/dir
    +

    VFS File Buffering

    +

    The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.

    +

    Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.

    +

    This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.

    +

    The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

    +

    VFS File Caching

    +

    These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.

    +

    For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.

    +

    Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.

    +
    --cache-dir string                     Directory rclone will use for caching.
+--vfs-cache-mode CacheMode             Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration              Time to writeback files after last use when using cache (default 5s)
    +

    If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

    +

    The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

    +

    Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.

    +

    If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.

    +

    The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .

    +

    You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

    +

    --vfs-cache-mode off

    +

    In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.

    +

    This will mean some operations are not possible

    + +

    --vfs-cache-mode minimal

    +

    This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

    +

    These operations are not possible

    + +

    --vfs-cache-mode writes

    +

    In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.

    +

    This mode should support all normal file system operations.

    +

    If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.

    +

    --vfs-cache-mode full

    +

    In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.

    +

    In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.

    +

    So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.

    +

    This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.

    +

    When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.

    +

    When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.

    +

    IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.

    +

    Fingerprinting

    +

    Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:

    + +

    where available on an object.

    +

    On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).

    +

    For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.

    +

    If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.

    +

    If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.

    +

    Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.

    +

    VFS Chunked Reading

    +

    When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.

    +

    These flags control the chunking:

    +
    --vfs-read-chunk-size SizeSuffix        Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix  Max chunk doubling size (default off)
    +

    Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.

    +

    With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

    +

    Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

    +

    VFS Performance

    +

    These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.

    +

    In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.

    +
    --no-checksum     Don't compare checksums on up/download.
+--no-modtime      Don't read/write the modification time (can speed things up).
+--no-seek         Don't allow seeking in files.
+--read-only       Only allow read-only access.
    +

    Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.

    +
    --vfs-read-wait duration   Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration  Time to wait for in-sequence write before giving error (default 1s)
    +

    When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).

    +
    --transfers int  Number of file transfers to run in parallel (default 4)
    +

    VFS Case Sensitivity

    +

    Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.

    +

    File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.

    +

    Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.

    +

    The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.

    +

    The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.

    +

    Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.

    +

    If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".

    +

    VFS Disk Options

    +

    This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.

    +
    --vfs-disk-space-total-size    Manually set the total disk space size (example: 256G, default: -1)
    +

    Alternate report of used bytes

    +

    Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.

    +

    WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.

    Auth Proxy

    If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.

    PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.

    @@ -5113,7 +5677,7 @@ htpasswd -B htpasswd anotherUser

    Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.

    This can be used to build general purpose proxies to any kind of backend that rclone supports.

    rclone serve webdav remote:path [flags]
    -

    Options

    +

    Options

          --addr stringArray                       IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
       --allow-origin string                    Origin which cross-domain request (CORS) can be executed from
       --auth-proxy string                      A program to use to create the backend from the auth
@@ -5157,10 +5721,11 @@ htpasswd -B htpasswd anotherUser
    --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi) --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off) --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms) + --vfs-refresh Refreshes the directory cache recursively on start --vfs-used-is-size rclone size Use the rclone size algorithm for Used size --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s) --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s) -

    Filter Options

    +

    Filter Options

    Flags for filtering directory listings.

          --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
@@ -5185,13 +5750,13 @@ htpasswd -B htpasswd anotherUser
    --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off) --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    SEE ALSO

    rclone settier

    Changes storage class/tier of objects in remote.

    -

    Synopsis

    +

    Synopsis

    rclone settier changes storage tier or class at remote if supported. Few cloud storage services provides different storage classes on objects, for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc.

    Note that, certain tier changes make objects not available to access immediately. For example tiering to archive in azure blob storage makes objects in frozen state, user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object inaccessible.true

    You can use it to tier single object

    @@ -5201,25 +5766,25 @@ htpasswd -B htpasswd anotherUser

    Or just provide remote directory and all files in directory will be tiered

    rclone settier tier remote:path/dir
    rclone settier tier remote:path [flags]
    -

    Options

    +

    Options

      -h, --help   help for settier

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    SEE ALSO

    rclone test

    Run a test command

    -

    Synopsis

    +

    Synopsis

    Rclone test is used to run test commands.

    Select which test command you want with the subcommand, eg

    rclone test memory remote:

    Each subcommand has its own options which you can see in their help.

    NB Be careful running these commands, they may do strange things so reading their documentation first is recommended.

    -

    Options

    +

    Options

      -h, --help   help for test

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    SEE ALSO

    Note that value of --timestamp is in UTC. If you want local time then add the --localtime flag.

    rclone touch remote:path [flags]
    -

    Options

    +

    Options

      -h, --help               help for touch
       --localtime          Use localtime for timestamp, not UTC
   -C, --no-create          Do not create the file if it does not exist (implied with --recursive)
@@ -5348,7 +5913,7 @@ htpasswd -B htpasswd anotherUser
      -n, --dry-run         Do a trial run with no permanent changes
   -i, --interactive     Enable interactive mode
   -v, --verbose count   Print lots more stuff (repeat for more)
    -

    Filter Options

    +

    Filter Options

    Flags for filtering directory listings.

          --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
@@ -5377,13 +5942,13 @@ htpasswd -B htpasswd anotherUser
          --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
       --fast-list           Use recursive list if available; uses more memory but fewer transactions

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    SEE ALSO

    rclone tree

    List the contents of the remote in a tree like fashion.

    -

    Synopsis

    +

    Synopsis

    rclone tree lists the contents of a remote in a similar way to the unix tree command.

    For example

    $ rclone tree remote:path
@@ -5400,7 +5965,7 @@ htpasswd -B htpasswd anotherUser

    The tree command has many options for controlling the listing which are compatible with the tree command, for example you can include file sizes with --size. Note that not all of them have short options as they conflict with rclone's short options.

    For a more interactive navigation of the remote see the ncdu command.

    rclone tree remote:path [flags]
    -

    Options

    +

    Options

      -a, --all             All files are listed (list . files too)
   -d, --dirs-only       List directories only
       --dirsfirst       List directories before files (-U disables)
@@ -5420,7 +5985,7 @@ htpasswd -B htpasswd anotherUser
    -r, --sort-reverse Reverse the order of the sort -U, --unsorted Leave files unsorted --version Sort files alphanumerically by version -

    Filter Options

    +

    Filter Options

    Flags for filtering directory listings.

          --delete-excluded                     Delete files on dest excluded from sync
       --exclude stringArray                 Exclude files matching pattern
@@ -5449,7 +6014,7 @@ htpasswd -B htpasswd anotherUser
          --default-time Time   Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
       --fast-list           Use recursive list if available; uses more memory but fewer transactions

    See the global flags page for global options not listed here.

    -

    SEE ALSO

    +

    SEE ALSO

    @@ -5570,6 +6135,7 @@ rclone sync --interactive /path/to/files remote:current-backup

    Rclone only supports a one-time sync of metadata. This means that metadata will be synced from the source object to the destination object only when the source object has changed and needs to be re-uploaded. If the metadata subsequently changes on the source object without changing the object itself then it won't be synced to the destination object. This is in line with the way rclone syncs Content-Type without the --metadata flag.

    Using --metadata when syncing from local to local will preserve file attributes such as file mode, owner, extended attributes (not Windows).

    Note that arbitrary metadata may be added to objects using the --metadata-set key=value flag when the object is first uploaded. This flag can be repeated as many times as necessary.

    +

    The --metadata-mapper flag can be used to pass the name of a program in which can transform metadata when it is being copied from source to destination.

    Types of metadata

    Metadata is divided into two type. System metadata and User metadata.

    Metadata which the backend uses itself is called system metadata. For example on the local backend the system metadata uid will store the user ID of the file when used on a unix based platform.

    @@ -5647,26 +6213,31 @@ rclone sync --interactive /path/to/files remote:current-backup 2006-01-02T15:04:05.999999999Z07:00 +utime +Time of file upload: RFC 3339 +2006-01-02T15:04:05.999999999Z07:00 + + cache-control Cache-Control header no-cache - + content-disposition Content-Disposition header inline - + content-encoding Content-Encoding header gzip - + content-language Content-Language header en-US - + content-type Content-Type header text/plain @@ -5675,7 +6246,7 @@ rclone sync --interactive /path/to/files remote:current-backup

    The metadata keys mtime and content-type will take precedence if supplied in the metadata over reading the Content-Type or modification time of the source object.

    Hashes are not included in system metadata as there is a well defined way of reading those already.

    -

    Options

    +

    Options

    Rclone has a number of options to control its behaviour.

    Options that take parameters can have the values passed in two ways, --option=value or --option value. However boolean (true/false) options behave slightly differently to the other options in that --boolean sets the option to true and the absence of the flag sets it to false. It is also possible to specify --boolean=false or --boolean=true. Note that --boolean false is not valid - this is parsed as --boolean and the false is parsed as an extra command line argument for rclone.

    Time or duration options

    @@ -5925,7 +6496,7 @@ See the dedupe command for more information as to what these options mean.

  • ftp
  • sftp
    • -

    Without --inplace (the default) rclone will first upload to a temporary file with an extension like this where XXXXXX represents a random string.

    +

    Without --inplace (the default) rclone will first upload to a temporary file with an extension like this, where XXXXXX represents a random string and .partial is --partial-suffix value (.partial by default).

    original-file-name.XXXXXX.partial

    (rclone will make sure the final name is no longer than 100 characters by truncating the original-file-name part if necessary).

    When the upload is complete, rclone will rename the .partial file to the correct name, overwriting any existing file at that point. If the upload fails then the .partial file will be deleted.

    @@ -6006,9 +6577,81 @@ y/n/s/!/q> n

    Specifying --cutoff-mode=soft will stop starting new transfers when Rclone reaches the limit.

    Specifying --cutoff-mode=cautious will try to prevent Rclone from reaching the limit. Only applicable for --max-transfer

    -M, --metadata

    -

    Setting this flag enables rclone to copy the metadata from the source to the destination. For local backends this is ownership, permissions, xattr etc. See the #metadata for more info.

    +

    Setting this flag enables rclone to copy the metadata from the source to the destination. For local backends this is ownership, permissions, xattr etc. See the metadata section for more info.

    +

    --metadata-mapper SpaceSepList

    +

    If you supply the parameter --metadata-mapper /path/to/program then rclone will use that program to map metadata from source object to destination object.

    +

    The argument to this flag should be a command with an optional space separated list of arguments. If one of the arguments has a space in then enclose it in ", if you want a literal " in an argument then enclose the argument in " and double the ". See CSV encoding for more info.

    +
    --metadata-mapper "python bin/test_metadata_mapper.py"
+--metadata-mapper 'python bin/test_metadata_mapper.py "argument with a space"'
+--metadata-mapper 'python bin/test_metadata_mapper.py "argument with ""two"" quotes"'
    +

    This uses a simple JSON based protocol with input on STDIN and output on STDOUT. This will be called for every file and directory copied and may be called concurrently.

    +

    The program's job is to take a metadata blob on the input and turn it into a metadata blob on the output suitable for the destination backend.

    +

    Input to the program (via STDIN) might look like this. This provides some context for the Metadata which may be important.

    + +
    {
+    "SrcFs": "gdrive:",
+    "SrcFsType": "drive",
+    "DstFs": "newdrive:user",
+    "DstFsType": "onedrive",
+    "Remote": "test.txt",
+    "Size": 6,
+    "MimeType": "text/plain; charset=utf-8",
+    "ModTime": "2022-10-11T17:53:10.286745272+01:00",
+    "IsDir": false,
+    "ID": "xyz",
+    "Metadata": {
+        "btime": "2022-10-11T16:53:11Z",
+        "content-type": "text/plain; charset=utf-8",
+        "mtime": "2022-10-11T17:53:10.286745272+01:00",
+        "owner": "user1@domain1.com",
+        "permissions": "...",
+        "description": "my nice file",
+        "starred": "false"
+    }
+}
    +

    The program should then modify the input as desired and send it to STDOUT. The returned Metadata field will be used in its entirety for the destination object. Any other fields will be ignored. Note in this example we translate user names and permissions and add something to the description:

    +
    {
+    "Metadata": {
+        "btime": "2022-10-11T16:53:11Z",
+        "content-type": "text/plain; charset=utf-8",
+        "mtime": "2022-10-11T17:53:10.286745272+01:00",
+        "owner": "user1@domain2.com",
+        "permissions": "...",
+        "description": "my nice file [migrated from domain1]",
+        "starred": "false"
+    }
+}
    +

    Metadata can be removed here too.

    +

    An example python program might look something like this to implement the above transformations.

    +
    import sys, json
+
+i = json.load(sys.stdin)
+metadata = i["Metadata"]
+# Add tag to description
+if "description" in metadata:
+    metadata["description"] += " [migrated from domain1]"
+else:
+    metadata["description"] = "[migrated from domain1]"
+# Modify owner
+if "owner" in metadata:
+    metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
+o = { "Metadata": metadata }
+json.dump(o, sys.stdout, indent="\t")
    +

    You can find this example (slightly expanded) in the rclone source code at bin/test_metadata_mapper.py.

    +

    If you want to see the input to the metadata mapper and the output returned from it in the log you can use -vv --dump mapper.

    +

    See the metadata section for more info.

    --metadata-set key=value

    -

    Add metadata key = value when uploading. This can be repeated as many times as required. See the #metadata for more info.

    +

    Add metadata key = value when uploading. This can be repeated as many times as required. See the metadata section for more info.

    --modify-window=TIME

    When checking whether a file has been modified, this is the maximum allowed time difference that a file can have and still be considered equivalent.

    The default is 1ns unless this is overridden by a remote. For example OS X only stores modification times to the nearest second so if you are reading and writing to an OS X filing system this will be 1s by default.

    @@ -6083,7 +6726,7 @@ y/n/s/!/q> n
  • --order-by name - send the files with alphabetically by path first

    • If the --order-by flag is not supplied or it is supplied with an empty string then the default ordering will be used which is as scanned. With --checkers 1 this is mostly alphabetical, however with the default --checkers 8 it is somewhat random.

    -

    Limitations

    +

    Limitations

    The --order-by flag does not do a separate pass over the data. This means that it may transfer some files out of the order specified if

    Rclone will do its best to transfer the best file it has so in practice this should not cause a problem. Think of --order-by as being more of a best efforts flag rather than a perfect ordering.

    If you want perfect ordering then you will need to specify --check-first which will find all the files which need transferring first before transferring any.

    +

    --partial-suffix

    +

    When --inplace is not used, it causes rclone to use the --partial-suffix as suffix for temporary files.

    +

    Suffix length limit is 16 characters.

    +

    The default is .partial.

    --password-command SpaceSepList

    This flag supplies a program which should supply the config password when run. This is an alternative to rclone prompting for the password or setting the RCLONE_CONFIG_PASS variable.

    The argument to this should be a command with a space separated list of arguments. If one of the arguments has a space in then enclose it in ", if you want a literal " in an argument then enclose the argument in " and double the ". See CSV encoding for more info.

    Eg

    -
    --password-command echo hello
---password-command echo "hello with space"
---password-command echo "hello with ""quotes"" and space"
    +
    --password-command "echo hello"
+--password-command 'echo "hello with space"'
+--password-command 'echo "hello with ""quotes"" and space"'

    See the Configuration Encryption for more info.

    See a Windows PowerShell example on the Wiki.

    -P, --progress

    @@ -6220,18 +6867,12 @@ y/n/s/!/q> n

    Specifying --delete-during will delete files while checking and uploading files. This is the fastest option and uses the least memory.

    Specifying --delete-after (the default value) will delay deletion of files until all new/updated files have been successfully transferred. The files to be deleted are collected in the copy pass then deleted after the copy pass has completed successfully. The files to be deleted are held in memory so this mode may use more memory. This is the safest mode as it will only delete files if there have been no errors subsequent to that. If there have been errors before the deletions start then you will get the message not deleting files as there were IO errors.

    --fast-list

    -

    When doing anything which involves a directory listing (e.g. sync, copy, ls - in fact nearly every command), rclone normally lists a directory and processes it before using more directory lists to process any subdirectories. This can be parallelised and works very quickly using the least amount of memory.

    -

    However, some remotes have a way of listing all files beneath a directory in one (or a small number) of transactions. These tend to be the bucket-based remotes (e.g. S3, B2, GCS, Swift).

    -

    If you use the --fast-list flag then rclone will use this method for listing directories. This will have the following consequences for the listing:

    - -

    rclone should always give identical results with and without --fast-list.

    -

    If you pay for transactions and can fit your entire sync listing into memory then --fast-list is recommended. If you have a very big sync to do then don't use --fast-list otherwise you will run out of memory.

    -

    If you use --fast-list on a remote which doesn't support it, then rclone will just ignore it.

    +

    When doing anything which involves a directory listing (e.g. sync, copy, ls - in fact nearly every command), rclone has different strategies to choose from.

    +

    The basic strategy is to list one directory and processes it before using more directory lists to process any subdirectories. This is a mandatory backend feature, called List, which means it is supported by all backends. This strategy uses small amount of memory, and because it can be parallelised it is fast for operations involving processing of the list results.

    +

    Some backends provide the support for an alternative strategy, where all files beneath a directory can be listed in one (or a small number) of transactions. Rclone supports this alternative strategy through an optional backend feature called ListR. You can see in the storage system overview documentation's optional features section which backends it is enabled for (these tend to be the bucket-based ones, e.g. S3, B2, GCS, Swift). This strategy requires fewer transactions for highly recursive operations, which is important on backends where this is charged or heavily rate limited. It may be faster (due to fewer transactions) or slower (because it can't be parallelized) depending on different parameters, and may require more memory if rclone has to keep the whole listing in memory.

    +

    Which listing strategy rclone picks for a given operation is complicated, but in general it tries to choose the best possible. It will prefer ListR in situations where it doesn't need to store the listed files in memory, e.g. for unlimited recursive ls command variants. In other situations it will prefer List, e.g. for sync and copy, where it needs to keep the listed files in memory, and is performing operations on them where parallelization may be a huge advantage.

    +

    Rclone is not able to take all relevant parameters into account for deciding the best strategy, and therefore allows you to influence the choice in two ways: You can stop rclone from using ListR by disabling the feature, using the --disable option (--disable ListR), or you can allow rclone to use ListR where it would normally choose not to do so due to higher memory usage, using the --fast-list option. Rclone should always produce identical results either way. Using --disable ListR or --fast-list on a remote which doesn't support ListR does nothing, rclone will just ignore it.

    +

    A rule of thumb is that if you pay for transactions and can fit your entire sync listing into memory, then --fast-list is recommended. If you have a very big sync to do, then don't use --fast-list, otherwise you will run out of memory. Run some tests and compare before you decide, and if in doubt then just leave the default, let rclone decide, i.e. not use --fast-list.

    --timeout=TIME

    This sets the IO idle timeout. If a transfer has started but then becomes idle for this long it is considered broken and disconnected.

    The default is 5m. Set to 0 to disable.

    @@ -6348,6 +6989,8 @@ export RCLONE_CONFIG_PASS

    This dumps a list of the running go-routines at the end of the command to standard output.

    --dump openfiles

    This dumps a list of the open files at the end of the command. It uses the lsof command to do that so you'll need that installed to use it.

    +

    --dump mapper

    +

    This shows the JSON blobs being sent to the program supplied with --metadata-mapper and received from it. It can be useful for debugging the metadata mapper interface.

    --memprofile=FILE

    Write memory profile to file. This can be analysed with go tool pprof.

    Filtering

    @@ -6414,7 +7057,7 @@ export RCLONE_CONFIG_PASS

    Environment Variables

    Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.

    -

    Options

    +

    Options

    Every option in rclone can have its default set by environment variable.

    To find the name of the environment variable, first, take the long option name, strip the leading --, change - to _, make upper case and prepend RCLONE_.

    For example, to always set --stats 5s, set the environment variable RCLONE_STATS=5s. If you set stats on the command line this will override the environment variable setting.

    @@ -7777,6 +8420,40 @@ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"Cache

    The result is as returned from rclone about --json

    See the about command for more information on the above.

    Authentication is required for this call.

    +

    operations/check: check the source and destination are the same

    +

    Checks the files in the source and destination match. It compares sizes and hashes and logs a report of files that don't match. It doesn't alter the source or destination.

    +

    This takes the following parameters:

    + +

    If you supply the download flag, it will download the data from both remotes and check them against each other on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.

    +

    If you supply the size-only global flag, it will only compare the sizes not the hashes as well. Use this for a quick check.

    +

    If you supply the checkFileHash option with a valid hash name, the checkFileFs:checkFileRemote must point to a text file in the SUM format. This treats the checksum file as the source and dstFs as the destination. Note that srcFs is not used and should not be supplied in this case.

    +

    Returns:

    + +

    Authentication is required for this call.

    operations/cleanup: Remove trashed files in the remote or path

    This takes the following parameters: