From 8d78768aaad75e8ff634981458990a66820093fd Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Tue, 24 Sep 2024 15:47:01 +0100 Subject: [PATCH] Version v1.68.1 --- MANUAL.html | 283 +++++++++++++---------- MANUAL.md | 163 +++++++------ MANUAL.txt | 178 ++++++++------ docs/content/changelog.md | 19 ++ docs/content/commands/rclone.md | 13 +- docs/content/commands/rclone_mount.md | 10 +- docs/content/commands/rclone_nfsmount.md | 10 +- docs/content/commands/rclone_rcd.md | 2 +- docs/content/flags.md | 13 +- docs/content/pikpak.md | 59 +---- rclone.1 | 241 +++++++++++++------ 11 files changed, 592 insertions(+), 399 deletions(-) diff --git a/MANUAL.html b/MANUAL.html index d4b683fec..1327ff595 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -81,7 +81,7 @@

rclone(1) User Manual

Nick Craig-Wood

-

Sep 08, 2024

+

Sep 24, 2024

Rclone syncs your files to cloud storage

rclone logo

@@ -2964,7 +2964,9 @@ rclone mount remote:path/to/files \\cloud\remote

When running in background mode the user will have to stop the mount manually:

# Linux
 fusermount -u /path/to/local/mount
-# OS X
+#... or on some systems
+fusermount3 -u /path/to/local/mount
+# OS X or Linux when using nfsmount
 umount /path/to/local/mount

The umount operation can fail, for example when the mountpoint is busy. When that happens, it is the user's responsibility to stop the mount manually.

The size of the mounted file system will be set according to information retrieved from the remote, the same as returned by the rclone about command. Remotes with unlimited storage may report the used size only, then an additional 1 PiB of free space is assumed. If the remote does not support the about feature at all, then 1 PiB is set as both the total and the free size.

@@ -3048,7 +3050,7 @@ sudo ln -s /opt/local/lib/libfuse.2.dylib

Note that all the rclone filters can be used to select a subset of the files to be visible in the mount.

systemd

When running rclone mount as a systemd service, it is possible to use Type=notify. In this case the service will enter the started state after the mountpoint has been successfully set up. Units having the rclone mount service specified as a requirement will see all files and folders immediately in this mode.

-

Note that systemd runs mount units without any environment variables including PATH or HOME. This means that tilde (~) expansion will not work and you should provide --config and --cache-dir explicitly as absolute paths via rclone arguments. Since mounting requires the fusermount program, rclone will use the fallback PATH of /bin:/usr/bin in this scenario. Please ensure that fusermount is present on this PATH.

+

Note that systemd runs mount units without any environment variables including PATH or HOME. This means that tilde (~) expansion will not work and you should provide --config and --cache-dir explicitly as absolute paths via rclone arguments. Since mounting requires the fusermount or fusermount3 program, rclone will use the fallback PATH of /bin:/usr/bin in this scenario. Please ensure that fusermount/fusermount3 is present on this PATH.

Rclone as Unix mount helper

The core Unix program /bin/mount normally takes the -t FSTYPE argument then runs the /sbin/mount.FSTYPE helper program passing it mount options as -o key=val,... or --opt=.... Automount (classic or systemd) behaves in a similar way.

rclone by default expects GNU-style flags --key val. To run it as a mount helper you should symlink rclone binary to /sbin/mount.rclone and optionally /usr/bin/rclonefs, e.g. ln -s /usr/bin/rclone /sbin/mount.rclone. rclone will detect it and translate command-line arguments appropriately.

@@ -3482,7 +3484,9 @@ rclone nfsmount remote:path/to/files \\cloud\remote

When running in background mode the user will have to stop the mount manually:

# Linux
 fusermount -u /path/to/local/mount
-# OS X
+#... or on some systems
+fusermount3 -u /path/to/local/mount
+# OS X or Linux when using nfsmount
 umount /path/to/local/mount

The umount operation can fail, for example when the mountpoint is busy. When that happens, it is the user's responsibility to stop the mount manually.

The size of the mounted file system will be set according to information retrieved from the remote, the same as returned by the rclone about command. Remotes with unlimited storage may report the used size only, then an additional 1 PiB of free space is assumed. If the remote does not support the about feature at all, then 1 PiB is set as both the total and the free size.

@@ -3566,7 +3570,7 @@ sudo ln -s /opt/local/lib/libfuse.2.dylib

Note that all the rclone filters can be used to select a subset of the files to be visible in the mount.

systemd

When running rclone nfsmount as a systemd service, it is possible to use Type=notify. In this case the service will enter the started state after the mountpoint has been successfully set up. Units having the rclone nfsmount service specified as a requirement will see all files and folders immediately in this mode.

-

Note that systemd runs mount units without any environment variables including PATH or HOME. This means that tilde (~) expansion will not work and you should provide --config and --cache-dir explicitly as absolute paths via rclone arguments. Since mounting requires the fusermount program, rclone will use the fallback PATH of /bin:/usr/bin in this scenario. Please ensure that fusermount is present on this PATH.

+

Note that systemd runs mount units without any environment variables including PATH or HOME. This means that tilde (~) expansion will not work and you should provide --config and --cache-dir explicitly as absolute paths via rclone arguments. Since mounting requires the fusermount or fusermount3 program, rclone will use the fallback PATH of /bin:/usr/bin in this scenario. Please ensure that fusermount/fusermount3 is present on this PATH.

Rclone as Unix mount helper

The core Unix program /bin/mount normally takes the -t FSTYPE argument then runs the /sbin/mount.FSTYPE helper program passing it mount options as -o key=val,... or --opt=.... Automount (classic or systemd) behaves in a similar way.

rclone by default expects GNU-style flags --key val. To run it as a mount helper you should symlink rclone binary to /sbin/mount.rclone and optionally /usr/bin/rclonefs, e.g. ln -s /usr/bin/rclone /sbin/mount.rclone. rclone will detect it and translate command-line arguments appropriately.

@@ -4056,7 +4060,7 @@ htpasswd -B htpasswd anotherUser

RC Options

Flags to control the Remote Control API

      --rc                                 Enable the remote control server
-      --rc-addr stringArray                IPaddress:Port or :Port to bind server to (default ["localhost:5572"])
+      --rc-addr stringArray                IPaddress:Port or :Port to bind server to (default localhost:5572)
       --rc-allow-origin string             Origin which cross-domain request (CORS) can be executed from
       --rc-baseurl string                  Prefix for URLs - leave blank for root
       --rc-cert string                     TLS PEM key (concatenation of certificate and CA certificate)
@@ -7849,6 +7853,38 @@ export RCLONE_CONFIG_PASS

Verbosity is slightly different, the environment variable equivalent of --verbose or -v is RCLONE_VERBOSE=1, or for -vv, RCLONE_VERBOSE=2.

The same parser is used for the options and the environment variables so they take exactly the same form.

The options set by environment variables can be seen with the -vv flag, e.g. rclone version -vv.

+

Options that can appear multiple times (type stringArray) are treated slighly differently as environment variables can only be defined once. In order to allow a simple mechanism for adding one or many items, the input is treated as a CSV encoded string. For example

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableEquivalent options
RCLONE_EXCLUDE="*.jpg"--exclude "*.jpg"
RCLONE_EXCLUDE="*.jpg,*.png"--exclude "*.jpg" --exclude "*.png"
RCLONE_EXCLUDE='"*.jpg","*.png"'--exclude "*.jpg" --exclude "*.png"
RCLONE_EXCLUDE='"/directory with comma , in it /**"'`--exclude "/directory with comma , in it /**"
+

If stringArray options are defined as environment variables and options on the command line then all the values will be used.

Config file

You can set defaults for values in the config file on an individual remote basis. The names of the config items are documented in the page for each backend.

To find the name of the environment variable, you need to set, take RCLONE_CONFIG_ + name of remote + _ + name of config file option and make it all uppercase. Note one implication here is the remote's name must be convertible into a valid environment variable name, so it can only contain letters, digits, or the _ (underscore) character.

@@ -11932,7 +11968,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total --tpslimit float Limit HTTP transactions per second to this --tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --use-cookies Enable session cookiejar - --user-agent string Set the user-agent to a specified string (default "rclone/v1.68.0") + --user-agent string Set the user-agent to a specified string (default "rclone/v1.68.1")

Performance

Flags helpful for increasing performance.

      --buffer-size SizeSuffix   In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -12033,7 +12069,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
 
RC

 
Flags to control the Remote Control API.

 
      --rc                                 Enable the remote control server
-      --rc-addr stringArray                IPaddress:Port or :Port to bind server to (default ["localhost:5572"])
+      --rc-addr stringArray                IPaddress:Port or :Port to bind server to (default localhost:5572)
       --rc-allow-origin string             Origin which cross-domain request (CORS) can be executed from
       --rc-baseurl string                  Prefix for URLs - leave blank for root
       --rc-cert string                     TLS PEM key (concatenation of certificate and CA certificate)
@@ -12063,7 +12099,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
       --rc-web-gui-update                  Check and update to latest version of web gui

 
Metrics

 
Flags to control the Metrics HTTP endpoint..

-
      --metrics-addr stringArray                IPaddress:Port or :Port to bind metrics server to (default [""])
+
      --metrics-addr stringArray                IPaddress:Port or :Port to bind metrics server to
       --metrics-allow-origin string             Origin which cross-domain request (CORS) can be executed from
       --metrics-baseurl string                  Prefix for URLs - leave blank for root
       --metrics-cert string                     TLS PEM key (concatenation of certificate and CA certificate)
@@ -12551,21 +12587,18 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
       --pcloud-token string                                 OAuth Access Token as a JSON blob
       --pcloud-token-url string                             Token server url
       --pcloud-username string                              Your pcloud username
-      --pikpak-auth-url string                              Auth server URL
       --pikpak-chunk-size SizeSuffix                        Chunk size for multipart uploads (default 5Mi)
-      --pikpak-client-id string                             OAuth Client Id
-      --pikpak-client-secret string                         OAuth Client Secret
       --pikpak-description string                           Description of the remote
+      --pikpak-device-id string                             Device ID used for authorization
       --pikpak-encoding Encoding                            The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot)
       --pikpak-hash-memory-limit SizeSuffix                 Files bigger than this will be cached on disk to calculate hash if required (default 10Mi)
       --pikpak-pass string                                  Pikpak password (obscured)
       --pikpak-root-folder-id string                        ID of the root folder
-      --pikpak-token string                                 OAuth Access Token as a JSON blob
-      --pikpak-token-url string                             Token server url
       --pikpak-trashed-only                                 Only show files that are in the trash
       --pikpak-upload-concurrency int                       Concurrency for multipart uploads (default 5)
       --pikpak-use-trash                                    Send files to the trash instead of deleting permanently (default true)
       --pikpak-user string                                  Pikpak username
+      --pikpak-user-agent string                            HTTP user agent for pikpak (default "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:129.0) Gecko/20100101 Firefox/129.0")
       --pixeldrain-api-key string                           API key for your pixeldrain account
       --pixeldrain-api-url string                           The API endpoint to connect to. In the vast majority of cases it's fine to leave (default "https://pixeldrain.com/api")
       --pixeldrain-description string                       Description of the remote
@@ -14529,6 +14562,18 @@ y/e/d>

 
By default, rclone will HEAD every object it uploads. It does this to check the object got uploaded correctly.

 
You can disable this with the --s3-no-head option - see there for more details.

 
Setting this flag increases the chance for undetected upload failures.

+
Increasing performance

+
Using server-side copy

+
If you are copying objects between S3 buckets in the same region, you should use server-side copy. This is much faster than downloading and re-uploading the objects, as no data is transferred.

+
For rclone to use server-side copy, you must use the same remote for the source and destination.

+
rclone copy s3:source-bucket s3:destination-bucket

+
When using server-side copy, the performance is limited by the rate at which rclone issues API requests to S3. See below for how to increase the number of API requests rclone makes.

+
Increasing the rate of API requests

+
You can increase the rate of API requests to S3 by increasing the parallelism using --transfers and --checkers options.

+
Rclone uses a very conservative defaults for these settings, as not all providers support high rates of requests. Depending on your provider, you can increase significantly the number of transfers and checkers.

+
For example, with AWS S3, if you can increase the number of checkers to values like 200. If you are doing a server-side copy, you can also increase the number of transfers to 200.

+
rclone sync --transfers 200 --checkers 200 --checksum s3:source-bucket s3:destination-bucket

+
You will need to experiment with these values to find the optimal settings for your setup.

 
Versions

 
When bucket versioning is enabled (this can be done with rclone with the rclone backend versioning command) when rclone uploads a new version of a file it creates a new version of it Likewise when you delete a file, the old version will be marked hidden and still be available.

 
Old versions of files, where available, are visible using the --s3-versions flag.

@@ -18322,7 +18367,7 @@ cos                  s3

 
For Netease NOS configure as per the configurator rclone config setting the provider Netease. This will automatically set force_path_style = false which is necessary for it to run properly.

 
Petabox

 
Here is an example of making a Petabox configuration. First run:

-
rclone config

+
rclone config

 
This will guide you through an interactive setup process.

 
No remotes found, make a new one?
 n) New remote
@@ -29660,75 +29705,75 @@ rclone rc vfs/refresh recursive=true

 
Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are "read", "write", "read,write", and "off" (the default). "write" supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use "read,write" instead of "write" if you wish to update/remove permissions.

 
Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business.

 
Example for OneDrive Personal:

-
[
-    {
-        "id": "1234567890ABC!123",
-        "grantedTo": {
-            "user": {
-                "id": "ryan@contoso.com"
-            },
-            "application": {},
-            "device": {}
-        },
-        "invitation": {
-            "email": "ryan@contoso.com"
-        },
-        "link": {
-            "webUrl": "https://1drv.ms/t/s!1234567890ABC"
-        },
-        "roles": [
-            "read"
-        ],
-        "shareId": "s!1234567890ABC"
-    }
-]

+
[
+    {
+        "id": "1234567890ABC!123",
+        "grantedTo": {
+            "user": {
+                "id": "ryan@contoso.com"
+            },
+            "application": {},
+            "device": {}
+        },
+        "invitation": {
+            "email": "ryan@contoso.com"
+        },
+        "link": {
+            "webUrl": "https://1drv.ms/t/s!1234567890ABC"
+        },
+        "roles": [
+            "read"
+        ],
+        "shareId": "s!1234567890ABC"
+    }
+]

 
Example for OneDrive Business:

-
[
-    {
-        "id": "48d31887-5fad-4d73-a9f5-3c356e68a038",
-        "grantedToIdentities": [
-            {
-                "user": {
-                    "displayName": "ryan@contoso.com"
-                },
-                "application": {},
-                "device": {}
-            }
-        ],
-        "link": {
-            "type": "view",
-            "scope": "users",
-            "webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s"
-        },
-        "roles": [
-            "read"
-        ],
-        "shareId": "u!LKj1lkdlals90j1nlkascl"
-    },
-    {
-        "id": "5D33DD65C6932946",
-        "grantedTo": {
-            "user": {
-                "displayName": "John Doe",
-                "id": "efee1b77-fb3b-4f65-99d6-274c11914d12"
-            },
-            "application": {},
-            "device": {}
-        },
-        "roles": [
-            "owner"
-        ],
-        "shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U"
-    }
-]

+
[
+    {
+        "id": "48d31887-5fad-4d73-a9f5-3c356e68a038",
+        "grantedToIdentities": [
+            {
+                "user": {
+                    "displayName": "ryan@contoso.com"
+                },
+                "application": {},
+                "device": {}
+            }
+        ],
+        "link": {
+            "type": "view",
+            "scope": "users",
+            "webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s"
+        },
+        "roles": [
+            "read"
+        ],
+        "shareId": "u!LKj1lkdlals90j1nlkascl"
+    },
+    {
+        "id": "5D33DD65C6932946",
+        "grantedTo": {
+            "user": {
+                "displayName": "John Doe",
+                "id": "efee1b77-fb3b-4f65-99d6-274c11914d12"
+            },
+            "application": {},
+            "device": {}
+        },
+        "roles": [
+            "owner"
+        ],
+        "shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U"
+    }
+]

 
To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this.

 
When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous".

 
Example request to add a "read" permission with --metadata-mapper:

-
{
-    "Metadata": {
-        "permissions": "[{\"grantedToIdentities\":[{\"user\":{\"id\":\"ryan@contoso.com\"}}],\"roles\":[\"read\"]}]"
-    }
-}

+
{
+    "Metadata": {
+        "permissions": "[{\"grantedToIdentities\":[{\"user\":{\"id\":\"ryan@contoso.com\"}}],\"roles\":[\"read\"]}]"
+    }
+}

 
Note that adding a permission can fail if a conflicting permission already exists for the file/folder.

 
To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed.

 
To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that the owner role will be ignored, as it cannot be removed.

@@ -32200,54 +32245,24 @@ y/e/d> y

Advanced options

Here are the Advanced options specific to pikpak (PikPak).

-

--pikpak-client-id

-

OAuth Client Id.

-

Leave blank normally.

+

--pikpak-device-id

+

Device ID used for authorization.

Properties:

-

--pikpak-client-secret

-

OAuth Client Secret.

-

Leave blank normally.

+

--pikpak-user-agent

+

HTTP user agent for pikpak.

+

Defaults to "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:129.0) Gecko/20100101 Firefox/129.0" or "--pikpak-user-agent" provided on command line.

Properties:

-

--pikpak-token

-

OAuth Access Token as a JSON blob.

-

Properties:

- -

--pikpak-auth-url

-

Auth server URL.

-

Leave blank to use the provider defaults.

-

Properties:

- -

--pikpak-token-url

-

Token server url.

-

Leave blank to use the provider defaults.

-

Properties:

-

--pikpak-root-folder-id

ID of the root folder. Leave blank normally.

@@ -36949,6 +36964,34 @@ $ tree /tmp/c
  • "error": return an error based on option value

    • Changelog

    +

    v1.68.1 - 2024-09-24

    +

    See commits

    +

    v1.68.0 - 2024-09-08

    See commits