alexvanin/rclone
1
0
Fork 0
forked from TrueCloudLab/rclone
Code Pull requests Activity

[#1] Add frostfs backend docs

Browse source 
Signed-off-by: Aleksey Kravchenko <al.kravchenko@yadro.com>
This commit is contained in:
Aleksey Kravchenko 2025-01-10 19:00:25 +03:00
parent 5385ce33a5
commit 0255757831
9 changed files with 37339 additions and 16471 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37880
MANUAL.html generated
View file

File diff suppressed because it is too large Load diff

485
MANUAL.md generated
View file

@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Nov 15, 2024
% Jan 10, 2025
# Rclone syncs your files to cloud storage
@ -120,6 +120,7 @@ WebDAV or S3, that work out of the box.)
- Enterprise File Fabric
- Fastmail Files
- Files.com
- FrostFS
- FTP
- Gofile
- Google Cloud Storage
@ -20497,7 +20498,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2-beta.8331.25cf42493.feature/add-frostfs-support")
```
@ -20934,6 +20935,21 @@ Backend-only flags (these can be set in the config file also).
--filescom-password string The password used to authenticate with Files.com (obscured)
--filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com)
--filescom-username string The username used to authenticate with Files.com
--frostfs-address string Address of account
--frostfs-ape-cache-invalidation-duration Duration APE cache invalidation duration (default 8s)
--frostfs-ape-cache-invalidation-timeout Duration APE cache invalidation timeout (default 24s)
--frostfs-ape-chain-check-interval Duration The interval for verifying that the APE chain is saved in FrostFS (default 500ms)
--frostfs-connection-timeout Duration FrostFS connection timeout (default 4s)
--frostfs-container-creation-policy string Container creation policy for new containers (default "private")
--frostfs-description string Description of the remote
--frostfs-endpoint string Endpoints to connect to FrostFS node
--frostfs-password string Password to decrypt wallet
--frostfs-placement-policy string Placement policy for new containers (default "REP 3")
--frostfs-rebalance-interval Duration FrostFS rebalance connections interval (default 15s)
--frostfs-request-timeout Duration FrostFS request timeout (default 12s)
--frostfs-rpc-endpoint string Endpoint to connect to Neo rpc node
--frostfs-session-expiration int FrostFS session expiration epoch (default 4294967295)
--frostfs-wallet string Path to wallet
--ftp-ask-password Allow asking for FTP password when needed
--ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
@ -34604,6 +34620,467 @@ Properties:
# FrostFS
Rclone FrostFS support is provided using the
[git.frostfs.info/TrueCloudLab/frostfs-sdk-go](https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go) package
## Configuration
To create an FrostFS configuration named `remote`, run
rclone config
Rclone config guides you through an interactive setup process. A minimal
rclone FrostFS remote definition only requires endpoint and path to FrostFS user wallet.
```
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
Enter name for new remote.
name> remote
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3)
5 / Backblaze B2
\ (b2)
6 / Better checksums for other remotes
\ (hasher)
7 / Box
\ (box)
8 / Cache a remote
\ (cache)
9 / Citrix Sharefile
\ (sharefile)
10 / Combine several remotes into one
\ (combine)
11 / Compress a remote
\ (compress)
12 / Distributed, decentralized object storage FrostFS
\ (frostfs)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Files.com
\ (filescom)
18 / Gofile
\ (gofile)
19 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
20 / Google Drive
\ (drive)
21 / Google Photos
\ (google photos)
22 / HTTP
\ (http)
23 / Hadoop distributed file system
\ (hdfs)
24 / HiDrive
\ (hidrive)
25 / ImageKit.io
\ (imagekit)
26 / In memory object storage system.
\ (memory)
27 / Internet Archive
\ (internetarchive)
28 / Jottacloud
\ (jottacloud)
29 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
30 / Linkbox
\ (linkbox)
31 / Local Disk
\ (local)
32 / Mail.ru Cloud
\ (mailru)
33 / Mega
\ (mega)
34 / Microsoft Azure Blob Storage
\ (azureblob)
35 / Microsoft Azure Files
\ (azurefiles)
36 / Microsoft OneDrive
\ (onedrive)
37 / OpenDrive
\ (opendrive)
38 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
\ (swift)
39 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
40 / Pcloud
\ (pcloud)
41 / PikPak
\ (pikpak)
42 / Pixeldrain Filesystem
\ (pixeldrain)
43 / Proton Drive
\ (protondrive)
44 / Put.io
\ (putio)
45 / QingCloud Object Storage
\ (qingstor)
46 / Quatrix by Maytech
\ (quatrix)
47 / SMB / CIFS
\ (smb)
48 / SSH/SFTP
\ (sftp)
49 / Sia Decentralized Cloud
\ (sia)
50 / Storj Decentralized Cloud Storage
\ (storj)
51 / Sugarsync
\ (sugarsync)
52 / Transparently chunk/split large files
\ (chunker)
53 / Uloz.to
\ (ulozto)
54 / Union merges the contents of several upstream fs
\ (union)
55 / Uptobox
\ (uptobox)
56 / WebDAV
\ (webdav)
57 / Yandex Disk
\ (yandex)
58 / Zoho
\ (zoho)
59 / premiumize.me
\ (premiumizeme)
60 / seafile
\ (seafile)
Storage> frostfs
Option endpoint.
Endpoints to connect to FrostFS node
Choose a number from below, or type in your own value.
1 / One endpoint.
\ (s01.frostfs.devenv:8080)
2 / Multiple endpoints to form pool.
\ (s01.frostfs.devenv:8080 s02.frostfs.devenv:8080)
3 / Multiple endpoints with priority (less value is higher priority). Until s01 is healthy all request will be send to it.
\ (s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2)
4 / Multiple endpoints with priority and weights. After s01 is unhealthy requests will be send to s02 and s03 in proportions 10% and 90% respectively.
\ (s01.frostfs.devenv:8080,1,1 s02.frostfs.devenv:8080,2,1 s03.frostfs.devenv:8080,2,9)
endpoint> s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2
Option connection_timeout.
FrostFS connection timeout
Enter a value of type Duration. Press Enter for the default (4s).
connection_timeout>
Option request_timeout.
FrostFS request timeout
Enter a value of type Duration. Press Enter for the default (4s).
request_timeout>
Option rebalance_interval.
FrostFS rebalance connections interval
Enter a value of type Duration. Press Enter for the default (15s).
rebalance_interval>
Option session_expiration.
FrostFS session expiration epoch
Enter a signed integer. Press Enter for the default (4294967295).
session_expiration>
Option ape_cache_invalidation_duration.
APE cache invalidation duration
Enter a value of type Duration. Press Enter for the default (8s).
ape_cache_invalidation_duration>
Option ape_cache_invalidation_timeout.
APE cache invalidation timeout
Enter a value of type Duration. Press Enter for the default (16s).
ape_cache_invalidation_timeout>
Option ape_chain_check_interval.
The interval for verifying that the APE chain is saved in FrostFS.
Enter a value of type Duration. Press Enter for the default (500ms).
ape_chain_check_interval>
Option rpc_endpoint.
Endpoint to connect to Neo rpc node
Enter a value. Press Enter to leave empty.
rpc_endpoint>
Option wallet.
Path to wallet
Enter a value.
wallet> /wallets/wallet.conf
Option address.
Address of account
Enter a value. Press Enter to leave empty.
address>
Option password.
Password to decrypt wallet
Enter a value. Press Enter to leave empty.
password>
Option placement_policy.
Placement policy for new containers
Choose a number from below, or type in your own value of type string.
Press Enter for the default (REP 3).
1 / Container will have 3 replicas
\ (REP 3)
placement_policy> REP 1
Option container_creation_policy.
Container creation policy for new containers
Choose a number from below, or type in your own value of type string.
Press Enter for the default (private).
1 / Public container, anyone can read and write
\ (public-read-write)
2 / Public container, owner can read and write, others can only read
\ (public-read)
3 / Private container, only owner has access to it
\ (private)
container_creation_policy>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
- type: frostfs
- endpoint: s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2
- password: M@zPGpWDravchenko/develop/go/playground/play_with_ape/cfg/wallet_akrav.js
- placement_policy: REP 1
Keep this "remote" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
```
As a remote root directory, you can use either the container identifier or its user-friendly name.
If you choose to use a container name, rclone will create a new container with that name when
it's used for the first time.
For example, both of the following commands would be valid if there was a `~/test-copy` directory and a container with
the identifier `23fk3Bcw5mPZ4YtYkTLJbQebtM2WXHz4HL8FgsrTJkSf`:
rclone copy ~/test-copy remote:23fk3Bcw5mPZ4YtYkTLJbQebtM2WXHz4HL8FgsrTJkSf/test-copy
rclone copy ~/test-copy remote:container-name/test-copy
### Standard options
Here are the Standard options specific to frostfs (Distributed, decentralized object storage FrostFS).
#### --frostfs-endpoint
Endpoints to connect to FrostFS node
Properties:
- Config: endpoint
- Env Var: RCLONE_FROSTFS_ENDPOINT
- Type: string
- Required: true
- Examples:
- "s01.frostfs.devenv:8080"
- One endpoint.
- "s01.frostfs.devenv:8080 s02.frostfs.devenv:8080"
- Multiple endpoints to form pool.
- "s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2"
- Multiple endpoints with priority (lower value has higher priority). Until s01 is healthy, all requests will be sent to it.
- "s01.frostfs.devenv:8080,1,1 s02.frostfs.devenv:8080,2,1 s03.frostfs.devenv:8080,2,9"
- Multiple endpoints with priority and weights. After the s01 endpoint is unhealthy, requests will be sent to the s02 and s03 endpoints in proportions of 10% and 90%, respectively.
#### --frostfs-connection-timeout
FrostFS connection timeout
Properties:
- Config: connection_timeout
- Env Var: RCLONE_FROSTFS_CONNECTION_TIMEOUT
- Type: Duration
- Default: 4s
#### --frostfs-request-timeout
FrostFS request timeout
Properties:
- Config: request_timeout
- Env Var: RCLONE_FROSTFS_REQUEST_TIMEOUT
- Type: Duration
- Default: 12s
#### --frostfs-rebalance-interval
FrostFS rebalance connections interval
Properties:
- Config: rebalance_interval
- Env Var: RCLONE_FROSTFS_REBALANCE_INTERVAL
- Type: Duration
- Default: 15s
#### --frostfs-session-expiration
FrostFS session expiration epoch
Properties:
- Config: session_expiration
- Env Var: RCLONE_FROSTFS_SESSION_EXPIRATION
- Type: int
- Default: 4294967295
#### --frostfs-ape-cache-invalidation-duration
APE cache invalidation duration
Properties:
- Config: ape_cache_invalidation_duration
- Env Var: RCLONE_FROSTFS_APE_CACHE_INVALIDATION_DURATION
- Type: Duration
- Default: 8s
#### --frostfs-ape-cache-invalidation-timeout
APE cache invalidation timeout
Properties:
- Config: ape_cache_invalidation_timeout
- Env Var: RCLONE_FROSTFS_APE_CACHE_INVALIDATION_TIMEOUT
- Type: Duration
- Default: 24s
#### --frostfs-ape-chain-check-interval
The interval for verifying that the APE chain is saved in FrostFS.
Properties:
- Config: ape_chain_check_interval
- Env Var: RCLONE_FROSTFS_APE_CHAIN_CHECK_INTERVAL
- Type: Duration
- Default: 500ms
#### --frostfs-rpc-endpoint
Endpoint to connect to Neo rpc node
Properties:
- Config: rpc_endpoint
- Env Var: RCLONE_FROSTFS_RPC_ENDPOINT
- Type: string
- Required: false
#### --frostfs-wallet
Path to wallet
Properties:
- Config: wallet
- Env Var: RCLONE_FROSTFS_WALLET
- Type: string
- Required: true
#### --frostfs-address
Address of account
Properties:
- Config: address
- Env Var: RCLONE_FROSTFS_ADDRESS
- Type: string
- Required: false
#### --frostfs-password
Password to decrypt wallet
Properties:
- Config: password
- Env Var: RCLONE_FROSTFS_PASSWORD
- Type: string
- Required: false
#### --frostfs-placement-policy
Placement policy for new containers
Properties:
- Config: placement_policy
- Env Var: RCLONE_FROSTFS_PLACEMENT_POLICY
- Type: string
- Default: "REP 3"
- Examples:
- "REP 3"
- Container will have 3 replicas
#### --frostfs-container-creation-policy
Container creation policy for new containers
Properties:
- Config: container_creation_policy
- Env Var: RCLONE_FROSTFS_CONTAINER_CREATION_POLICY
- Type: string
- Default: "private"
- Examples:
- "public-read-write"
- Public container, anyone can read and write
- "public-read"
- Public container, owner can read and write, others can only read
- "private"
- Private container, only owner has access to it
### Advanced options
Here are the Advanced options specific to frostfs (Distributed, decentralized object storage FrostFS).
#### --frostfs-description
Description of the remote.
Properties:
- Config: description
- Env Var: RCLONE_FROSTFS_DESCRIPTION
- Type: string
- Required: false
# FTP
FTP is the File Transfer Protocol. Rclone FTP support is provided using the
@ -34727,13 +35204,13 @@ therefore works even in Windows Command Prompt:
rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM
rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM:
### Implicit TLS
### Implicit TLS[util.go](../../backend/frostfs/util.go)
Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to
be enabled in the FTP backend config for the remote, or with
[`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and
can be set with [`--ftp-port`](#ftp-port).
[util.go](../../backend/frostfs/util.go)
### Restricted filename characters
In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)

715
MANUAL.txt generated
View file

@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Nov 15, 2024
Jan 10, 2025
Rclone syncs your files to cloud storage
@ -109,6 +109,7 @@ S3, that work out of the box.)
- Enterprise File Fabric
- Fastmail Files
- Files.com
- FrostFS
- FTP
- Gofile
- Google Cloud Storage
@ -1202,11 +1203,11 @@ what happened to it. These are reminiscent of diff files.
- = path means path was found in source and destination and was
identical
- `- path` means path was missing on the source, so only in the
- - path means path was missing on the source, so only in the
destination
- `+ path` means path was missing on the destination, so only in the
- + path means path was missing on the destination, so only in the
source
- `* path` means path was present in source and destination but
- * path means path was present in source and destination but
different.
- ! path means there was an error reading or hashing the source or
dest.
@ -1714,11 +1715,11 @@ what happened to it. These are reminiscent of diff files.
- = path means path was found in source and destination and was
identical
- `- path` means path was missing on the source, so only in the
- - path means path was missing on the source, so only in the
destination
- `+ path` means path was missing on the destination, so only in the
- + path means path was missing on the destination, so only in the
source
- `* path` means path was present in source and destination but
- * path means path was present in source and destination but
different.
- ! path means there was an error reading or hashing the source or
dest.
@ -2919,11 +2920,11 @@ what happened to it. These are reminiscent of diff files.
- = path means path was found in source and destination and was
identical
- `- path` means path was missing on the source, so only in the
- - path means path was missing on the source, so only in the
destination
- `+ path` means path was missing on the destination, so only in the
- + path means path was missing on the destination, so only in the
source
- `* path` means path was present in source and destination but
- * path means path was present in source and destination but
different.
- ! path means there was an error reading or hashing the source or
dest.
@ -4023,11 +4024,11 @@ what happened to it. These are reminiscent of diff files.
- = path means path was found in source and destination and was
identical
- `- path` means path was missing on the source, so only in the
- - path means path was missing on the source, so only in the
destination
- `+ path` means path was missing on the destination, so only in the
- + path means path was missing on the destination, so only in the
source
- `* path` means path was present in source and destination but
- * path means path was present in source and destination but
different.
- ! path means there was an error reading or hashing the source or
dest.
@ -13174,9 +13175,9 @@ Will get their own names
Valid remote names
Remote names are case sensitive, and must adhere to the following rules:
- May contain number, letter, _, -, ., +, @ and space. - May not start
with - or space. - May not end with space.
Remote names are case sensitive, and must adhere to the following
rules: - May contain number, letter, _, -, ., +, @ and space. - May not
start with - or space. - May not end with space.
Starting with rclone version 1.61, any Unicode numbers and letters are
allowed, while in older versions it was limited to plain ASCII (0-9,
@ -16096,28 +16097,49 @@ into regular expressions.
Filter pattern examples
---------------------------------------------------------------------------------------------
Description Pattern Matches Does not match
--------------- ---------------- ------------------------------- ------------------
------------------- ---------------- ------------------------------- ------------------------
Wildcard *.jpg /file.jpg /file.png
/dir/file.jpg /dir/file.png
Rooted /*.jpg /file.jpg /file.png
/file2.jpg /dir/file.jpg
Alternates *.{jpg,png} /file.jpg /file.gif
/dir/file.png /dir/file.gif
Path Wildcard dir/** /dir/anyfile file.png
/subdir/dir/subsubdir/anyfile /subdir/file.png
Any Char *.t?t /file.txt /file.qxt
/dir/file.tzt /dir/file.png
Range *.[a-z] /file.a /file.0
/dir/file.b /dir/file.1
Escape *.\?\?\? /file.??? /file.abc
/dir/file.??? /dir/file.def
Class *.\d\d\d /file.012 /file.abc
/dir/file.345 /dir/file.def
Regexp *.{{jpe?g}} /file.jpeg /file.png
/dir/file.jpg /dir/file.jpeeg
Rooted Regexp /{{.*\.jpe?g}} /file.jpeg /file.png
/file.jpg /dir/file.jpg
---------------------------------------------------------------------------------------------
How filter rules are applied to files
@ -19242,58 +19264,121 @@ Features
Here is an overview of the major features of each cloud storage system.
Name Hash ModTime Case Insensitive Duplicate Files MIME Type Metadata
------------------------------- ------------------- --------- ------------------ ----------------- ----------- ----------
--------------------------------------------------------------------------------------
Name Hash ModTime Case Duplicate MIME Metadata
Insensitive Files Type
----------------- -------------- --------- ------------- ----------- ------ ----------
1Fichier Whirlpool - No Yes R -
Akamai Netstorage MD5, SHA256 R/W No No R -
Amazon S3 (or S3 compatible) MD5 R/W No No R/W RWU
Amazon S3 (or S3 MD5 R/W No No R/W RWU
compatible)
Backblaze B2 SHA1 R/W No No R/W -
Box SHA1 R/W Yes No - -
Citrix ShareFile MD5 R/W Yes No - -
Dropbox DBHASH ¹ R Yes No - -
Enterprise File Fabric - R/W Yes No R/W -
Enterprise File - R/W Yes No R/W -
Fabric
Files.com MD5, CRC32 DR/W Yes No R -
FTP - R/W ¹⁰ No No - -
Gofile MD5 DR/W No Yes R -
Google Cloud Storage MD5 R/W No No R/W -
Google Drive MD5, SHA1, SHA256 DR/W No Yes R/W DRWU
Google Cloud MD5 R/W No No R/W -
Storage
Google Drive MD5, SHA1, DR/W No Yes R/W DRWU
SHA256
Google Photos - - No Yes R -
HDFS - R/W No No - -
HiDrive HiDrive ¹² R/W No No - -
HTTP - R No No R -
Internet Archive MD5, SHA1, CRC32 R/W ¹¹ No No - RWU
Internet Archive MD5, SHA1, R/W ¹¹ No No - RWU
CRC32
Jottacloud MD5 R/W Yes No R RW
Koofr MD5 - Yes No - -
Linkbox - R No No - -
Mail.ru Cloud Mailru ⁶ R/W Yes No - -
Mega - - No Yes - -
Memory MD5 R/W No No - -
Microsoft Azure Blob Storage MD5 R/W No No R/W -
Microsoft Azure Files Storage MD5 R/W Yes No R/W -
Microsoft OneDrive QuickXorHash ⁵ DR/W Yes No R DRW
Microsoft Azure MD5 R/W No No R/W -
Blob Storage
Microsoft Azure MD5 R/W Yes No R/W -
Files Storage
Microsoft QuickXorHash ⁵ DR/W Yes No R DRW
OneDrive
OpenDrive MD5 R/W Yes Partial ⁸ - -
OpenStack Swift MD5 R/W No No R/W -
Oracle Object Storage MD5 R/W No No R/W -
Oracle Object MD5 R/W No No R/W -
Storage
pCloud MD5, SHA1 ⁷ R/W No No W -
PikPak MD5 R No No R -
Pixeldrain SHA256 R/W No No R RW
premiumize.me - - Yes No R -
put.io CRC-32 R/W No Yes R -
Proton Drive SHA1 R/W No No R -
QingStor MD5 - ⁹ No No R/W -
Quatrix by Maytech - R/W No No - -
Quatrix by - R/W No No - -
Maytech
Seafile - - No No - -
SFTP MD5, SHA1 ² DR/W Depends No - -
Sia - - No No - -
SMB - R/W Yes No - -
SugarSync - - No No - -
Storj - R No No - -
Uloz.to MD5, SHA256 ¹³ - No Yes - -
Uptobox - - No Yes - -
WebDAV MD5, SHA1 ³ R ⁴ Depends No - -
Yandex Disk MD5 R/W No No R -
Zoho WorkDrive - - No No - -
The local filesystem All DR/W Depends No - DRWU
The local All DR/W Depends No - DRWU
filesystem
--------------------------------------------------------------------------------------
¹ Dropbox supports its own custom hash. This is an SHA256 sum of all the
4 MiB block SHA256s.
@ -20081,7 +20166,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2-beta.8331.25cf42493.feature/add-frostfs-support")
Performance
@ -20488,6 +20573,21 @@ Backend-only flags (these can be set in the config file also).
--filescom-password string The password used to authenticate with Files.com (obscured)
--filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com)
--filescom-username string The username used to authenticate with Files.com
--frostfs-address string Address of account
--frostfs-ape-cache-invalidation-duration Duration APE cache invalidation duration (default 8s)
--frostfs-ape-cache-invalidation-timeout Duration APE cache invalidation timeout (default 24s)
--frostfs-ape-chain-check-interval Duration The interval for verifying that the APE chain is saved in FrostFS (default 500ms)
--frostfs-connection-timeout Duration FrostFS connection timeout (default 4s)
--frostfs-container-creation-policy string Container creation policy for new containers (default "private")
--frostfs-description string Description of the remote
--frostfs-endpoint string Endpoints to connect to FrostFS node
--frostfs-password string Password to decrypt wallet
--frostfs-placement-policy string Placement policy for new containers (default "REP 3")
--frostfs-rebalance-interval Duration FrostFS rebalance connections interval (default 15s)
--frostfs-request-timeout Duration FrostFS request timeout (default 12s)
--frostfs-rpc-endpoint string Endpoint to connect to Neo rpc node
--frostfs-session-expiration int FrostFS session expiration epoch (default 4294967295)
--frostfs-wallet string Path to wallet
--ftp-ask-password Allow asking for FTP password when needed
--ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
@ -22714,7 +22814,7 @@ include-style filters for Windows user directories below.
Filters file writing guidelines
1. Begin with excluding directory trees:
- e.g. `- /AppData/`
- e.g. - /AppData/
- ** on the end is not necessary. Once a given directory level is
excluded then everything beneath it won't be looked at by
rclone.
@ -22729,22 +22829,22 @@ Filters file writing guidelines
-- or --
- Exclude select directories and therefore include everything else
3. Include select directories:
- Add lines like: `+ /Documents/PersonalFiles/**` to select which
- Add lines like: + /Documents/PersonalFiles/** to select which
directories to include in the sync.
- ** on the end specifies to include the full depth of the
specified tree.
- With Include-style filters, files at the Path1/Path2 root are
not included. They may be included with `+ /*`.
not included. They may be included with + /*.
- Place RCLONE_TEST files within these included directory trees.
They will only be looked for in these directory trees.
- Finish by excluding everything else by adding `- **` at the end
of the filters file.
- Finish by excluding everything else by adding - ** at the end of
the filters file.
- Disregard step 4.
4. Exclude select directories:
- Add more lines like in step 1. For example:
-/Desktop/tempfiles/, or `- /testdir/. Again, a**` on the end
is not necessary.
- Do not add a `- **` in the file. Without this line, everything
-/Desktop/tempfiles/, or - /testdir/. Again, a ** on the end is
not necessary.
- Do not add a - ** in the file. Without this line, everything
will be included that has not been explicitly excluded.
- Disregard step 3.
@ -22769,7 +22869,7 @@ This Windows include-style example is based on the sync root (Path1) set
to C:\Users\MyLogin. The strategy is to select specific directories to
be synched with a network drive (Path2).
- `- /AppData/` excludes an entire tree of Windows stored stuff that
- - /AppData/ excludes an entire tree of Windows stored stuff that
need not be synched. In my case, AppData has >11 GB of stuff I don't
care about, and there are some subdirectories beneath AppData that
are not accessible to my user login, resulting in bisync critical
@ -22778,11 +22878,12 @@ be synched with a network drive (Path2).
NTUSER at C:\Users\MyLogin. These files may be dynamic, locked, and
are generally don't care.
- There are just a few directories with my data that I do want
synched, in the form of `+
/. By selecting only the directory trees I want to avoid the dozen plus directories that various apps make atC:`.
synched, in the form of + /<path>. By selecting only the directory
trees I want to avoid the dozen plus directories that various apps
make at C:\Users\MyLogin\Documents.
- Include files in the root of the sync point, C:\Users\MyLogin, by
adding the `+ /*` line.
- This is an Include-style filters file, therefore it ends with `- **`
adding the + /* line.
- This is an Include-style filters file, therefore it ends with - **
which excludes everything not explicitly included.
- /AppData/
@ -22809,14 +22910,14 @@ This noise can be quashed by adding --quiet to the bisync command line.
Example exclude-style filters files for use with Dropbox
- Dropbox disallows synching the listed temporary and
configuration/data files. The `- ` filters exclude these files where
ever they may occur in the sync tree. Consider adding similar
exclusions for file types you don't need to sync, such as core dump
and software build files.
configuration/data files. The - <filename> filters exclude these
files where ever they may occur in the sync tree. Consider adding
similar exclusions for file types you don't need to sync, such as
core dump and software build files.
- bisync testing creates /testdir/ at the top level of the sync tree,
and usually deletes the tree after the test. If a normal sync should
run while the /testdir/ tree exists the --check-access phase may
fail due to unbalanced RCLONE_TEST files. The `- /testdir/` filter
fail due to unbalanced RCLONE_TEST files. The - /testdir/ filter
blocks this tree from being synched. You don't need this exclusion
if you are not doing bisync development testing.
- Everything else beneath the Path1/Path2 root will be synched.
@ -23190,10 +23291,14 @@ Notes about testing
these directories when validating and debugging a given test. These
directories will be flushed before running another test, independent
of the -no-cleanup usage.
- You will likely want to add `-
/testdir/to your normal bisync--filters-fileso that normal syncs do not attempt to sync the test temporary directories, which may haveRCLONE_TESTmiscompares in some testcases which would otherwise trip the--check-accesssystem. The--check-accessmechanism is hard-coded to ignoreRCLONE_TESTfiles beneathbisync/testdata`,
so the test cases may reside on the synched tree even if there are
check file mismatches in the test tree.
- You will likely want to add - /testdir/ to your normal bisync
--filters-file so that normal syncs do not attempt to sync the test
temporary directories, which may have RCLONE_TEST miscompares in
some testcases which would otherwise trip the --check-access system.
The --check-access mechanism is hard-coded to ignore RCLONE_TEST
files beneath bisync/testdata, so the test cases may reside on the
synched tree even if there are check file mismatches in the test
tree.
- Some Dropbox tests can fail, notably printing the following message:
src and dst identical but can't set mod time without deleting and re-uploading
This is expected and happens due to the way Dropbox handles
@ -23340,8 +23445,8 @@ characters (which factors into load time and RAM required).
References
rclone's bisync implementation was derived from the rclonesync-V2
project, including documentation and test mechanisms, with
[@cjnaz](https://github.com/cjnaz)'s full support and encouragement.
project, including documentation and test mechanisms, with @cjnaz's full
support and encouragement.
rclone bisync is similar in nature to a range of other projects:
@ -33978,6 +34083,472 @@ Properties:
- Type: string
- Required: false
FrostFS
Rclone FrostFS support is provided using the
git.frostfs.info/TrueCloudLab/frostfs-sdk-go package
Configuration
To create an FrostFS configuration named remote, run
rclone config
Rclone config guides you through an interactive setup process. A minimal
rclone FrostFS remote definition only requires endpoint and path to
FrostFS user wallet.
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
Enter name for new remote.
name> remote
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3)
5 / Backblaze B2
\ (b2)
6 / Better checksums for other remotes
\ (hasher)
7 / Box
\ (box)
8 / Cache a remote
\ (cache)
9 / Citrix Sharefile
\ (sharefile)
10 / Combine several remotes into one
\ (combine)
11 / Compress a remote
\ (compress)
12 / Distributed, decentralized object storage FrostFS
\ (frostfs)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Files.com
\ (filescom)
18 / Gofile
\ (gofile)
19 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
20 / Google Drive
\ (drive)
21 / Google Photos
\ (google photos)
22 / HTTP
\ (http)
23 / Hadoop distributed file system
\ (hdfs)
24 / HiDrive
\ (hidrive)
25 / ImageKit.io
\ (imagekit)
26 / In memory object storage system.
\ (memory)
27 / Internet Archive
\ (internetarchive)
28 / Jottacloud
\ (jottacloud)
29 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
30 / Linkbox
\ (linkbox)
31 / Local Disk
\ (local)
32 / Mail.ru Cloud
\ (mailru)
33 / Mega
\ (mega)
34 / Microsoft Azure Blob Storage
\ (azureblob)
35 / Microsoft Azure Files
\ (azurefiles)
36 / Microsoft OneDrive
\ (onedrive)
37 / OpenDrive
\ (opendrive)
38 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
\ (swift)
39 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
40 / Pcloud
\ (pcloud)
41 / PikPak
\ (pikpak)
42 / Pixeldrain Filesystem
\ (pixeldrain)
43 / Proton Drive
\ (protondrive)
44 / Put.io
\ (putio)
45 / QingCloud Object Storage
\ (qingstor)
46 / Quatrix by Maytech
\ (quatrix)
47 / SMB / CIFS
\ (smb)
48 / SSH/SFTP
\ (sftp)
49 / Sia Decentralized Cloud
\ (sia)
50 / Storj Decentralized Cloud Storage
\ (storj)
51 / Sugarsync
\ (sugarsync)
52 / Transparently chunk/split large files
\ (chunker)
53 / Uloz.to
\ (ulozto)
54 / Union merges the contents of several upstream fs
\ (union)
55 / Uptobox
\ (uptobox)
56 / WebDAV
\ (webdav)
57 / Yandex Disk
\ (yandex)
58 / Zoho
\ (zoho)
59 / premiumize.me
\ (premiumizeme)
60 / seafile
\ (seafile)
Storage> frostfs
Option endpoint.
Endpoints to connect to FrostFS node
Choose a number from below, or type in your own value.
1 / One endpoint.
\ (s01.frostfs.devenv:8080)
2 / Multiple endpoints to form pool.
\ (s01.frostfs.devenv:8080 s02.frostfs.devenv:8080)
3 / Multiple endpoints with priority (less value is higher priority). Until s01 is healthy all request will be send to it.
\ (s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2)
4 / Multiple endpoints with priority and weights. After s01 is unhealthy requests will be send to s02 and s03 in proportions 10% and 90% respectively.
\ (s01.frostfs.devenv:8080,1,1 s02.frostfs.devenv:8080,2,1 s03.frostfs.devenv:8080,2,9)
endpoint> s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2
Option connection_timeout.
FrostFS connection timeout
Enter a value of type Duration. Press Enter for the default (4s).
connection_timeout>
Option request_timeout.
FrostFS request timeout
Enter a value of type Duration. Press Enter for the default (4s).
request_timeout>
Option rebalance_interval.
FrostFS rebalance connections interval
Enter a value of type Duration. Press Enter for the default (15s).
rebalance_interval>
Option session_expiration.
FrostFS session expiration epoch
Enter a signed integer. Press Enter for the default (4294967295).
session_expiration>
Option ape_cache_invalidation_duration.
APE cache invalidation duration
Enter a value of type Duration. Press Enter for the default (8s).
ape_cache_invalidation_duration>
Option ape_cache_invalidation_timeout.
APE cache invalidation timeout
Enter a value of type Duration. Press Enter for the default (16s).
ape_cache_invalidation_timeout>
Option ape_chain_check_interval.
The interval for verifying that the APE chain is saved in FrostFS.
Enter a value of type Duration. Press Enter for the default (500ms).
ape_chain_check_interval>
Option rpc_endpoint.
Endpoint to connect to Neo rpc node
Enter a value. Press Enter to leave empty.
rpc_endpoint>
Option wallet.
Path to wallet
Enter a value.
wallet> /wallets/wallet.conf
Option address.
Address of account
Enter a value. Press Enter to leave empty.
address>
Option password.
Password to decrypt wallet
Enter a value. Press Enter to leave empty.
password>
Option placement_policy.
Placement policy for new containers
Choose a number from below, or type in your own value of type string.
Press Enter for the default (REP 3).
1 / Container will have 3 replicas
\ (REP 3)
placement_policy> REP 1
Option container_creation_policy.
Container creation policy for new containers
Choose a number from below, or type in your own value of type string.
Press Enter for the default (private).
1 / Public container, anyone can read and write
\ (public-read-write)
2 / Public container, owner can read and write, others can only read
\ (public-read)
3 / Private container, only owner has access to it
\ (private)
container_creation_policy>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
- type: frostfs
- endpoint: s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2
- password: M@zPGpWDravchenko/develop/go/playground/play_with_ape/cfg/wallet_akrav.js
- placement_policy: REP 1
Keep this "remote" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
As a remote root directory, you can use either the container identifier
or its user-friendly name. If you choose to use a container name, rclone
will create a new container with that name when it's used for the first
time.
For example, both of the following commands would be valid if there was
a ~/test-copy directory and a container with the identifier
23fk3Bcw5mPZ4YtYkTLJbQebtM2WXHz4HL8FgsrTJkSf:
rclone copy ~/test-copy remote:23fk3Bcw5mPZ4YtYkTLJbQebtM2WXHz4HL8FgsrTJkSf/test-copy
rclone copy ~/test-copy remote:container-name/test-copy
Standard options
Here are the Standard options specific to frostfs (Distributed,
decentralized object storage FrostFS).
--frostfs-endpoint
Endpoints to connect to FrostFS node
Properties:
- Config: endpoint
- Env Var: RCLONE_FROSTFS_ENDPOINT
- Type: string
- Required: true
- Examples:
- "s01.frostfs.devenv:8080"
- One endpoint.
- "s01.frostfs.devenv:8080 s02.frostfs.devenv:8080"
- Multiple endpoints to form pool.
- "s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2"
- Multiple endpoints with priority (lower value has higher
priority). Until s01 is healthy, all requests will be sent
to it.
- "s01.frostfs.devenv:8080,1,1 s02.frostfs.devenv:8080,2,1
s03.frostfs.devenv:8080,2,9"
- Multiple endpoints with priority and weights. After the s01
endpoint is unhealthy, requests will be sent to the s02 and
s03 endpoints in proportions of 10% and 90%, respectively.
--frostfs-connection-timeout
FrostFS connection timeout
Properties:
- Config: connection_timeout
- Env Var: RCLONE_FROSTFS_CONNECTION_TIMEOUT
- Type: Duration
- Default: 4s
--frostfs-request-timeout
FrostFS request timeout
Properties:
- Config: request_timeout
- Env Var: RCLONE_FROSTFS_REQUEST_TIMEOUT
- Type: Duration
- Default: 12s
--frostfs-rebalance-interval
FrostFS rebalance connections interval
Properties:
- Config: rebalance_interval
- Env Var: RCLONE_FROSTFS_REBALANCE_INTERVAL
- Type: Duration
- Default: 15s
--frostfs-session-expiration
FrostFS session expiration epoch
Properties:
- Config: session_expiration
- Env Var: RCLONE_FROSTFS_SESSION_EXPIRATION
- Type: int
- Default: 4294967295
--frostfs-ape-cache-invalidation-duration
APE cache invalidation duration
Properties:
- Config: ape_cache_invalidation_duration
- Env Var: RCLONE_FROSTFS_APE_CACHE_INVALIDATION_DURATION
- Type: Duration
- Default: 8s
--frostfs-ape-cache-invalidation-timeout
APE cache invalidation timeout
Properties:
- Config: ape_cache_invalidation_timeout
- Env Var: RCLONE_FROSTFS_APE_CACHE_INVALIDATION_TIMEOUT
- Type: Duration
- Default: 24s
--frostfs-ape-chain-check-interval
The interval for verifying that the APE chain is saved in FrostFS.
Properties:
- Config: ape_chain_check_interval
- Env Var: RCLONE_FROSTFS_APE_CHAIN_CHECK_INTERVAL
- Type: Duration
- Default: 500ms
--frostfs-rpc-endpoint
Endpoint to connect to Neo rpc node
Properties:
- Config: rpc_endpoint
- Env Var: RCLONE_FROSTFS_RPC_ENDPOINT
- Type: string
- Required: false
--frostfs-wallet
Path to wallet
Properties:
- Config: wallet
- Env Var: RCLONE_FROSTFS_WALLET
- Type: string
- Required: true
--frostfs-address
Address of account
Properties:
- Config: address
- Env Var: RCLONE_FROSTFS_ADDRESS
- Type: string
- Required: false
--frostfs-password
Password to decrypt wallet
Properties:
- Config: password
- Env Var: RCLONE_FROSTFS_PASSWORD
- Type: string
- Required: false
--frostfs-placement-policy
Placement policy for new containers
Properties:
- Config: placement_policy
- Env Var: RCLONE_FROSTFS_PLACEMENT_POLICY
- Type: string
- Default: "REP 3"
- Examples:
- "REP 3"
- Container will have 3 replicas
--frostfs-container-creation-policy
Container creation policy for new containers
Properties:
- Config: container_creation_policy
- Env Var: RCLONE_FROSTFS_CONTAINER_CREATION_POLICY
- Type: string
- Default: "private"
- Examples:
- "public-read-write"
- Public container, anyone can read and write
- "public-read"
- Public container, owner can read and write, others can only
read
- "private"
- Private container, only owner has access to it
Advanced options
Here are the Advanced options specific to frostfs (Distributed,
decentralized object storage FrostFS).
--frostfs-description
Description of the remote.
Properties:
- Config: description
- Env Var: RCLONE_FROSTFS_DESCRIPTION
- Type: string
- Required: false
FTP
FTP is the File Transfer Protocol. Rclone FTP support is provided using
@ -34097,13 +34668,12 @@ even in Windows Command Prompt:
rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM
rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM:
Implicit TLS
Implicit TLSutil.go
Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to be
enabled in the FTP backend config for the remote, or with --ftp-tls. The
default FTPS port is 990, not 21 and can be set with --ftp-port.
Restricted filename characters
default FTPS port is 990, not 21 and can be set with --ftp-port. util.go
### Restricted filename characters
In addition to the default restricted characters set the following
characters are also replaced:
@ -35946,15 +36516,23 @@ The conversion must result in a file with the same extension when the
Here are some examples for allowed and prohibited conversions.
-------------------------------------------------------------------------
export-formats import-formats Upload Ext Document Ext Allowed
---------------- ---------------- ------------ -------------- ---------
----------------- ----------------- ------------ -------------- ---------
odt odt odt odt Yes
odt docx,odt odt odt Yes
docx docx docx Yes
odt odt docx No
odt,docx docx,odt docx odt No
docx,odt docx,odt docx docx Yes
docx,odt docx,odt odt docx No
-------------------------------------------------------------------------
This limitation can be disabled by specifying
--drive-allow-import-name-change. When using this flag, rclone can
@ -42575,8 +43153,9 @@ Properties:
Upload chunk size.
Note that this is stored in memory and there may be up to "--transfers"
* "--azureblob-upload-concurrency" chunks stored at once in memory.
Note that this is stored in memory and there may be up to
"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
in memory.
Properties:
@ -43482,8 +44061,9 @@ Properties:
Upload chunk size.
Note that this is stored in memory and there may be up to "--transfers"
* "--azurefile-upload-concurrency" chunks stored at once in memory.
Note that this is stored in memory and there may be up to
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.
Properties:
@ -47846,8 +48426,9 @@ Chunk size for multipart uploads.
Large files will be uploaded in chunks of this size.
Note that this is stored in memory and there may be up to "--transfers"
* "--pikpak-upload-concurrency" chunks stored at once in memory.
Note that this is stored in memory and there may be up to
"--transfers" * "--pikpak-upload-concurrency" chunks stored at once in
memory.
If you are transferring large files over high-speed links and you have
enough memory, then increasing this will speed up the transfers.
@ -62767,7 +63348,7 @@ Twitter
You can also follow Nick on twitter for rclone announcements:
- [@njcw](https://twitter.com/njcw)
- @njcw
Email

1
bin/make_manual.py
View file

@ -42,6 +42,7 @@ docs = [
"dropbox.md",
"filefabric.md",
"filescom.md",
"frostfs.md",
"ftp.md",
"gofile.md",
"googlecloudstorage.md",

1
docs/content/_index.md
View file

@ -123,6 +123,7 @@ WebDAV or S3, that work out of the box.)
{{< provider name="Enterprise File Fabric" home="https://storagemadeeasy.com/about/" config="/filefabric/" >}}
{{< provider name="Fastmail Files" home="https://www.fastmail.com/" config="/webdav/#fastmail-files" >}}
{{< provider name="Files.com" home="https://www.files.com/" config="/filescom/" >}}
{{< provider name="FrostFS" home="https://git.frostfs.info/TrueCloudLab/" config="/frostfs/" >}}
{{< provider name="FTP" home="https://en.wikipedia.org/wiki/File_Transfer_Protocol" config="/ftp/" >}}
{{< provider name="Gofile" home="https://gofile.io/" config="/gofile/" >}}
{{< provider name="Google Cloud Storage" home="https://cloud.google.com/storage/" config="/googlecloudstorage/" >}}

17
docs/content/commands/rclone.md
View file

@ -296,6 +296,21 @@ rclone [flags]
-f, --filter stringArray Add a file filtering rule
--filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
--fix-case Force rename of case insensitive dest to match source
--frostfs-address string Address of account
--frostfs-ape-cache-invalidation-duration Duration APE cache invalidation duration (default 8s)
--frostfs-ape-cache-invalidation-timeout Duration APE cache invalidation timeout (default 24s)
--frostfs-ape-chain-check-interval Duration The interval for verifying that the APE chain is saved in FrostFS (default 500ms)
--frostfs-connection-timeout Duration FrostFS connection timeout (default 4s)
--frostfs-container-creation-policy string Container creation policy for new containers (default "private")
--frostfs-description string Description of the remote
--frostfs-endpoint string Endpoints to connect to FrostFS node
--frostfs-password string Password to decrypt wallet
--frostfs-placement-policy string Placement policy for new containers (default "REP 3")
--frostfs-rebalance-interval Duration FrostFS rebalance connections interval (default 15s)
--frostfs-request-timeout Duration FrostFS request timeout (default 12s)
--frostfs-rpc-endpoint string Endpoint to connect to Neo rpc node
--frostfs-session-expiration int FrostFS session expiration epoch (default 4294967295)
--frostfs-wallet string Path to wallet
--fs-cache-expire-duration Duration Cache remotes for this long (0 to disable caching) (default 5m0s)
--fs-cache-expire-interval Duration Interval to check for expired remotes (default 1m0s)
--ftp-ask-password Allow asking for FTP password when needed
@ -929,7 +944,7 @@ rclone [flags]
--use-json-log Use json log format
--use-mmap Use mmap allocator (see docs)
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2-beta.8331.25cf42493.feature/add-frostfs-support")
-v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)

17
docs/content/flags.md
View file

@ -115,7 +115,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2-beta.8331.25cf42493.feature/add-frostfs-support")
```
@ -552,6 +552,21 @@ Backend-only flags (these can be set in the config file also).
--filescom-password string The password used to authenticate with Files.com (obscured)
--filescom-site string Your site subdomain (e.g. mysite) or custom domain (e.g. myfiles.customdomain.com)
--filescom-username string The username used to authenticate with Files.com
--frostfs-address string Address of account
--frostfs-ape-cache-invalidation-duration Duration APE cache invalidation duration (default 8s)
--frostfs-ape-cache-invalidation-timeout Duration APE cache invalidation timeout (default 24s)
--frostfs-ape-chain-check-interval Duration The interval for verifying that the APE chain is saved in FrostFS (default 500ms)
--frostfs-connection-timeout Duration FrostFS connection timeout (default 4s)
--frostfs-container-creation-policy string Container creation policy for new containers (default "private")
--frostfs-description string Description of the remote
--frostfs-endpoint string Endpoints to connect to FrostFS node
--frostfs-password string Password to decrypt wallet
--frostfs-placement-policy string Placement policy for new containers (default "REP 3")
--frostfs-rebalance-interval Duration FrostFS rebalance connections interval (default 15s)
--frostfs-request-timeout Duration FrostFS request timeout (default 12s)
--frostfs-rpc-endpoint string Endpoint to connect to Neo rpc node
--frostfs-session-expiration int FrostFS session expiration epoch (default 4294967295)
--frostfs-wallet string Path to wallet
--ftp-ask-password Allow asking for FTP password when needed
--ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s)
--ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited

466
docs/content/frostfs.md Normal file
View file

@ -0,0 +1,466 @@
---
title: "FrostFS"
description: "Rclone docs for FrostFS backend"
versionIntroduced: "---"
---
# {{< icon "fa fa-file" >}} FrostFS
Rclone FrostFS support is provided using the
[git.frostfs.info/TrueCloudLab/frostfs-sdk-go](https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go) package
## Configuration
To create an FrostFS configuration named `remote`, run
rclone config
Rclone config guides you through an interactive setup process. A minimal
rclone FrostFS remote definition only requires endpoint and path to FrostFS user wallet.
```
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
Enter name for new remote.
name> remote
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3)
5 / Backblaze B2
\ (b2)
6 / Better checksums for other remotes
\ (hasher)
7 / Box
\ (box)
8 / Cache a remote
\ (cache)
9 / Citrix Sharefile
\ (sharefile)
10 / Combine several remotes into one
\ (combine)
11 / Compress a remote
\ (compress)
12 / Distributed, decentralized object storage FrostFS
\ (frostfs)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Files.com
\ (filescom)
18 / Gofile
\ (gofile)
19 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
20 / Google Drive
\ (drive)
21 / Google Photos
\ (google photos)
22 / HTTP
\ (http)
23 / Hadoop distributed file system
\ (hdfs)
24 / HiDrive
\ (hidrive)
25 / ImageKit.io
\ (imagekit)
26 / In memory object storage system.
\ (memory)
27 / Internet Archive
\ (internetarchive)
28 / Jottacloud
\ (jottacloud)
29 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
30 / Linkbox
\ (linkbox)
31 / Local Disk
\ (local)
32 / Mail.ru Cloud
\ (mailru)
33 / Mega
\ (mega)
34 / Microsoft Azure Blob Storage
\ (azureblob)
35 / Microsoft Azure Files
\ (azurefiles)
36 / Microsoft OneDrive
\ (onedrive)
37 / OpenDrive
\ (opendrive)
38 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
\ (swift)
39 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
40 / Pcloud
\ (pcloud)
41 / PikPak
\ (pikpak)
42 / Pixeldrain Filesystem
\ (pixeldrain)
43 / Proton Drive
\ (protondrive)
44 / Put.io
\ (putio)
45 / QingCloud Object Storage
\ (qingstor)
46 / Quatrix by Maytech
\ (quatrix)
47 / SMB / CIFS
\ (smb)
48 / SSH/SFTP
\ (sftp)
49 / Sia Decentralized Cloud
\ (sia)
50 / Storj Decentralized Cloud Storage
\ (storj)
51 / Sugarsync
\ (sugarsync)
52 / Transparently chunk/split large files
\ (chunker)
53 / Uloz.to
\ (ulozto)
54 / Union merges the contents of several upstream fs
\ (union)
55 / Uptobox
\ (uptobox)
56 / WebDAV
\ (webdav)
57 / Yandex Disk
\ (yandex)
58 / Zoho
\ (zoho)
59 / premiumize.me
\ (premiumizeme)
60 / seafile
\ (seafile)
Storage> frostfs
Option endpoint.
Endpoints to connect to FrostFS node
Choose a number from below, or type in your own value.
1 / One endpoint.
\ (s01.frostfs.devenv:8080)
2 / Multiple endpoints to form pool.
\ (s01.frostfs.devenv:8080 s02.frostfs.devenv:8080)
3 / Multiple endpoints with priority (less value is higher priority). Until s01 is healthy all request will be send to it.
\ (s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2)
4 / Multiple endpoints with priority and weights. After s01 is unhealthy requests will be send to s02 and s03 in proportions 10% and 90% respectively.
\ (s01.frostfs.devenv:8080,1,1 s02.frostfs.devenv:8080,2,1 s03.frostfs.devenv:8080,2,9)
endpoint> s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2
Option connection_timeout.
FrostFS connection timeout
Enter a value of type Duration. Press Enter for the default (4s).
connection_timeout>
Option request_timeout.
FrostFS request timeout
Enter a value of type Duration. Press Enter for the default (12s).
request_timeout>
Option rebalance_interval.
FrostFS rebalance connections interval
Enter a value of type Duration. Press Enter for the default (15s).
rebalance_interval>
Option session_expiration.
FrostFS session expiration epoch
Enter a signed integer. Press Enter for the default (4294967295).
session_expiration>
Option ape_cache_invalidation_duration.
APE cache invalidation duration
Enter a value of type Duration. Press Enter for the default (8s).
ape_cache_invalidation_duration>
Option ape_cache_invalidation_timeout.
APE cache invalidation timeout
Enter a value of type Duration. Press Enter for the default (24s).
ape_cache_invalidation_timeout>
Option ape_chain_check_interval.
The interval for verifying that the APE chain is saved in FrostFS.
Enter a value of type Duration. Press Enter for the default (500ms).
ape_chain_check_interval>
Option rpc_endpoint.
Endpoint to connect to Neo rpc node
Enter a value. Press Enter to leave empty.
rpc_endpoint>
Option wallet.
Path to wallet
Enter a value.
wallet> /wallets/wallet.conf
Option address.
Address of account
Enter a value. Press Enter to leave empty.
address>
Option password.
Password to decrypt wallet
Enter a value. Press Enter to leave empty.
password>
Option placement_policy.
Placement policy for new containers
Choose a number from below, or type in your own value of type string.
Press Enter for the default (REP 3).
1 / Container will have 3 replicas
\ (REP 3)
placement_policy> REP 1
Option container_creation_policy.
Container creation policy for new containers
Choose a number from below, or type in your own value of type string.
Press Enter for the default (private).
1 / Public container, anyone can read and write
\ (public-read-write)
2 / Public container, owner can read and write, others can only read
\ (public-read)
3 / Private container, only owner has access to it
\ (private)
container_creation_policy>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
- type: frostfs
- endpoint: s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2
- password: M@zPGpWDravchenko/develop/go/playground/play_with_ape/cfg/wallet_akrav.js
- placement_policy: REP 1
Keep this "remote" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
```
As a remote root directory, you can use either the container identifier or its user-friendly name.
If you choose to use a container name, rclone will create a new container with that name when
it's used for the first time.
For example, both of the following commands would be valid if there was a `~/test-copy` directory and a container with
the identifier `23fk3Bcw5mPZ4YtYkTLJbQebtM2WXHz4HL8FgsrTJkSf`:
rclone copy ~/test-copy remote:23fk3Bcw5mPZ4YtYkTLJbQebtM2WXHz4HL8FgsrTJkSf/test-copy
rclone copy ~/test-copy remote:container-name/test-copy
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/frostfs/frostfs.go then run make backenddocs" >}}
### Standard options
Here are the Standard options specific to frostfs (Distributed, decentralized object storage FrostFS).
#### --frostfs-endpoint
Endpoints to connect to FrostFS node
Properties:
- Config: endpoint
- Env Var: RCLONE_FROSTFS_ENDPOINT
- Type: string
- Required: true
- Examples:
- "s01.frostfs.devenv:8080"
- One endpoint.
- "s01.frostfs.devenv:8080 s02.frostfs.devenv:8080"
- Multiple endpoints to form pool.
- "s01.frostfs.devenv:8080,1 s02.frostfs.devenv:8080,2"
- Multiple endpoints with priority (lower value has higher priority). Until s01 is healthy, all requests will be sent to it.
- "s01.frostfs.devenv:8080,1,1 s02.frostfs.devenv:8080,2,1 s03.frostfs.devenv:8080,2,9"
- Multiple endpoints with priority and weights. After the s01 endpoint is unhealthy, requests will be sent to the s02 and s03 endpoints in proportions of 10% and 90%, respectively.
#### --frostfs-connection-timeout
FrostFS connection timeout
Properties:
- Config: connection_timeout
- Env Var: RCLONE_FROSTFS_CONNECTION_TIMEOUT
- Type: Duration
- Default: 4s
#### --frostfs-request-timeout
FrostFS request timeout
Properties:
- Config: request_timeout
- Env Var: RCLONE_FROSTFS_REQUEST_TIMEOUT
- Type: Duration
- Default: 12s
#### --frostfs-rebalance-interval
FrostFS rebalance connections interval
Properties:
- Config: rebalance_interval
- Env Var: RCLONE_FROSTFS_REBALANCE_INTERVAL
- Type: Duration
- Default: 15s
#### --frostfs-session-expiration
FrostFS session expiration epoch
Properties:
- Config: session_expiration
- Env Var: RCLONE_FROSTFS_SESSION_EXPIRATION
- Type: int
- Default: 4294967295
#### --frostfs-ape-cache-invalidation-duration
APE cache invalidation duration
Properties:
- Config: ape_cache_invalidation_duration
- Env Var: RCLONE_FROSTFS_APE_CACHE_INVALIDATION_DURATION
- Type: Duration
- Default: 8s
#### --frostfs-ape-cache-invalidation-timeout
APE cache invalidation timeout
Properties:
- Config: ape_cache_invalidation_timeout
- Env Var: RCLONE_FROSTFS_APE_CACHE_INVALIDATION_TIMEOUT
- Type: Duration
- Default: 24s
#### --frostfs-ape-chain-check-interval
The interval for verifying that the APE chain is saved in FrostFS.
Properties:
- Config: ape_chain_check_interval
- Env Var: RCLONE_FROSTFS_APE_CHAIN_CHECK_INTERVAL
- Type: Duration
- Default: 500ms
#### --frostfs-rpc-endpoint
Endpoint to connect to Neo rpc node
Properties:
- Config: rpc_endpoint
- Env Var: RCLONE_FROSTFS_RPC_ENDPOINT
- Type: string
- Required: false
#### --frostfs-wallet
Path to wallet
Properties:
- Config: wallet
- Env Var: RCLONE_FROSTFS_WALLET
- Type: string
- Required: true
#### --frostfs-address
Address of account
Properties:
- Config: address
- Env Var: RCLONE_FROSTFS_ADDRESS
- Type: string
- Required: false
#### --frostfs-password
Password to decrypt wallet
Properties:
- Config: password
- Env Var: RCLONE_FROSTFS_PASSWORD
- Type: string
- Required: false
#### --frostfs-placement-policy
Placement policy for new containers
Properties:
- Config: placement_policy
- Env Var: RCLONE_FROSTFS_PLACEMENT_POLICY
- Type: string
- Default: "REP 3"
- Examples:
- "REP 3"
- Container will have 3 replicas
#### --frostfs-container-creation-policy
Container creation policy for new containers
Properties:
- Config: container_creation_policy
- Env Var: RCLONE_FROSTFS_CONTAINER_CREATION_POLICY
- Type: string
- Default: "private"
- Examples:
- "public-read-write"
- Public container, anyone can read and write
- "public-read"
- Public container, owner can read and write, others can only read
- "private"
- Private container, only owner has access to it
### Advanced options
Here are the Advanced options specific to frostfs (Distributed, decentralized object storage FrostFS).
#### --frostfs-description
Description of the remote.
Properties:
- Config: description
- Env Var: RCLONE_FROSTFS_DESCRIPTION
- Type: string
- Required: false
{{< rem autogenerated options stop >}}

14092
rclone.1 generated
View file

File diff suppressed because it is too large Load diff