KurlesHS/rclone
1
0
Fork 0
forked from TrueCloudLab/rclone
Code Pull requests Activity

docs/vfs: Merge duplicate chunked reading documentation from mount docs

Browse source
This commit is contained in:
albertony 2021-08-19 19:29:41 +02:00 • committed by GitHub
parent 60323dc5e2
commit 63708d73be
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 43 additions and 44 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
cmd/mountlib/help.go
View file

@ -283,20 +283,4 @@ to use Type=notify. In this case the service will enter the started state
after the mountpoint has been successfully set up. after the mountpoint has been successfully set up.
Units having the rclone @ service specified as a requirement Units having the rclone @ service specified as a requirement
will see all files and folders immediately in this mode. will see all files and folders immediately in this mode.
### chunked reading
|--vfs-read-chunk-size| will enable reading the source objects in parts.
This can reduce the used download quota for some remotes by requesting only chunks
from the remote that are actually read at the cost of an increased number of requests.
When |--vfs-read-chunk-size-limit| is also specified and greater than
|--vfs-read-chunk-size|, the chunk size for each open file will get doubled
for each chunk read, until the specified value is reached. A value of |-1| will disable
the limit and the chunk size will grow indefinitely.
With |--vfs-read-chunk-size 100M| and |--vfs-read-chunk-size-limit 0|
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
When |--vfs-read-chunk-size-limit 500M| is specified, the result would be
0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
` `

71
vfs/help.go
View file

@ -100,8 +100,8 @@ The higher the cache mode the more compatible rclone becomes at the
cost of using disk space. cost of using disk space.
Note that files are written back to the remote only when they are Note that files are written back to the remote only when they are
closed and if they haven't been accessed for --vfs-write-back closed and if they haven't been accessed for !--vfs-write-back!
second. If rclone is quit or dies with files that haven't been seconds. If rclone is quit or dies with files that haven't been
uploaded, these will be uploaded next time rclone is run with the same uploaded, these will be uploaded next time rclone is run with the same
flags. flags.
@ -170,27 +170,55 @@ their full size in the cache, but they will be sparse files with only
the data that has been downloaded present in them. the data that has been downloaded present in them.
This mode should support all normal file system operations and is This mode should support all normal file system operations and is
otherwise identical to --vfs-cache-mode writes. otherwise identical to !--vfs-cache-mode! writes.
When reading a file rclone will read --buffer-size plus When reading a file rclone will read !--buffer-size! plus
--vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory !--vfs-read-ahead! bytes ahead. The !--buffer-size! is buffered in memory
whereas the --vfs-read-ahead is buffered on disk. whereas the !--vfs-read-ahead! is buffered on disk.
When using this mode it is recommended that --buffer-size is not set When using this mode it is recommended that !--buffer-size! is not set
too big and --vfs-read-ahead is set large if required. too large and !--vfs-read-ahead! is set large if required.
**IMPORTANT** not all file systems support sparse files. In particular **IMPORTANT** not all file systems support sparse files. In particular
FAT/exFAT do not. Rclone will perform very badly if the cache FAT/exFAT do not. Rclone will perform very badly if the cache
directory is on a filesystem which doesn't support sparse files and it directory is on a filesystem which doesn't support sparse files and it
will log an ERROR message if one is detected. will log an ERROR message if one is detected.
### VFS Chunked Reading
When rclone reads files from a remote it reads them in chunks. This
means that rather than requesting the whole file rclone reads the
chunk specified. This can reduce the used download quota for some
remotes by requesting only chunks from the remote that are actually
read, at the cost of an increased number of requests.
These flags control the chunking:
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
Rclone will start reading a chunk of size !--vfs-read-chunk-size!,
and then double the size for each read. When !--vfs-read-chunk-size-limit! is
specified, and greater than !--vfs-read-chunk-size!, the chunk size for each
open file will get doubled only until the specified value is reached. If the
value is "off", which is the default, the limit is disabled and the chunk size
will grow indefinitely.
With !--vfs-read-chunk-size 100M! and !--vfs-read-chunk-size-limit 0!
the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
When !--vfs-read-chunk-size-limit 500M! is specified, the result would be
0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting !--vfs-read-chunk-size! to !0! or "off" disables chunked reading.
### VFS Performance ### VFS Performance
These flags may be used to enable/disable features of the VFS for These flags may be used to enable/disable features of the VFS for
performance or other reasons. performance or other reasons. See also the [chunked reading](#vfs-chunked-reading)
feature.
In particular S3 and Swift benefit hugely from the --no-modtime flag In particular S3 and Swift benefit hugely from the !--no-modtime! flag
(or use --use-server-modtime for a slightly different effect) as each (or use !--use-server-modtime! for a slightly different effect) as each
read of the modification time takes a transaction. read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download. --no-checksum Don't compare checksums on up/download.
@ -198,19 +226,6 @@ read of the modification time takes a transaction.
--no-seek Don't allow seeking in files. --no-seek Don't allow seeking in files.
--read-only Mount read-only. --read-only Mount read-only.
When rclone reads files from a remote it reads them in chunks. This
means that rather than requesting the whole file rclone reads the
chunk specified. This is advantageous because some cloud providers
account for reads being all the data requested, not all the data
delivered.
Rclone will keep doubling the chunk size requested starting at
--vfs-read-chunk-size with a maximum of --vfs-read-chunk-size-limit
unless it is set to "off" in which case there will be no limit.
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default "off")
Sometimes rclone is delivered reads or writes out of order. Rather Sometimes rclone is delivered reads or writes out of order. Rather
than seeking rclone will wait a short time for the in sequence read or than seeking rclone will wait a short time for the in sequence read or
write to come in. These flags only come into effect when not using an write to come in. These flags only come into effect when not using an
@ -219,9 +234,9 @@ on disk cache file.
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms) --vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 20ms)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s) --vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
When using VFS write caching (--vfs-cache-mode with value writes or full), When using VFS write caching (!--vfs-cache-mode! with value writes or full),
the global flag --transfers can be set to adjust the number of parallel uploads of the global flag !--transfers! can be set to adjust the number of parallel uploads of
modified files from cache (the related global flag --checkers have no effect on mount). modified files from cache (the related global flag !--checkers! have no effect on mount).
--transfers int Number of file transfers to run in parallel. (default 4) --transfers int Number of file transfers to run in parallel. (default 4)
@ -236,7 +251,7 @@ to create the file is preserved and available for programs to query.
It is not allowed for two files in the same directory to differ only by case. It is not allowed for two files in the same directory to differ only by case.
Usually file systems on macOS are case-insensitive. It is possible to make macOS Usually file systems on macOS are case-insensitive. It is possible to make macOS
file systems case-sensitive but that is not the default file systems case-sensitive but that is not the default.
The !--vfs-case-insensitive! mount flag controls how rclone handles these The !--vfs-case-insensitive! mount flag controls how rclone handles these
two cases. If its value is "false", rclone passes file names to the mounted two cases. If its value is "false", rclone passes file names to the mounted