TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
5438 commits 181 branches 114 tags 335 MiB
Go 98.6%
HTML 0.4%
Shell 0.3%
Python 0.3%
JavaScript 0.2%
Other 0.1%
Find a file
Leo Luan 8f23cae1c0 vfs: Add cache reset for --vfs-cache-max-size handling at cache poll interval 
The vfs-cache-max-size parameter is probably confusing to many users.
The cache cleaner checks cache size periodically at the --vfs-cache-poll-interval
(default 60 seconds) interval and remove cache items in the following order.

(1) cache items that are not in use and with age > vfs-cache-max-age
(2) if the cache space used at this time still is larger than
vfs-cache-max-size, the cleaner continues to remove cache items that are
not in use.

The cache cleaning process does not remove cache items that are currently in use.
If the total space consumed by in-use cache items exceeds vfs-cache-max-size, the
periodical cache cleaner thread does not do anything further and leaves the in-use
cache items alone with a total space larger than vfs-cache-max-size.

A cache reset feature was introduced in 1.53 which resets in-use (but not dirty,
i.e., not being updated) cache items when additional cache data incurs an ENOSPC
error.  But this code was not activated in the periodical cache cleaning thread.

This patch adds the cache reset step in the cache cleaner thread during cache
poll to reset cache items until the total size of the remaining cache items is
below vfs-cache-max-size.
2021-04-26 17:55:52 +01:00
.github build: Only run event-based workflow scripts under rclone repo with manual override 2021-04-26 17:52:03 +01:00
backend b2: don't include the bucket name in public link file prefixes 2021-04-26 16:56:41 +01:00
bin make_manual.py: fix missing comma for doc build after uptobox merge 2021-04-19 16:18:18 +01:00
cmd test changenotify: add command to help debugging changenotify 2021-04-21 10:52:05 +01:00
contrib/docker Contrib: Add sample WebDAV server Docker Compose manifest. 2019-09-15 16:06:54 +01:00
docs Add Jeffrey Tolar to contributors 2021-04-26 16:57:21 +01:00
fs config: treat any config file paths with filename notfound as memory-only config (#5235) 2021-04-18 00:09:03 +02:00
fstest uptobox: integration tests 2021-04-13 17:46:07 +02:00
graphics build: Add file properties and icon to Windows executable (fixes #4304) 2020-06-11 09:26:14 +01:00
lib oauthutil: fix #5265 old authorize result not recognised 2021-04-23 01:20:52 +08:00
vfs vfs: Add cache reset for --vfs-cache-max-size handling at cache poll interval 2021-04-26 17:55:52 +01:00
.gitattributes build: add azure pipelines build 2019-08-06 10:31:32 +01:00
.gitignore cmd: rclone selfupdate (#5080) 2021-03-11 22:39:30 +03:00
.golangci.yml build: move linter build tags into Makefile to fix golangci-lint 2019-04-12 15:48:36 +01:00
CONTRIBUTING.md contributing.md: update setup instructions for go1.16 2021-04-04 09:10:43 +01:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile build: add an rclone user to the Docker image but don't use it by default 2021-02-11 17:45:44 +00:00
go.mod sftp: Fix performance regression by re-enabling concurrent writes #5197 2021-04-26 09:24:28 +01:00
go.sum vfs: Add cache reset for --vfs-cache-max-size handling at cache poll interval 2021-04-26 17:55:52 +01:00
MAINTAINERS.md compress: finish docs 2020-12-02 16:30:02 +01:00
Makefile build: add missing BUILD_FLAGS to compile_only to speed up other_os build 2021-03-15 19:22:07 +00:00
MANUAL.html Version v1.55.0 2021-03-31 19:12:08 +01:00
MANUAL.md Version v1.55.0 2021-03-31 19:12:08 +01:00
MANUAL.txt Version v1.55.0 2021-03-31 19:12:08 +01:00
notes.txt docs: spelling: etc. 2020-10-28 18:16:23 +00:00
rclone.1 Version v1.55.0 2021-03-31 19:12:08 +01:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00
README.md HDFS (Hadoop Distributed File System) implementation - #42 2021-01-07 09:48:51 +00:00
RELEASE.md build: update notes on how to build the release manually with docker 2021-03-24 14:22:27 +00:00
VERSION Start v1.56.0-DEV development 2021-03-31 19:51:43 +01:00

README.md

rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional transparent compression (Compress)
  • Optional encryption (Crypt)
  • Optional cache (Cache)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of MIT the license (check the COPYING file included in this package).