"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
c665201b85
This patch provides the support of synchronous cache space recovery to allow read threads to recover from ENOSPC errors when cache space can be recovered from cache items that are not in use or safe to be reset/emptied . The patch complements the existing cache cleaning process in two ways. Firstly, the existing cache cleaning process is time-driven that runs periodically. The cache space can run out while the cache cleaner thread is still waiting for its next scheduled run. The io threads encountering ENOSPC return an internal error to the applications in this case even when cache space can be recovered to avoid this error. This patch addresses this problem by having the read threads kick the cache cleaner thread in this condition to recover cache space preventing unnecessary ENOSPC errors from being seen by the applications. Secondly, this patch enhances the cache cleaner to support cache item reset. Currently the cache purge process removes cache items that are not in use. This may not be sufficient when the total size of the working set exceeds the cache directory's capacity. Like in the current code, this patch starts the purge process by removing cache files that are not in use. Cache items whose access times are older than vfs-cache-max-age are removed first. After that, other not-in-use items are removed in LRU order until vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota) is still not reached at this time, this patch adds a cache reset step to reset/empty cache files that are still in use but not dirtied. This enables application processes to continue without seeing an error even when the working set depletes the cache space as long as there is not a large write working set hoarding the entire cache space. By design this patch does not add ENOSPC error recovery for write IOs. Rclone does not empty a write cache item until the file data is written back to the backend upon close. Allowing more cache space to be consumed by dirty cache items when the cache space is already running low would increase the risk of exhausting the cache space in a way that the vfs mount becomes unreadable. |
||
|---|---|---|
| .github | ||
| backend | ||
| bin | ||
| cmd | ||
| contrib/docker | ||
| docs | ||
| fs | ||
| fstest | ||
| graphics | ||
| lib | ||
| vfs | ||
| .gitattributes | ||
| .gitignore | ||
| .golangci.yml | ||
| CONTRIBUTING.md | ||
| COPYING | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| MAINTAINERS.md | ||
| Makefile | ||
| MANUAL.html | ||
| MANUAL.md | ||
| MANUAL.txt | ||
| notes.txt | ||
| rclone.1 | ||
| rclone.go | ||
| README.md | ||
| RELEASE.md | ||
| VERSION | ||
Website | Documentation | Download | Contributing | Changelog | Installation | Forum
Rclone
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Storage providers
- 1Fichier 📄
- Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄
- Amazon Drive 📄 (See note)
- Amazon S3 📄
- Backblaze B2 📄
- Box 📄
- Ceph 📄
- Citrix ShareFile 📄
- DigitalOcean Spaces 📄
- Dreamhost 📄
- Dropbox 📄
- FTP 📄
- GetSky 📄
- Google Cloud Storage 📄
- Google Drive 📄
- Google Photos 📄
- HTTP 📄
- Hubic 📄
- Jottacloud 📄
- IBM COS S3 📄
- Koofr 📄
- Mail.ru Cloud 📄
- Memset Memstore 📄
- Mega 📄
- Memory 📄
- Microsoft Azure Blob Storage 📄
- Microsoft OneDrive 📄
- Minio 📄
- Nextcloud 📄
- OVH 📄
- OpenDrive 📄
- OpenStack Swift 📄
- Oracle Cloud Storage 📄
- ownCloud 📄
- pCloud 📄
- premiumize.me 📄
- put.io 📄
- QingStor 📄
- Rackspace Cloud Files 📄
- Scaleway 📄
- Seafile 📄
- SFTP 📄
- StackPath 📄
- SugarSync 📄
- Tardigrade 📄
- Wasabi 📄
- WebDAV 📄
- Yandex Disk 📄
- The local filesystem 📄
Please see the full list of all storage providers and their features
Features
- MD5/SHA-1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Check mode to check for file hash equality
- Can sync to and from network, e.g. two different cloud accounts
- Optional large file chunking (Chunker)
- Optional encryption (Crypt)
- Optional cache (Cache)
- Optional FUSE mount (rclone mount)
- Multi-threaded downloads to local disk
- Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
Installation & documentation
Please see the rclone website for:
Downloads
License
This is free software under the terms of MIT the license (check the COPYING file included in this package).