"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
38e8415e77
This is what I wrote to Digital Ocean support on July 10, 2020 - alas it didn't result in the rate limits dropping, so reluctantly I'm going to remove DO from the integration tests since they never pass and have no hope of ever passing while this rate limit is in effect. ---- Somewhere towards the end of June 2020 or the start of July 2020 my integration tests between rclone ( https://rclone.org ) and Digital Ocean started failing. I tried moving the tests to different regions (currently they are using AMS1 because I'm in Europe) with no improvement. Rclone seems to be hitting this rate limit as documented here: https://www.digitalocean.com/docs/spaces/#limits - 2 COPYs per 5 minutes on any individual object in a Space Rclone creates small objects about 100 bytes in size and renames them a few times - this involves using the COPY call as S3 does not have a rename API. The tests do this more than twice per object so hit the 5 minute timeout I think. Rclone does exponential backoff and fails after 10 retries not having reached 5 minutes delay after 10 retries. Having a 5 minute lockout on an S3 compatible API is surprising! Rclone integration tests with about 30 other providers, none of which have a rate limit like this. I understand the need for a COPY rate limit as server side copying large files can be resource intensive. However a 5 minute lockout for copying 100 byte files seems excessive! Might I humbly suggest that you reduce or eliminate this rate limit for small files? ---- This was the reply Unfortunately it is not possible to raise this limit or remove it currently on our platform. I do see how this would interfere with type of applications that need to copy many small files and will be happy to take the feedback to our engineering team to see how we can improve the spaces system in the future |
||
|---|---|---|
| .github | ||
| backend | ||
| bin | ||
| cmd | ||
| contrib/docker | ||
| docs | ||
| fs | ||
| fstest | ||
| graphics | ||
| lib | ||
| vfs | ||
| .gitattributes | ||
| .gitignore | ||
| .golangci.yml | ||
| CONTRIBUTING.md | ||
| COPYING | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| MAINTAINERS.md | ||
| Makefile | ||
| MANUAL.html | ||
| MANUAL.md | ||
| MANUAL.txt | ||
| notes.txt | ||
| rclone.1 | ||
| rclone.go | ||
| README.md | ||
| RELEASE.md | ||
| VERSION | ||
Website | Documentation | Download | Contributing | Changelog | Installation | Forum
Rclone
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Storage providers
- 1Fichier 📄
- Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄
- Amazon Drive 📄 (See note)
- Amazon S3 📄
- Backblaze B2 📄
- Box 📄
- Ceph 📄
- Citrix ShareFile 📄
- DigitalOcean Spaces 📄
- Dreamhost 📄
- Dropbox 📄
- FTP 📄
- GetSky 📄
- Google Cloud Storage 📄
- Google Drive 📄
- Google Photos 📄
- HTTP 📄
- Hubic 📄
- Jottacloud 📄
- IBM COS S3 📄
- Koofr 📄
- Mail.ru Cloud 📄
- Memset Memstore 📄
- Mega 📄
- Memory 📄
- Microsoft Azure Blob Storage 📄
- Microsoft OneDrive 📄
- Minio 📄
- Nextcloud 📄
- OVH 📄
- OpenDrive 📄
- OpenStack Swift 📄
- Oracle Cloud Storage 📄
- ownCloud 📄
- pCloud 📄
- premiumize.me 📄
- put.io 📄
- QingStor 📄
- Rackspace Cloud Files 📄
- Scaleway 📄
- Seafile 📄
- SFTP 📄
- StackPath 📄
- SugarSync 📄
- Tardigrade 📄
- Wasabi 📄
- WebDAV 📄
- Yandex Disk 📄
- The local filesystem 📄
Please see the full list of all storage providers and their features
Features
- MD5/SHA-1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Check mode to check for file hash equality
- Can sync to and from network, e.g. two different cloud accounts
- Optional large file chunking (Chunker)
- Optional encryption (Crypt)
- Optional cache (Cache)
- Optional FUSE mount (rclone mount)
- Multi-threaded downloads to local disk
- Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
Installation & documentation
Please see the rclone website for:
Downloads
License
This is free software under the terms of MIT the license (check the COPYING file included in this package).