TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
3843 commits 181 branches 114 tags 335 MiB
Go 98.6%
HTML 0.4%
Shell 0.3%
Python 0.3%
JavaScript 0.2%
Other 0.1%
Find a file
Nick Craig-Wood 7e6fac8b1e s3: re-implement multipart upload to fix memory issues 
There have been quite a few reports of problems with the multipart
uploader using too much memory and not retrying possible errors.

Before this change the multipart uploader used the s3manager
abstraction in the AWS SDK.  There are numerous bug reports of this
using up too much memory.

This change re-implements a much simplified version of the s3manager
code specialized for rclone's purposes.

This should use much less memory and retry chunks properly.

See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563
See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405
See: https://forum.rclone.org/t/big-file-upload-to-s3-fails/13575
2020-01-03 22:19:28 +00:00
.github build: pin actions/checkout to v1 to fix build failure 2019-12-04 13:48:03 +00:00
backend s3: re-implement multipart upload to fix memory issues 2020-01-03 22:19:28 +00:00
bin bin/get-github-release: support tar.bz2 files 2019-10-01 16:23:36 +01:00
cmd Fix completion with an encrypted config 2019-11-29 14:48:12 +00:00
contrib/docker Contrib: Add sample WebDAV server Docker Compose manifest. 2019-09-15 16:06:54 +01:00
docs Add Thomas Kriechbaumer to contributors 2020-01-03 22:18:23 +00:00
fs fs/accounting: add option to delete stats 2020-01-03 17:44:05 +00:00
fstest chunker: fix integration tests and hashsum issues 2019-10-13 10:43:46 +01:00
graphics graphics: update rclone logos to new design 2019-08-24 09:31:33 +01:00
lib config: do not open browser on headless if google fs 2019-10-30 14:12:42 +00:00
vendor vendor: update all dependencies 2019-11-14 21:51:34 +00:00
vfs vfs: when renaming files in the cache, rename the cache item in memory too 2019-12-12 13:31:10 +00:00
.gitattributes build: add azure pipelines build 2019-08-06 10:31:32 +01:00
.gitignore chore: update .gitignore 2019-06-19 11:59:46 +01:00
.golangci.yml build: move linter build tags into Makefile to fix golangci-lint 2019-04-12 15:48:36 +01:00
CONTRIBUTING.md docs: Add notes on how to find out the encodings used in a backend 2019-09-30 22:00:25 +01:00
COPYING Initial commit - some small parts working 2012-11-18 17:32:31 +00:00
Dockerfile Dockerfile fixes 2019-09-17 10:12:32 +01:00
go.mod vendor: update all dependencies 2019-11-14 21:51:34 +00:00
go.sum vendor: update all dependencies 2019-11-14 21:51:34 +00:00
MAINTAINERS.md Add Ivan Andreev to maintainers 2019-10-20 00:33:16 +03:00
Makefile build: make replacement of new rclone binary atomic on build 2019-10-28 22:11:38 +00:00
MANUAL.html Version v1.50.0 2019-10-26 11:04:54 +01:00
MANUAL.md Version v1.50.0 2019-10-26 11:04:54 +01:00
MANUAL.txt Version v1.50.0 2019-10-26 11:04:54 +01:00
notes.txt Replace test_all.sh with test_all.go which is cross platform and parallel 2015-12-30 09:26:34 +00:00
rclone.1 Version v1.50.0 2019-10-26 11:04:54 +01:00
rclone.go lib: add plugin support 2019-09-29 11:05:10 +01:00
README.md docs: add GetSky to list of supported providers 2020-01-02 15:37:33 +01:00
RELEASE.md Start v1.50.1-DEV development 2019-11-02 15:26:01 +00:00
VERSION Start v1.50.2-DEV development 2019-11-19 16:51:32 +00:00

README.md

rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Windows Build Status Build Status CircleCI Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional encryption (Crypt)
  • Optional cache (Cache)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of MIT the license (check the COPYING file included in this package).