This replaces the `sync.Pool` allocator with lib/pool. This
implements a pool of buffers of up to 64MB which can be re-used but is
flushed every 5 seconds.
If `--use-mmap` is set then rclone will use mmap for memory
allocations which is much better at returning memory to the OS.
The --no-traverse flag was not implemented when the new sync routines
(using the march package) was implemented.
This re-implements --no-traverse in march by trying to find a match
for each object with NewObject rather than from a directory listing.
This change allows remotes to be created on the fly without a config
file by using the remote type prefixed with a : as the remote name, Eg
:s3: to make an s3 remote.
This assumes the user is supplying the backend config via command line
flags or environment variables.
--max-backlog controls the queue length.
Add statistics for the check/upload/rename queues.
This means that checking can complete before the uploads which will
give rclone the ability to show exactly what is outstanding.
* Add Mkdir, Rmdir, Purge, Delete, SetModTime, Copy, Move, DirMove
* Update file size after upload
* Add Open seek
* Set private permission for new folder and uploaded file
* Add docs
* Update List function
* Fix UserSessionInfo struct
* Fix socket leaks
* Don’t close resp.Body in Open method
* Get hash when listing files
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.
Still to do
* docs - need finishing
* tests
Add a period at the end of each sentence for consistency.
Change the remaining verbs to their imperative form (again, for consistency).
The default `rclone lsl` output is size, modification time and path, so reword the command description to reflect that.
Correct various typos.
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain
* headers - HTTP headers as before
* bodies - HTTP bodies as before
* requests - HTTP request bodies
* responses - HTTP response bodies
* auth - HTTP auth
* filters - Filter rexeps
Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
* Fixup bitrot (rclone and Azure library)
* Implement Copy
* Add modtime to metadata under mtime key as RFC3339Nano
* Make multipart upload work
* Make it pass the integration tests
* Fix uploading of zero length blobs
* Rename to azureblob as it seems likely we will do azurefile
* Add docs
Add new package qingstor to support QingStor API.
Add new unit test for its and tested through; But I commented
on some tests case because of some of the features of QingStor.
Add new docs for it.
* Fix remaining problems
* Refactor to make testing easier and add a test suite
* Make path parsing more robust.
* Add single file operations
* Add MimeType reading for objects
* Add documentation
* Note go1.7+ is required to build
* -vv or --log-level DEBUG
* -v or --log-level INFO
* --log-level NOTICE (default)
* -q --log-level ERROR
Replace Config.Verbose and Config.Quiet with Config.LogLevel
Fixes#739Fixes#1108Fixes#1000
- Change the --bwlimit command line parameter to accept both a limit (as
before) or a full timetable (formatted as "hh:mm,limit
hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
limits.
This resolves#221.
* put the most up to date docs into the code
* generate command docs using rclone gendocs
* put command docs into own directory
* remake them into MANUAL.md
* Make move command check for overlapping remotes and refuse to run
* Do copy/delete rather than all the copies then all the deletes
* Doesn't purge the source - this was unexpected behaviour see #512 and #416
* Add -list-retries flag to test suite to control retries
This changes the semantics of `move` slightly. However it now errs on
the side of not deleting stuff.
Refactor sync/copy/move
* Don't load the src listing unless doing a sync and --delete-before
* Don't load the dst listing if doing copy/move and --no-traverse is set
`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.
This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.
Fixes#8Fixes#544Fixes#546
* Now removes identical copies without asking
* Now obeys `--dry-run`
* Implement `--dedupe-mode` for non interactive running
* `--dedupe-mode interactive` - interactive the default.
* `--dedupe-mode skip` - removes identical files then skips anything left.
* `--dedupe-mode first` - removes identical files then keeps the first one.
* `--dedupe-mode newest` - removes identical files then keeps the newest one.
* `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
* `--dedupe-mode rename` - removes identical files then renames the rest to be different.
* Add tests which will only run on Google Drive.
See #317 for details.
Use `rclone config` to add/change/remove password.
Tests that loads the default configuration will now fail with a better error message, and add a switch that makes it possible to disable password prompts and fail instead.
Make it possible to use the "RCLONE_CONFIG_PASS" environment variable as password for configuration.
The option name mirrors the 'wget' option (also `--no-check-certificate`). The cURL equivalent is called `--insecure`, which is a bit unclear.
Put in the "developers" section in documentation with proper warnings.
Fixes#168
* Implement include/exclude
* Implement rsync compatible file globbing
* Implement command line filtering flags
* --delete-excluded - Delete files on dest excluded from sync
* --filter - Add a file-filtering rule
* --filter-from - Read filtering patterns from a file
* --exclude - Exclude files matching pattern
* --exclude-from - Read exclude patterns from file
* --include - Include files matching pattern
* --include-from - Read include patterns from file
* --files-from - Read list of source-file nam
* --min-size - Don't transfer any file smaller than this in k or suffix k|M|G
* --max-size - Don't transfer any file larger than this in k or suffix k|M|G
* Document
