From 63f6827a0deceb72338a1b6c89a2edbd77e5af10 Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Wed, 13 Jul 2016 12:26:22 +0100 Subject: [PATCH] Version v1.31 --- MANUAL.html | 234 +++++++++++++++-- MANUAL.md | 395 +++++++++++++++++++++++++---- MANUAL.txt | 388 ++++++++++++++++++++++++---- docs/content/changelog.md | 38 ++- docs/content/downloads.md | 42 +-- fs/version.go | 2 +- rclone.1 | 521 ++++++++++++++++++++++++++++++++++---- 7 files changed, 1421 insertions(+), 199 deletions(-) diff --git a/MANUAL.html b/MANUAL.html index ca1d0b4be..b1e1d2c83 100644 --- a/MANUAL.html +++ b/MANUAL.html @@ -12,7 +12,7 @@

Rclone

Logo

@@ -66,7 +66,17 @@ sudo chmod 755 /usr/sbin/rclone #install manpage sudo mkdir -p /usr/local/share/man/man1 sudo cp rclone.1 /usr/local/share/man/man1/ -sudo mandb +sudo mandb +

Installation with Ansible

+

This can be done with Stefan Weichinger's ansible role.

+

Instructions

+
    +
  1. git clone https://github.com/stefangweichinger/ansible-rclone.git into your local roles-directory
    2. +
  2. add the role to the hosts you want rclone installed to:
    3. +
+
    - hosts: rclone-hosts
+      roles:
+          - rclone

Configure

First you'll need to configure rclone. As the object storage systems have quite complicated authentication these are kept in a config file .rclone.conf in your home directory by default. (You can use the --config option to choose a different config file.)

The easiest way to make the config is to run rclone with the config option:

@@ -108,6 +118,7 @@ destpath/two.txt 
destpath/sourcepath/one.txt
 destpath/sourcepath/two.txt

If you are familiar with rsync, rclone always works as if you had written a trailing / - meaning "copy the contents of this directory". This applies to all commands and whether you are talking about the source or destination.

+

See the --no-traverse option for controlling whether rclone lists the destination directory or not.

rclone sync source:path dest:path

Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.

Important: Since this can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted.

@@ -115,9 +126,9 @@ destpath/sourcepath/two.txt

It is always the contents of the directory that is synced, not the directory so when source:path is a directory, it's the contents of source:path that are copied, not the directory name and contents. See extended explanation in the copy command above if unsure.

If dest:path doesn't exist, it is created and the source:path contents go there.

move source:path dest:path

-

Moves the source to the destination.

-

If there are no filters in use this is equivalent to a copy followed by a purge, but may use server side operations to speed it up if possible.

-

If filters are in use then it is equivalent to a copy followed by delete, followed by an rmdir (which only removes the directory if empty). The individual file moves will be moved with server side operations if possible.

+

Moves the contents of the source directory to the destination directory. Rclone will error if the source and destination overlap.

+

If no filters are in use and if possible this will server side move source:path into dest:path. After this source:path will no longer longer exist.

+

Otherwise for each file in source:path selected by the filters (if any) this will move it into dest:path. If possible a server side move will be used, otherwise it will copy it (server side if possible) into dest:path then delete the original (if no errors on copy) in source:path.

Important: Since this can cause data loss, test first with the --dry-run flag.

rclone ls remote:path

List all the objects in the path with size and path.

@@ -149,6 +160,8 @@ rclone --dry-run --min-size 100M delete remote:path

rclone check source:path dest:path

Checks the files in the source and destination match. It compares sizes and MD5SUMs and prints a report of files which don't match. It doesn't alter the source or destination.

--size-only may be used to only compare the sizes, not the MD5SUMs.

+

rclone cleanup remote:path

+

Clean up the remote if possible. Empty the trash or delete old file versions. Not supported by all remotes.

rclone dedupe remote:path

By default dedup interactively finds duplicate files and offers to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.

The dedupe command will delete all but one of any identical (same md5sum) files it finds without confirmation. This means that for most duplicated files the dedupe command will not be interactive. You can use --dry-run to see what would happen without doing anything.

@@ -209,6 +222,16 @@ two-3.txt: renamed from: two.txt

Enter an interactive configuration session.

rclone help

Prints help on rclone commands and options.

+

Copying single files

+

rclone normally syncs or copies directories. However if the source remote points to a file, rclone will just copy that file. The destination remote must point to a directory - rclone will give the error Failed to create file system for "remote:file": is a file not a directory if it isn't.

+

For example, suppose you have a remote with a file in called test.jpg, then you could copy just that file like this

+
rclone copy remote:test.jpg /tmp/download
+

The file test.jpg will be placed inside /tmp/download.

+

This is equivalent to specifying

+
rclone copy --no-traverse --files-from /tmp/files remote: /tmp/download
+

Where /tmp/files contains the single line

+
test.jpg
+

It is recommended to use copy when copying single files not sync. They have pretty much the same effect but copy will use a lot less memory.

Quoting and the shell

When you are typing commands to your computer you are using something called the command line shell. This interprets various characters in an OS specific way.

Here are some gotchas which may help users unfamiliar with the shell rules

@@ -291,6 +314,9 @@ rclone sync /path/to/files remote:current-backup

--no-gzip-encoding

Don't set Accept-Encoding: gzip. This means that rclone won't ask the server for compressed files automatically. Useful if you've set the server to return files with Content-Encoding: gzip but you uploaded compressed files.

There is no need to set this in normal operation, and doing so will decrease the network transfer efficiency of rclone.

+

--no-update-modtime

+

When using this flag, rclone won't update modification times of remote files if they are incorrect as it would normally.

+

This can be used if the remote is being synced with another tool also (eg the Google Drive client).

-q, --quiet

Normally rclone outputs stats and a completion message. If you set this flag it will make as little output as possible.

--retries int

@@ -306,8 +332,8 @@ rclone sync /path/to/files remote:current-backup

The default is 1m. Use 0 to disable.

--delete-(before,during,after)

This option allows you to specify when files on your destination are deleted when you sync folders.

-

Specifying the value --delete-before will delete all files present on the destination, but not on the source before starting the transfer of any new or updated files.

-

Specifying --delete-during (default value) will delete files while checking and uploading files. This is usually the fastest option.

+

Specifying the value --delete-before will delete all files present on the destination, but not on the source before starting the transfer of any new or updated files. This uses extra memory as it has to store the source listing before proceeding.

+

Specifying --delete-during (default value) will delete files while checking and uploading files. This is usually the fastest option. Currently this works the same as --delete-after but it may change in the future.

Specifying --delete-after will delay deletion of files until all new/updated files have been successfully transfered.

--timeout=TIME

This sets the IO idle timeout. If a transfer has started but then becomes idle for this long it is considered broken and disconnected.

@@ -377,6 +403,11 @@ c/u/q>

--no-check-certificate controls whether a client verifies the server's certificate chain and host name. If --no-check-certificate is true, TLS accepts any certificate presented by the server and any host name in that certificate. In this mode, TLS is susceptible to man-in-the-middle attacks.

This option defaults to false.

This should be used only for testing.

+

--no-traverse

+

The --no-traverse flag controls whether the destination file system is traversed when using the copy or move commands.

+

If you are only copying a small number of files and/or have a large number of files on the destination then --no-traverse will stop rclone listing the destination and save time.

+

However if you are copying a large number of files, escpecially if you are doing a copy where lots of the files haven't changed and won't need copying then you shouldn't use --no-traverse.

+

It can also be used to reduce the memory usage of rclone when copying - rclone --no-traverse copy src dst won't load either the source or destination listings into memory so will use the minimum amount of memory.

Filtering

For the filtering options

-

A similar process is done on directory entries before recursing into them. This only works on remotes which have a concept of directory (Eg local, drive, onedrive, amazon cloud drive) and not on bucket based remotes (eg s3, swift, google compute storage, b2).

+

A similar process is done on directory entries before recursing into them. This only works on remotes which have a concept of directory (Eg local, google drive, onedrive, amazon drive) and not on bucket based remotes (eg s3, swift, google compute storage, b2).

Adding filtering rules

Filtering rules are added with the following command line flags.

--exclude - Exclude files matching pattern

@@ -579,6 +610,24 @@ file2.avi file1.jpg file2.jpg

Then use as --files-from files-from.txt. This will only transfer file1.jpg and file2.jpg providing they exist.

+

For example, let's say you had a few files you want to back up regularly with these absolute paths:

+
/home/user1/important
+/home/user1/dir/file
+/home/user2/stuff
+

To copy these you'd find a common subdirectory - in this case /home and put the remaining files in files-from.txt with or without leading /, eg

+
user1/important
+user1/dir/file
+user2/stuff
+

You could then copy these to a remote like this

+
rclone copy --files-from files-from.txt /home remote:backup
+

The 3 files will arrive in remote:backup with the paths as in the files-from.txt.

+

You could of course choose / as the root too in which case your files-from.txt might look like this.

+
/home/user1/important
+/home/user1/dir/file
+/home/user2/stuff
+

And you would transfer it like this

+
rclone copy --files-from files-from.txt / remote:backup
+

In this case there will be an extra home directory on the remote.

--min-size - Don't transfer any file smaller than this

This option controls the minimum size file which will be transferred. This defaults to kBytes but a suffix of k, M, or G can be used.

For example --min-size 50k means no files smaller than 50kByte will be transferred.

@@ -1161,6 +1210,40 @@ region = other-v2-signature ], }

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as xxxxxx/xxxx it will work fine.

+

Minio

+

Minio is an object storage server built for cloud application developers and devops.

+

It is very easy to install and provides an S3 compatible server which can be used by rclone.

+

To use it, install Minio following the instructions from the web site.

+

When it configures itself Minio will print something like this

+
AccessKey: WLGDGYAQYIGI833EV05A  SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF Region: us-east-1
+
+Minio Object Storage:
+     http://127.0.0.1:9000
+     http://10.0.0.3:9000
+
+Minio Browser:
+     http://127.0.0.1:9000
+     http://10.0.0.3:9000
+

These details need to go into rclone config like this. Note that it is important to put the region in as stated above.

+
env_auth> 1
+access_key_id> WLGDGYAQYIGI833EV05A
+secret_access_key> BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF   
+region> us-east-1
+endpoint> http://10.0.0.3:9000
+location_constraint> 
+server_side_encryption>
+

Which makes the config file look like this

+
[minio]
+env_auth = false
+access_key_id = WLGDGYAQYIGI833EV05A
+secret_access_key = BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
+region = us-east-1
+endpoint = http://10.0.0.3:9000
+location_constraint = 
+server_side_encryption =
+

Minio doesn't support all the features of S3 yet. In particular it doesn't support MD5 checksums (ETags) or metadata. This means rclone can't check MD5SUMs or store the modified date. However you can work around this with the --size-only flag of rclone.

+

So once set up, for example to copy files into a bucket

+
rclone --size-only copy /path/to/files minio:bucket

Swift

Swift refers to Openstack Object Storage. Commercial implementations of that being: