diff --git a/docs/content/filtering.md b/docs/content/filtering.md index 2074d1a63..b34411ae9 100644 --- a/docs/content/filtering.md +++ b/docs/content/filtering.md @@ -1,90 +1,87 @@ --- -title: "Filtering" -description: "Filtering, includes and excludes" +title: "Rclone Filtering" +description: "Rclone filtering, includes and excludes" --- -# Filtering, includes and excludes # +# Filtering, includes and excludes -Rclone has a sophisticated set of include and exclude rules. Some of -these are based on patterns and some on other things like file size. +Filter flags determine which files rclone `sync`, `move`, `ls`, `lsl`, +`md5sum`, `sha1sum`, `size`, `delete`, `check` and similar commands +apply to. -The filters are applied for the `copy`, `sync`, `move`, `ls`, `lsl`, -`md5sum`, `sha1sum`, `size`, `delete` and `check` operations. -Note that `purge` does not obey the filters. +They are specified in terms of path/file name patterns; path/file +lists; file age and size, or presence of a file in a directory. Bucket +based remotes without the concept of directory apply filters to object +key, age and size in an analogous way. -Each path as it passes through rclone is matched against the include -and exclude rules like `--include`, `--exclude`, `--include-from`, -`--exclude-from`, `--filter`, or `--filter-from`. The simplest way to -try them out is using the `ls` command, or `--dry-run` together with -`-v`. `--filter-from`, `--exclude-from`, `--include-from`, `--files-from`, -`--files-from-raw` understand `-` as a file name to mean read from standard -input. +Rclone `purge` does not obey filters. -## Patterns ## +To test filters without risk of damage to data, apply them to `rclone +ls`, or with the `--dry-run` and `-vv` flags. -The patterns used to match files for inclusion or exclusion are based -on "file globs" as used by the unix shell. +Rclone filter patterns can only be used in filter command line options, not +in the specification of a remote. -If the pattern starts with a `/` then it only matches at the top level -of the directory tree, **relative to the root of the remote** (not -necessarily the root of the local drive). If it doesn't start with `/` -then it is matched starting at the **end of the path**, but it will -only match a complete path element: +E.g. `rclone copy "remote:dir*.jpg" /path/to/dir` does not have a filter effect. +`rclone copy remote:dir /path/to/dir --include "*.jpg"` does. - file.jpg - matches "file.jpg" - - matches "directory/file.jpg" - - doesn't match "afile.jpg" - - doesn't match "directory/afile.jpg" - /file.jpg - matches "file.jpg" in the root directory of the remote - - doesn't match "afile.jpg" - - doesn't match "directory/file.jpg" +**Important** Avoid mixing any two of `--include...`, `--exclude...` or +`--filter...` flags in an rclone command. The results may not be what +you expect. Instead use a `--filter...` flag. -**Important** Note that you must use `/` in patterns and not `\` even -if running on Windows. +## Patterns for matching path/file names -A `*` matches anything but not a `/`. +### Pattern syntax - *.jpg - matches "file.jpg" - - matches "directory/file.jpg" - - doesn't match "file.jpg/something" +Rclone matching rules follow a glob style: -Use `**` to match anything, including slashes (`/`). + `*` matches any sequence of non-separator (`/`) characters + `**` matches any sequence of characters including `/` separators + `?` matches any single non-separator (`/`) character + `[` [ `!` ] { character-range } `]` + character class (must be non-empty) + `{` pattern-list `}` + pattern alternatives + c matches character c (c != `*`, `**`, `?`, `\`, `[`, `{`, `}`) + `\` c matches character c - dir/** - matches "dir/file.jpg" - - matches "dir/dir1/dir2/file.jpg" - - doesn't match "directory/file.jpg" - - doesn't match "adir/file.jpg" +character-range: -A `?` matches any character except a slash `/`. + c matches character c (c != `\\`, `-`, `]`) + `\` c matches character c + lo `-` hi matches character c for lo <= c <= hi - l?ss - matches "less" - - matches "lass" - - doesn't match "floss" +pattern-list: -A `[` and `]` together make a character class, such as `[a-z]` or -`[aeiou]` or `[[:alpha:]]`. See the [go regexp -docs](https://golang.org/pkg/regexp/syntax/) for more info on these. + pattern { `,` pattern } + comma-separated (without spaces) patterns - h[ae]llo - matches "hello" - - matches "hallo" - - doesn't match "hullo" +character classes (see [Go regular expression reference](https://golang.org/pkg/regexp/syntax/)) include: -A `{` and `}` define a choice between elements. It should contain a -comma separated list of patterns, any of which might match. These -patterns can contain wildcards. + Named character classes (e.g. [\d], [^\d], [\D], [^\D]) + Perl character classes (e.g. \s, \S, \w, \W) + ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]]) - {one,two}_potato - matches "one_potato" - - matches "two_potato" - - doesn't match "three_potato" - - doesn't match "_potato" +If the filter pattern starts with a `/` then it only matches +at the top level of the directory tree, +**relative to the root of the remote** (not necessarily the root +of the drive). If it does not start with `/` then it is matched +starting at the **end of the path/file name** but it only matches +a complete path element - it must match from a `/` +separator or the beginning of the path/file. -Special characters can be escaped with a `\` before them. + file.jpg - matches "file.jpg" + - matches "directory/file.jpg" + - doesn't match "afile.jpg" + - doesn't match "directory/afile.jpg" + /file.jpg - matches "file.jpg" in the root directory of the remote + - doesn't match "afile.jpg" + - doesn't match "directory/file.jpg" - \*.jpg - matches "*.jpg" - \\.jpg - matches "\.jpg" - \[one\].jpg - matches "[one].jpg" +**Important** Use `/` in path/file name patterns and not `\` even if +running on Microsoft Windows. -Patterns are case sensitive unless the `--ignore-case` flag is used. +Simple patterns are case sensitive unless the `--ignore-case` flag is used. Without `--ignore-case` (default) @@ -96,82 +93,9 @@ With `--ignore-case` potato - matches "potato" - matches "POTATO" -Note also that rclone filter globs can only be used in one of the -filter command line flags, not in the specification of the remote, so -`rclone copy "remote:dir*.jpg" /path/to/dir` won't work - what is -required is `rclone --include "*.jpg" copy remote:dir /path/to/dir` +## How filter rules are applied to files -### Directories ### - -Rclone keeps track of directories that could match any file patterns. - -Eg if you add the include rule - - /a/*.jpg - -Rclone will synthesize the directory include rule - - /a/ - -If you put any rules which end in `/` then it will only match -directories. - -Directory matches are **only** used to optimise directory access -patterns - you must still match the files that you want to match. -Directory matches won't optimise anything on bucket based remotes (e.g. -s3, swift, google compute storage, b2) which don't have a concept of -directory. - -### Differences between rsync and rclone patterns ### - -Rclone implements bash style `{a,b,c}` glob matching which rsync doesn't. - -Rclone always does a wildcard match so `\` must always escape a `\`. - -## How the rules are used ## - -Rclone maintains a combined list of include rules and exclude rules. - -Each file is matched in order, starting from the top, against the rule -in the list until it finds a match. The file is then included or -excluded according to the rule type. - -If the matcher fails to find a match after testing against all the -entries in the list then the path is included. - -For example given the following rules, `+` being include, `-` being -exclude, - - - secret*.jpg - + *.jpg - + *.png - + file2.avi - - * - -This would include - - * `file1.jpg` - * `file3.png` - * `file2.avi` - -This would exclude - - * `secret17.jpg` - * non `*.jpg` and `*.png` - -A similar process is done on directory entries before recursing into -them. This only works on remotes which have a concept of directory -(Eg local, google drive, onedrive, amazon drive) and not on bucket -based remotes (e.g. s3, swift, google compute storage, b2). - -## Adding filtering rules ## - -Filtering rules are added with the following command line flags. - -### Repeating options ## - -You can repeat the following options to add more than one rule of that -type. +Rclone path / file name filters are made up of one or more of the following flags: * `--include` * `--include-from` @@ -179,108 +103,282 @@ type. * `--exclude-from` * `--filter` * `--filter-from` - * `--filter-from-raw` -**Important** You should not use `--include*` together with `--exclude*`. -It may produce different results than you expected. In that case try to use: `--filter*`. +There can be more than one instance of individual flags. -Note that all the options of the same type are processed together in -the order above, regardless of what order they were placed on the -command line. +Rclone internally uses a combined list of all the include and exclude +rules. The order in which rules are processed can influence the result +of the filter. -So all `--include` options are processed first in the order they -appeared on the command line, then all `--include-from` options etc. +All flags of the same type are processed together in the order +above, regardless of what order the different types of flags are +included on the command line. -To mix up the order includes and excludes, the `--filter` flag can be -used. +Multiple instances of the same flag are processed from left +to right according to their position in the command line. -### `--exclude` - Exclude files matching pattern ### +To mix up the order of processing includes and excludes use `--filter...` +flags. -Add a single exclude rule with `--exclude`. +Within `--include-from`, `--exclude-from` and `--filter-from` flags +rules are processed from top to bottom of the referenced file.. -This flag can be repeated. See above for the order the flags are +If there is an `--include` or `--include-from` flag specified, rclone +implies a `- **` rule which it adds to the bottom of the internal rule +list. Specifying a `+` rule with a `--filter...` flag does not imply +that rule. + +Each path/file name passed through rclone is matched against the +combined filter list. At first match to a rule the path/file name +is included or excluded and no further filter rules are processed for +that path/file. + +If rclone does not find a match, after testing against all rules +(including the implied rule if appropriate), the path/file name +is included. + +Any path/file included at that stage is processed by the rclone +command. + +`--files-from` and `--files-from-raw` flags over-ride and cannot be +combined with other filter options. + +To see the internal combined rule list, in regular expression form, +for a command add the `--dump filters` flag. Running an rclone command +with `--dump filters` and `-vv` flags lists the internal filter elements +and shows how they are applied to each source path/file. There is not +currently a means provided to pass regular expression filter options into +rclone directly though character class filter rules contain character +classes. [Go regular expression reference](https://golang.org/pkg/regexp/syntax/) + +### How filter rules are applied to directories + +Rclone commands filter, and are applied to, path/file names not +directories. The entire contents of a directory can be matched +to a filter by the pattern `directory/*` or recursively by +`directory/**`. + +Directory filter rules are defined with a closing `/` separator. + +E.g. `/directory/subdirectory/` is an rclone directory filter rule. + +Rclone commands can use directory filter rules to determine whether they +recurse into subdirectories. This potentially optimises access to a remote +by avoiding listing unnecessary directories. Whether optimisation is +desirable depends on the specific filter rules and source remote content. + +Optimisation occurs if either: + +* A source remote does not support the rclone `ListR` primitive. `local`, +`sftp`, `Microsoft OneDrive` and `WebDav` do not support `ListR`. Google +Drive and most bucket type storage do. [Full list](https://rclone.org/overview/#optional-features) + +* On other remotes, if the rclone command is not naturally recursive, +provided it is not run with the `--fast-list` flag. `ls`, `lsf -R` and +`size` are recursive but `sync`, `copy` and `move` are not. + +* Whenever the `--disable ListR` flag is applied to an rclone command. + +Rclone commands imply directory filter rules from path/file filter +rules. To view the directory filter rules rclone has implied for a +command specify the `--dump filters` flag. + +E.g. for an include rule + + /a/*.jpg + +Rclone implies the directory include rule + + /a/ + +Directory filter rules specified in an rclone command can limit +the scope of an rclone command but path/file filters still have +to be specified. + +E.g. `rclone ls remote: --include /directory/` will not match any +files. Because it is an `--include` option the `--exclude **` rule +is implied, and the `\directory\` pattern serves only to optimise +access to the remote by ignoring everything outside of that directory. + +E.g. `rclone ls remote: --filter-from filter-list.txt` with a file +`filter-list.txt`: + + - /dir1/ + - /dir2/ + + *.pdf + - ** + +All files in directories `dir1` or `dir2` or their subdirectories +are completely excluded from the listing. Only files of suffix +`'pdf` in the root of `remote:` or its subdirectories are listed. +The `- **` rule prevents listing of any path/files not previously +matched by the rules above. + +Option `exclude-if-present` creates a directory exclude rule based +on the presence of a file in a directory and takes precedence over +other rclone directory filter rules. + +### `--exclude` - Exclude files matching pattern + +Excludes path/file names from an rclone command based on a single exclude +rule. + +This flag can be repeated. See above for the order filter flags are processed in. -Eg `--exclude *.bak` to exclude all bak files from the sync. +`--exclude` should not be used with `--include`, `--include-from`, +`--filter` or `--filter-from` flags. -### `--exclude-from` - Read exclude patterns from file ### +`--exclude` has no effect when combined with `--files-from` or +`--files-from-raw` flags. -Add exclude rules from a file. +E.g. `rclone ls remote: --exclude *.bak` excludes all .bak files +from listing. -This flag can be repeated. See above for the order the flags are -processed in. +E.g. `rclone size remote: "--exclude /dir/**"` returns the total size of +all files on `remote:` excluding those in root directory `dir` and sub +directories. -Prepare a file like this `exclude-file.txt` +E.g. on Microsoft Windows `rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"` +lists the files in `remote:` with `[JP]` or `[KR]` or `[HK]` in +their name. The single quotes prevent the shell from interpreting the `\` +characters. The `\` characters escape the `[` and `]` so ran clone filter +treats them literally rather than as a character-range. The `{` and `}` +define an rclone pattern list. For other operating systems single quotes are +required ie `rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'` + +### `--exclude-from` - Read exclude patterns from file + +Excludes path/file names from an rclone command based on rules in a +named file. The file contains a list of remarks and pattern rules. + +For an example `exclude-file.txt`: # a sample exclude rule file *.bak file2.jpg -Then use as `--exclude-from exclude-file.txt`. This will sync all -files except those ending in `bak` and `file2.jpg`. +`rclone ls remote: --exclude-from exclude-file.txt` lists the files on +`remote:` except those named `file2.jpg` or with a suffix `.bak`. That is +equivalent to `rclone ls remote: --exclude file2.jpg --exclude "*.bak"`. -This is useful if you have a lot of rules. - -### `--include` - Include files matching pattern ### - -Add a single include rule with `--include`. - -This flag can be repeated. See above for the order the flags are +This flag can be repeated. See above for the order filter flags are processed in. -Eg `--include *.{png,jpg}` to include all `png` and `jpg` files in the -backup and no others. +The `--exclude-from` flag is useful where multiple exclude filter rules +are applied to an rclone command. -This adds an implicit `--exclude *` at the very end of the filter -list. This means you can mix `--include` and `--include-from` with the -other filters (e.g. `--exclude`) but you must include all the files you -want in the include statement. If this doesn't provide enough -flexibility then you must use `--filter-from`. +`--exclude-from` should not be used with `--include`, `--include-from`, +`--filter` or `--filter-from` flags. -### `--include-from` - Read include patterns from file ### +`--exclude-from` has no effect when combined with `--files-from` or +`--files-from-raw` flags. -Add include rules from a file. +`--exclude-from` followed by `-` reads filter rules from standard input. -This flag can be repeated. See above for the order the flags are +### `--include` - Include files matching pattern + +Adds a single include rule based on path/file names to an rclone +command. + +This flag can be repeated. See above for the order filter flags are processed in. -Prepare a file like this `include-file.txt` +`--include` has no effect when combined with `--files-from` or +`--files-from-raw` flags. + +`--include` implies `--exclude **` at the end of an rclone internal +filter list. Therefore if you mix `--include` and `--include-from` +flags with `--exclude`, `--exclude-from`, `--filter` or `--filter-from`, +you must use include rules for all the files you want in the include +statement. For more flexibility use the `--filter-from` flag. + +E.g. `rclone ls remote: --include "*.{png,jpg}"` lists the files on +`remote:` with suffix `.png` and `.jpg`. All other files are excluded. + +E.g. multiple rclone copy commands can be combined with `--include` and a +pattern-list. + + rclone copy /vol1/A remote:A + rclone copy /vol1/B remote:B + +is equivalent to: + + rclone copy /vol1 remote: --include "{A,B}/**" + +E.g. `rclone ls remote:/wheat --include "??[^[:punct:]]*"` lists the +files `remote:` directory `wheat` (and subdirectories) whose third +character is not punctuation. This example uses +an [ASCII character class](https://golang.org/pkg/regexp/syntax/). + +### `--include-from` - Read include patterns from file + +Adds path/file names to an rclone command based on rules in a +named file. The file contains a list of remarks and pattern rules. + +For an example `include-file.txt`: # a sample include rule file *.jpg - *.png file2.avi -Then use as `--include-from include-file.txt`. This will sync all -`jpg`, `png` files and `file2.avi`. +`rclone ls remote: --include-from include-file.txt` lists the files on +`remote:` with name `file2.avi` or suffix `.jpg`. That is equivalent to +`rclone ls remote: --include file2.avi --include "*.jpg"`. -This is useful if you have a lot of rules. - -This adds an implicit `--exclude *` at the very end of the filter -list. This means you can mix `--include` and `--include-from` with the -other filters (e.g. `--exclude`) but you must include all the files you -want in the include statement. If this doesn't provide enough -flexibility then you must use `--filter-from`. - -### `--filter` - Add a file-filtering rule ### - -This can be used to add a single include or exclude rule. Include -rules start with `+ ` and exclude rules start with `- `. A special -rule called `!` can be used to clear the existing rules. - -This flag can be repeated. See above for the order the flags are +This flag can be repeated. See above for the order filter flags are processed in. -Eg `--filter "- *.bak"` to exclude all bak files from the sync. +The `--include-from` flag is useful where multiple include filter rules +are applied to an rclone command. -### `--filter-from` - Read filtering patterns from a file ### +`--include-from` implies `--exclude **` at the end of an rclone internal +filter list. Therefore if you mix `--include` and `--include-from` +flags with `--exclude`, `--exclude-from`, `--filter` or `--filter-from`, +you must use include rules for all the files you want in the include +statement. For more flexibility use the `--filter-from` flag. -Add include/exclude rules from a file. +`--exclude-from` has no effect when combined with `--files-from` or +`--files-from-raw` flags. -This flag can be repeated. See above for the order the flags are +`--exclude-from` followed by `-` reads filter rules from standard input. + +### `--filter` - Add a file-filtering rule + +Specifies path/file names to an rclone command, based on a single +include or exclude rule, in `+` or `-` format. + +This flag can be repeated. See above for the order filter flags are processed in. -Prepare a file like this `filter-file.txt` +`--filter +` differs from `--include`. In the case of `--include` rclone +implies an `--exclude *` rule which it adds to the bottom of the internal rule +list. `--filter...+` does not imply +that rule. + +`--filter` has no effect when combined with `--files-from` or +`--files-from-raw` flags. + +`--filter` should not be used with `--include`, `--include-from`, +`--exclude` or `--exclude-from` flags. + +E.g. `rclone ls remote: --filter "- *.bak"` excludes all `.bak` files +from a list of `remote:`. + +### `--filter-from` - Read filtering patterns from a file + +Adds path/file names to an rclone command based on rules in a +named file. The file contains a list of remarks and pattern rules. Include +rules start with `+ ` and exclude rules with `- `. `!` clears existing +rules. Rules are processed in the order they are defined. + +This flag can be repeated. See above for the order filter flags are +processed in. + +Arrange the order of filter rules with the most restrictive first and +work down. + +E.g. For `filter-file.txt`: # a sample filter rule file - secret*.jpg @@ -292,225 +390,262 @@ Prepare a file like this `filter-file.txt` # exclude everything else - * -Then use as `--filter-from filter-file.txt`. The rules are processed -in the order that they are defined. +`rclone ls remote: --filter-from filter-file.txt` lists the path/files on +`remote:` including all `jpg` and `png` files, excluding any +matching `secret*.jpg` and including `file2.avi`. It also includes +everything in the directory `dir` at the root of `remote`, except +`remote:dir/Trash` which it excludes. Everything else is excluded. -This example will include all `jpg` and `png` files, exclude any files -matching `secret*.jpg` and include `file2.avi`. It will also include -everything in the directory `dir` at the root of the sync, except -`dir/Trash` which it will exclude. Everything else will be excluded -from the sync. -### `--files-from` - Read list of source-file names ### +E.g. for an alternative `filter-file.txt`: -This reads a list of file names from the file passed in and **only** -these files are transferred. The **filtering rules are ignored** -completely if you use this option. + - secret*.jpg + + *.jpg + + *.png + + file2.avi + - * -`--files-from` expects a list of files as its input. Leading / trailing -whitespace is stripped from the input lines and lines starting with `#` -and `;` are ignored. +Files `file1.jpg`, `file3.png` and `file2.avi` are listed whilst +`secret17.jpg` and files without the suffix .jpg` or `.png` are excluded. -Rclone will traverse the file system if you use `--files-from`, -effectively using the files in `--files-from` as a set of filters. -Rclone will not error if any of the files are missing. +E.g. for an alternative `filter-file.txt`: -If you use `--no-traverse` as well as `--files-from` then rclone will -not traverse the destination file system, it will find each file -individually using approximately 1 API call. This can be more -efficient for small lists of files. + + *.jpg + + *.gif + ! + + 42.doc + - * -This option can be repeated to read from more than one file. These -are read in the order that they are placed on the command line. +Only file 42.doc is listed. Prior rules are cleared by the `!`. -Paths within the `--files-from` file will be interpreted as starting -with the root specified in the command. Leading `/` characters are -ignored. See [--files-from-raw](#files-from-raw-read-list-of-source-file-names-without-any-processing) -if you need the input to be processed in a raw manner. +### `--files-from` - Read list of source-file names -For example, suppose you had `files-from.txt` with this content: +Adds path/files to an rclone command from a list in a named file. +Rclone processes the path/file names in the order of the list, and +no others. + +Other filter flags (`--include`, `--include-from`, `--exclude`, +`--exclude-from`, `--filter` and `--filter-from`) are ignored when +`--files-from` is used. + +`--files-from` expects a list of files as its input. Leading or +trailing whitespace is stripped from the input lines. Lines starting +with `#` or `;` are ignored. + +Rclone commands with a `--files-from` flag traverse the remote, +treating the names in `--files-from` as a set of filters. + +If the `--no-traverse` and `--files-from` flags are used together +an rclone command does not traverse the remote. Instead it addresses +each path/file named in the file individually. For each path/file name, that +requires typically 1 API call. This can be efficient for a short `--files-from` +list and a remote containing many files. + +Rclone commands do not error if any names in the `--files-from` file are +missing from the source remote. + +The `--files-from` flag can be repeated in a single rclone command to +read path/file names from more than one file. The files are read from left +to right along the command line. + +Paths within the `--files-from` file are interpreted as starting +with the root specified in the rclone command. Leading `/` separators are +ignored. See [--files-from-raw](#files-from-raw-read-list-of-source-file-names-without-any-processing) if +you need the input to be processed in a raw manner. + +E.g. for a file `files-from.txt`: # comment file1.jpg subdir/file2.jpg -You could then use it like this: - - rclone copy --files-from files-from.txt /home/me/pics remote:pics - -This will transfer these files only (if they exist) +`rclone copy --files-from files-from.txt /home/me/pics remote:pics` +copies the following, if they exist, and only those files. /home/me/pics/file1.jpg → remote:pics/file1.jpg /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg -To take a more complicated example, let's say you had a few files you -want to back up regularly with these absolute paths: +E.g. to copy the following files referenced by their absolute paths: - /home/user1/important - /home/user1/dir/file - /home/user2/stuff + /home/user1/42 + /home/user1/dir/ford + /home/user2/prefect -To copy these you'd find a common subdirectory - in this case `/home` +First find a common subdirectory - in this case `/home` and put the remaining files in `files-from.txt` with or without leading `/`, e.g. - user1/important - user1/dir/file - user2/stuff + user1/42 + user1/dir/ford + user2/prefect -You could then copy these to a remote like this +Then copy these to a remote: rclone copy --files-from files-from.txt /home remote:backup -The 3 files will arrive in `remote:backup` with the paths as in the -`files-from.txt` like this: +The three files are transferred as follows: - /home/user1/important → remote:backup/user1/important - /home/user1/dir/file → remote:backup/user1/dir/file - /home/user2/stuff → remote:backup/user2/stuff + /home/user1/42 → remote:backup/user1/important + /home/user1/dir/ford → remote:backup/user1/dir/file + /home/user2/prefect → remote:backup/user2/stuff -You could of course choose `/` as the root too in which case your -`files-from.txt` might look like this. +Alternatively if `/` is chosen as root `files-from.txt` would be: - /home/user1/important - /home/user1/dir/file - /home/user2/stuff + /home/user1/42 + /home/user1/dir/ford + /home/user2/prefect -And you would transfer it like this +The copy command would be: rclone copy --files-from files-from.txt / remote:backup -In this case there will be an extra `home` directory on the remote: +Then there will be an extra `home` directory on the remote: - /home/user1/important → remote:backup/home/user1/important - /home/user1/dir/file → remote:backup/home/user1/dir/file - /home/user2/stuff → remote:backup/home/user2/stuff + /home/user1/42 → remote:backup/home/user1/42 + /home/user1/dir/ford → remote:backup/home/user1/dir/ford + /home/user2/prefect → remote:backup/home/user2/prefect -### `--files-from-raw` - Read list of source-file names without any processing ### -This option is same as `--files-from` with the only difference being that the input -is read in a raw manner. This means that lines with leading/trailing whitespace and -lines starting with `;` or `#` are read without any processing. [rclone lsf](/commands/rclone_lsf/) -has a compatible format that can be used to export file lists from remotes, which -can then be used as an input to `--files-from-raw`. +### `--files-from-raw` - Read list of source-file names without any processing -### `--min-size` - Don't transfer any file smaller than this ### +This flag is the same as `--files-from` except that input is read in a +raw manner. Lines with leading / trailing whitespace, and lines starting +with `;` or `#` are read without any processing. [rclone lsf](/commands/rclone_lsf/) has +a compatible format that can be used to export file lists from remotes for +input to `--files-from-raw`. -This option controls the minimum size file which will be transferred. -This defaults to `kBytes` but a suffix of `k`, `M`, or `G` can be -used. +### `--ignore-case` - make searches case insensitive -For example `--min-size 50k` means no files smaller than 50kByte will be -transferred. +By default rclone filter patterns are case sensitive. The `--ignore-case` +flag makes all of the filters patterns on the command line case +insensitive. -### `--max-size` - Don't transfer any file larger than this ### +E.g. `--include "zaphod.txt"` does not match a file `Zaphod.txt`. With +`--ignore-case` a match is made. -This option controls the maximum size file which will be transferred. -This defaults to `kBytes` but a suffix of `k`, `M`, or `G` can be -used. +## Quoting shell metacharacters -For example `--max-size 1G` means no files larger than 1GByte will be -transferred. +Rclone commands with filter patterns containing shell metacharacters may +not as work as expected in your shell and may require quoting. -### `--max-age` - Don't transfer any file older than this ### +E.g. linux, OSX (`*` metacharacter) -This option controls the maximum age of files to transfer. Give in -seconds or with a suffix of: + * `--include \*.jpg` + * `--include '*.jpg'` + * `--include='*.jpg'` + +Microsoft Windows expansion is done by the command, not shell, so +`--include *.jpg` does not require quoting. + +If the rclone error +`Command .... needs .... arguments maximum: you provided .... non flag arguments:` +is encountered, the cause is commonly spaces within the name of a +remote or flag value. The fix then is to quote values containing spaces. + +## Other filters + +### `--min-size` - Don't transfer any file smaller than this + +Controls the minimum size file within the scope of an rclone command. +Default units are `kBytes` but abbreviations `k`, `M`, or `G` are valid. + +E.g. `rclone ls remote: --min-size 50k` lists files on `remote:` of 50kByte +size or larger. + +### `--max-size` - Don't transfer any file larger than this + +Controls the maximum size file within the scope of an rclone command. +Default units are `kBytes` but abbreviations `k`, `M`, or `G` are valid. + +E.g. `rclone ls remote: --max-size 1G` lists files on `remote:` of 1GByte +size or smaller. + +### `--max-age` - Don't transfer any file older than this + +Controls the maximum age of files within the scope of an rclone command. +Default units are seconds or the following abbreviations are valid: * `ms` - Milliseconds - * `s` - Seconds - * `m` - Minutes - * `h` - Hours - * `d` - Days - * `w` - Weeks - * `M` - Months - * `y` - Years + * `s` - Seconds + * `m` - Minutes + * `h` - Hours + * `d` - Days + * `w` - Weeks + * `M` - Months + * `y` - Years -For example `--max-age 2d` means no files older than 2 days will be -transferred. - -This can also be an absolute time in one of these formats +`--max-age` can also be specified as an absolute time in the following +formats: - RFC3339 - e.g. "2006-01-02T15:04:05Z07:00" - ISO8601 Date and time, local timezone - "2006-01-02T15:04:05" - ISO8601 Date and time, local timezone - "2006-01-02 15:04:05" - ISO8601 Date - "2006-01-02" (YYYY-MM-DD) -### `--min-age` - Don't transfer any file younger than this ### +`--max-age` applies only to files and not to directories. -This option controls the minimum age of files to transfer. Give in -seconds or with a suffix (see `--max-age` for list of suffixes) +E.g. `rclone ls remote: --max-age 2d` lists files on `remote:` of 2 days +old or less. -For example `--min-age 2d` means no files younger than 2 days will be -transferred. +### `--min-age` - Don't transfer any file younger than this -### `--delete-excluded` - Delete files on dest excluded from sync ### +Controls the minimum age of files within the scope of an rclone command. +(see `--max-age` for valid formats) -**Important** this flag is dangerous - use with `--dry-run` and `-v` first. +`--min-age` applies only to files and not to directories. -When doing `rclone sync` this will delete any files which are excluded -from the sync on the destination. +E.g. `rclone ls remote: --min-age 2d` lists files on `remote:` of 2 days +old or more. -If for example you did a sync from `A` to `B` without the `--min-size 50k` flag +## Other flags - rclone sync -i A: B: +### `--delete-excluded` - Delete files on dest excluded from sync -Then you repeated it like this with the `--delete-excluded` +**Important** this flag is dangerous to your data - use with `--dry-run` +and `-v` first. + +In conjunction with `rclone sync` the `--delete-excluded deletes any files +on the destination which are excluded from the command. + +E.g. the scope of `rclone sync -i A: B:` can be restricted: rclone --min-size 50k --delete-excluded sync A: B: -This would delete all files on `B` which are less than 50 kBytes as -these are now excluded from the sync. +All files on `B:` which are less than 50 kBytes are deleted +because they are excluded from the rclone sync command. -Always test first with `--dry-run` and `-v` before using this flag. +### `--dump filters` - dump the filters to the output -### `--dump filters` - dump the filters to the output ### - -This dumps the defined filters to the output as regular expressions. +Dumps the defined filters to standard output in regular expression +format. Useful for debugging. -### `--ignore-case` - make searches case insensitive ### +## Exclude directory based on a file -Normally filter patterns are case sensitive. If this flag is supplied -then filter patterns become case insensitive. +The `--exclude-if-present` flag controls whether a directory is +within the scope of an rclone command based on the presence of a +named file within it. -Normally a `--include "file.txt"` will not match a file called -`FILE.txt`. However if you use the `--ignore-case` flag then -`--include "file.txt"` this will match a file called `FILE.txt`. +This flag has a priority over other filter flags. -## Quoting shell metacharacters ## - -The examples above may not work verbatim in your shell as they have -shell metacharacters in them (e.g. `*`), and may require quoting. - -Eg linux, OSX - - * `--include \*.jpg` - * `--include '*.jpg'` - * `--include='*.jpg'` - -In Windows the expansion is done by the command not the shell so this -should work fine - - * `--include *.jpg` - -## Exclude directory based on a file ## - -It is possible to exclude a directory based on a file, which is -present in this directory. Filename should be specified using the -`--exclude-if-present` flag. This flag has a priority over the other -filtering flags. - -Imagine, you have the following directory structure: +E.g. for the following directory structure: dir1/file1 dir1/dir2/file2 dir1/dir2/dir3/file3 dir1/dir2/dir3/.ignore -You can exclude `dir3` from sync by running the following command: +The command `rclone ls --exclude-if-present .ignore dir1` does +not list `dir3`, `file3` or `.ignore`. - rclone sync -i --exclude-if-present .ignore dir1 remote:backup +`--exclude-if-present` can only be used once in an rclone command. + +## Common pitfalls + +The most frequent filter support issues on +the [rclone forum](https://https://forum.rclone.org/) are: + +* Not using paths relative to the root of the remote +* Not using `/` to match from the root of a remote +* Not using `**` to match the contents of a directory -Currently only one filename is supported, i.e. `--exclude-if-present` -should not be used multiple times.