Commit graph

357 commits

Author SHA1 Message Date
alankrit
6b17044f8e fs:Added multiple ca certificate support. 2023-01-17 12:16:11 +00:00
Nick Craig-Wood
3a6f1f5cd7 filter: add metadata filters --metadata-include/exclude/filter and friends
Fixes #6353
2022-12-17 10:21:11 +00:00
albertony
8e6a469f98 fspath: allow unicode numbers and letters in remote names
Previously it was limited to plain ASCII (0-9, A-Z, a-z).

Implemented by adding \p{L}\p{N} alongside the \w in the regex,
even though these overlap it means we can be sure it is 100%
backwards compatible.

Fixes #6618
2022-12-12 13:24:32 +00:00
albertony
f650a543ef docs: remote names may not start or end with space 2022-12-12 13:24:32 +00:00
Kevin Verstaen
c2dfc3e5b3
fs: Add global flag '--color' to control terminal colors
* fs: add TerminalColorMode type
* fs: add new config(flags) for TerminalColorMode
* lib/terminal: use TerminalColorMode to determine how to handle colors
* Add documentation for '--terminal-color-mode'
* tree: remove obsolete --color replaced by global --color

This changes the default behaviour of tree. It now displays colors by
default instead of only displaying them when the flag -C/--color was
active. Old behaviour (no color) can be achieved by setting --color to
'never'.

Fixes: #6604
2022-12-06 12:07:06 +00:00
Manoj Ghosh
b75c207208 oracle-object-storage: overview, docs update 2022-10-21 12:47:03 +01:00
Nick Craig-Wood
4e79de106a hubic: remove backend as service has now shutdown - fixes #6481 2022-10-05 13:33:37 +01:00
Lesmiscore
2aa264b33c smb: backend to support SMB - fixes #2042 2022-09-30 16:10:57 +01:00
Nick Craig-Wood
be53dcc9c9 docs: add more information about --track-renames
See: https://forum.rclone.org/t/feature-question-how-does-rclone-track-renames-and-moves/32911/4
2022-09-12 11:54:35 +01:00
Josh Soref
ce3b65e6dc all: fix spelling across the project
* abcdefghijklmnopqrstuvwxyz
* accounting
* additional
* allowed
* almost
* already
* appropriately
* arise
* bandwidth
* behave
* bidirectional
* brackets
* cached
* characters
* cloud
* committing
* concatenating
* configured
* constructs
* current
* cutoff
* deferred
* different
* directory
* disposition
* dropbox
* either way
* error
* excess
* experiments
* explicitly
* externally
* files
* github
* gzipped
* hierarchies
* huffman
* hyphen
* implicitly
* independent
* insensitive
* integrity
* libraries
* literally
* metadata
* mimics
* missing
* modification
* multipart
* multiple
* nightmare
* nonexistent
* number
* obscure
* ourselves
* overridden
* potatoes
* preexisting
* priority
* received
* remote
* replacement
* represents
* reproducibility
* response
* satisfies
* sensitive
* separately
* separator
* specifying
* string
* successful
* synchronization
* syncing
* šenfeld
* take
* temporarily
* testcontents
* that
* the
* themselves
* throttling
* timeout
* transaction
* transferred
* unnecessary
* using
* webbrowser
* which
* with
* workspace

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-08-30 11:16:26 +02:00
Nick Craig-Wood
fa49971d49 docs: move time/duration option docs to the main docs 2022-08-05 16:42:30 +01:00
Nick Naumann
8c02fe7b89 sync: update docs and error messages to reflect fixes to overlap checks 2022-07-13 16:04:53 +01:00
Nick Craig-Wood
55bbff6346 operations: add --server-side-across-configs global flag for any backend 2022-07-11 18:17:42 +01:00
Ovidiu Victor Tatar
b4d847cadd new backend: hidrive - fixes #1069 2022-07-08 18:24:54 +01:00
Nick Craig-Wood
c4451bc43a fs: add --metadata-set flag to specify metadata for uploads 2022-06-29 14:29:36 +01:00
Nick Craig-Wood
6a0e021dac fs: implement optional Metadata interface for Objects #111
This implements integration tests for the feature also.
2022-06-29 11:21:29 +01:00
mirekphd
32006033e6
docs: note wider impact of --checkers=N on parallelism #6280 2022-06-27 19:33:07 +01:00
albertony
f4f0e444bf filter: allow multiple --exclude-if-present flags - fixes #6219 2022-06-08 17:11:52 +02:00
Nick Craig-Wood
4b358ff43b combine: backend to combine multiple remotes in one directory tree
Fixes #5600
2022-06-08 14:57:25 +01:00
Eric Wolf
7ffab5d998
docs: clarify use of verbosity environment variable (#6208)
RCLONE_VERBOSE does not use a true/false setting, instead using a 0,1,2 setting.
2022-06-01 20:00:43 +02:00
Lesmiscore
598364ad0f backend/internetarchive: add support for Internet Archive
This adds support for Internet Archive (archive.org) Items.
2022-04-28 10:25:38 +01:00
Zsolt Ero
211dbe9aee
docs: Add --multi-thread-streams note to --transfers. 2022-04-27 18:40:39 +01:00
albertony
f065a267f6 docs: fix some links to command pages 2022-04-07 15:50:41 +02:00
Nil Alexandrov
65652f7a75 Add Akamai Netstorage as a new backend. 2022-03-09 12:42:22 +00:00
jaKa
fd453f2c7b koofr: renamed digistorage to exclude the romania part. 2022-03-08 22:39:23 +00:00
jaKa
5d06a82c5d koofr: add digistorage service as a koofr provider. 2022-03-08 10:36:18 +00:00
Nick Craig-Wood
f22b703a51 storj: rename tardigrade backend to storj backend #5616
This adds an alias for backwards compatibility and leaves a stub
documentation page to redirect people to the new documentation.
2022-02-11 11:04:15 +00:00
Nick Craig-Wood
19a5e1d63b docs: document --disable-http2 #5253 2022-02-07 12:29:36 +00:00
Nick Craig-Wood
0ae39bda8d docs: fix and reword --update docs
After discussion on the forum with @bandwidth, this rewords the
--update docs to be correct and easier to understand.

See: https://forum.rclone.org/t/help-understanding-update/28937
2022-02-01 11:07:51 +00:00
Atílio Antônio
c08d48a50d
docs: improve grammar and fix typos (#5361)
This alters some comments in source files, but is interested mainly in documentation files and help messages.
2021-11-04 12:50:43 +01:00
albertony
39e2af7974
config: allow dot in remote names (#5606) 2021-11-01 20:50:06 +01:00
Ivan Andreev
074234119a bisync: documentation #5164
Co-authored-by: Chris Nelson <stuff@cjnaz.com>
2021-11-01 21:00:27 +03:00
albertony
c9d7248d85 docs/jottacloud: update description of whitelabel services 2021-11-01 12:57:51 +00:00
albertony
da8f9be84b docs: describe the --human-readable option in more detail 2021-11-01 12:55:52 +00:00
Ivan Andreev
0d7426a2dd hasher: backend documentation #5587 2021-10-20 19:11:54 +03:00
Ivan Andreev
6d02530f9d sia: finish documentation #4514
Also rename stuttering `--sia-sia-user-agent` to `--sia-user-agent`
2021-10-19 22:55:27 +03:00
albertony
0ffdca42d5 docs: document --cache-dir flag 2021-10-11 15:08:35 +02:00
albertony
dbb6f94d95 config: make temporary directory user-configurable
See #5591
2021-10-11 15:08:35 +02:00
Ivan Andreev
68be24c88d log: optionally print pid in logs - #5593
This option is useful to troubleshoot `rclone mount --daemon`
2021-10-02 23:45:16 +03:00
albertony
26b6c83e49 docs: extend documentation on valid remote names 2021-10-01 15:18:04 +02:00
albertony
b5a27b1c75 docs: cleanup header levels 2021-09-30 17:54:57 +02:00
albertony
f1f923a986 Change byte unit format from MiByte to MiB 2021-08-31 09:57:27 +02:00
Nathan Collins
c32d5dd1f3 fs: move with --ignore-existing will not delete skipped files - #5463 2021-08-01 17:46:45 +01:00
Ole Frost
5e91b93e59 cmdtest: end-to-end test for commands, flags and environment variables
There was no easy way to automatically test the end-to-end functionality
of commands, flags, environment variables etc.

The need for end-to-end testing was highlighted by the issues fixed
in #5341. There was no automated test to continually verify current
behaviour, nor a framework to quickly test the correctness of the fixes.

This change adds an end-to-end testing framework in the cmdtest folder.
It has some simple examples in func TestCmdTest in cmdtest_test.go. The
tests should be readable by anybody familiar with rclone and look like
this:

    // Test the rclone version command with debug logging (-vv)
    out, err = rclone("version", "-vv")
    if assert.NoError(t, err) {
        assert.Contains(t, out, "rclone v")
        assert.Contains(t, out, "os/version:")
        assert.Contains(t, out, " DEBUG : ")
    }

The end-to-end tests are executed just like the Go unit tests, that is:

    go test ./cmdtest -v

The change also contains a thorough test of environment variables in
environment_test.go.

Thanks to @ncw for encouragement and introduction to the TestMain trick.
2021-07-05 16:38:20 +01:00
Ole Frost
58c99427b3 config: fixed issues with flags/options set by environment vars.
Some environment variables didn’t behave like their corresponding
command line flags. The affected flags were --stats, --log-level,
--separator, --multi-tread-streams, --rc-addr, --rc-user and --rc-pass.
Example:

    RCLONE_STATS='10s'
    rclone check remote: remote: --progress
    # Expected: rclone check remote: remote: --progress –-stats=10s
    # Actual: rclone check remote: remote: --progress

Remote specific options set by environment variables was overruled by
less specific backend options set by environment variables. Example:

    RCLONE_DRIVE_USE_TRASH='false'
    RCLONE_CONFIG_MYDRIVE_USE_TRASH='true'
    rclone deletefile myDrive:my-test-file
    # Expected: my-test-file is recoverable in the trash folder
    # Actual: my-test-file is permanently deleted (not recoverable)

Backend specific options set by environment variables was overruled by
general backend options set by environment variables. Example:

    RCLONE_SKIP_LINKS='true'
    RCLONE_LOCAL_SKIP_LINKS='false'
    rclone lsd local:
    # Expected result: Warnings when symlinks are skipped
    # Actual result: No warnings when symlinks are skipped
    # That is RCLONE_SKIP_LINKS takes precedence

The above issues have been fixed.

The debug logging (-vv) has been enhanced to show when flags are set by
environment variables.

The documentation has been enhanced with details on the precedence of
configuration options.

See pull request #5341 for more information.
2021-07-05 16:38:20 +01:00
albertony
fee0abf513 docs: add note about use of user and logname environment variables for current username 2021-07-05 16:31:16 +01:00
Ivan Andreev
062919e08c
deprecate cache backend (#5382) 2021-06-10 19:52:55 +03:00
Tyson Moore
2188fe38e5 docs: add caveat about DSCP on Windows 2021-05-28 13:43:38 +01:00
albertony
07f2f3a62e docs: fix link to paths on windows section 2021-05-19 22:11:17 +02:00
Romeo Kienzler
88aabd1f71
docs: corrected spelling
from "Check the integrity of an encrypted remote." to "Check the integrity of a crypted remote."
2021-04-28 22:50:55 +01:00
albertony
2925e1384c Use binary prefixes for size and rate units
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
2021-04-27 02:25:52 +03:00
albertony
98579608ec docs: cleanup spelling of size and rate units 2021-04-27 02:25:52 +03:00
albertony
f8d56bebaf
config: delay load config file (#5258)
Restructuring of config code in v1.55 resulted in config
file being loaded early at process startup. If configuration
file is encrypted this means user will need to supply the password,
even when running commands that does not use config.
This also lead to an issue where mount with --deamon failed to
decrypt the config file when it had to prompt user for passord.

Fixes #5236
Fixes #5228
2021-04-26 23:37:49 +02:00
albertony
3544e09e95
config: treat any config file paths with filename notfound as memory-only config (#5235) 2021-04-18 00:09:03 +02:00
albertony
d8711cf7f9 config: create config file in windows appdata directory by default (#5226)
Use %AppData% as primary default for configuration file on Windows,
which is more in line with Windows standards, while existing default
of using home directory is more Unix standards - though that made rclone
more consistent accross different OS.

Fixes #4667
2021-04-13 19:25:09 +03:00
buengese
cd69f9e6e8 uptobox: add docs 2021-04-13 17:46:07 +02:00
albertony
b96ebfc40b docs: less confusing example with config path option 2021-04-12 18:17:19 +02:00
albertony
ddb7eb6e0a
docs: fixed some typos 2021-04-08 10:19:03 +02:00
Nick Craig-Wood
c0c74003f2 fs/cache: add --fs-cache-expire-duration to control the fs cache
This commit makes the previously statically configured fs cache configurable.

It introduces two parameters `--fs-cache-expire-duration` and
`--fs-cache-expire-interval` to control the caching of the items.

It also adds new interfaces to lib/cache to set these.
2021-03-30 12:46:47 +01:00
albertony
9c9186183d
docs: add short description of configuration file format (#5142)
Fixes #572
2021-03-27 17:26:01 +01:00
albertony
f76c6cc893 docs: describe how to bypass loading of config file
Fixes #5125
2021-03-16 14:25:00 +00:00
Nick Craig-Wood
a12b2746b4 fs: make sure backends with additional config have a different name #4996
Backends for which additional config is detected (in the config string
or on the command line or as environment variables) will gain a suffix
`{XXXXX}` where `XXXX` is a base64 encoded md5hash of the config
string.

This fixes backend caching with config string remotes.

This much requested feature now works properly:

    rclone copy -vv drive,shared_with_me:file.txt drive:
2021-03-15 19:22:07 +00:00
Nick Craig-Wood
8a46dd1b57 fspath: Implement a connection string parser #4996
This is implemented as a state machine parser so it can emit sensible
error messages.

It does not use the connection strings elsewhere in rclone yet - see
subsequent commits.

An optional fuzzer is implemented for the Parse function.
2021-03-15 19:22:07 +00:00
Nick Craig-Wood
bb0b6432ae config: --config "" or "/notfound" for in memory config only #4996
If `--config` is set to empty string or the special value `/notfound`
then rclone will keep the config file in memory only.
2021-03-14 16:03:35 +00:00
albertony
18cd2064ec mount: docs: add note about volume path syntax on windows 2021-03-11 17:09:22 +00:00
pvalls
cc08f66dc1 docs: singular/plural duplicity for MByte{s} 2021-02-23 11:34:32 +00:00
pvalls
6b8da24eb8 docs: uppercase for MBytes
MBytes is written as Mbytes and MBytes interchangeably.
Use uppercase consistently across all docs.md
2021-02-23 11:34:32 +00:00
Max Sum
edfe183ba2 fshttp: add DSCP support with --dscp for QoS with differentiated services 2021-02-10 18:29:18 +00:00
Nick Craig-Wood
0a932dc1f2 Add --bwlimit for upload and download #1873 2021-01-30 17:39:30 +00:00
Nick Craig-Wood
c57af26de9 docs: add note about --check-first being useful with --order-by 2021-01-26 12:12:26 +00:00
Nick Craig-Wood
e775328523 ftp,sftp: Make --tpslimit apply - fixes #4906 2021-01-08 10:29:57 +00:00
Yury Stankevich
71edc75ca6 HDFS (Hadoop Distributed File System) implementation - #42
This includes an HDFS docker image to use with the integration tests.

Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2021-01-07 09:48:51 +00:00
buengese
66c3f2f31f new backend: zoho workdrive - fixes #4533 2020-12-30 17:56:08 +00:00
buengese
d5181118cc compress: finish docs 2020-12-02 16:30:02 +01:00
Nick Craig-Wood
979bb07c86 filefabric: Implement the Enterprise File Fabric backend
Missing features
- M-Stream support
- Oauth-like flow (soon being changed to oauth)
2020-11-25 21:11:29 +00:00
Josh Soref
3e1cb8302a docs: spelling: etc.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
LaSombra
aac2406e19
cmd: add --progress-terminal-title to print ETA to terminal title
Adds a flag, --progress-terminal-title, that when used with --progress,
will print the string `ETA: %s` to the terminal title.

This also adds WriteTerminalTitle to lib/terminal
2020-10-06 16:34:26 +01:00
Nick Craig-Wood
db4bbf9521 operations: fix use of --suffix without --backup-dir
As part of the original work adding this feature it was overlooked
that this didn't actually work for full rclone copy/sync.

This commit fixes the problem and adds a test to make sure it stays
working.

See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526
2020-10-04 16:40:20 +01:00
Nick Craig-Wood
e2426ea87b docs: note --log-file does append 2020-09-08 16:13:33 +01:00
Evan Harris
841edc729c docs: Updated --track-renames docs 2020-09-02 11:52:01 +01:00
Nick Craig-Wood
26001d520a fs: add --bwlimit-file flag to limit speeds of individual file transfers 2020-07-28 11:46:24 +01:00
Nick Craig-Wood
8f42532b6d sync: add --track-renames-strategy leaf
See: https://forum.rclone.org/t/how-to-minimize-bandwith-w-r-t-renames-during-sync/16928/22
2020-07-28 11:34:27 +01:00
Nick Craig-Wood
0f72aa8a5f docs: add section about config variable precedence
See: https://forum.rclone.org/t/precedence-rules-for-config/17707
2020-07-07 11:12:35 +01:00
Nick Craig-Wood
3efdf5e095 operations: implement --refresh-times flag to set modtimes on hashless backends 2020-06-17 10:48:13 +01:00
jtagcat
f8b6727190 docs: document valid remote names 2020-06-16 15:02:15 +01:00
Nick Craig-Wood
844b903595 docs: promote the use of -i/--interactive and "rclone sync -i" everywhere #1574 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
2020-06-10 12:33:53 +01:00
Nick Craig-Wood
b62d08d136 config: set RCLONE_CONFIG_DIR for use in config files and subprocesses
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
2020-06-03 17:39:08 +01:00
Nick Craig-Wood
e62c032184 docs: remove manually set dates and use git dates instead 2020-06-01 13:07:46 +01:00
Martin Michlmayr
7a5b531bd0 docs: fix cosmetics issues 2020-05-26 13:49:09 +01:00
Martin Michlmayr
f204d95a8b docs: fix range
This is a range so there should be two dots (as in the other ranges).
2020-05-25 11:25:23 +01:00
Martin Michlmayr
7a5b814e59 docs: fix cosmetic issues 2020-05-25 11:25:23 +01:00
Martin Michlmayr
aa6c06d751 docs: use code elements more consistently 2020-05-25 11:25:23 +01:00
Nick Craig-Wood
518d39815c sync,copy,move: add --check-first to do all checking before starting transfers
See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/
2020-05-22 17:50:07 +01:00
Nick Craig-Wood
147f97d1f7 sync: allow --max-backlog to be -ve meaning as large as possible 2020-05-22 17:50:07 +01:00
Martin Michlmayr
a36ef8582f doc: use consistent capitalization 2020-05-20 15:54:51 +01:00
Martin Michlmayr
fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
Nick Craig-Wood
610f40f700 local: implement --local-no-sparse flag for disabling sparse files #2469
This also introduces a one time warning for sparse files and updates
the docs to warn about them.
2020-05-19 10:16:43 +01:00