Commit graph

5937 commits

Author SHA1 Message Date
Nick Craig-Wood
4b1ada2d51 filefabric: allow integration tests double time as they keep timing out #5734 2021-10-21 09:54:29 +01:00
albertony
f589dbc077 docs: don't show fictional example values for options as multiple choice items
See #5538
2021-10-20 22:56:19 +02:00
albertony
8efead1ee6 docs: update guide for contributing documentation
See #5538
2021-10-20 22:56:19 +02:00
albertony
9a17b32b5d docs: automatically remove ending punctuation from first line of backend option help string when used for flag usage
See #5538
2021-10-20 22:56:19 +02:00
albertony
8b65c55711 cmd/config: improve option prompt
See #5538
2021-10-20 22:56:19 +02:00
albertony
e2f47ecdeb docs: punctuation cleanup
See #5538
2021-10-20 22:56:19 +02:00
albertony
b868561951 jottacloud: return direct download link from rclone link command
If the shared path is a directory, then the download will be a zip archive.

Fixes #5391

See #5399
2021-10-20 19:54:29 +02:00
albertony
78db3dba0e jottacloud: add support for UserInfo feature
See #5399
2021-10-20 19:54:29 +02:00
albertony
41876dd669 touch: add support for touching files in directory, with options for recursive, filtering and dry-run/interactive
Fixes #5301
2021-10-20 19:24:57 +02:00
Nick Craig-Wood
2e72ec96c1 qingstor: ignore harmless failing integration test #5734
The test TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR fails in
the integration test because there is a broken bucket in the test
account which support haven't been able to remove.
2021-10-20 17:51:06 +01:00
Nick Craig-Wood
9742648fce fichier: allow more list retries for the integration tests #5734 2021-10-20 17:45:54 +01:00
Nick Craig-Wood
d73264572b putio: allow integration tests double time as they keep timing out #5734 2021-10-20 17:38:30 +01:00
Nick Craig-Wood
ff801e8e17 test_all: allow configuring a multiplier for the timeout #5734 2021-10-20 17:38:30 +01:00
Nick Craig-Wood
72c013c6f4 vfs: increase time to wait for writers in tests to 30s
In some backends (eg putio) this deadline was consistently missed at
10s so this patch increases it to 30s.

See: #5734
2021-10-20 17:38:30 +01:00
Nick Craig-Wood
1eff0eef7a premiumizeme: attempt to fix integration tests
This tries to fix the integration tests by only allowing one
premiumizeme test to run at once, in the hope it will stop rclone
hitting the rate limits and breaking the tests.

See: #5734
2021-10-20 17:38:30 +01:00
Nick Craig-Wood
5a5318720a onedrive: stop public link test complaining on non-business account
The TestIntegration/FsMkdir/FsPutFiles/PublicLink test doesn't work on
a standard onedrive account, it returns

    accessDenied: accountUpgradeRequired: Account Upgrade is required for this operation.

See: #5734
2021-10-20 17:38:30 +01:00
Nick Craig-Wood
e253b44882 dropbox: stop public link test complaining on non-enterprise account
The TestIntegration/FsMkdir/FsPutFiles/PublicLink test doesn't work on
a standard dropbox account, only on an enterprise account because it
sets expiry dates.

See: #5734
2021-10-20 17:38:30 +01:00
Ivan Andreev
0d7426a2dd hasher: backend documentation #5587 2021-10-20 19:11:54 +03:00
Ivan Andreev
f102ef2161 hasher: add hasher backend #5587 2021-10-20 19:11:54 +03:00
Ivan Andreev
57c7fde864 lib/kv: add unit tests, fix races #5587
After testing concurrent calling of `kv.Start` and `db.Stop` I had to restrict
more parts of these under mutex to make results deterministic without Sleep's
in the test body. It's more safe but has potential to lock Start for up to
2 seconds due to `db.open`.
2021-10-20 19:11:54 +03:00
Ivan Andreev
50df8cec9c lib/kv: add key-value database api #5587
Add bolt-based key-value database support.

Quick API description:
https://github.com/rclone/rclone/pull/5587#issuecomment-942174768
2021-10-20 19:11:54 +03:00
Nick Craig-Wood
8cd3251b57 fstests: Relax MimeType support checking #5587
Before this change we checked that features.ReadMimeTime was set if
and only if the Object.MimeType method was implemented.

However this test is overly general - we don't care if Objects
advertise MimeType when features.ReadMimeTime is set provided that
they always return an empty string (which is what a wrapping backend
might do).

This patch implements that logic.
2021-10-20 19:11:54 +03:00
Nick Craig-Wood
cc2f6f722c filefabric: make backoff exponential for error_background to fix errors
Before this change the backoff for the error_background error was 6
seconds. This means that if it wasn't resolved in 60 seconds (with the
default 10 low level retries) then an error was reported.

This error was being reported frequently in the integration tests, so
is likely affecting real users too.

This patch changes the backoff into an exponential backoff
1,2,4,8...1024 seconds to make sure we wait long enough for the
background operation to complete.

See #5734
2021-10-20 15:41:09 +01:00
Ivan Andreev
6cda4c2c3c Add Ian Levesque to contributors 2021-10-19 23:02:51 +03:00
Ivan Andreev
023b666863 Add Filip Rysavy to contributors 2021-10-19 23:01:47 +03:00
Ivan Andreev
2a4c6ad0e7 Add Matthew Sevey to contributors 2021-10-19 23:00:16 +03:00
Ivan Andreev
6d02530f9d sia: finish documentation #4514
Also rename stuttering `--sia-sia-user-agent` to `--sia-user-agent`
2021-10-19 22:55:27 +03:00
Ivan Andreev
c5bc857f9b sia: fix and enable integration tests #4514
- setup correct path encoding (fixes backend test FsEncoding)
- ignore range option if file is empty (fixes VFS test TestFileReadAtZeroLength)
- cleanup stray files left after failed upload (fixes test FsPutError)
- rebase code on master, adapt backend for rclone context passing
- translate Siad errors to rclone native FS errors in sia errorHandler
- TestSia: return proper backend options from the script
- TestSia: use uptodate AntFarm image, nebulouslabs/siaantfarm is stale
2021-10-19 22:55:27 +03:00
Matthew Sevey
0d1e017e09 sia: setup docker with sia-antfarm for test #4514
Always pull the latest Sia Antfarm docker image
Add wait for Sia renter to become upload ready

Co-authored-by: Filip Rysavy <fil@siasky.net>
2021-10-19 22:55:27 +03:00
Ian Levesque
3351b1e6ae sia: add backend for sia decentralized cloud #4514 2021-10-19 22:55:27 +03:00
Fred
b085aa1a3f
seafile: fix error when not configured for 2fa (#5665) 2021-10-19 20:53:35 +01:00
Nick Craig-Wood
eb0c8284f1 azureblob: fix incorrect size after --azureblob-no-head-object patch
In

05f128868f azureblob: add --azureblob-no-head-object

we incorrectly parsed the size of the object as the Content-Length of
the returned header. This is incorrect in the presense of Range
requests.

This fixes the problem by parsing the Content-Range header if
avaialble to read the correct length from if a Range request was
issued.

See: #5734
2021-10-19 20:12:17 +01:00
Nick Craig-Wood
f5c7c597ba s3: Use a combination of SDK retries and rclone retries - fixes #5509
This reverts commit

dc06973796 Revert "s3: use rclone's low level retries instead of AWS SDK to fix listing retries"

Which in turn reverted

5470d34740 "backend/s3: use low-level-retries as the number of SDK retries"

So we are back where we started.

It then modifies it to set the AWS SDK to `--low-level-retries`
retries, but set the rclone retries to 2 so that directory listings
can be retried.
2021-10-19 20:12:17 +01:00
Nick Craig-Wood
3cef84aabe Add r0kk3rz to contributors 2021-10-19 20:12:17 +01:00
Nick Craig-Wood
93afd5c346 Add Rajat Goel to contributors 2021-10-19 20:12:17 +01:00
Alex Chen
1c3c8babd3
docs: mention make for building and cmount tag for macos (#5487) 2021-10-19 12:18:06 +08:00
Ivan Andreev
690a7ac783 chunker: fix md5all test for no-meta test remotes 2021-10-18 18:04:07 +03:00
Ivan Andreev
bbcc9a45fe serve/docker: allow to customize proxy settings of docker plugin 2021-10-18 18:03:06 +03:00
albertony
16949fde09 Do not override mime types from os defaults
https://forum.rclone.org/t/rclone-serve-http-save-as/26672
2021-10-18 13:28:22 +01:00
r0kk3rz
8e4b87ae03
s3: Add AWS Snowball Edge to providers examples - fixes #5720 2021-10-18 12:52:59 +01:00
Rajat Goel
db6002952e dropbox: upgrade sdk version 2021-10-16 10:55:02 +01:00
Nick Craig-Wood
96e14bf456 sftp: fix initialization bug introduced by fs.ErrorIsDir return
3fbaa4c0b0 backends: make NewObject return fs.ErrorIsDir if possible
2021-10-16 10:38:24 +01:00
Nick Craig-Wood
54da6154c4 operations: fix lsjson --stat on root directories of bucket based backends 2021-10-16 10:38:24 +01:00
Ivan Andreev
f50537b64b ftp: add option to disable tls13 2021-10-15 20:24:37 +03:00
Ivan Andreev
f37b25a2df ftp: enable tls session cache by default 2021-10-15 19:54:25 +03:00
albertony
29b8c71522
build: force utf8 when updating backend docs from python script (#5721) 2021-10-15 18:51:57 +02:00
Nick Craig-Wood
7b66ca132d build: increase timeout for golangci-lint to 10 minutes 2021-10-15 15:58:52 +01:00
Nick Craig-Wood
9ce0df3242 dropbox: add --dropbox-batch-commit-timeout to control batch timeout
This also adds an Debug message log showing how long each batch took

See: #5491
2021-10-15 15:32:40 +01:00
Nick Craig-Wood
f4c5f1f185 box: retry operation_blocked_temporary errors #5545 2021-10-15 15:28:54 +01:00
Nick Craig-Wood
825f7826f5 box: add --box-owned-by to only show items owned by the login passed #5545 2021-10-15 15:28:54 +01:00