Commit graph

4967 commits

Author SHA1 Message Date
Nick Craig-Wood
816e68a274 Add Laurens Janssen to contributors 2020-12-11 10:00:51 +00:00
Laurens Janssen
6ab6c8eefa gcs: Storage class object header support - fixes #3043 2020-12-10 20:06:49 +00:00
Nick Craig-Wood
cb16f42075 b2: Make NewObject use less expensive API calls
Before this change when NewObject was called the b2 backend would list
the directory that the object was in in order to find it.

Unfortunately list calls are Class C transactions and cost more.

This patch switches to using HEAD requests instead. These are Class B
transactions. It is then necessary to parse the headers from response
back into the data that we get from the listing. However B2 returns
exactly the same data, just in a different form.

Rclone will use the old directory listing method when looking for
files with versions as these can't be found via a HEAD request.

This change will particularly benefit --files-from, rclone serve
restic but most operations will see some benefit.
2020-12-09 20:00:22 +00:00
Nick Craig-Wood
7ae84a3c91 Add James Lim to contributors 2020-12-09 20:00:22 +00:00
James Lim
2fd543c989 azure: add support for service principals - fixes #3230
Before: users can only connect to Azure blob containers using the access keys
from the storage account.

After: users can additionally choose connect to Azure blob containers
using service principals. This uses OAuth2 under the hood to exchange
a client ID and client secret for a short-lived access token.

Ref:
- https://github.com/rclone/rclone/issues/3230
- https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app?tabs=dotnet#well-known-values-for-authentication-with-azure-ad
- https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization#available-authentication-types-and-methods
- https://gist.github.com/ItalyPaleAle/ec6498bfa81a96f9ca27a2da6f60a770
2020-12-09 17:52:15 +00:00
Nick Craig-Wood
50cf97fc72 sugarsync: fix NewObject for files that differ in case #4830 2020-12-07 17:38:22 +00:00
Nick Craig-Wood
4acd68188b box: fix NewObject for files that differ in case #4830 2020-12-07 17:38:22 +00:00
Nick Craig-Wood
b81b6da3fc fstest: add test for reading object in a case insensitive way #4830 2020-12-07 17:38:22 +00:00
Nick Craig-Wood
56ad6aac4d test_all: remove duplicated config for filefabric backend 2020-12-07 17:38:22 +00:00
Caleb Case
1efb8ea280 backend/tardigrade: Upgrade to uplink v1.4.1
Uplink v1.4.1 provides two important improvements for rclone:

* Fix for a connection handling issue where an open project could
  potentially become unusable because the underlying connection had
  failed.
* Fix for concurrent use issue in drpc.
2020-12-06 11:45:47 +00:00
Matteo Pietro Dazzi
9cfc01f791
build: upgrade docker buildx action 2020-12-06 11:43:34 +00:00
Nick Craig-Wood
86014cebd7 dedupe: add --dedupe-mode list to just list dupes, changing nothing 2020-12-02 16:52:12 +00:00
Nick Craig-Wood
507f861c67 dedupe: add --by-hash to dedupe on hash not file name - fixes #1674 2020-12-02 16:52:12 +00:00
Nick Craig-Wood
e073720a8f dropbox: enable short lived access tokens #4792
Starting September 30th, 2021, the Dropbox OAuth flow will no longer
return long-lived access tokens. It will instead return short-lived
access tokens, and optionally return refresh tokens.

This patch adds the token_access_type=offline parameter which causes
dropbox to return short lived tokens now.
2020-12-02 16:50:16 +00:00
Nick Craig-Wood
ce7cdadb71 Add zhucan to contributors 2020-12-02 16:50:16 +00:00
zhucan
a223b78872 fs: support multi-threads to head dst object
Signed-off-by: zhuc <zhucan.k8s@gmail.com>
2020-12-02 16:26:37 +00:00
buengese
d5181118cc compress: finish docs 2020-12-02 16:30:02 +01:00
buengese
886b3abac1 compress: fix broken tests 2020-12-02 16:30:02 +01:00
Nick Craig-Wood
250f8d9371 drive: allow shortcut resolution and creation to be retried
This was an oversight in the original code - these operations should
always have been retriable.
2020-12-02 15:28:38 +00:00
Anagh Kumar Baranwal
8a429d12cf s3: Added error handling for error code 429 indicating too many requests
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-12-01 18:13:31 +00:00
Nick Craig-Wood
8bf4697dc2 serve http/webdav: redirect requests to the base url without the /
When using `--baseurl` before this patch, if a request was made to the
base URL without a trailing / then rclone would return a 404 error.

Unfortunately GVFS / Nautilus makes the request without the /
regardless of what the user put in.

This patch redirects the request to the base URL with a /. So if the
user was using `--baseurl rclone` then a request to
http://localhost/rclone would be redirected with a 308 response to
http://localhost/rclone/

Fixes #4814
2020-12-01 18:00:07 +00:00
Nick Craig-Wood
584523672c dropbox: test file name length before upload to fix upload loop
Before this change rclone would upload the whole of multipart files
before receiving a message from dropbox that the path was too long.

This change hard codes the 255 rune limit and checks that before
uploading any files.

Fixes #4805
2020-12-01 17:56:36 +00:00
Nick Craig-Wood
a9585efd64 dropbox: make malformed_path errors from too long files not retriable
Before this change, rclone would retry files with filenames that were
too long again and again.

This changed recognises the malformed_path error that is returned and
marks it not to be retried which stops unnecessary retrying of the file.

See #4805
2020-12-01 17:56:36 +00:00
Nick Craig-Wood
f6b1f05e0f dropbox: tidy repeated error message 2020-12-01 17:56:36 +00:00
Nick Craig-Wood
cc8538e0d1 gcs: fix server side copy of large objects - fixes #3724
Before this change rclone was using the copy endpoint to copy large objects.

This can fail for large objects with this error:

    Error 413: Copy spanning locations and/or storage classes could
    not complete within 30 seconds. Please use the Rewrite method

This change makes Copy use the Rewrite method as suggested by the
error message which should be good for any size of copy.
2020-11-30 16:20:30 +00:00
Nick Craig-Wood
f7d9b15707 cmount: don't call host.Umount if a signal has been received
Before this change cgofuse and libatexit would race to see who could
unmount the file system with unpredicatable results. On Linux it could
report an error or not, depending.

This change checks to see if umount is beng called from a signal and
if so leaves the unmounting to cgofuse/libfuse.

See #4804
2020-11-29 17:44:00 +00:00
Nick Craig-Wood
83406bc473 atexit: add Signalled() function - set if called from a signal #4804 2020-11-29 17:44:00 +00:00
Nick Craig-Wood
1cfce703b2 mountlib: make sure we don't call umount more than once #4804
Before this change when using CTRL-C with rclone cmount the
mount would be unmounted twice.
2020-11-29 17:44:00 +00:00
Nick Craig-Wood
3b24a4cada yandex: set Features.WriteMimeType=false as Yandex ignores mime types
Yandex appears to ignore mime types set as part of the PUT request or
as part of a PATCH request.

The docs make no mention of being able to set a mime type, so set
WriteMimeType=false indicating the backend can't set mime types on
uploaded files.
2020-11-29 17:22:43 +00:00
Nick Craig-Wood
135adb426e filefabric: set Features.Read/WriteMimeType as both supported 2020-11-29 17:22:43 +00:00
Nick Craig-Wood
987dac9fe5 fichier: set Features.ReadMimeType=true as Object.MimeType is supported 2020-11-29 17:22:43 +00:00
Nick Craig-Wood
7fde48a805 dropbox: set Features.ReadMimeType=false as Object.MimeType not supported 2020-11-29 17:22:43 +00:00
Nick Craig-Wood
ce9028bb5b chunker: set Features.ReadMimeType=false as Object.MimeType not supported 2020-11-29 17:22:43 +00:00
buengese
52688a63c6 jottacloud: don't erroniously report support for writing mime types - fixes #4817 2020-11-29 18:11:43 +01:00
Durval Menezes
8904e81cdf Fixed verbal tense
"which are uploaded" -> "which were uploaded" (minor peeve)
2020-11-29 17:32:49 +03:00
Nick Craig-Wood
bcbe393af3 sftp: implement Shutdown method 2020-11-27 17:35:01 +00:00
Nick Craig-Wood
47aada16a0 fs: add Shutdown optional method for backends 2020-11-27 17:35:01 +00:00
Nick Craig-Wood
c22d04aa30 filter: deglobalise to put filter config into the context #4685 2020-11-27 17:28:42 +00:00
Nick Craig-Wood
354b4f19ec rc: add context to flag Reload function #4685 2020-11-27 17:28:42 +00:00
Durval Menezes
0ed1857fa9
webdav: updated docs to show streaming to nextcloud is working 2020-11-27 16:57:43 +00:00
Nick Craig-Wood
dfadd98969 azureblob,memory,pcloud: fix setting of mime types
Before this change the backend was reading the mime type of the
destination object instead of the source object when uploading.

This changes fixes the problem and introduces an integration test for
it.

See: https://forum.rclone.org/t/is-there-a-way-to-get-rclone-copy-to-preserve-metadata/20682/2
2020-11-27 14:40:05 +00:00
edwardxml
19a8b66cee
docs: update rclone about docs
Create a full loop of documentation for rclone about, backends overview
and individual backend pages.

Discussion:
https://github.com/rclone/rclone/pull/4774 relates

Previously pull was requested, in part, under ref
https://github.com/rclone/rclone/pull/4801

Notes:
Introduce a tentative draft see-link format the end of sections to try
rather than lots of in-para links.

Update about.go incl link to list of backends not supporting about.

In list of backends not supporting about, include link to about command
reference.

I appreciate there may be decisions to make going forward about whether
command links should be code formatted, and using proper pretty url
links, but I have fudged that for now.

Update backend pages that do not support about with wording used
previously for ftp - it is in passive voice but I can live with it. (my
own wording and fault). The note is applied to a limitations section. If
one does not already exist it is created (even if there are other
limitations with their own sections)
2020-11-27 14:08:52 +00:00
Anagh Kumar Baranwal
07dee18d6b cmount: Add optional brew tag to throw an error when using mount in the
binaries installed via Homebrew - Fixes #4775

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2020-11-26 22:56:59 +00:00
Maciej Zimnoch
70e8b11805
accounting: fix data race in Transferred()
startedTransfers is accessed by multiple threads, and it wasn't
protected by the mutex call in Transferred() func.

Fixes #4799
2020-11-26 22:54:38 +00:00
Nick Craig-Wood
9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config #4685
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
2020-11-26 16:40:12 +00:00
Nick Craig-Wood
506342317b s3: update docs with a Reducing Costs section - Fixes #2889 2020-11-26 15:05:26 +00:00
Nick Craig-Wood
979bb07c86 filefabric: Implement the Enterprise File Fabric backend
Missing features
- M-Stream support
- Oauth-like flow (soon being changed to oauth)
2020-11-25 21:11:29 +00:00
Nick Craig-Wood
dfeae0e70a Revert "sharefile: fix backend due to API swapping integers for strings"
The API seems to have reverted to what it was before

This reverts commit 095c7bd801.
2020-11-25 20:52:57 +00:00
Nick Craig-Wood
f43a9ac17e pcloud: only use SHA1 hashes in EU region
Apparently only SHA1 hashes are supported in the EU region for
pcloud. This has been confirmed by pCloud support. The EU regions also
support SHA256 hashes which we don't support yet.

https://forum.rclone.org/t/pcloud-to-local-no-hashes-in-common/19440
2020-11-25 20:46:38 +00:00