Commit graph

1076 commits

Author SHA1 Message Date
Rauno Ots
9808a53416 webdav: add support for sharepoint with NTLM authentication (#2921)
Add new option option "sharepoint-ntlm" for the vendor setting.
Use it when your hosted Sharepoint is not tied to the OneDrive
accounts and uses NTLM authentication.
Also add documentation and integration test.

Fixes: #2171
2021-02-23 23:27:30 +00:00
buengese
333faa6c68 zoho: fix custom client id's 2021-02-23 11:27:05 +00:00
Nick Craig-Wood
d809930e1d union: fix mkdir at root with remote:/
Before the this fix if you specified remote:/ then the union backend
would fail to notice the root directory existed.

This was fixed by stripping the trailing / from the root.

See: https://forum.rclone.org/t/upgraded-from-1-45-to-1-54-now-cant-create-new-directory-within-union-mount/22284/
2021-02-17 12:11:34 +00:00
Dmitry Chepurovskiy
1605f9e14d
s3: Fix shared_credentials_file auth
S3 backend shared_credentials_file option wasn't working neither from
config option nor from command line option. This was caused cause
shared_credentials_file_provider works as part of chain provider, but in
case user haven't specified access_token and access_key we had removed
(set nil) to credentials field, that may contain actual credentials got
from ChainProvider.

AWS_SHARED_CREDENTIALS_FILE env varible as far as i understood worked,
cause aws_sdk code handles it as one of default auth options, when
there's not configured credentials.
2021-02-17 12:04:26 +00:00
Vesnyx
4ea7c7aa47
crypt: add option to not encrypt data #1077 (#2981)
Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
2021-02-17 03:40:37 +03:00
Nick Craig-Wood
c81311722e ftp: close idle connections after --ftp-idle-timeout (1m by default)
This fixes a problem where ftp backends live on forever when using
the rc and use more and more connections.
2021-02-16 12:39:05 +00:00
Nick Craig-Wood
843ddd9136 ftp: implement Shutdown method 2021-02-16 12:39:05 +00:00
Nick Craig-Wood
a3fcadddc8 sftp: close idle connections after --sftp-idle-timeout (1m by default)
This fixes a problem where sftp backends live on forever when using
the rc and use more and more connections.

Fixes #4883
2021-02-16 12:39:05 +00:00
Nick Craig-Wood
37e630178e dropbox: add scopes to oauth request and optionally "members.read"
This change adds the scopes rclone wants during the oauth request.
Previously rclone left these blank to get a default set.

This allows rclone to add the "members.read" scope which is necessary
for "impersonate" to work, but only when it is in use as it require
authorisation from a Team Admin.

See: https://forum.rclone.org/t/dropbox-no-members-read/22223/3
2021-02-13 12:35:24 +00:00
David Sze
2d7afe8690 local: Add flag --no-preallocate - #3207
Some virtual filesystems (such as Google Drive File Stream) may
incorrectly set the actual file size equal to the preallocated space,
causing checksum and file size checks to fail.

This flag can be used to disable preallocation for local backends of
this type.
2021-02-11 17:25:28 +00:00
Nick Craig-Wood
53aa4b87fd b2: fix failed to create file system with application key limited to a prefix
Before this change, if an application key limited to a prefix was in
use, with trailing `/` marking the folders then rclone would HEAD the
path without a trailing `/` to work out if it was a file or a folder.
This returned a permission denied error, which rclone returned to the
user.

    Failed to create file system for "b2:bucket/path/":
        failed to HEAD for download: Unknown 401  (401 unknown)

This change assumes any errors on HEAD will make rclone assume the
object does not exist and the path is referring to a directory.

See: https://forum.rclone.org/t/b2-error-on-application-key-limited-to-a-prefix/22159/
2021-02-11 15:13:19 +00:00
Max Sum
edfe183ba2 fshttp: add DSCP support with --dscp for QoS with differentiated services 2021-02-10 18:29:18 +00:00
Nick Craig-Wood
e32f08f37b drive: refer to Shared Drives instead of Team Drives 2021-02-07 12:30:21 +00:00
Nick Craig-Wood
bbe791a886 swift: update github.com/ncw/swift to v2.0.0
The update to v2 of the swift library introduces a context parameter
to each function. This required a lot of mostly mechanical changes
adding context parameters.

See: https://github.com/ncw/swift/issues/159
See: https://github.com/ncw/swift/issues/161
2021-02-03 20:23:37 +00:00
Nick Craig-Wood
1545ace8f2 build: remove go1.13 build constraints 2021-02-03 17:46:08 +00:00
Nick Craig-Wood
bcac8fdc83 Use http.NewRequestWithContext where possible after go1.13 minimum version 2021-02-03 17:41:27 +00:00
Nick Craig-Wood
9710ded60f b2: automatically raise upload cutoff to avoid spurious error
Before this change, if --b2-chunk-size was raised above 200M then this
error would be produced:

    b2: upload cutoff: 200M is less than chunk size 1G

This change automatically reaises --b2-upload-cutoff to be the value
of --b2-chunk-size if it is below it, which stops this error being
generated.

Fixes #4475
2021-02-03 16:29:32 +00:00
Nick Craig-Wood
8b41dfa50a s3: add --s3-no-head parameter to minimise transactions on upload
See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
2021-02-02 10:07:48 +00:00
Nick Craig-Wood
f66928a846 drive: fix copyid command with a bare filename: can't use empty string as a path
Before this change, running

    rclone backend copyid drive: ID file.txt

Failed with the error

    command "copyid" failed: failed copying "ID" "file.txt": can't use empty string as a path

This fixes the problem.
2021-02-01 15:17:25 +00:00
Nick Craig-Wood
3b1122c888 azureblob: require go1.14+ to compile due to SDK changes 2021-01-30 18:01:12 +00:00
Nick Craig-Wood
189ef5f257 azureblob: fix memory usage by upgrading the SDK and implementing a TransferManager
In the Azure SDK there was a bug which caused excessive memory use
when doing repeated transfers:

https://github.com/Azure/azure-storage-blob-go/issues/233

This patch updates the SDK to v0.13.0 which allowed us to implement a
custom TransferManager which integrates with rclone's memory pool.

This fixes the excessive memory consumption.

See: https://forum.rclone.org/t/ask-for-settings-recommendation-for-azureblob/21505/
2021-01-30 17:26:59 +00:00
NyaMisty
15da53696e
onedrive: add support for china region operated by 21vianet #4963 (#4963)
fixes #3804
obsoletes #3973
obsoletes #4072
2021-01-29 23:04:21 +03:00
Pau Rodriguez-Estivill
f0bf9cfda1 drive: add xdg office icons to xdg desktop files 2021-01-28 17:12:48 +00:00
Nguyễn Hữu Luân
671dd047f7
swift: ensure partially uploaded large files are uploaded unless --swift-leave-parts-on-error
This makes sure that partially uploaded large files are removed
unless the `--swift-leave-parts-on-error` flag is supplied.

- refactor swift.go
- add unit test for swift with chunk
- add unit test for large object with fail case
- add "-" to white list char during encode.
2021-01-28 17:09:41 +00:00
Nicolas Rueff
f5af761466
gphotos: new flag --gphotos-include-archived - fixes #4728
New flag --gphotos-include-archived to download and view archived media when needed.
2021-01-28 16:51:31 +00:00
Nick Craig-Wood
e6a9f005d6 sftp: implement --sftp-use-fstat
See: https://forum.rclone.org/t/sftp-fails-to-sync-to-local-failed-to-copy-file-does-not-exist/21759
2021-01-28 16:07:26 +00:00
Yury Stankevich
b569dc11a0 hdfs: support kerberos authentication #42 2021-01-27 18:16:58 +00:00
Riccardo Iaconelli
df4e6079f1 local: new flag --local-zero-size-links to fix sync on some virtual filesystems
Assume the Stat size of links is zero (and read them instead)

On some virtual filesystems (such ash LucidLink), reading a link size via a
Stat call always returns 0.
However, on unix it reads as the length of the text in the link. This may
cause errors like this when syncing:

    Failed to copy: corrupted on transfer: sizes differ 0 vs 13

Setting this flag causes rclone to read the link and use that as the size of
the link instead of 0 which in most cases fixes the problem.

Fixes #4950

Signed-off-by: Riccardo Iaconelli <riccardo@kde.org>
2021-01-27 18:13:16 +00:00
Louis Koo
cdaea62932
s3: fix copy multipart with v2 auth failing with 'SignatureDoesNotMatch'
Signed-off-by: zhuc <zhucan.k8s@gmail.com>
2021-01-27 14:43:02 +00:00
Nick Craig-Wood
78afe01d15 filefabric: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
4eac88babf premiumizeme: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
b4217fabd3 opendrive: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood
3e188495f5 sugarsync: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood
acb9e17eb3 box: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
lluuaapp
35b2ca642c b2: fixed possible crash when accessing Backblaze b2 remote 2021-01-25 17:48:40 +00:00
Nick Craig-Wood
a774f6bfdb qingstor: fix rclone cleanup
This patch changes to using the default page limit for listing
unfinished multpart uploads rather than 1000. 1000 is the maximum
specified in the docs, but setting anything larger than 200 gives an
error.
2021-01-21 17:35:31 +00:00
Nick Craig-Wood
d7cd35e2ca qingstor: fix error propagation in CleanUp
Before this change errors cleaning multiple buckets were passing silently
2021-01-21 17:35:31 +00:00
buengese
eb090d3544 compress: check type assertion in SetTier - fixes #4941 2021-01-20 22:59:14 +00:00
Nick Craig-Wood
0be69018b8 drive: log that emptying the trash can take some time - fixes #4915 2021-01-19 18:09:36 +00:00
Nick Craig-Wood
9b9ab5f3e8 gcs: Fix Entry doesn't belong in directory "" (same as directory) - ignoring
This change allows directory markers to be non-zero in size.

See: https://forum.rclone.org/t/public-gcs-bucket-and-entry-doesnt-belong-in-directory-same-as-directory/21753/
2021-01-19 16:50:37 +00:00
Nick Craig-Wood
072464cbdb gcs: fix anonymous client to use rclone's HTTP client 2021-01-19 16:50:37 +00:00
buengese
45b57822d5 compress: improve testing 2021-01-18 21:42:58 +01:00
buengese
d8984cd37f compress: correctly handle wrapping of remotes without PutStream
Also fixes ObjectInfo wrapping for Hash and Size - fixes #4928
2021-01-18 21:42:58 +01:00
Patrik Nordlén
80e63af470
jottacloud: Add support for Telia Cloud (#4930) 2021-01-17 02:38:57 +01:00
Nick Craig-Wood
cef51d58ac jottacloud: fix token refresh failed: is not a regular file error
Before this change the jottacloud token renewer would run and give the
error:

    Token refresh failed: is not a regular file

This is because the refresh runs on the root and it isn't a file.

This was fixed by ignoring that specific error.

See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
2021-01-12 17:09:44 +00:00
Nick Craig-Wood
e0b5a13a13 jottacloud: fix token renewer to fix long uploads
See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
2021-01-11 16:44:11 +00:00
Ivan Andreev
35a4de2030 chunker: fix case-insensitive NewObject, test metadata detection #4902
- fix test case FsNewObjectCaseInsensitive (PR #4830)
- continue PR #4917, add comments in metadata detection code
- add warning about metadata detection in user documentation
- change metadata size limits, make room for future development
- hide critical chunker parameters from command line
2021-01-10 22:29:24 +03:00
Ivan Andreev
847625822f chunker: improve detection of incompatible metadata #4917
Before this patch chunker required that there is at least one
data chunk to start checking for a composite object.

Now if chunker finds at least one potential temporary or control
chunk, it marks found files as a suspected composite object.
When later rclone tries a concrete operation on the object,
it performs postponed metadata read and decides: is this a native
composite object, incompatible metadata or just garbage.
2021-01-10 21:55:15 +03:00
Nick Craig-Wood
3877df4e62 s3: update help for --s3-no-check-bucket #4913 2021-01-10 17:54:19 +00:00
Denis Neuling
ec73d2fb9a azure-blob-storage: utilize streaming capabilities - #1614 2021-01-10 17:02:42 +00:00